Commit Graph
910 Commits
Author SHA1 Message Date
Devin FoleyandPaperclip b7922570e1 fix(runtime): verify GitHub bridge health before agent startup (#21537)
Verify the inactive GitHub credential endpoint from the execution target
before publishing launcher settings. If HTTP/2 fails during startup,
stop it and verify one file-bridge attempt. Stop an unverified final
bridge before returning isolated anonymous launchers.

Regression tests cover early transport loss, delayed loss reporting,
credential isolation, and failed file-bridge cleanup. The master update
keeps the current MCP approval fixture.

Verification: 96 focused tests and server typecheck pass after conflict
resolution. Full CI passes on 843cd9945e. Greptile is 5/5 with no
unresolved review threads.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 19:41:17 -08:00
DottaandClaude Opus 5.5 f6c5986b61 feat(connections): machine sign-in (MCP OAuth client credentials) for organization connections (#21473)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through connections to remote MCP servers
> - Today an organization connection must sign in as a person (browser
sign-in) or use a static key
> - Service accounts and CI-style integrations have no person who can
sign in
> - The MCP OAuth Client Credentials extension (2026-07-28) defines
machine-to-machine sign-in for this case
> - This pull request adds "Machine sign-in" for organization
connections, with a Paperclip-held signing key (recommended) or a client
secret
> - The benefit is that organizations can connect automation-owned MCP
servers safely, with credentials only in the vault and bound to one
sign-in service

## Linked Issues or Issue Description

**Subsystem affected**
Connections: remote MCP OAuth (server `tool-access/oauth`, tool gateway
token refresh) and the custom MCP server setup flow in the UI.

**Problem or motivation**
Organization-owned automation (service accounts, CI integrations) cannot
connect an MCP server that requires OAuth, because no person can
complete a browser sign-in. The MCP OAuth Client Credentials extension
(`io.modelcontextprotocol/oauth-client-credentials`) solves this with
`grant_type=client_credentials` and RFC 7523 signed client assertions.

**Proposed solution**
Add machine sign-in for organization connections only. Paperclip creates
a keypair per connection, keeps the private key in the company vault,
and shows the public key (PEM and JWK) for the admin to register. A
client secret is also supported. The sign-in service (issuer and token
endpoint) is pinned at setup. Tokens use the existing organization grant
refresh path, so caching, concurrent dedupe and 401 re-request reuse the
existing lease and single flight.

**Alternatives considered**
Reuse the deployment-wide `PAPERCLIP_TOOL_OAUTH_*` client env path that
already sent `client_credentials`. Rejected: it is not per connection,
it keeps secrets in env, and it has no issuer pin. Route machine sign-in
through `connectGalleryApp`. Rejected: its rollback deletes the draft on
a failed test, which would lose a key the admin already registered.

**Roadmap alignment**
Phase 1 of the enterprise MCP plan (no enterprise single sign-on
needed). OIDC login and enterprise-managed authorization (ID-JAG) are
later phases.

## What Changed

- `oauth/client-credentials.ts`: keypair generation (RS256, or ES256
when the service requires it), RFC 7638 key IDs, and RFC 7523 assertions
(`iss` = `sub` = client ID, `aud` = token endpoint, 60 s `exp`, unique
`jti`) with Node `crypto` only.
- `oauth/client.ts`: the token exchange accepts a signed assertion and
then sends only `client_assertion_type` and `client_assertion` (no
`client_id` or secret), as the extension specifies.
- `oauth/client-credentials-service.ts`: discovery of the sign-in
service (protected-resource metadata, then RFC 8414/OIDC metadata with
issuer check, or a token URL entered directly), issuer pinning,
vault-only key and secret handling, draft creation, test-and-activate,
and no-downtime rotation (test the new secret first; stage a new key and
switch only after the provider accepts it).
- `oauth/grant-refresh.ts` and `oauth/connection-refresh.ts`: machine
sign-in requests a new access token with the connection's own
credentials instead of needing a refresh token. It never falls back to
deployment-wide OAuth clients.
- `catalog/discovery.ts`: a rejected token on a machine sign-in
connection never re-runs discovery or rewrites the pinned endpoints.
- `mcp-http.ts` and the tool gateway: declare
`io.modelcontextprotocol/oauth-client-credentials` in client
capabilities only for connections that use it.
- Routes (with OpenAPI and activity logging): `POST
/companies/{companyId}/tools/machine-sign-in`, `GET|POST
/tool-connections/{id}/machine-sign-in[/test|/rotate|/rotate/complete]`.
Removal activity now records `machineSignIn: true`. The runner API
treats these routes as sensitive.
- Shared validators and types for machine sign-in.
- UI: "Machine sign-in" under Advanced authentication in the custom MCP
server flow (organization connections only), `MachineSignInSetup`
(choose Paperclip key or client secret, register the public key, test),
and Storybook stories for each setup state.
- Docs: `doc/connections/MCP-CLIENT-CREDENTIALS.md`, linked from
`doc/connections/README.md` and `GENERIC-REMOTE-MCP.md`.

## Verification

- `server/src/__tests__/tool-access-client-credentials.test.ts`: 16/16
pass. It uses a fake sign-in service that verifies real signatures
against registered keys, and a fake MCP server. It covers RS256 and
ES256 assertion claims and signatures, assertion-only client
authentication, the secret flow (`client_secret_basic`), discovery and
direct token URL, issuer pinning (refuses a different sign-in service
and sends it nothing), the pinned endpoint surviving a hostile
challenge, the token cache (0 token requests while live), concurrent
dedupe (5 callers on 2 service instances produce 1 token request), 401
re-request, key and secret rotation, a refused personal connection,
secret redaction in responses, rows and activity, and vault cleanup on
removal.
- Red/green: with the grant-refresh machine sign-in branch and the
discovery pin guard disabled, 5 runtime tests fail (pin, cache/dedupe,
401 re-request, both rotations). Restored: 16/16 pass. The other tests
cover new routes and services that do not exist on `master`.
- `ui/src/features/connections/MachineSignInSetup.test.tsx`: 3/3.
`ConnectionSetupFlow.architecture` and `.oauth` tests: 19/19.
- `openapi-routes.test.ts`: 51/51. `runner-api.test.ts`: 2153/2153.
`mcp-http.test.ts`: 9/9. Existing `tool-access-service.test.ts` refresh
and legacy client-credentials tests: 54/54 (filtered).
- Server package `tsc --noEmit`: clean. UI `tsc`: clean except the
pre-existing generated Storybook catalog imports.
- Not run locally (machine under heavy load): full suites, repo-wide
typecheck, browser check of the live app. CI covers breadth.

## Risks

- No migration. New config lives in `config.oauth` (`grantType:
"client_credentials"` plus `clientAuth`). Older env-configured
client-credentials connections have no `clientAuth` and keep their
original path.
- The capability declaration adds one extension key only for machine
sign-in connections. Other requests are unchanged.
- Rotation has no settings-page control yet; it is available through the
API. This is a follow-up.
- Low risk to agent boot and time to first token: a live token is read
exactly like a browser sign-in token.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:10:32 +00:00
DottaandClaude Opus 5.5 1dee321844 perf(run-events): read the wake's transaction id and the replay probe in the lock read (#21528)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every native event that announces work to another part of the server
(chat progress, a final response, an evidence outcome, recovery)
registers a wake in its transaction before it writes the event
> - Such an event takes six statements: BEGIN, the run lock read, a
replay probe, a read of the transaction id for the wake, the append, and
COMMIT. A controller checkpoint save takes five: BEGIN, the owner lock,
the transaction id, the write, COMMIT
> - In the agent scaling load model these two paths are now the largest
per-agent cost after #21511 and #21520, and other events fall behind
first at 600 agents
> - This pull request returns the transaction id from the lock read that
each path already sends, and moves the replay probe into the run lock
read when it is safe
> - The benefit is four statements for such an event and four for a
checkpoint save, with the same locks, the same wake order and the same
replay rule

## Linked Issues or Issue Description

Refs #21511 and #21520 (the delta batch path). Refs #21418 (agent
scaling plan and load model).

**What existing behavior does this improve?**
The statement count of native events that register a wake
(`appendNativeSourceEvent`) and of controller checkpoint saves
(`nativeRunnerCheckpointWriter`).

**Subsystem affected**
`packages/db/src/work-signals.ts` (one new export),
`server/src/services/heartbeat-run-events.ts`,
`server/src/services/native-runtime/native-runner-checkpoint.ts`.

**Current behavior**
`signalDatabaseWork` sends `select pg_current_xact_id()` for the first
wake of each transaction. A native event with a wake reads the run lock,
then probes for its replay in a separate statement, because a statement
that waits for the lock keeps its earlier snapshot.

**Proposed behavior**
- The run lock read and the checkpoint owner lock read also return
`pg_current_xact_id()::text`. Both already lock a row, so the
transaction has an id at that point. `primeDatabaseWorkTransactionId`
gives it to the transaction's wake state, and the wake sends no
statement for it.
- For an event that may register a progress, response, reassessment or
retention wake, the run lock read also reads `next_event_seq` from its
own snapshot and checks for a replay in that snapshot. Every event
append advances `next_event_seq` in its transaction. So when the locked
row has the same `next_event_seq` as the snapshot, no append committed
in between, and the probe result is current. Otherwise, or when the
probe finds a replay, the separate probe runs as before.

**Reason and benefit**
Fewer round trips while a pool connection is held in a transaction. In
the load model these paths hold the main pool and the three-connection
checkpoint pool longest per event.

**Breaking changes**
None. No schema change.

## What Changed

- `primeDatabaseWorkTransactionId(connection, xid)` (`work-signals.ts`,
exported from `@paperclipai/db`): inside a transaction whose id is not
yet known, it records the id. Outside a transaction it does nothing. An
uncertain commit still resolves with `pg_xact_status` on the same id.
- `appendNativeSourceEvent`: the lock read returns the transaction id
when the event has any wake. For a probing wake it is one statement: a
CTE that reads `next_event_seq` from the snapshot, a CTE that locks the
run row (`for no key update`, unchanged), and `exists (...)` for the
replay. The wake is still registered after the replay check and before
the append.
- `lockNativeRunnerCheckpointOwner` returns the transaction id with its
owner lock and primes the wake state. Its return value and every caller
are unchanged.

## Verification

- Red/green on embedded PostgreSQL; each new assertion failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "registers a native
event's wakes before its write and only when the event is new": four
statements (before: six), the wake's intent and settlement are
published, and a replay publishes no wake.
- New: "finds a replay that another writer committed while the lock read
waited, and registers no wake for it". Another connection holds the run
row (the test waits until PostgreSQL reports the append waiting for that
lock), stores the same event and moves `next_event_seq`; the waiting
append returns `duplicate`, runs the separate probe, and publishes no
wake. With the `next_event_seq` comparison removed, this test fails.
- New `native-runner-checkpoint-save.test.ts`: a save is BEGIN, the
owner lock with the transaction id, the write, COMMIT (before: five
statements), and its recovery wake is published; a save after the lease
expired writes nothing and fails with
`native_runner_checkpoint_owner_lost`.
- `work-signals.test.ts`: a primed transaction sends no id statement for
its wakes; a primed transaction whose connection is killed after its
write stays pending and then resolves through `reconcileDatabaseWork`,
with no row stored.
- Added after merging `origin/master` (with #21511, #21520 and #21436)
into this branch:
- "gives the wake the transaction id of the transaction that took the
run lock": inside a caller's transaction, an event with a progress wake
sends no separate `pg_current_xact_id()` statement; the transaction's
connection is then killed, and `reconcileDatabaseWork` checks
`pg_xact_status` with exactly the id that `pg_current_xact_id()` returns
on that same transaction. Priming a wrong id makes it fail.
- "primes nothing when the lock read fails, so the transaction is not
left uncertain": a lock read that fails on `lock_timeout` leaves no
pending work; after the same failure inside a savepoint, the next signal
on that transaction still sends its own `pg_current_xact_id()`. Priming
before the lock read makes it fail.
- Both call sites pass only the value returned by the statement they
just ran on the same `tx`: `appendNativeSourceEvent` (`tx.execute(...)`
then `primeDatabaseWorkTransactionId(tx, run.xid)`) and
`lockNativeRunnerCheckpointOwner` (`tx.with(locked).select(...)` then
`primeDatabaseWorkTransactionId(tx, xid)`).
- Retest on the combined tree, one file at a time: atomic-append 28
(including the equivalence test and both race tests), the port suite
with conformance 105, checkpoint-save 2, `live-recovery-work` 51,
`work-signals` 25, runner `native-session-runtime` 297; server typecheck
passes. No code change was needed for the combination.
- Other focused suites pass, one file at a time: the port suite with its
conformance suite (100), `chat-native-progress-work` (35),
`native-observer-replay` (1), `heartbeat-run-event-sequencing` (5),
`native-event-boundary-golden` (1), `native-recovered-owner-followup`
(97), `native-restart-recovery-isolation` (1),
`heartbeat-native-restart-suspension` (115), `live-recovery-work` (51),
`work-signals` (25).
- `pnpm --filter @paperclipai/server run typecheck` and `pnpm
check:module-boundaries` pass.
- Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in
#21418). Stacked: this pull request merged locally with #21511, and the
model's local `DELTA_COALESCE=1` mode that mirrors #21520 (the model
applies the port's batching rules itself; that mode is not in any pull
request). Default pool of 10 connections; embedded PostgreSQL on the
same, shared machine; runs one after the other with a pause between
them. No run had a failed operation, a failed checkpoint save or a lost
lease. "Before" rows are `origin/master` from the same session.

| Agents | Code | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 /
12,000 | 278 / 3,895 |
| 200 | stacked | 60 s | 7.9 | 0.31 | 2,725 | 102 / 253 | 12,000 /
12,000 | 1 / 70 |
| 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 /
24,000 | 1,904 / 60,154 |
| 400 | stacked | 60 s | 8.6 | 0.60 | 5,609 | 117 / 955 | 24,000 /
24,000 | 2 / 1,274 |
| 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 /
36,000 | 10,822 / 52,171 |
| 600 | #21511 + #21520 only | 120 s | 11.4 | 0.72 | 6,730 | 1,147 /
6,335 | 61,158 / 72,000 | 11,678 / 21,212 |
| 600 | stacked | 120 s | 9.6 | 0.93 | 10,290 | 656 / 4,233 | 71,895 /
72,000 | 1,136 / 4,980 |
| 600 | stacked (repeat) | 120 s | 9.8 | 0.90 | 9,579 | 648 / 4,509 |
71,958 / 72,000 | 1,319 / 5,712 |
| 800 | stacked | 60 s | 9.7 | 0.98 | 10,971 | 920 / 3,305 | 46,244 /
48,000 | 1,974 / 3,835 |

With the three changes, one process finishes 99.9% of the offered work
at 600 agents, but its event-loop utilization is 0.90–0.93, above the
0.75 target, and p99 lateness is about 5 s. It saturates between 600 and
800 agents.
- Statements: an event with a wake 6 → 4; a checkpoint save 5 → 4 (6 → 5
where commit-time notifications are on).

## Risks

- The transaction id is read in a statement that already assigns one (a
row lock), so no transaction gets an id it did not have before.
- The probe in the lock read is used only when `next_event_seq` did not
change between the snapshot and the lock. Every writer of
`heartbeat_run_events` moves `next_event_seq` in the same transaction
(native and ordinary appends, the run coordinator store, the retry
queue). A future writer that inserts events without it would make this
probe miss a concurrent replay; the append statement then still finds
the replay and stores nothing, but the wake would already be registered
(a spurious wake, not a lost or duplicate event).
- This pull request and #21511 both edit `heartbeat-run-events.ts`, in
different functions.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:48:17 -06:00
DottaandClaude Opus 5.5 1f388ed385 Start follow-up runs normally when a warm sandbox's window ends during its idle close (#21504)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm sandbox keeps an agent's Runner alive between runs, so the
next run on the task starts fast
> - When a run ends, the server closes the idle warm Runner in-process.
A separate SQL worker stops the sandbox when its warm window ends
(#21410, merged today)
> - Both steps start at the same moment. The sandbox stop wins, so the
idle close fails. The failed owner stays registered, and the next run
throws `native_instruction_materialization_handoff_unavailable` before
any provider work
> - Recovery must just work: a follow-up run must never fail because a
predecessor's warm copy is gone
> - This pull request lets the idle close finish before the sandbox
stops, prepares instructions fresh when the warm copy cannot be handed
off, and releases an owner whose sandbox the host already gave up
> - The benefit is that automatic follow-up runs start normally after a
warm window ends

## Linked Issues or Issue Description

Refs #21472 (base of this stacked PR), #21410 (introduced the
warm-window stop).

**What happened?**
In a Daytona two-server chaos wave (`w-20261113T200824-s7106`), three
automatic follow-up runs failed before any provider work with
`native_instruction_materialization_handoff_unavailable`: run
`5f68e194`, run `d6bd5a97` and run `d1198e40`. None of them was at a
release swap. In each case, the predecessor's idle close had failed
first with `provider_state_unreadable` / "Daytona sandbox lease … is no
longer active". The cause: the warm-window stop from #21410 stopped the
sandbox at the same moment that the idle close started. For the first
run's predecessor, the close was requested at 20:30:34 and the sandbox
got SIGTERM at 20:30:37.8.

**What did you expect?**
The follow-up run prepares its instructions fresh and continues. The
warm-window stop does not interrupt the host's own idle close.

**Steps to reproduce**
1. Run a warm Daytona task until it ends, so its warm window starts.
2. Let the warm window end while the idle timer closes the Runner.
3. Start a follow-up run on the same task. It fails with
`native_instruction_materialization_handoff_unavailable`.

Two of the three tasks then ended stranded. That downstream part (the
claim-gate cancellation with `execution_reconciliation_required` and the
stranded notice) is owned and fixed by another session. It is **not** in
this PR.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
`claimWarmNativeInstructionCopy`: after it retires a stale warm owner
whose copy cannot be adopted, it returns `null`. The caller then
prepares instructions fresh. Before, it threw
`native_instruction_materialization_handoff_unavailable`.
- `server/src/services/warm-sandbox-expiry.ts`: the warm-window stop now
waits `WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS` (2 minutes) past the window.
The in-process idle close (close grace plus the 30 s proof read)
finishes first.
- `native-session-executor.ts` idle timer: when an idle close fails, it
reads the lease rows for that sandbox. If every row is no longer
`active` or `retained` (the host gave the sandbox up), it releases the
in-process owner. The next run then prepares fresh.
- `doc/execution-semantics.md`: one paragraph about the stop margin and
fresh preparation.

## Verification

Red → green (each test fails on the base `35ba76d0ef` and passes on this
branch):

- `native-session-executor.test.ts` › "hands off collection before
probing or retiring a warm owner: unavailable" — red: throws
`handoff_unavailable`; green: returns `null` (fresh preparation).
- `native-session-executor.test.ts` › "forgets a warm owner whose idle
close failed only once the host gave up its sandbox: pending_cleanup /
released / retained" — red on all three; green: the owner is released
for `pending_cleanup` and `released`, and kept (retired by the
successor) for `retained`.
- `warm-sandbox-expiry.test.ts` › "leaves the host's idle close of the
warm runner time to finish before stopping its sandbox" — red: the
sandbox stops 3 s after the window; green: it stops only after the
margin. Two existing tests ("keeps a sandbox still inside its warm
window, and wakes when the window ends", "reconciles a warm sandbox left
running across a restart") now expect the deadline with the margin.

Local runs on this branch:

- `server`: `tsc --noEmit` clean.
- `vitest run src/__tests__/warm-sandbox-expiry.test.ts
src/services/heartbeat/instruction-working-copy.test.ts`: 24/24 passed.
- `vitest run
src/services/native-runtime/native-session-executor.test.ts -t "hands
off collection before probing or retiring a warm owner|forgets a warm
owner whose idle close failed"`: 10/10 passed.

Not run: a Daytona wave with this change.

## Risks

- Warm sandboxes now stop up to 2 minutes later than the configured
window. This is a small cost increase per warm sandbox. I chose a fixed
margin over an in-process "is a close running?" check because the stop
is pure SQL. The margin works when two servers run at the same time (the
server that runs the stop may not be the server that owns the Runner),
and it needs no busy-loop or cross-server signal.
- The margin does not cover a re-armed idle timer or a close that takes
longer than 2 minutes. Fresh preparation (fix 1) and the owner release
(fix 3) cover those cases.
- Not run on Daytona. The evidence is from the wave above and the unit
tests.
- In the wave, a `restore_failed` copy-back onto the just-restarted
sandbox was also seen. I did not find its root cause. It is not changed
here.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test runs) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:47:42 -06:00
DottaandClaude Opus 5.5 a82fa96540 perf(run-events): collect deltas that arrive while a batch commits into one next batch (#21520)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane port commits streamed text deltas of a run in
batches (#21460): the first text at once, later deltas after at most 250
ms
> - Each batch is cut when its timer fires, even when the previous batch
of the same run has not committed yet
> - When the database is slow, every run then queues one small batch per
250 ms behind its slow commit. The number of statements stays the same
exactly when the database has the least room for them, and the queue
grows
> - This pull request cuts a batch only when the previous batch of the
run has committed. Deltas that arrive while a batch commits all join the
next batch
> - The benefit is fewer, larger statements under load, with no change
when the database keeps up, and no longer wait for any delta

## Linked Issues or Issue Description

Refs #21460 (batched deltas). Refs #21511 (one statement per delta
batch). Refs #21418 (agent scaling plan and load model).

**What existing behavior does this improve?**
How `PaperclipControlPlanePort` groups queued text deltas into batches
when a batch commit is slower than the batch time.

**Subsystem affected**
Server run event ingest:
`server/src/services/native-runtime/paperclip-control-plane-port.ts`.

**Current behavior**
`#flushStreamingEvents` takes the queued deltas when it runs (on the 250
ms timer, at 100 deltas, or before any other event) and chains their
commit behind the previous batch. If one commit takes two seconds, the
run queues about eight batches, each one statement, behind it.

**Proposed behavior**
`#flushStreamingEvents` chains one cut behind the previous batch and
takes the queued deltas when that cut runs. While a cut waits, later
flush requests join it. A run has at most one batch committing and one
cut waiting.

**Reason and benefit**
Under load, the number of delta statements per run falls to about one
per commit time instead of one per 250 ms. When commits are faster than
250 ms, nothing changes.

**Breaking changes**
None. Rows, receipts, order and the first-text rule are the same.

## What Changed

- `#flushStreamingEvents` queues one cut on the run's commit chain
(`#streamingFlushQueued`). The cut takes every delta queued at that
time, sets the first-text flag, commits the batch, and then releases the
pending count of that batch.
- A flush while a cut waits returns the same chain, so a caller that
waits for it (every other event, `checkpointSession`, `completeRun`,
`replayEvents`, `provesNoRunProviderWork`) still waits for every delta
queued before it.
- A batch can now hold more than 100 deltas when commits are slow. The
100-delta rule still asks for a flush; the batch is one statement with a
fixed number of bind parameters (#21460, #21511).

## Verification

- Red/green: new port test "collects the deltas that arrive while a
batch commits into one next batch, before the next other event"
(`paperclip-control-plane-port.test.ts`). A transaction holds the run
row, so the first batch waits. Six more deltas arrive 15 ms apart with a
5 ms batch time, then a non-streaming event. Before: 8 inserts (one per
timer). After: 3 inserts (the waiting batch, one batch of six, the other
event). Rows are stored 1–9 in order and every receipt resolves.
- The whole port suite with its conformance suite passes: 101 tests.
Other suites, one file at a time: `native-observer-replay` (1),
`paperclip-semantic-conformance` (1), `native-interaction-bridge` (7),
`chat-native-progress-work` (35). I ran the four streaming port tests
two more times; they pass each time.
- `pnpm --filter @paperclipai/server run typecheck` passes. `pnpm
check:module-boundaries` passes.
- Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in
#21418). The model applies the port's batching rules itself, so for
these runs I changed my local copy only (not in this pull request): with
`DELTA_COALESCE=1`, `flushDeltas` queues one cut on the agent's append
chain and takes the queued deltas when the cut runs, as the port now
does. All rows run with #21511. Default pool of 10 connections; embedded
PostgreSQL on the same, shared machine; runs one after the other with a
pause between them. No run had a failed operation, a failed checkpoint
save or a lost lease.

| Agents | Batching | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | this change | 60 s | 9.7 | 0.51 | 3,832 | 151 / 3,087 | 12,000 /
12,000 | 3 / 3,994 |
| 400 | cut on timer (before) | 60 s | 12.7 | 0.74 | 5,465 | 1,129 /
6,040 | 18,719 / 24,000 | 5,721 / 15,238 |
| 400 | this change | 60 s | 12.5 | 0.78 | 7,010 | 892 / 3,131 | 23,993
/ 24,000 | 1,896 / 4,193 |
| 600 | cut on timer (before) | 60 s | 10.0 | 0.69 | 6,158 | 2,357 /
9,483 | 18,771 / 36,000 | 2,280 / 30,985 |
| 600 | this change | 120 s | 11.4 | 0.72 | 6,730 | 1,147 / 6,335 |
61,158 / 72,000 | 11,678 / 21,212 |
| 600 | this change (repeat) | 120 s | 13.6 | 0.74 | 6,498 | 1,225 /
6,057 | 58,984 / 72,000 | 11,465 / 25,361 |
| 800 | this change | 60 s | 11.5 | 0.80 | 6,228 | 2,054 / 7,186 |
24,877 / 48,000 | 24,544 / 31,331 |

The process does more work per second with this change (more other
events finish), so statements per second rise; delta lateness falls. 600
agents are still not carried reliably: other events fall behind. #21528
removes two statements from each such event and one from each checkpoint
save; its description has the stacked results.

## Risks

- A delta never waits longer than before: it already waited for the
previous batch of its run. When the database keeps up, batches are the
same as before.
- Under load, one batch can be large. The live view then gets one larger
step instead of several small steps behind the same wait.
- A failed batch now rejects more deltas at once (every delta that
joined it). The runner resends them, as before; each resend is
idempotent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:00:08 -06:00
DottaandClaude Opus 5.5 0bba04675c perf(run-events): append in-order native events in one statement (#21511)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - After #21460, text deltas commit in batches, but each batch still
takes five statements: BEGIN, the run lock, a replay probe, one insert,
and COMMIT. A single event without a wake takes four
> - In the agent scaling load model, the process issues about 36
statements per agent per second and saturates its one thread between 200
and 400 agents
> - This pull request writes a batch of in-order native events, or one
such event, with one statement that commits on its own
> - The benefit is about half the statements per agent, less CPU per
agent on the main thread, and less time on a pool connection, with the
same rows, receipts, cursor and locks

## Linked Issues or Issue Description

Refs #21460 (batched deltas). Refs #21435 (source cursor). Refs #21418
(agent scaling plan and load model).

**What existing behavior does this improve?**
The append of native source events that continue their source's stored
cursor: batched text deltas, the first delta after another event, and
every other native event that registers no wake.

**Subsystem affected**
Server run event ingest: `server/src/services/heartbeat-run-events.ts`,
and the port comment in `paperclip-control-plane-port.ts`.

**Current behavior**
A delta batch takes BEGIN, a run lock read (`for no key update`), a
replay probe, one insert, and COMMIT: five statements. One delta that
commits at once takes four. In the load model at 200 agents the process
sends about 6,700 statements per second at event-loop utilization 0.72.

**Proposed behavior**
The same append is one statement. Its update of the run row takes the
same row lock. It requires the stored cursor to end just before the
batch, requires the `next_event_seq` that its own snapshot read, and
finds no replayed event. Then it moves the sequence and the cursor and
inserts every row. If any check fails, it writes nothing and the append
takes the existing locked transaction.

**Reason and benefit**
Fewer statements free the main thread and the shared pool for other
agents. In the load model at 200 agents, statements per second fall from
about 6,700 to about 3,300 and event-loop utilization falls from 0.72 to
0.56.

**Breaking changes**
None. No schema change. Rows, receipts and the cursor are the same.
Servers from before this change keep working on the same runs.

## What Changed

- `appendNewNativeSourceEvents` (`heartbeat-run-events.ts`) is one
statement: a CTE reads `next_event_seq` from the statement snapshot, an
`update heartbeat_runs ... returning` locks the run row and moves
`next_event_seq` and the source cursor, and an `insert ... select from
jsonb_array_elements($9)` writes every row. The text is fixed, with nine
bind parameters for any batch size.
- **Why the `next_event_seq` check.** A statement that waits for the run
row lock keeps the snapshot it took before the wait. Its replay probe
cannot see events that the lock holder committed. Every event append
(native, ordinary, the run coordinator store, the retry queue) moves
`next_event_seq` in its own transaction. So the update requires the
value that its snapshot read. PostgreSQL re-checks that condition on the
newest row version after the wait, and any append committed meanwhile
makes the statement write nothing. The unique indexes on `(run_id,
source_event_id)` and `(run_id, source_instance_id, source_seq)` are a
second guard: a unique violation from this statement also falls back.
- A source without a cursor starts at 0. The probe still finds any
stored event of that source, so a run that an older server wrote falls
back and recounts.
- `appendHeartbeatRunEventBatch` and `appendHeartbeatRunEvent` try the
one statement first for native events without a wake, outside a caller's
transaction. Every other case (a replay, a gap, an event with a wake, a
cursor that lags, a run changed meanwhile, a binding mismatch, a run
deleted meanwhile, a caller's transaction) takes the existing
transaction, unchanged.
- Inside a caller's transaction the append keeps its SAVEPOINT path. A
failed statement there would abort the caller's transaction, and the
fallback could not run.
- `assertWritable` runs before the statement is sent and after it
returns (see Risks).

## Verification

- Red/green on embedded PostgreSQL. I ran the new tests against the
previous `heartbeat-run-events.ts` and saw 9 of them fail; all 25 tests
in the file pass with the change
(`server/src/__tests__/heartbeat-run-event-atomic-append.test.ts`):
- One statement for an in-order event and for a batch (before: 4 and 5).
- A writer revoked before the send sends nothing; a statement that fails
on its third row stores no row and no sequence; the resend commits.
- A writer revoked while the statement runs: the rows stay, the receipt
is rejected, and the resend is a replay with the same rows.
- A batch waits for Stop on the run row and appends after it; a batch
and a single event that wait for a run deletion fail with
`heartbeat_run_event_binding_mismatch` and store nothing.
- An older server stores events 2 and 3 while it holds the run row,
without the cursor: the waiting batch 2–4 falls back, returns duplicate,
duplicate, committed, and the cursor ends at 4.
- Two connections append the same 200-event stream in batches of 10:
each event is stored once, in source order, sequences 1–200 without a
gap, one `committed` and one `duplicate` receipt per event.
- A 7,000-event batch (more than any bind parameter limit at 10
parameters per row) is one statement. Before: it fell back to one
statement per event and timed out at 15 s.
- A run without a cursor starts at its first event; a run with events
from an older server replays and then continues.
- Equivalence: 12 random streams of 30 steps (in-order batches, gaps,
replays of earlier ranges, fills of gaps, ordinary events, conflicting
replays). Each step goes to two runs, one through the new path and one
forced through the one-by-one locked path. Receipts, errors, rows,
`next_event_seq` and cursors are equal at every step.
- Mutation check: with the `next_event_seq` check and the
unique-violation fallback removed, the older-server race test fails.
- The port suite (`paperclip-control-plane-port.test.ts`, with the
package conformance suite) passes: 125 tests. Two tests changed: the
streamed-event test now expects one statement and the new revocation
behavior, and the batch test checks that the five deltas are one
statement sent before the boundary event.
- Other focused suites pass, one file at a time:
`heartbeat-run-event-sequencing` (5), `native-event-boundary-golden`
(1), `native-observer-replay` (1), `paperclip-semantic-conformance` (1),
`native-interaction-bridge` (7), `chat-native-progress-work` (19),
`heartbeat-native-restart-suspension` (115), `live-recovery-work` (51).
In 2 of 3 runs with this change, all 51 tests passed but the file's
`afterAll` (closing the side pools and the embedded database) hit its 30
s timeout on this busy machine; the third run passed cleanly, and one
run on `origin/master` passed. The native events in that file all
register a wake, so they do not take the new path.
- `pnpm --filter @paperclipai/server run typecheck` and `pnpm
check:module-boundaries` pass.
- Load model: `scripts/benchmark-agent-scaling-load.ts` as merged in
#21418 (per-source counts stop when the window closes). I did not change
its batching rules for this pull request. Model rates: 20 deltas/s, 1
other event/s, 0.3 API requests/s per agent; default pool of 10
connections; embedded PostgreSQL on the same, shared machine. I ran
"before" (`origin/master`) and "after" one after the other, with a pause
between runs, and I give the load average at the start of each run. The
machine was busy with other work (load 10–15 on 18 cores), so compare
rows that ran close together. No run had a failed operation, a failed
checkpoint save or a lost lease.

| Agents | Code | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 /
12,000 | 278 / 3,895 |
| 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 /
24,000 | 1,904 / 60,154 |
| 400 | after | 60 s | 12.7 | 0.74 | 5,465 | 1,129 / 6,040 | 18,719 /
24,000 | 5,721 / 15,238 |
| 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 /
36,000 | 10,822 / 52,171 |
| 600 | after | 60 s | 10.0 | 0.69 | 6,158 | 2,357 / 9,483 | 18,771 /
36,000 | 2,280 / 30,985 |
| 600 | after, `DELTA_BATCH_MS=0` | 60 s | 10.8 | 0.68 | 8,014 | 36,135
/ 52,222 | 36,000 / 36,000 | 698 / 3,422 |

At 200 agents an earlier pair gave 0.61 → 0.43 event-loop utilization
and 7,051 → 3,210 statements/s. One process with this change alone still
does not carry 600 agents: at 400 and above the 10-connection pool and
the shared database are the limit (event-loop utilization stays below
0.75 while other events fall behind). #21520 and a third pull request
for the remaining statements of other events and checkpoint saves
continue from here; their description has the stacked results.
- Statements: a delta batch 5 → 1; one delta or one native event without
a wake 4 → 1. Unchanged: an event with a wake (6) and a checkpoint save
(5).

## Risks

- **`assertWritable` window (read first).** Before, the writer check ran
inside the transaction, before the first statement and before COMMIT. A
revocation while the statements ran rolled the append back. Now a
revocation while the one statement runs does not roll it back: the rows
commit, and the caller gets the error instead of the receipt, so no
observer runs and the runner gets no ack. This check guards a controller
that hands its run to a successor at restart
(`nativeRunsDetachingForRestart`). I think this is acceptable: (1) the
earlier check was not atomic with COMMIT either, so a revocation during
the COMMIT round trip already committed; the window grows from one
COMMIT round trip to one statement round trip plus its wait for a pool
connection; (2) the append is idempotent by source event id and payload
hash, so the successor sees the committed row as a replay of the event
that the runner resends; (3) the old controller never acts on the
rejected receipt. A new test covers exactly this.
- A crash loses nothing new: the statement commits or not as a whole,
and a resend after an unknown outcome is a replay.
- The check on `next_event_seq` makes the statement write nothing
whenever any other event of the run committed after its snapshot. Under
heavy contention on one run, more appends take the fallback. The
fallback is the previous code path.
- The statement text is fixed SQL sent through the driver
(`$client.unsafe`), like the rest of the event stream statements.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:59:01 -06:00
DottaandClaude Opus 5.5 2178c3df88 fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a recovery read to a remote Runner goes unanswered, the next
attempt of the same run carries `replaceUnansweredRunner`
> - That attempt verified the Runner was alive and then refused to adopt
it anyway. On a resumed sandbox lease the reuse check then failed with
`resumed_not_suspended` and left the Runner running. Otherwise a second
Runner was launched, and its launch fence killed the busy Runner, which
lost the provider turn.
> - After a late reconnect a Runner is often slow, not dead: it is still
replaying the events it saved with no controller attached
> - This pull request makes the attempt check progress first. It adopts
a live Runner that is making progress and replaces only a Runner that
made no progress.
> - The benefit is that a slow but working Runner keeps its turn, and
the retry no longer causes the failure it was meant to fix.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13. The retry after a
first-attempt timeout failed with these codes (counts from the traced
sample of 136 failures):

- `runner_ingress_unavailable: runnerd process failed:
runner_harness_state_mismatch: resumed_not_suspended` (part of 40
`ingress` failures)
- `runner_remote_process_identity_unavailable` (25): the Runner was gone
by the time the retry checked, because the launch fence or a previous
retry had killed it
- "restored the conversation after process loss, but…" /
`provider_turn_lost_on_restore` (10): the replacement Runner restored
from the old state and the provider turn was gone

Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9
failed. Over 420 s: 14 of 15 failed. The process ID of the Runner
changed inside the same sandbox in the failed runs, which shows the
replacement.

Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt
with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit`
verifies the Runner is alive and then returns `undefined` because of
that flag, so `adoptedProcess` is `undefined`.

**Expected behavior**

A live Runner that is making progress is adopted. A Runner that made no
progress is stopped with proof and then restored from its own journal.

**Steps to reproduce**

1. Reattach a remote run whose previous attempt left a recovery read
unanswered (`replaceUnansweredRunner: true`), with the Runner alive and
its saved event sequence past the controller's last acknowledgement.
2. On master the attempt launches a second Runner, or fails with
`resumed_not_suspended` on a resumed sandbox lease.
3. The new tests in `native-session-executor.test.ts` and
`native-runner-restart-recovery.integration.test.ts` do this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): with
`replaceUnansweredRunner`, the attempt adopts the live Runner, launching
nothing and signalling nothing, when:
- the Runner's saved event sequence is past the controller's last
acknowledgement, or
- its saved state (event and command sequences, generation, lifecycle)
changes within a bounded wait of about 10 s.
- The baseline is the first successful state read; every successful read
is also compared against the controller's acknowledgement. A Runner
whose state cannot be read at all during the wait is adopted, not
replaced: missing evidence is not proof of a wedge. A Runner whose
identity changes during the wait fails closed.
- A settled exit during the wait keeps the attempt allowance that
admitted the replacement, so it also restores on the extra fourth
attempt.
- Only a Runner that made no progress is replaced. The relaunch's
existing launch fence stops it and its process lineage with proof, and
the new Runner restores its own journal. A resumed sandbox lease no
longer refuses that relaunch with `resumed_not_suspended`.
- A Runner that exits during the wait follows the existing exited-Runner
rules: a settled suspension restores, and a mid-turn exit stays
fail-closed.
- The wait runs only on this recovery path, never on a normal run start.
- `doc/execution-semantics.md`: describes adopt versus replace.
- The claim flag `replaceUnansweredRunner` is set in claim logic that
another session owns. This PR does not change it; it changes only what
the executor does with it.

## Verification

Red on master, green on this branch:

- `native-session-executor.test.ts`
- `adopts a live Runner that emitted events past the controller's last
acknowledgement`
- `adopts a live Runner whose saved state advances within the bounded
wait`
- `hands a live Runner that made no progress to the relaunch, whose
launch fence stops it, without signalling it here`
- `restores from the journal of a Runner that suspended and exited
during the wait`
- `stays fail-closed for a Runner that exited mid-turn during the wait`
- `native-runner-restart-recovery.integration.test.ts`: `reattaches a
live %s Runner whose recovery read went unanswered on a resumed sandbox
lease` (progressing, wedged)
- Review fixes, `native-runner-restart-recovery.integration.test.ts`
(the first four were red before the fix):
- `adopts a live Runner whose first state read failed and whose later
reads advance`
- `adopts a live Runner whose first readable state is already past the
controller's acknowledgement`
- `does not replace a live Runner whose state could not be read at all
during the wait`
- `restores from the journal of a Runner that suspended and exited
during the wait on the admitted fourth attempt`
- `fails closed when the Runner's identity changes during the wait`
(pins existing behavior)
- `native-remote-retry-admission.test.ts`: unchanged, still passes.

Local runs on this branch: `tsc --noEmit` is clean for `server` on this
branch (master c6b497d831, which contains #21442).
`native-runner-restart-recovery.integration.test.ts`, the
adopt-or-replace block: 12 of 12 pass. The full
`native-session-executor.test.ts` and
`native-remote-retry-admission.test.ts` runs did not finish on this
shared machine (load 20 to 96); CI ran them on this head and they
passed.

Conflict resolved: master added a relaunch branch for a Runner that
exited for a fresh bootstrap (#21374) at the same place. Both branches
are kept; the cheaper check for an unanswered Runner that made no
progress comes first.

Daytona results that exist. They ran on a combined branch with this PR,
#21450, #21442 (merged) and #21362 (merged), so they cannot be credited
to this PR alone:

- G5 baseline (master d1d71a7755; 480 s delay injected on the first
identity write per run): one Claude run reproduced the staging chain:
authentication 419 ms, identity +423.5 s, attempt 1 `native session
provider recovery timed out after 900000ms`, then the retry failed in 6
s with `runner_harness_state_mismatch: resumed_not_suspended`.
- G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished
hands-off. The chain did not recur. All runs that crossed swap 2 were
adopted on the same Runner process ID.
- Caveat: the adopt-or-replace retry path that this PR changes was never
exercised in a cell. In G5c no first attempt failed, because all
identity writes started together 4 to 5 s after the claim. G5c also ran
before the phase timing fix in #21450.

## Risks

- Not proven on Daytona: the adopt-or-replace path itself (see the
caveat above).
- Without #21442 (merged) an adopted Runner can still time out on its
first command while it replays a backlog. This PR stops the replacement;
#21442 (merged) makes the adopted Runner answer.
- A Runner whose state cannot be read during the whole wait is adopted.
If it is in fact wedged, the adopted attempt times out again and the
attempt limit bounds it.
- A wedged Runner that still emits a few events counts as progressing
and is adopted. The existing attempt limit bounds this.
- The progress check adds a wait of up to about 10 s on this recovery
path only. A normal run start has no added wait.
- The claim construction (`native-restart-recovery.ts`) is owned by
another session and is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:56:11 -06:00
DottaandClaude Opus 5.5 47a2c82d35 fix(heartbeat): reach the previous server's live runs first after a release swap (#21450)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - During a release swap, the outgoing server stops and the incoming
server takes over the live runs on remote Runners
> - The incoming server reached those runs late: it waited up to 60 s
for the outgoing server's controller lease to lapse, re-verified each
run's whole log from object storage, and deferred a reattach when the
agent directory was busy. Queued starts competed with the reattaches for
the same resources.
> - On hosted staging, the later a live run was reached, the more likely
it was to fail. Runs reached in under 150 s never failed; runs reached
after 420 s almost always failed.
> - This pull request hands live runs over at the start of shutdown,
makes a reattach skip work that the same run already did, and holds
queued starts while handed-over runs are being reached, with a 180 s
cap.
> - The benefit is that live runs are reached within seconds of a swap
instead of minutes, so they continue instead of failing.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:

- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed

In one traced run the incoming server waited on
`remote_controller_lease_live` three times (09:14:21, 09:15:18,
09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of
that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the
run's whole log from object storage. A control run spent 23 s in
`task.prepare`. Queued starts used the same preparation path, the same 8
object-I/O slots and the same database pool, and nothing gave a reattach
priority. The first-attempt failures were `PRP command session.snapshot
timed out` and `codex_history_read_failed`. The retry then failed with
`runner_harness_state_mismatch: resumed_not_suspended`,
`runner_remote_process_identity_unavailable` or
`provider_turn_lost_on_restore`.

**Expected behavior**

Every live run is reached within about 30 s of the swap. Queued work
starts after that, or after a bounded wait.

**Steps to reproduce**

1. Run a two-server release swap with live remote runs and queued runs.
2. Observe `native.recovery.transition` "waiting" events with
`remote_controller_lease_live`, then the claim, then a long
`heartbeat.prepare.run_output` span.
3. The new tests in `reattach-phase.test.ts` and
`heartbeat-native-early-hand-off.test.ts` model this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- Early hand-off at shutdown (`heartbeat/recovery.ts`,
`server/src/index.ts`): in deploy-overlap mode, the outgoing server
releases the remote controller lease of its live runs at SIGTERM
(`recoveryState` `awaiting_runner_reattach`). A run another server
already took over is left alone. The incoming server no longer waits for
the lease to lapse.
- Reattach fast path:
- `heartbeat/run-output.ts`, `native-runtime/native-run-log-export.ts`:
a reattach starts without waiting for the existing log to be
re-verified.
- `heartbeat/agent-directory-busy.ts`, `heartbeat.ts`: a reattach that
meets a busy agent directory waits in process (at most 30 s) instead of
being deferred to a full re-preparation. An ordinary start is still
deferred at once, with no added wait.
- Reattach phase lifecycle (`heartbeat/reattach-phase.ts`): the watch is
a worker on the shared delivery coordinator, queue `reattach-phase`.
- Trigger: one startup listing of a previous server's live runs; then
the hand-over notification (`notifyReattachHandOver`, sent in the
hand-over transaction); the earliest lease deadline it watches (a killed
owner); and a 2 s re-check only while handed-over runs are still
pending, which ends with the phase (cap 180 s). The bounded re-check
exists because a run can also end on another path that sends no
notification to this queue.
- Source of truth: `native_run_finalizations` and `heartbeat_runs` rows.
- Quiet condition: nothing pending and no watched lease, so no deadline
and no SQL. It stops with the coordinator at shutdown.
- Reached hook for every driver (`native-session-executor.ts`): a
reattached run counts as reached when the native session runtime hands
the executor its recovered or opened session (`onSession`). Every driver
reaches that point only after the Runner answered this server's attach
and recovery commands, including an adopted `RunnerdExternalDriver`
Runner, which records no new process identity. The identity hook stays
as an earlier signal.
- Claim wake: when the reattach-phase worker finds runs that just became
claimable (hand-over notification or a killed owner's lease deadline),
it starts this server's restart-recovery claim for them at once. A
validation cell showed the claim otherwise waited about 21.5 s after the
hand-over.
- `recovery/assigned-work-sweep.ts`: the `deferred_reattach_phase`
return now sits before every action path of `act()`, except starting an
overdue saved-work recovery (it resumes an existing run).
- The in-transaction admission re-check of a queued claim
(`confirmRunAdmissionStillOpen`) also honours the reattach phase. A
phase that begins during a claim returns the run to the queue through
the existing rollback.
- A reattach whose log re-verification runs in the background awaits it
before the log is finalized (`settleDeferredLogRecovery`, both
finalization sites in `run-completion.ts`).
- Reattach phase (`heartbeat/reattach-phase.ts`,
`heartbeat/run-admission.ts`): `isReattachPhaseActive()` is true while
runs that a previous controller handed over (lease released or observed
expired, or claimed by this server's restart recovery) are claimable but
not yet reached. A run is reached when its process identity is recorded.
Authentication is logged as a mark. The cap (180 s,
`PAPERCLIP_REATTACH_PHASE_CAP_MS`) counts from the moment the runs
became claimable. Later batches join or re-arm the phase. Runs reached
after 150 s are counted and logged.
- Gates on the phase:
- `claimQueuedRun`, start-next, resume-queued and dispatch in
`heartbeat/queue.ts`, `run-lifecycle.ts` and
`native-runner-execution-driver.ts` leave queued runs queued.
- `execution-recovery-resolution.ts`: one line, the first line of
`deliverReconciledExecutions`.
- `recovery/assigned-work-sweep.ts`: one line, outcome
`deferred_reattach_phase`.
- `heartbeat/recovery.ts`: a one-line claim hook
(`noteReattachRunsClaimed`) after restart recovery claims runs.
  - The owner of those three files agreed to these one-line gates.
- `tests/restart-chaos/daytona-image.ts`:
`PAPERCLIP_CHAOS_TEST_ACCEPT_IMAGE_CONTENT_ID` lets a local wave run a
controller-only branch against an existing Daytona image. It accepts a
content-ID mismatch only for the exact content ID of the named image,
and it prints a TEST OVERRIDE warning. This is harness-only.

## Verification

Red on master, green on this branch:

- `heartbeat/reattach-phase.test.ts` (27 tests). Five of the six
two-server scenarios were red on the earlier phase timing:
- `does not hold queued starts while the previous owner is alive and not
handing over`
- `closes the gate when the previous owner hands its runs over, and
counts reach latency from the hand-over`
- `runs the cap from the hand-over, so a wedged run ends the phase
without the overlap eating it`
- `begins when a killed owner's lease is observed expired, without any
hand-over`
- `re-arms for a later batch, including runs the previous owner started
after discovery`
- `closes the gate at once when this server claims a run between two
lease reads, dated from the hand-over`
- plus `claimQueuedRun leaves a queued run queued and unstarted, in the
admission gate's shape`, `is the same gate the assigned-work sweep and
reconciled continuation delivery read`, `ends at the cap, counted from
when the runs became claimable, so one wedged Runner cannot hold run
starts`
- `heartbeat-native-early-hand-off.test.ts`: `releases the remote
controller lease of %i live runs at once` (1, 20, 40 runs; 40 runs in 56
ms), `leaves a run another server already took over alone`, `holds the
incoming server's queued starts only from the hand-over, not while the
outgoing server still drives its runs`
- `heartbeat/run-output.test.ts`: `lets a reattach start before its
existing log is re-verified`
- `heartbeat/agent-directory-busy.test.ts`: `defers an ordinary start at
once, with no wait added to the run-start path`, `lets a reattach wait
its turn in process instead of a deferred full re-preparation`, `still
defers a reattach once its bounded wait runs out`
- `tests/restart-chaos/daytona-image.test.ts`: `accepts a mismatch only
for the exact image content ID named by the test override`
- Review fixes (red before the fix, green after):
`reattach-phase.test.ts` `starts the claim of handed-over runs within a
second of the hand-over notification`; `assigned-work-invariant.test.ts`
`takes no action while handed-over live runs are being reached, and acts
once the phase ends`; `heartbeat-native-early-hand-off.test.ts` `never
lets the outgoing server claim back a run it handed off, while the
incoming server may take it over`; `reattach-phase.test.ts` `retries a
failed first listing instead of going quiet, and still finds the
previous owner's runs`, `keeps a hand-over notification whose listing
failed until a listing succeeds`, `moves the cap earlier when a claimed
run turns out to have been handed over before its claim`;
`queue.test.ts` `returns the claim to the queue when a reattach phase
begins while the claim is being checked`; `run-completion.test.ts`
`finalizes the run log in completeRun|failRun only after a reattach's
deferred log recovery settles`. `native-session-executor.test.ts` `ends
the reattach phase once a reattached %s session is driven by this server
again, without the cap or manual settling` over codex_app_server,
opencode_server, acpx_runtime, muse_external, grok_bot_mcp and
openai_dot_mcp (6 of 6 red without the hook; the test drives the
executor with a mocked session runtime, not each real driver). Lifecycle
tests for the new worker: `does no database work and keeps no deadline
on an instance with nothing to reattach`, `waits for the hand-over
notification or the lease deadline instead of polling while the previous
owner drives its runs`, `begins at a killed owner's lease deadline,
without any hand-over or notification`, `stops with the delivery
coordinator at shutdown and ignores a later hand-over`.

Local runs on this branch: `tsc --noEmit` is clean for `server`. Passing
on this branch: `reattach-phase.test.ts`,
`heartbeat-native-early-hand-off.test.ts`, `run-output.test.ts` (53
tests together), `agent-directory-busy.test.ts`,
`heartbeat-native-restart-suspension.test.ts` (115 of 115, alone), the
new cases in `queue.test.ts` and `run-completion.test.ts`. Under load 30
to 96 on this shared machine, the full `queue.test.ts` and
`recovery-wake-herd.test.ts` runs stalled in their embedded-Postgres
setup; CI supplies them.

Daytona results that exist (local servers with Daytona sandboxes,
release-swap cells). They ran on a combined branch with this PR, #21462,
#21442 and #21362 (merged), so they cannot be credited to this PR alone:

- G5 baseline (master d1d71a7755): SIGTERM to PRP authentication 21.5 to
38 s at swap 1.
- G5c (combined branch 588c6ba267): SIGTERM to authentication 6.2 to 7.7
s at swap 1. All runs that crossed swap 2 were adopted on the same
Runner process. The staging chain (`provider recovery timed out after
900000ms`, then `resumed_not_suspended`) did not recur.
- Caveat: G5c ran before the phase timing fix in this PR. In G5c the
phase opened at process start, its 180 s cap ran out during the overlap,
and swap 2 ended `cap_reached` with 0 runs reached, 1 s before the lease
claims. The timing fix (the phase begins when the runs become claimable)
has not run on Daytona.
- G6c (12 live runs and 20 queued, two graceful swaps, no injection,
combined branch 588c6ba267): 0 failed runs at either swap; 11 of 11
survivors at both.
- Swap 1: hand-off of 11 runs in 291 ms; claims 0.3 to 2.9 s after
hand-off; authentication 7.6 to 12.5 s and identity 8.3 to 13.3 s after
SIGTERM. The 30 s target was met.
- Swap 2: hand-off took 8,562 ms and `hot_restart_prepare` timed out at
19 s, so the outgoing server drained gracefully. The incoming server did
not claim on the hand-over; it claimed only when the outgoing server
exited, 33 s after SIGTERM. Authentication was 46 to 54 s and identity
47 to 57 s after SIGTERM. The target was missed. Why the claims waited
for the exit is not explained.
- The host load was 49 to 70 from other work. 7 of 32 tasks timed out in
the queue or budget after the swaps; we suspect the overload.

## Risks

- Product trade-off: while handed-over live runs are being reached,
queued starts wait, up to the 180 s cap. On an instance with no
handed-over runs, nothing waits.
- Not proven: two-server behavior with the phase timing fix and the
notification-driven watch (no cell has run them); warm standby becoming
active; runs that the outgoing server starts after discovery.
- Early hand-off at SIGTERM orphans a server-side exec tool that is in
flight. With #21442 the next controller settles it as interrupted with
an unknown outcome; it is never run again.
- Swap 2 in G6c: the incoming server claimed only at the outgoing
server's exit. That is unexplained.
- Cross-session ownership: the gates in
`execution-recovery-resolution.ts` and
`recovery/assigned-work-sweep.ts`, and the claim hook in
`heartbeat/recovery.ts`, are in files that the restart-recovery work
owns. They are one-line gates agreed with that owner. That owner said
the gate did not need to be inside the claim transaction; that was
before the phase could begin during a claim. After review, the
in-transaction re-check in `heartbeat/queue.ts` also reads the phase,
through the existing rollback. #21427 (open) also edits
`execution-recovery-resolution.ts`, `heartbeat/queue.ts` and
`recovery/assigned-work-sweep.ts`; whichever merges second needs a
rebase. `native-runtime/native-restart-recovery.ts` (claim logic, owned
by the restart-recovery work): a 3-line guard in
`claimNativeRestartRecoveries` returns a run this process handed off at
shutdown as `blocked` / `handed_off_to_next_controller` before takeover
is evaluated, so the outgoing server never claims back a run it handed
off. #21478 (merged) added a sweep path that starts an overdue
saved-work recovery; it is deliberately not gated by the reattach phase,
because it resumes an existing run (reattach-like work), which the phase
makes room for rather than competes with.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:55:31 -06:00
Devin FoleyandPaperclip 089748fbf9 fix(recovery): settle uncertain runner API receipts instead of fencing them forever (#21522)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs change Paperclip through the runner `call_api`
tool. Each mutation reserves a receipt before dispatch, so a crash
cannot replay a mutation that may have committed.
> - When the response of a mutation is lost, the receipt records an
unknown outcome. `priorRecoveryApiReceipt` then fences every later
mutation with the same operation id on that task, in every later run.
> - Nothing can settle that fence. The agent gets "Reconcile the
existing operation", but it has no tool to do that. The task stays stuck
for agents.
> - This pull request records the server's own final status for each run
mutation, and settles an unknown receipt from that record. A later
change by a board user on the task also lifts the fence.
> - The benefit is that a lost response no longer blocks a task's
mutations forever, and the fence still holds when the outcome is really
unknown.

## Linked Issues or Issue Description

**What happened?**
A routine run sent `PATCH /api/issues/{id}` to hand the task to a board
user. The server committed the change, but the server was slow and the
response took longer than the 30 second client timeout. The receipt
recorded `api_transport_failure` with `outcome: "unknown"`. Hours later,
the board user commented and reassigned the task to the agent. The new
run did the work, but both of its hand-off PATCH calls returned
`api_outcome_unknown` within 2 seconds and never reached the route.
Every later run on that task would get the same result.

**Expected behavior**
When the server has a record of the earlier call's result, the uncertain
receipt settles from that record. When a person has changed the task
after the uncertain run, the agent can act on the current state.

**Steps to reproduce**
1. In a native run, send a `call_api` mutation that commits on the
server, but let the client time out before the response.
2. Start a later run in the same retry or continuation ancestry.
3. Send any mutation with the same operation id. It returns
`api_outcome_unknown` and is not dispatched.

**Relevant logs or output**
```
call_api PATCH /api/issues/{id} -> { ok: false, status: null, error: "api_outcome_unknown", outcome: "unknown",
  guidance: "Reconcile the existing operation before repeating it. ..." }
```

No related public PR exists. I searched open and closed PRs for this
fence.

## What Changed

- `runner-api-client.ts`: `executeRunnerApi` sends the receipt key in
`X-Paperclip-Runner-Call` with mutations. Reads do not send it.
- `paperclip-runner-tool-authority.ts`: `#callApi` passes the receipt
key of the reserved mutation to the client.
- `middleware/runner-call-outcome.ts` (new): for an agent JWT request
with a run id and a valid call key, it records the final HTTP status as
a `runner.api_completed` activity row of the run. It records when the
handler ends the response, so it also records after the client has gone
away. It records one time per request.
- `recovery-effect-receipts.ts`: `priorRecoveryApiReceipt` settles an
unknown receipt:
- A recorded 2xx is an applied effect. An exact retry from the ancestry
gets an applied result and is not dispatched again. Other calls
continue.
- A recorded definite 4xx rejection (not 408) no longer fences, the same
as a completed 4xx receipt today.
  - A recorded 5xx, or no record, keeps the fence.
- A board user's `issue.updated` or `issue.comment_added` on the task
after the uncertain run finished lifts the fence of that run. Agent,
system, read and inbox activity do not.
- The fenced result now tells the agent to read the current state and
not to repeat a change that is already there.

## Verification

- `pnpm --filter @paperclipai/server run typecheck` passes.
- `npx vitest run src/middleware/runner-call-outcome.test.ts
src/services/task-retry.test.ts
src/services/native-runtime/runner-api.test.ts
src/services/native-runtime/runner-api.integration.test.ts
src/services/native-runtime/runner-api-artifacts.test.ts
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/__tests__/teams-catalog-service.test.ts
src/__tests__/primary-agent-guidance.test.ts
src/__tests__/skill-import-discovery.test.ts` (in `server/`): 2418
passed, 2 skipped.
- `npx vitest run src/__tests__/runner-api-recovery.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts
src/services/native-runtime/confirmation-comment-api.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/runner-api-log-redaction.test.ts
src/services/native-runtime/runner-api-artifacts.integration.test.ts
src/services/native-runtime/native-case-tools.integration.test.ts
src/__tests__/connection-management.integration.test.ts`: 404 passed, 1
failed. The failure is `runner-api-artifacts.integration.test.ts >
preserves exact HTTP bytes in the immutable asset and downloaded helper
input` ("Artifact installation is unavailable to this run or provider").
It fails the same way with the unchanged `master` files in my
environment, so it is not from this change. CI covers it.
- Red before the fix: with the `master` version of
`recovery-effect-receipts.ts`, the new settlement and supersession cases
in `task-retry.test.ts` fail.
- New tests:
- `task-retry.test.ts`: settlement by recorded status 200, 422, 500, and
a record of a different call only; supersession by a later user, agent
or system change, including a change before the run finished and a read
marker.
- `runner-call-outcome.test.ts`: records the final status one time;
ignores reads, a missing or malformed key, agent API keys, board actors
and non-UUID run ids.
- `runner-api.test.ts`: the call key header is sent with mutations only.

## Risks

- Each run mutation that carries a call key adds one activity row
(`runner.api_completed`, entity `heartbeat_run`). The row has no request
body and no response body. Only the status and the receipt key hash.
- The settlement trusts the server's own final status. Routes commit
before they respond, so a 2xx means the effect is applied. A route that
answers 2xx before it commits its effect is still treated as applied.
- A run can write its own outcome record with its own token. This can
only settle that run's own receipts, so it does not give an agent new
access.
- Receipts made before this change have no record. They keep the fence
until a board user changes the task.
- No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 16:45:51 -08:00
DottaandClaude Opus 5.5 e82c4f0a31 fix(runner): prove a warm Runner's suspension on a slow sandbox host so the next run does not fail (#21472)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**

This PR extends the failed-close retry that #21442 (merged) added. It is
based on `master`, which contains #21442.

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote sandbox is reused by the next run of the same task.
Before the next run starts, the previous run's Runner must be retired:
suspended, with durable proof.
> - An idle warm close had 10 s for everything. On a loaded sandbox host
one liveness probe or one read of the durable state used that up. The
suspend was never sent, or it completed but was never proven.
> - The next run then replayed that cached close failure, failed before
any provider work, and the task showed a failed run with Retry
> - For a follow-up on a task blocked on its subtasks, the automatic
continuation was refused and the Retry stayed until another wake
> - This pull request bounds each probe, gives the suspend proof its own
read window, lets a later close prove the suspension again, and lets
that follow-up continue by itself
> - The benefit is that a slow host no longer turns a completed
suspension into a failed task, and the person sees no Retry for it

## Linked Issues or Issue Description

**What happened?**

Local release-swap cells on Daytona (G5c and G6c, 2026-11-13) each
showed one Retry banner: "previous run's session could not be retired
(runner did not durably suspend before checkpoint)". G6c had no fault
injection, so this is a real class.

- In the G6c case, the Runner completed its suspend and exited. The
server's proof read (a sandbox exec under load) ran out of the remaining
0.6 s of the close grace, so the suspension was "not proven".
- In the G5c case (an adopted Runner), the liveness probe
`adoptedRunner.isAlive()` took about 220 s and used the whole close
deadline. The suspend was never queued. Five adopted sessions failed the
same way in the same second.
- The successor run's retirement called `transport.close()`, which
returned the cached rejection of the predecessor's failed close. It did
not send a new suspend and did not read the state again.
- The successor failed before provider work. Its fresh-sandbox
continuation (`interrupted_preparation`) was refused as "blocked after
recovery", because the parent task was blocked on its subtasks before
the follow-up started. The task showed a failed run with Retry for 27 s,
until a blocker-stall wake.

**Expected behavior**

A completed suspend is proven, even on a slow host. A follow-up that
fails before any provider work on a task that was already blocked
continues by itself, and the task shows as working while that
continuation is queued.

**Steps to reproduce**

1. Close an idle warm Runner whose durable state read takes longer than
the close grace, or whose liveness probe stalls.
2. On master the close fails and the next close replays the same
failure.
3. The new tests in `runnerd-codex-transport-close-proof.test.ts` and
`heartbeat-native-predispatch-retry.test.ts` do this.

**Paperclip version or commit**

`master` at c6b497d831 (contains #21442).

## What Changed

- `runnerd-codex-transport.ts` (controller-side TypeScript):
- Each liveness probe in the close is bounded. A probe that does not
answer counts as possibly alive, never as gone, and the suspend goes
over the live connection. A connected Runner's probe gets a 1 s slice,
so drain and suspend keep their budget.
- Once the suspend command's completion is observed, what the
durable-state read shows decides how long the proof may take:

    | Read after the completion | Proof window | Maximum close time |
    | --- | --- | --- |
    | Suspended, right Runner | proven at once | unchanged |
| Another run's or Runner's state | ends at the close deadline, as on
master | close grace (10 s; 45 s for Pi) |
| Right Runner, not yet `suspended` | settle window of 3 s from the
first such read (`RUNNER_SUSPENSION_PROOF_SETTLE_MS`, test option
`closeProofSettleMs`) | close grace + 3 s |
| Unreadable or slow read | full proof window of 30 s from the
completion (`RUNNER_SUSPENSION_PROOF_READ_MS`, `closeProofReadMs`) |
close grace + 30 s |

All rows add the existing bounded exit-probe slices. runner-core saves
the `suspended` lifecycle before it delivers the suspend result
(`persist_lifecycle_before_command_delivery` in `durable/runner.rs`), so
a readable non-suspended state after the completion can only be a stale
remote read; the settle window covers a few read attempts and is not a
write-ordering wait.
- The settlement diagnostic names the branch that ended the window
(`proofWindowEnd`: `proven`, `wrong_identity`,
`not_suspended_after_settle`, `unreadable_after_window`,
`suspend_not_completed`).
- A close that failed only because readable proof was missing is retried
by the next close: it reads the durable state again with a fresh bound
and finishes settled. A close whose proof was refuted stays memoized, as
on master. This extends the failed-close retry from #21442.
  - The close failure carries per-step timings.
- `native-runtime/predecessor-retirement.ts`,
`heartbeat/run-completion.ts`: the predecessor-retirement fallback
records the task's status when the run failed.
- `modules/run-dispatch/domain/policy.ts`,
`modules/run-dispatch/adapters/postgres.ts`: a continuation of a run
that did no provider work, on a task that was already blocked then, is
not treated as blocked after recovery. Its dependency check follows the
wake it continues. The continuation keeps a person's comment wake.
- `task-interruption-presentation.ts`: while that run's failure handling
queues the continuation (up to 30 s), the task shows as working with no
Retry. The window ends when the continuation is scheduled, when the run
records `continuationScheduling: "refused"` (written by
`run-completion.ts` when the fallback does not schedule it), or after 30
s. The fallback record alone no longer ends it.

## Verification

Red on master, green on this branch:

- `runnerd-codex-transport-close-proof.test.ts`
- `proves a completed suspend on its own read window when the durable
state read outlasts the close grace`
- `suspends five adopted idle Runners whose liveness probes stall
together, instead of spending the close on the probe`
- `heartbeat-native-predispatch-retry.test.ts`
- `continues a follow-up on a blocked task whose predecessor could not
be retired`
- `does not continue a task that was blocked only after its run failed`
- `task-interruption-presentation.test.ts`
- `shows the task as working while its fresh-sandbox continuation is
being queued`
- `keeps the task working after the fallback is recorded while its
continuation is still being scheduled` (review fix)
  - `offers Retry when the continuation was refused or never queued`
- `runnerd-codex-transport.test.ts` (review fixes)
- `gives a suspend that completes near the close deadline its whole
proof window from the completion`
- `bounds the proof window from the observed completion, so a proof read
that never answers still ends`
- `proves a suspension whose suspended state appears 1.5 s after the
command completed`
- `fails a suspension whose state stays ready after the settle window,
not after the whole proof window`
- `fails at the close deadline when the durable state belongs to another
run`
- The three existing tests in
`runnerd-codex-transport-suspension.test.ts` that expect a conclusive
negative to fail within the close grace pass unchanged.

Local runs on this branch: `tsc --noEmit` is clean for `server` and
`packages/paperclip-runner`. Passing on this branch:
`runnerd-codex-transport-suspension`, `-close-proof`, `-adoption`,
`-remote-retention`, `runnerd-detach-durability` (all files), the
proof-window cases of `runnerd-codex-transport.test.ts`,
`task-interruption-presentation.test.ts` (72), `run-dispatch` policy and
Postgres adapter tests and `heartbeat-native-predispatch-retry.test.ts`
(166 together).

Conflict resolved: `task-interruption-presentation.ts` gained
`WAITING_CONTINUATION_MOVE_FORWARD_MS` on master (#21385). Both
constants are kept.

Runner content ID: the Runner change is controller-side TypeScript under
`packages/paperclip-runner/src` only. There is no Rust, `protocol/` or
`generated/` change. Existing Runner images work with this controller.

Daytona: this fix has not run on Daytona at all. The two banners above
are the evidence of the class (G5c and G6c ran on 588c6ba267, without
this fix).

## Risks

- Not proven on Daytona.
- A close can now take up to 30 s longer when the proof read is
unreadable or slow, and up to 3 s longer when the read is stale. This is
on the close path only, not before `turn.start`.
- A probe that does not answer now counts as possibly alive, so the
close sends a suspend instead of treating the Runner as gone.
- Ownership: this PR changes
`server/src/modules/run-dispatch/domain/policy.ts`,
`server/src/modules/run-dispatch/adapters/postgres.ts` and
`server/src/services/heartbeat/run-completion.ts`. CODEOWNERS does not
list these files. Their recent changes come from the restart-recovery
and Retry work (#21348, #21256, #20863, #21302). That area should review
the dispatch-policy change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:32:22 -06:00
Nicky LeachandClaude Opus 5.5 6d8ebc4a85 fix(secrets): mask secret-bound environment values in run output (#21341)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can bind a company secret to an environment variable for an
agent, an environment, a project, or a routine
> - Core resolves these values for each run and gives them to the agent
process or to the Runner
> - Core stores the run output: the run log, the run excerpts, adapter
events, and native (Runner protocol) transcript events
> - The run-output redactor did not know the resolved secret values.
Only failure diagnostics and the `meta.env` keys of `adapter.invoke`
were masked
> - So when an agent ran `env` or `echo $STRIPE_KEY`, Core stored the
secret in plaintext, and every person who can view the run could read it
> - This pull request gives the resolved values of secret-bound keys to
the run-output redactor and to the native control-plane port
> - The benefit is that an accidental print of a secret does not put the
secret into stored run output, for local and native runs

## Linked Issues or Issue Description

**What happened?**
An agent printed a secret-bound environment variable during a run (for
example with `env` or `echo $STRIPE_KEY`). Core stored the value in
plaintext in the run log, in `stdoutExcerpt` and `stderrExcerpt`, and in
the transcript events. Anyone who can view the run can read the value.

**Expected behavior**
Core masks the value of each secret-bound environment variable before it
stores run output. Core does not mask plain (non-secret) configured
values.

**Steps to reproduce**
1. Create a company secret and bind it to an agent environment variable
with `secret_ref`.
2. Configure the agent so that its command prints the variable.
3. Run the agent and open the run log. The value shows in plaintext.

**Related**
Refs #21330. That PR keeps configured secrets out of OpenCode's own
storage and model requests inside the Runner. This PR is the Core side:
it masks secrets in the run output that Core stores, for every run type.
The two changes are independent.

## What Changed

- `secretEnvironmentValues(env, secretKeys)` in
`agent-identity-redaction.ts` selects the raw values of keys in
`secretKeys` (the keys that came from `secret_ref` or `user_secret_ref`
bindings). `secretOutputForms` derives the forms to mask: the full
value, the value without outer whitespace, the JSON-escaped form, and
each line of a multi-line value (PEM armor lines are not added). It
skips a form that has fewer than 8 characters after trimming. This is
the same 8-character limit that diagnostic redaction uses for unknown
values.
- The output redactor keeps these secrets in a separate set, beside the
existing agent private-key values. The private-key values are masked
everywhere, as before. Environment secrets are masked only in output
text. They are not masked in protocol fields: `schema`, `type`, `kind`,
`status`, `state`, `outcome`, `stream`, `channel`, `role`, `level`, ids,
refs, codes, revisions, versions, digests, timestamps, and names that
end in these words (for example `eventType`, `turnTerminalState`,
`reportedWorkDisposition`). So a secret that equals a protocol word (for
example `completed`) cannot change an event type or a terminal state.
- The streaming match now scans left to right. At each position it tries
the longest value first. If a longer value could still start at that
position, it holds the text until the next chunk, also when a shorter
value already matches there. So the tail of a longer secret cannot leak
when a shorter secret is its prefix.
- `prepareHeartbeatExecutionConfiguration` adds the resolved secrets to
the run's output redactor immediately after it resolves them. This one
redactor already masks the local adapter log stream, the excerpts, live
log events, adapter events, `adapter.invoke` metadata, the adapter
result, and the canonical native log export.
- For a native run, `executeHeartbeatRuntime` stores the delivered
secrets (only values that have a form to mask) encrypted under a new
context key, `paperclipOutputSecretRedactions`, in one transaction. It
then adds all stored values of the run to the redactor. A run that
reconnects to a live Runner after a `"latest"` secret rotation then
masks the value that the Runner holds and the new value. This store
reuses the encryption and fingerprint logic of the run redaction
registry, but it is separate: API read redaction does not apply its
values, so a secret cannot change ids, event names or terminal states in
API responses. API reads hide the new key like the existing registry
key.
- `executeHeartbeatRuntime` gives the native executor the run's secrets
and the observability credentials (`observabilitySecretValues`).
- The native executor gives the secrets to its failure-diagnostic
redactor and to `PaperclipControlPlanePort` (`secrets`). The port masks
them in each PRP event before it persists the event, with the existing
delta-aware handling for values that are split across `item.delta`
events. This covers native runs on the local Runner and on plugin-hosted
agent computers.
- After a controller restart, a new control-plane port cannot know the
text that the old port held back, because that text is never stored. On
the first delta of each stream it has not seen, the port masks leading
text that matches the tail of a secret (4 or more characters). It skips
text that can start a secret, and it keeps checking the stream while a
delta falls entirely inside such a tail.
- The native executor classifies failures from the original error. It
masks only the agent private key in place, as before. It masks
environment secrets in a separate redactor that it uses only for text
that it writes or logs (`failureDetail`, failure messages, stderr
diagnostics). Core masks the rethrown error before it stores it.
- `coalesceIntoActiveExecutionRun` (wake queue) writes the merged
context with `withRunRedactionHistory`, which takes
`paperclipSecretRedactions` and `paperclipOutputSecretRedactions` from
the locked current row. Before, a coalesced wake built from an older
context could delete either key. This also protects the existing
registry. This is a change outside the secrets module, in the one place
that replaces a running run's context concurrently.
- When an adapter or native run fails, the heartbeat keeps the original
error for retry decisions and rethrows it. The copies that it saves in
`workspace_finalize` metadata (`errorMessage`) and in a failed session
goal action go through the run's output redactor first.
- `docs/deploy/secrets.md` describes the mask and its limits.

## Verification

- `server/src/__tests__/run-output-secret-env-redaction.test.ts` (new,
embedded Postgres): a real `process` adapter run with a `secret_ref`
bound to `"latest"`. The agent prints the secret split across two
writes, in JSON, and on stderr. The stored log, run row, and run events
do not contain the value. The plain value and a 5-character secret are
not masked. The test then rotates the secret and runs again. The second
run masks the rotated value, and a SHA-256 digest in the output proves
the run received the rotated value. Red-green: it fails without the new
line in `execution-configuration.ts`.
-
`server/src/services/native-runtime/paperclip-control-plane-port.test.ts`:
the port masks a secret in a persisted tool event and in a value split
across two `item.delta` events; plain and short values stay (red-green:
fails without the port change). A new test binds secrets equal to
`interrupted`, `completed`, `succeeded` and `satisfied`, then appends a
`turn.interrupted` event, checkpoints, and completes the run. The event
type, payload status, terminal states and criterion status stay
unchanged and the result is saved (red-green: fails when protocol fields
are masked).
- `server/src/services/heartbeat/runtime-execution.test.ts`: a value
stored by an earlier attempt, the current value, and an observability
header value all reach native dispatch and the heartbeat redactor. Short
values are not stored. The shared run redaction registry stays empty,
and an API read hides the new key (red-green: fails without the store
read).
- `server/src/services/native-runtime/native-session-executor.test.ts`:
a native failure that contains the secret is masked in failure writes,
logs and the rethrown error, and the executor gives the secrets to the
control-plane port.
- `server/src/services/agent-identity-redaction.test.ts` (new): value
selection, the minimum length, multi-line and JSON-escaped forms, nested
values, every split point of a streamed value, overlapping secrets
(`abcdefgh` and `abcdefgh1234567890`) split at the shorter value,
protocol fields in events, terminal states and deltas, and secrets added
by a later attempt.
- Restart, retry and coalescing (each fails without its fix): a port
test restarts between two deltas of a secret and stores `echo ` and
`***REDACTED*** done`; an executor test binds `timed out` and keeps
`provider_session_open_timeout`; a wake-queue adapter test keeps both
redaction keys when a coalesced wake replaces the context.
- `server/src/services/heartbeat/run-output.test.ts`: the local output
path masks the log, excerpt, live event, adapter event, and adapter
metadata.
- `pnpm --filter @paperclipai/server typecheck` passes. The changed test
files and the other users of the redactor (`agent-identity`, `run-log`,
`heartbeat-case-handoff`, `native-run-log-export.integration`) pass in
full locally: 1986 passed, 7 skipped; the 6 run-redaction-registry test
files also pass (231 tests) (macOS, embedded Postgres). `pnpm verify`
exits 2 (broader verification required) because `docs/deploy/secrets.md`
is a global CI input, so required CI supplies broad coverage.

## Risks

- This is a guard against accidental leaks. It is not a security
boundary. An agent can change a value before it prints it (for example,
encode it in base64 or print one character at a time), and Core cannot
detect that.
- Core does not mask a secret value or line that has fewer than 8
characters. This prevents a short value from erasing ordinary output. A
secret that short is not masked in run output.
- Core does not mask environment secrets in protocol fields (see What
Changed). If a secret appears in such a field, it stays. These fields
hold protocol words and generated identifiers, not agent output.
- If a value is common text of 8 or more characters, Core masks that
text wherever it occurs in that run's output text. This only affects
runs that bind that secret.
- If a key is secret-bound in one source and a later source sets the
same key to a plain value, `secretKeys` still includes the key, so Core
masks the plain value. This is the existing meaning of `secretKeys`.
- For native runs, Core now stores the delivered secrets encrypted in
`contextSnapshot` under `paperclipOutputSecretRedactions`, with the same
local encryption as the existing run redaction registry. It adds one
transaction per native run that has secrets. API reads do not apply
these values and hide the key.
- After a controller restart in the middle of a secret, a tail shorter
than 4 characters can still be stored. A new stream that starts with the
tail of a secret (4 or more characters) is masked at its start even
without a restart. This is a small amount of extra masking.
- Core masks values at write time. Run output that Core stored before
this change is not changed.
- No schema change and no migration. The new fields are optional and
in-process only.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 16:14:41 -08:00
defed64af9 fix(native-runtime): retry a native failure write after a brief lock timeout (#21494)
An interrupted native turn records its same-run resume in one transaction
under a 1 s lock_timeout. When another transaction briefly held a row that
the write needs (for example the agent row that the new recovery action
references), the insert failed with 55P03 and the whole write rolled back.
The run kept its dead lease, and the task needed reconciliation instead of
the recorded automatic resume.

Retry the whole failure write on transient lock contention with the
existing bounded helper. A rolled-back attempt has no effect, so a fresh
attempt is safe. The regression test holds the agent row until one attempt
is cancelled by its lock timeout, so only a retry can complete the write.

Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:02:04 -08:00
Nicky LeachandClaude Opus 5.5 5df1ada9c1 feat(plugins): send configured environment on plugin task resume (#21352)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21335 lets a plugin task provider that restarts the Runner on
`resume` opt in (`resumeCredentialVersion: 1`) to receive the claimant
run's harness credential
> - A provider restarting the Runner also needs the agent's configured
environment, which Core sends on `resume` only during a session handoff
> - Without it, a Runner a provider moves to a new computer starts
without the agent's configured environment variables
> - This pull request widens that opt-in to both launch inputs, under a
name that says so, before any provider adopts the old name
> - The benefit is that a provider can restart a parked Runner on resume
with the same credential and environment a `submit` would carry

## Linked Issues or Issue Description

Refs #21289
Follows: #21335

**What existing behavior does this improve?**
The opt-in resume launch inputs added in #21335.

**Subsystem affected**
Plugin SDK environment task schema (`describe` capability), the plugin
Runner task owner's `resume`, and the plugin environment task docs.

**Current behavior**
With `resumeCredentialVersion: 1`, `resume` carries `credential`.
`configuredEnvironment` is sent on `resume` only for a session handoff.

**Proposed behavior**
The capability is renamed `resumeLaunchInputsVersion: 1`. With it,
`resume` carries `credential` as before and also `configuredEnvironment`
(same builder as `submit`) when the provider advertises
`configuredEnvironmentVersion: 1`. Session handoff is unchanged.

**Reason and benefit**
A provider that restarts the Runner on resume gets both launch inputs it
needs.

**Breaking changes**
The capability name changes from `resumeCredentialVersion` to
`resumeLaunchInputsVersion`. No provider advertises the old name yet (it
merged in #21335 and has no adopter), so nothing changes behavior.

## What Changed

- Rename `resumeCredentialVersion` → `resumeLaunchInputsVersion` in the
SDK `describe` result schema, executor plumbing, owner input, docs and
tests.
- With the capability, `resume` also adds `configuredEnvironment` when
the provider advertises `configuredEnvironmentVersion: 1`. A normal
resume does not mark the variables as delivered, since a live Runner
keeps the environment it started with.
- Docs: the "Release and resume" section describes both fields; the
"Configured environment" section points to it.
- The runner-e2e SDK public-input pin is updated.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts`: 68 passed.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts`: 153 passed, including a normal
resume with the capability (credential sent, variables added only with
`configuredEnvironmentVersion: 1`, delivered flag unset) and redaction
of both fields from metadata writes and sanitized errors.
- `native-session-executor.test.ts -t "uses a task-only"`: 5 passed.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts`: 46 passed.
- `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- A provider restarting the Runner gets the claimant run's current
variables, which can differ from the ones the parked Runner started
with. That is intended.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:46:37 -08:00
4cf04d64cb fix(native): retry a transient endpoint failure after a same-run bootstrap restart (#21488)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts a Runner in a remote sandbox for each
agent run. A bounded same-run retry (at most 3 startup attempts) handles
transient startup failures.
> - A server restart can stop a run during its first Runner bootstrap.
Restart recovery then claims `bootstrap_incomplete`. It stops and
quarantines the unfinished bootstrap and launches a fresh bootstrap on
the same run.
> - The executor builds its "unclaimed bootstrap" proof only when
`restartRecovery` is not set. Thus the fresh bootstrap of a
`bootstrap_incomplete` restart can never prove that a transient failure
is safe to retry.
> - A transient HTTP 502 from the Runner endpoint request in that fresh
bootstrap was then classified as `ambiguous_state`. The run stopped and
needed manual board recovery.
> - This pull request gives the unclaimed-bootstrap proof to a
`bootstrap_incomplete` restart as well as to a first attempt.
> - The benefit is that a transient endpoint failure after a restart
uses the bounded automatic retry and does not need a person to recover
the run.

## Linked Issues or Issue Description

Refs #20414 (related: it fixes executor admission of a
`bootstrap_incomplete` restart at state-root migration. It does not
change the retry evidence that this PR changes.)

**What happened?**
A server restart interrupted the first Runner bootstrap of a remote
native run. Restart recovery claimed `bootstrap_incomplete` and launched
a fresh bootstrap on the same run. The fresh bootstrap registered its
controller checkpoint. Then `runner.ingress.acquire` failed with HTTP
502, before any Runner connection or provider event. The run ended with
`nativePhase: terminal_failure`, `recoveryOwner: board`, attempt 1 of 3,
and the next action "resolve the ambiguous session state". The error was
`Runner connection endpoint could not be acquired (HTTP 502).`
(`runner_ingress_acquisition_failed`).

**What did you expect to happen?**
The transient 502 occurred before the Runner connected and before any
provider work. It should use the bounded same-run startup retry, as it
does on a first attempt.

**Steps to reproduce**
1. Start a native run on a remote sandbox target.
2. Restart the server during the first Runner bootstrap, so that restart
recovery claims `bootstrap_incomplete`.
3. Make the next `runner.ingress.acquire` call return HTTP 502.
4. See that the run stops as ambiguous state and does not schedule a
startup retry.

## What Changed

- Add `attemptLaunchesFreshBootstrap()` to `native-bootstrap-proof.ts`.
It returns true for a first attempt and for a `bootstrap_incomplete`
restart. It returns false for `reattach_existing_runner`,
`reattach_remote_runner` and `resume_dead_runner`, because these
continue an existing Runner or provider session.
- In `native-session-executor.ts`, use this helper in place of
`!input.restartRecovery` when the executor builds the
unclaimed-bootstrap evidence for a startup failure.
- Add `native-bootstrap-proof.test.ts` for the helper.
- Add a case to `native-startup-retry.integration.test.ts`. It runs the
real executor under a `bootstrap_incomplete` restart claim, and the
fresh bootstrap gets a typed HTTP 502 from the Runner endpoint before
the Runner connects.

## Verification

- `native-bootstrap-proof.test.ts`: the test "accepts a same-run restart
of an incomplete bootstrap" failed before the fix and passes after the
fix (5/5 tests pass, vitest 5.1.0).
- `native-startup-retry.integration.test.ts`, case "transient 502 after
a bootstrap_incomplete restart": the run must settle as
`retryable_failure` with `recoveryMode: "bootstrap_retry"`, attempt 1
and a set `nextAttemptAt`. Without the executor change, the executor
builds no bootstrap evidence and the run settles as `ambiguous_state`.
CI runs this suite; I could not run it locally.
- The other guards for a retry did not change. The checkpoint must be
unclaimed, there must be no provider events and no session checkpoint,
and the 3-attempt startup budget applies. If the earlier attempt opened
a provider session, provider events exist and the run still fails closed
as ambiguous.
- I could not run the full typecheck and the real-Runner integration
suites locally (no Rust toolchain in the sandbox). CI must run them.

## Risks

- Low risk. The change only affects a `bootstrap_incomplete` restart
whose fresh bootstrap fails during startup. Such a failure now gets the
same retry decision as a first attempt. All the other evidence checks
still apply, so a run with provider activity still stops for manual
recovery.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with
tool use (shell, file edit, and code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:37:24 -08:00
DottaandClaude Opus 5.5 342fb308bd perf(run-events): commit streaming text deltas in batches (#21460)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - Text deltas (`item.delta`) are about 85% of that process's SQL
statements: each delta is one transaction of four statements on the
shared 10-connection pool
> - At 50 agents and 20 deltas per second, the process falls behind and
unrelated agent requests wait for the pool
> - This pull request commits text deltas in batches: one transaction
for all deltas that arrive within 250 ms, with one multi-row insert
> - The benefit is about 7 times fewer transactions and 5–6 times fewer
statements per delta, with no delay for the first text of an item

## Linked Issues or Issue Description

Refs #21435 (this pull request is stacked on it). Refs #21131. Item 1a
of the agent scaling plan in #21418.

**What existing behavior does this improve?**
The persistence of native text deltas. `native-session-runtime` awaited
`PaperclipControlPlanePort.appendEvent` for each event, and the port ran
one transaction per event.

**Subsystem affected**
Server run event ingest (`paperclip-control-plane-port.ts`,
`heartbeat-run-events.ts`) and the runner package's turn consumer
(`native-session-runtime.ts`).

**Current behavior**
Each delta takes one transaction: BEGIN, run lock, append, COMMIT. In
the load model with 50 agents (deltas only), that is about 4.0
statements and 1.0 transaction per delta.

**Proposed behavior**
The turn consumer starts the next append without waiting for a delta's
receipt. The port collects deltas and commits them in one transaction
with one insert. In the same load model, that is about 0.7 statements
and 0.14 transactions per delta.

**Reason and benefit**
Fewer transactions free the shared pool and the main thread for other
agents' requests. This is the largest per-agent cost in the scaling
plan.

**Breaking changes**
None. The row shape does not change. A port without the new capability
keeps the earlier serial behavior.

## What Changed

- `appendHeartbeatRunEventBatch` (`heartbeat-run-events.ts`) appends
native source events of one run and one source in one transaction. When
the events continue the stored source cursor (from #21435), it sends
four statements for the whole batch: the run lock, one replay probe, one
`insert ... select from jsonb_array_elements(...)` of every row, and the
commit. Any other batch (a replay, a gap, an event with a wake, or a run
without a cursor) appends its events one by one in the same transaction,
with the single-append checks.
- **Storage choice: one row per delta, written by one statement per
batch.** I did not store one row per batch. The transcript, live events,
run log export, chat progress, replay (`replayEvents`), recovery cursors
and the UI all read one row per delta. One statement per batch keeps
every reader unchanged, and rows written before this change read the
same.
- `PaperclipControlPlanePort` sets `pipelinesStreamingEvents`. An
`item.delta` appended with the new `streaming` option goes into a queue:
- It commits at once until a delta with text commits after the latest
other event. So the first text of each item, and the time to first
token, never wait for the timer.
- Later deltas commit after at most 250 ms, or at once when the batch
has 100 deltas.
- Every other event, `checkpointSession`, `completeRun`, `replayEvents`
and `provesNoRunProviderWork` first commit the queue. A non-streaming
event is never committed before the deltas that preceded it.
- A delta appended without `streaming` commits at once, so callers that
wait for each receipt are not slowed.
- The receipt of a delta settles only after its batch commits and its
observers (live event, executor callback) run, in order.
- If a batch fails, every delta in it rejects and nothing from it is
stored.
- `native-session-runtime` `consumeTurn` appends deltas with `streaming`
and does not wait for their receipts (at most 256 pending). Before any
other event, any result, and the governed-wait settlement, it calls
`flushStreamingEvents()` and then waits for all pending receipts, so a
boundary never waits for the timer. A failed delta stops the turn before
a later event is appended. The consumer joins every pending delta append
before it settles, also when the provider stream fails.
- If one observer of a committed batch fails, only that delta's receipt
rejects. The other rows are still published and observed.
- `ControlPlanePort` contract: optional `pipelinesStreamingEvents`,
`flushStreamingEvents()` and `AppendControlPlaneEventOptions.streaming`.

## Verification

- Red/green. The new tests failed before the change and pass after it:
- Runner: "starts text delta appends without waiting and flushes them
before any other event" (`native-session-runtime.test.ts`). The port in
the test has no timer, so the turn finishes only if the boundary
flushes. Before: the appends were serial, and later the boundary waited
for the timer.
- Runner: "joins pending delta appends before a failed turn settles".
Before: the turn settled while a delta commit still ran.
- Port: "observes every committed delta of a batch when one observer
fails".
- Port: "commits the first text delta at once and later deltas in one
batch before the next other event". It checks one transaction and one
insert for five deltas, flush before a non-streaming event, receipts
only after commit, and immediate commit of the first delta with text.
Before: each delta committed alone.
- Port: "flushes queued deltas on the timer and before a replay read,
and rejects every delta of a failed batch". It checks the
crash-before-commit case: a revoked writer fails the batch, nothing is
stored, and the resent deltas commit idempotently (the stored one is a
duplicate).
- Store: four new tests in `heartbeat-run-event-atomic-append.test.ts`:
one lock, one probe and one insert per batch; replay idempotency and
payload conflict; a batch that fails before commit stores nothing and
its resend commits; the one-by-one path for no cursor and for gaps.
- Focused suites pass: `heartbeat-run-event-atomic-append`,
`heartbeat-run-event-sequencing`, `paperclip-control-plane-port`
(including the package conformance suite),
`paperclip-semantic-conformance`, `native-observer-replay`,
`native-event-boundary-golden`, `live-recovery-work`,
`native-interaction-bridge`, `chat-native-progress-work`, and
`native-session-runtime.test.ts` in the runner package.
- `pnpm --filter @paperclipai/paperclip-runner run typecheck:typescript`
and `pnpm --filter @paperclipai/server run typecheck` pass. `pnpm
check:module-boundaries` passes.
- Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418,
`AGENTS=50 SECONDS=20`, local embedded PostgreSQL). The load model calls
the store directly, so for this run I added a local `BATCH_MS` mode that
applies the port's batching rules (first delta at once, 250 ms timer,
100 cap, flush before other events) and does not wait for delta
receipts. That mode is not in this pull request. The machine was very
busy (load average 40–60), so absolute figures are noisy; per-delta
ratios are the stable result:

| Deltas only | Statements/s | Transactions/s | Deltas done of offered |
Statements per delta | Transactions per delta | Lateness p50 / p99 |
  | --- | --- | --- | --- | --- | --- | --- |
| One transaction per delta | 2,920 | 730 | 14,668 of 20,000 | 3.98 |
1.00 | 3,074 / 5,384 ms |
| Batched (250 ms) | 704 | 141 | 19,940 of 20,000 | 0.71 | 0.14 | 668 /
2,858 ms |
| One transaction per delta (repeat) | 1,712 | 428 | 8,575 of 20,000 |
3.99 | 1.00 | 6,558 / 11,464 ms |
| Batched (250 ms, repeat) | 856 | 172 | 20,000 of 20,000 | 0.86 | 0.17
| 187 / 556 ms |

With all sources on, the machine was saturated (event-loop utilization
0.96–0.98) in both modes and the results were not usable (764 and 12,800
of 20,000 deltas done).

## Risks

- **Ack and crash (read first).** In the live runnerd path the runner's
ack does not wait for the `heartbeat_run_events` row today.
`DurablePrpControlPlane.#event` acks after the in-memory journal commit.
Streaming events are not in the controller checkpoint. So a delta that
is acked but not yet appended is lost if the server crashes, and the
runner does not resend it. This is the earlier behavior. This pull
request makes that window up to 250 ms longer. The final text still
survives through `item.completed`, which commits immediately and in
order. This pull request does not change `durable-prp-control-plane.ts`.
- Live text in the UI arrives in steps of up to 250 ms after the first
text of an item.
- A delta failure now surfaces at the next non-delta event or at the end
of the turn, not at the delta. The turn stops before it appends a later
event.
- The governed-wait drain and the control-event replay still append one
event at a time. They do not carry deltas.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:34:01 -06:00
Devin FoleyandPaperclip 12aa08721c feat(ssh): use the agent AI account in new environments (#21248)
Use the agent-selected AI account for new SSH environments. Preserve
remote-machine authentication for existing environments and expose it as
an advanced option. Keep managed credentials private while preserving
supported conversation state and credential refreshes across turns.

Validated with focused route, adapter, lifecycle, and real-process tests;
final CI passed on d634944df1.
Greptile Apex reviewed that revision at 5/5 with no unresolved threads.
External neutral security reports remain documented in the pull request.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 13:20:13 -08:00
DottaandClaude Opus 5.5 b5adf35119 test(runner): guard that a relaunch after a lost launch acknowledgement leaves one Runner (#21477)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) that the
server launches in a sandbox.
> - A deploy can stop the server after the launch command reached the
sandbox but before the server read back the Runner's process identity (a
lost acknowledgement).
> - The database then has no PID for the run, so the successor server
launches the Runner again on the same state folder.
> - Two Runners must never serve the same run's state. #20852 proposed a
large launch-intent design for this window. Since then, #21079 added a
fence that stops a leftover Runner on the same state folder before each
launch.
> - This pull request adds a Linux regression test that drives the real
launcher through the lost-acknowledgement window.
> - The benefit is that a future change to the launch path cannot bring
back a second Runner without failing this test.

## Linked Issues or Issue Description

Refs #20852 (superseded: current master already prevents the duplicate
Runner through #21079). Refs #21079.

**What happened?**
#20852 claimed that a relaunch after a lost launch acknowledgement could
start a second Runner for the same run. Nothing proved whether current
master still has this problem.

**Expected behavior**
After a lost acknowledgement, the successor's relaunch stops the
orphaned Runner. Exactly one Runner serves the run's state folder, and
the process marker names it.

**Steps to reproduce**
1. Launch a remote Runner, and lose the launch RPC's acknowledgement
before the identity read (the old server stops).
2. Relaunch the same run on the same sandbox from a new launcher (the
successor).
3. Count the live `paperclip-runnerd` processes that serve the run's
`--state-dir`.

## What Changed

- New
`server/src/services/native-runtime/remote-runner-lost-launch-ack.test.ts`.
It is a test only and changes no product code.
- It runs only where `/proc` exists (Linux CI), like the existing #21079
fence tests.
- It drives the exported `createRemoteRunnerProcessLauncher` with a
local command runner. The old launcher's runner runs the launch command,
then throws `remote_launch_ack_lost`. After that, every command fails,
because the old server is gone.
- A small bash stand-in for `paperclip-runnerd` keeps the Runner's
argv[0] and `--state-dir`. Its `reap-lineage` reports a verified, empty
lineage.
- The test checks these results: the orphan was alive and had no PID on
the handle; the successor's Runner has a new PID; the orphan stops;
exactly one Runner serves the state folder; the marker names the
successor's PID.

## Verification

- Linux reproduction (Docker `node:24-bookworm-slim`). I used the exact
outer launch command and `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT`, which I
extracted from this head:
- master scripts: `runners after successor relaunch [61], orphan alive
false`. Result: one Runner (PASS).
- the same scripts without the #21079 stale-state fence
(counterfactual): `runners after successor relaunch [24, 56], orphan
alive true`. Result: duplicate Runner (FAIL).
- macOS: `npx vitest run
src/services/native-runtime/remote-runner-lost-launch-ack.test.ts`
reports the file as skipped (1 skipped), as intended, because there is
no `/proc`.
- The test runs for real only on Linux CI. CI on this head is the first
real run of the vitest file.

## Risks

- Low risk. The change is a test only.
- The test starts real local processes. Its teardown kills only the PIDs
it found on its own temporary state folder, and it never signals PID 0.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, Docker, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:46:24 -06:00
DottaandClaude Opus 5.5 11f4a76743 perf(native-runtime): share one frozen assigned tool catalog across sessions (#21482)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every native agent session. Each session
gets the app tools (MCP tools) that are assigned to its agent.
> - Each live or warm session kept two private copies of its assigned
tool catalog: the gateway descriptors and a canonical copy of every
input schema.
> - With 400 tools of about 5.7 KB each, that was about 7.6 MB of heap
for each session. With 800 tools, it was about 15 MB. Warm sessions stay
for 5 minutes after the last turn, so idle agents also hold this memory.
> - At 1,000 concurrent agents, this is 7 to 15 GB of heap, and the
server cannot get there.
> - This pull request keeps one deeply frozen catalog for all sessions
of the same company that get the same admitted tools and schemas.
> - The benefit is that each additional session on the same catalog adds
about 40 to 80 KB of heap instead of 7.6 to 15 MB, and session start is
faster.

## Linked Issues or Issue Description

Refs #21415 (releases a finished session's tool authority). Refs #21440
(adds the catalog memory benchmark that this change extends).

**What existing behavior does this improve?**
`createAssignedMcpTools` in
`server/src/services/native-runtime/assigned-mcp-tools.ts`. Each call
kept every gateway descriptor (with its provider metadata) and a
separate canonical clone of each input schema. Each `definitions()` call
also made a new `structuredClone` of every schema.

**Subsystem affected**
Native runtime: assigned app tools (MCP) for native agent sessions.

**Current behavior**
Retained heap grows by one full catalog for each session. Measured with
the real `createAssignedMcpTools` (5.7 KB schemas): 400 tools is 7.58 MB
per session, and 800 tools is 15.1 MB per session, for 1, 10, and 100
sessions. Each `definitions()` call takes 14 to 41 ms because it clones
every schema.

**Proposed behavior**
Sessions of the same company that get identical admitted tools share one
frozen catalog. The total for 100 sessions is about one catalog: 4.2 MB
at 400 tools and 8.1 MB at 800 tools. `definitions()` takes 0.01 ms.

**Reason and benefit**
The server must hold 1,000 concurrent agents in one process. The catalog
was the largest per-session heap cost. Revocation and authorization do
not change: every call is still authorized again by the gateway, and
every new session still reads a fresh listing.

**Breaking changes**
None in the API or data. The direct definitions that `definitions()`
returns are now frozen. A consumer that writes to a returned definition
or schema gets a `TypeError` and does not change other sessions. No
production consumer writes to them (see Risks).

## What Changed

- `server/src/services/native-runtime/assigned-mcp-tools.ts`:
- A session keeps only a small frozen record for each tool (gateway
name, display name, description, risk), plus the frozen schema and the
frozen definition. It does not keep the gateway descriptors.
- New per-process index of shared catalogs. The key is the SHA-256 of an
identity string: `[companyId, [[assignedName, gatewayName, displayName,
description, risk, sha256(JSON of the input schema)] for each listed
tool, in listing order]]`. On a hit, the code also compares the full
identity string, so a hash collision cannot give a wrong catalog.
- The identity is computed from the fresh listing that the gateway
returns for this session. The gateway applies all per-agent, per-run,
per-profile, and policy filtering, and the approval description suffix,
before it returns the listing. Thus two sessions share only if they were
given exactly the same tools, names, descriptions, risks, and schemas in
the same company. Without `companyId`, nothing is shared.
- Schemas, definitions, tool records, and the definition arrays are
deeply frozen. Search results still return cloned copies.
- Sessions that start at the same time with the same catalog wait for
one build and share it.
- The index holds each catalog through a `WeakRef`, so it never keeps a
catalog alive. The index has at most 512 entries and 256 MiB (accounted
schema and description bytes). The oldest entries are removed first. A
catalog that is larger than the byte limit is not indexed.
- New `close()` on the assigned tools. When the last session that holds
a catalog closes, the index entry is removed. A `FinalizationRegistry`
also removes entries whose catalog was collected.
- `definitions()` returns the shared frozen definitions for the
session's work mode. It no longer clones every schema on each call.
-
`server/src/services/native-runtime/paperclip-runner-tool-authority.ts`:
`close()` also closes the assigned tools. The session executor already
closes the authority when it revokes a session's tool authority.
- `server/src/services/native-runtime/native-session-executor.ts`: one
line. It passes the run binding's `companyId` to
`createAssignedMcpTools`.
- `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: passes
`companyId`, adds `--companies`, measures session start and
`definitions()` time, and returns fresh copies of fixed rows (like
database reads).
- Tests in `assigned-mcp-tools.test.ts` and
`paperclip-runner-tool-authority.test.ts`.

## Verification

Red/green tests in
`server/src/services/native-runtime/assigned-mcp-tools.test.ts` ("shared
assigned tool catalogs"):

- Retained heap: 20 sessions of one company on a 60-tool, 4 KB-schema
catalog. The heap for the other 19 sessions must be less than two
catalogs. Red on master: `expected 14025976 to be less than 3788112`.
Green with this change.
- Shared deeply frozen schemas, also for sessions that start together.
Red on master (`Object.is` equality fails).
- Mutation isolation: writes to a returned definition, schema property,
and `required` array, and a delete, throw `TypeError`. Changes to the
gateway's source rows and to a search result do not change any session.
Red on master (no throw).
- Release and bounds: the entry stays while one session holds it and
goes when the last session closes. `close()` is idempotent. A new
session after release builds a new catalog. 517 companies keep the index
at 512 entries or fewer. Red on master (no `close` or stats).
- Guard tests: two companies with identical catalogs do not share.
Sessions without a company do not share. Fewer tools, an edited schema,
an edited description, or a different risk do not share. A revoked or
edited tool is served from fresh discovery. On master these pass,
because master never shares. To prove they test the key, I removed
`companyId` and the schema digest from the identity: three guard tests
failed, and they pass with the full key.
- An existing test changed: "does not project gateway secrets..." now
expects a write to a returned schema to throw `TypeError` (before, the
write changed only a private clone). The isolation check after the write
is the same.
- `paperclip-runner-tool-authority.test.ts`: `authority.close()` closes
the assigned tools.

Heap (post-GC `heapUsed` delta, `node --expose-gc`, 5.7 KB schemas, one
company). Command: `cd server && node --expose-gc --import tsx
scripts/benchmark-assigned-mcp-tools-memory.ts --tools <N>
--schema-bytes 5700 --sessions <S> --turns 5`.

| Tools | Sessions | Total before | Per session before | Total after |
Per session after |
| --- | --- | --- | --- | --- | --- |
| 400 | 1 | 7.60 MB | 7.60 MB | 3.96 MB | 3.96 MB |
| 400 | 10 | 75.9 MB | 7.59 MB | 4.04 MB | 0.40 MB |
| 400 | 100 | 757.9 MB | 7.58 MB | 4.18 MB | 0.042 MB |
| 800 | 1 | 15.2 MB | 15.2 MB | 7.91 MB | 7.91 MB |
| 800 | 10 | 151.4 MB | 15.1 MB | 7.98 MB | 0.80 MB |
| 800 | 100 | 1,513 MB | 15.1 MB | 8.13 MB | 0.081 MB |

The first session also uses about half the heap of before, because the
session no longer keeps the gateway descriptors. When all sessions are
dropped without `close()`, the catalog is collected (0 bytes held after
release).

Session start, interleaved A/B in one process (old and new
`createAssignedMcpTools` alternate, 40 starts each, median of the last
35, includes a fresh copy of the listing; the machine had a load average
of 25 to 65):

| Tools | Listing copy only | Before | After |
| --- | --- | --- | --- |
| 400 | 22.6 ms | 68.0 ms | 37.6 ms |
| 800 | 30.5 ms | 72.5 ms | 41.2 ms |

A start that reuses a catalog does no schema clone and no admission. It
still serializes and hashes each schema to compute the key. The first
start of a new catalog does the same admission work as before (the cold
first start was about 600 to 750 ms in both versions on this busy
machine).

`definitions()` median: 14 to 41 ms before, 0.01 ms after.

Other checks:

- `cd server && npx vitest run
src/services/native-runtime/assigned-mcp-tools.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts
src/services/native-runtime/runner-api.integration.test.ts
src/services/native-runtime/runner-tool-results-prp.integration.test.ts
src/__tests__/deployment-connections.test.ts`: 204 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"assigned|tool authority|released|MCP"`: 31 passed.
- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "assigned"`: 10
passed.
- `cd server && npx tsc --noEmit`: pass.
- `pnpm check:module-boundaries`: pass.
- `pnpm verify` selects a broad plan (the benchmark script is a broad
input), so CI supplies the full suite.

## Risks

- Frozen definitions: the direct definitions now go to the runner
drivers as frozen objects. I checked the consumers.
`PaperclipRunnerToolAuthority.definitions()` filters and spreads them.
The Codex, Cursor Cloud, OpenCode, ACPX, and runnerd drivers read them,
spread them, clone them (`structuredClone` in ACPX and the OpenCode MCP
bridge), or serialize them. Their types are `readonly
Readonly<Record<string, unknown>>[]`. The first-party semantic tool
definitions are already shared module constants that go through the same
path without clones. If a consumer that I did not find writes to a
definition, it gets a `TypeError` instead of a silent change. Catalogs
over 256 tools or over the listing budget use the two on-demand tools
and are not affected.
- Authorization: the shared data is only the result of a listing that
the gateway already authorized for that session, keyed on the full
content plus the company. Execution still goes through the gateway with
the session's own token on every call. Search still intersects with a
fresh listing. A new session still reads a fresh listing, so a revoked
or edited tool takes effect for new sessions and for existing sessions
at the same points as before.
- Several server processes: the index is a per-process memo of the
listing content. Each process builds its own catalog from its own
listing. Nothing is stored or shared outside the process.
- The index is bounded (512 entries, 256 MiB) and holds catalogs only
weakly. Index eviction only stops future sharing. It never frees memory
that a session uses.
- No schema change. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs, heap and timing benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:30 -06:00
DottaandClaude Opus 5.5 754b1f36d3 perf(native): renew all of a process's execution leases in one transaction, without a recovery wake (#21444)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every running native agent. Each run
holds a native execution lease that the process must renew before it
expires (60 seconds for a remote run, 20 minutes for a local run)
> - Each run renewed its lease on its own timer, in its own transaction,
on a private pool of two connections. At 1,000 remote runs that is about
50 transactions each second through two connections
> - When renewals queue behind each other, a renewal can miss its 60
second deadline, and the run is lost although the server is healthy
> - Each renewal also woke six live-run recovery queues for its run. The
recovery owner then re-checked a healthy run about every 20 seconds,
although a renewal only moves the deadline later
> - This pull request renews all leases of one process with one shared
timer. All leases that are due go into one transaction of fixed size,
the same model that the legacy controller lease already uses. A renewal
no longer wakes recovery, as #20924 did for the legacy lease
> - The benefit is that the renewal cost no longer grows with the number
of runs, and all safety properties of a single renewal stay the same

## Linked Issues or Issue Description

Refs #20924 (the same change for the legacy controller lease), #21242.

**What existing behavior does this improve?**
Native execution lease renewal in
`server/src/services/native-runtime/native-execution-lease.ts`. Each run
had one `setInterval` and one transaction per renewal: statement
timeout, row lock, recovery notification, update, commit.

**Subsystem affected**
Server: native runtime, controller lease renewal on the liveness
connection pool.

**Current behavior**
The renewal cost grows with the number of runs. A local load model with
1,000 remote runs sends 35.7 lease transactions and 212.8 lease
statements each second, and the event-loop utilization is 0.34. All of
them use a pool of two connections.

**Proposed behavior**
One timer for each database client renews all due leases in one
transaction for each 200 runs. With 1,000 remote runs the same model
sends 0.29 lease transactions and 1.7 lease statements each second, and
the event-loop utilization is 0.07.

**Reason and benefit**
The plan for 1,000 concurrent agents needs renewals that do not scale
with the run count (item 1d). A renewal that misses its deadline loses
the run.

**Breaking changes**
None. The `NativeExecutionLeaseRenewal` contract (`stop()`,
`remainingMs()`) and `renewNativeSessionExecutionLease` do not change.

## What Changed

- `native-execution-lease.ts`: a renewal (batched or single) no longer
calls `notifyLiveRunRecovery`. A renewal only moves the lease deadline
later. Recovery already holds a deadline at the earlier expiry. At that
deadline it re-reads the row and moves to the new expiry. A lapsed lease
is still found at its expiry. This change does not touch the signals of
the writers that release, revoke or take over a lease, or that change
the run status. This also removes six notifications for each renewal,
which was 1,200 notifications for a batch of 200 runs during a deploy
overlap.
- `native-execution-lease.ts`: one shared schedule for each database
client replaces the timer of each run. When the first lease is due, all
leases that are due within a quarter of their own interval renew with
it. Thus the leases of a process converge on one batch for each
interval.
- A batch is one transaction on the liveness pool for each 200 runs:
`statement_timeout`, one `SELECT … FOR UPDATE SKIP LOCKED` of the
coordinator rows, and one `UPDATE … FROM (VALUES …)`.
- The batch keeps the guarantees of a single renewal:
- It locks each coordinator row before it reads the expiry, as before.
Thus an owner whose lease expired while another writer held the row is
never revived.
- It extends a row only for the exact owner: run, company, task, lease
owner, attempt, controller boot id, pid, process start, and a live
lease.
- The confirmed deadline is sampled after the locks and before the
update, as before.
- A run whose row was locked and not extended fails with
`native_session_lease_lost`. The other runs in the batch succeed.
- These rows do not renew in the batch. They renew alone through the
unchanged single-run path:
- A row that another writer holds (`SKIP LOCKED`). Thus one busy row
cannot hold the batch until the statement timeout.
- A row that the lock did not find (another owner, or no row). The
single path gives the same answer as before.
- A local lease that lapsed (host sleep). Only the single path may take
it back, under the run row lock.
- A batch that is cancelled (statement timeout `57014` or deadlock
`40P01`) extends nothing. Its runs retry after 1 second, and each run
keeps its own deadline timer. Before this change, a statement timeout
waited for the next 20 second tick.
- Any other batch failure, also an error thrown while the statement is
built, may come from one bad row. Each run of that batch then renews
alone at once through the single path, so a bad row can cost only its
own run. A run that fails alone while others in its batch renew keeps
renewing alone (out of later batches) until a single renewal succeeds.
During a database outage the single renewals also fail; those runs
return to the batch at their next interval, not after a 1 second retry,
so an outage causes at most one queued single transaction for each run
and interval.
- Runs that become due while a batch is in flight wait for that batch. A
batch that runs for more than 15 seconds no longer holds back the next
batch.
- A tick that passed while a renewal was in flight is skipped, as a
fixed interval skips it. Thus a slow database gets no extra load.
- The shared timer replaces the interval timer of each run. It exists
only while this process holds a lease, and it fires at the next renewal
deadline. With no held lease there is no timer and no SQL.
- New export `renewNativeExecutionLeasesNow(db)` renews all leases of
the process at once. Tests use it.

## Verification

- New PostgreSQL tests in `native-execution-lease-batch.test.ts`:
- 3 runs and 60 runs each renew in 1 transaction with the same number of
statements.
  - 450 runs renew in 3 transactions.
- In one batch, the runs whose lease was taken over, whose controller
boot changed, or whose lease expired fail with
`native_session_lease_lost`. Their rows do not change. The healthy runs
renew. A newer attempt under the same owner is not lost, as before.
- Interleaving: another transaction holds a coordinator row from before
its lease expires until after it expired. The healthy run in the same
batch renews at once. After the lock is released, the expired owner is
lost and its row stays expired.
- A takeover that commits before the batch: the batch extends nothing
and the run is lost.
- A local and a remote lease renew in one transaction, each for its own
lease length.
- A poison binding (an attempt outside the integer range) makes the
shared UPDATE fail with `22003`. The other 4 runs still renew in the
same tick, only the poison run is lost, and the next tick renews the 4
runs in 1 transaction again.
- An invalid date throws while the statement is built. The other runs
renew, the broken run is lost alone, and no lease stays busy (the next
call renews again in 1 transaction).
- New PostgreSQL test in `live-recovery-work.test.ts` with the real
registered recovery owners: another server's native run renews its
lease. No recovery pass runs after the renewal. At the old expiry
recovery re-reads the row, keeps the run running and moves its deadline
to the new expiry. When renewals stop, recovery acts on the run within
10 seconds after the end of the last lease.
- Red first: with a shim on `origin/master` that renews each lease
alone, the transaction test failed (`expected 3 to be 1`) and the
450-run test failed.
- The unit tests in `native-execution-lease.test.ts` now drive the
batch. New tests show that a non-cancelled batch failure renews the run
alone at once, and that an outage (batch and single renewal both fail)
does not retry every second and loses no run.
- `native-execution-lease-reacquire.test.ts`: the host-sleep tests now
go through the batch, which hands the lapsed local lease to the single
path.
- Load model (`startNativeSessionExecutionLeaseRenewal` for N remote
runs that start at random times over 20 seconds, 70 second window, local
PostgreSQL, machine load average about 50):

| Runs | Lease transactions/s before → after | Lease statements/s before
→ after | Event-loop utilization before → after | Smallest margin to the
60 s deadline | Lost |
| --- | --- | --- | --- | --- | --- |
| 50 | 1.81 → 0.11 | 10.9 → 0.69 | 0.015 → 0.095 (noise) | 40.0 s → 40.0
s | 0 → 0 |
| 1,000 | 35.67 → 0.29 | 212.8 → 1.71 | 0.341 → 0.065 | 39.0 s → 40.0 s
| 0 → 0 |

The window includes the 20 second ramp, so the steady rate before is
higher (one transaction each 20 seconds for each run, 50/s at 1,000
runs). After the change, one batch carries up to 200 runs.

## Risks

- **Behavior change: batch-wide error handling.** One shared transaction
now carries up to 200 runs, so one database error reaches all of them. A
cancelled batch only skips the extension and retries after 1 second. Any
other batch error sends each run to the single path at once, so one bad
row (a bad value, a cast error, an error while the statement is built)
costs only its own run. In a database outage, up to one single
transaction for each run waits on the two-connection pool at each
interval. Each run's own deadline is unchanged.
- **Behavior change: no recovery signal on renewal.** See below.
- During a deploy overlap the old server renews its runs one by one and
the new server renews in batches. Both use the same row lock and the
same conditions, so they do not conflict. Rows written by the old
version have the same columns.
- `SKIP LOCKED` sends a busy row to the single path, which waits for the
lock as before.
- **No recovery signal on renewal:** recovery of a healthy run of
another server now runs once for each lease period at the old expiry,
not after each renewal. A dead run is found at the same time as before:
at the expiry of its last lease.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:16 -06:00
DottaandClaude Opus 5.5 89f486cdab fix(heartbeat): run one dispatch pass per agent, back off refused claims, and enforce the run limit in the claim (#21433)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts an agent's queued runs. Each trigger (a
run finishes, a wake arrives, a sweep runs) asks for one dispatch pass
of that agent
> - An agent could run more runs than its `maxConcurrentRuns`, dispatch
passes could stack up without a limit, and every pass offered the same
unclaimable runs again
> - On hosted staging (e23549cd9c), one agent with limit 20 had 20–35
runs `running` and 170–270 queued. The logs showed 375 "offered a free
slot but could not be claimed" lines and 117 "claim was rejected" lines
in 10 minutes
> - This pull request runs one pass per agent at a time, folds triggers
into one follow-up pass, backs off refused claims, enforces the limit in
the claim transaction, and reads only the columns the pass needs
> - The benefit is that an agent never runs past its limit (also across
two servers during a deploy), and a burst of triggers costs one or two
passes, not one pass per trigger

## Linked Issues or Issue Description

Refs #21391, #21242, #21234, #21277, #21324, #21349

**What happened?**

On hosted staging (Core e23549cd9c), one agent ("Codie",
`maxConcurrentRuns` 20) had 170–270 queued runs and 20–35 runs
`running`, so it was over its limit. The logs showed a refused claim
offered again on every pass. This pull request is about correctness and
load. It is not the cause of the staging out-of-memory crashes: a
separate leak (`sessionToolAuthorityEpochs` in
`native-session-executor.ts`, about 3 MB per finished native run)
explains those, and another change fixes it.

Live again at 17:02Z on staging (Core 04f12fd2be, which includes
#21391): the same agent had 52 runs `running` against
`maxConcurrentRuns` 40. All 52 started between 16:55Z and 17:02Z, right
after a deploy released the backlog, with 112 runs still queued. So
overlapping passes exceed the limit with claimable runs, not only with
unclaimable ones.

**Why did it happen?**

1. `withAgentStartLock` chained each trigger behind the previous one and
measured its 30 s stale limit from when the predecessor was chained, not
from when it started. When triggers arrived faster than one per pass,
every trigger after about 30 s of backlog started its own pass next to
the others.
2. Each pass counted free slots by itself, outside any lock that other
passes or another server see. Overlapping passes and two servers during
a deploy swap each claimed up to the limit.
3. Each pass read every queued run with `select *`, including the large
`contextSnapshot`, and offered every refused run again with no backoff.

A local reproduction of the staging shape by another session (250
unclaimable queued runs with 8 KB snapshots, one trigger per second, on
e23549cd9c and again on master e7e28ba241): passes that run at the same
time went 1 → 12 → 39 → 79 → 139 at t = 30/45/75/116/178 s, only 4
passes finished in 178 s, and post-GC heap grew by about 2.8 MB per
stacked pass (519 MB → 1,061 MB at 190 passes, mostly the parsed
`heartbeat_runs` rows and the dispatcher facts each pass holds). 242 s
after the triggers stopped, 216 passes were still outstanding.

**Expected behavior**

At most one dispatch pass per agent runs at a time. Triggers that arrive
during a pass cause one follow-up pass. The `running` count never goes
above `maxConcurrentRuns`. A refused run is offered again after a
bounded wait, and no queued run is stranded.

**Steps to reproduce**

1. Give an agent a limit of 4, 12 aged queued runs that the chat-control
gate refuses, and 12 claimable runs.
2. Start a pass. While its first claim waits, send 40 triggers, each
after more than 30 s of the old lock's clock.
3. Count the passes, the refused offers, and the highest `running`
count.

## What Changed

- `server/src/services/agent-start-lock.ts`: replaces the stale-bypass
lock with `dispatchForAgent`. One pass runs per agent. Triggers during a
pass merge into one follow-up pass that all their callers share. A
trigger from inside the agent's own pass does not wait on itself. A pass
has a 60 s deadline on a monotonic clock, measured from when the pass
started (not from when a waiter was chained). At the deadline the pass
stops claiming (an abort signal) and the follow-up may start. A trigger
that arrives after the deadline also ends the pass, so a stalled timer
cannot keep an agent blocked. Post-pass effects run after the agent is
free, as before.
- `server/src/services/heartbeat/run-slot-admission.ts` (new):
`assertAgentRunSlotFree` counts the agent's `running` runs and compares
them with `maxConcurrentRunsSql()` inside the agent-scoped advisory lock
that every claim already takes (`withExternalAdmissionGuard`). It throws
`AgentRunSlotsFull` when the agent is full. The count leaves out the run
that the claimed run replaces (`retryOfRunId` or `previousRunId`), so a
successor that starts while its predecessor finishes is not refused.
- `server/src/services/heartbeat/queue.ts`:
- Every claim path (ordinary, queued comment, native review through
`native-review-dispatch.ts`) checks the slot inside its claim
transaction. A full agent leaves the run queued, and the pass stops.
- The pass reads only the columns the dispatcher needs, with `issueId`,
`wakeReason` and `previousRunId` taken from `contextSnapshot` in SQL. It
reads the full row only for the run it is about to claim.
- A refused claim (declined or 4xx) of a run that has waited more than
an hour (the existing aging limit) is not offered again for 5 s, then
10, 20, 40, and at most 60 s. A person's run of that age waits 2 s. A
run queued for less than an hour is never backed off, so a successor or
retry that waits for its predecessor starts as soon as the predecessor
ends. A backoff also ends when the run's task changes (status,
`updatedAt` or execution run) or when a run of that task ends in this
process. A backed-off run does not take the oldest-run share. The
queued-run sweep (`resumeQueuedRuns`, `resumeQueuedRunsAfterDrain`,
which the queued-run redrive calls) offers refused runs again regardless
of the backoff.
- A pass makes at most 24 claim attempts. When it reaches that limit it
asks for one follow-up pass that skips every run the chain already
tried, so the follow-up reaches runs further down the line. The chain
ends when a pass does not reach the limit, so refused runs cannot cause
a continuous chain of passes. Only this automatic follow-up carries the
tried list. A wake, a completion or another fresh trigger that merges
into it resets the list, so a run whose blocker cleared during the chain
is offered again.
- Admission, suppression and cutoff are read once per pass, not once per
trigger.
- Seam for a skip before the claim: in `runQueuedDispatchPass`
(`claimQueuedRunsForAgent`), right after the skip line (`if
(triedRunIds.has(candidate.id) || (!retryRefused &&
claimBackedOff(candidate.id, new Date(), taskStamp))) continue;`),
before the claim-budget check, `claimAttempts += 1` and the full-row
read. A `continue` there spends no claim attempt, reads no row, does not
enter the claim backoff or the aged-decline note, and does not call
`dispatcher.claimed`, so the run keeps its place and its oldest-run
share. The dispatcher exposes `dispatcher.personOrigin(run)` (the
person-origin fact it already derives) and `dispatcher.personRun(run)`.
The candidate rows keep `id`, `companyId`, `agentId`, `status`,
`invocationSource`, `retryOfRunId`, `wakeupRequestId`,
`scheduledRetryAt`, `scheduledRetryAttempt`, `scheduledRetryReason`,
`createdAt`, `startedAt`, and
`contextSnapshot.issueId/wakeReason/previousRunId`.
- Each pass is registered with the execution drain
(`drainActiveRunExecutions`) until it ends or reaches its deadline.
- Tests: new `server/src/services/agent-start-lock.test.ts` and
`server/src/__tests__/queued-run-dispatch-stacking.test.ts`.
`server/src/__tests__/heartbeat-start-lock.test.ts` is removed: it
asserted the 30 s stale bypass that this pull request removes.
`workspace-recovery-preparation-dispatch.test.ts` holds and counts
passes through the new function.

The chat-control fallback lookup from #21391 (`issueId = X OR
nativeIssueId = X OR contextSnapshot->>'issueId' = X`) was measured on
embedded Postgres with 200,000 runs in one company. Postgres uses a
BitmapOr of three existing indexes
(`heartbeat_runs_company_issue_created_idx`,
`heartbeat_runs_company_native_issue_contract_id_uq`,
`heartbeat_runs_company_ctx_issue_created_idx`), not a sequential scan:
0.33–0.45 ms per call for a recent previous run, a 30-day-old previous
run, and no previous run. It is not changed.

## Verification

Red/green, `server/src/__tests__/queued-run-dispatch-stacking.test.ts`
(embedded Postgres, real heartbeat queue, limit 4, 12 refused aged runs,
12 claimable runs, 41 triggers):

| Measure | master | this pull request |
|---|---|---|
| Passes | 42 | 2 |
| Peak passes at the same time | 40 | 1 |
| Peak claims in flight | 9 | 1 |
| Refused-claim offers per trigger | 12 | 0.29 (12 in total) |
| Peak `running` (limit 4) | 8 | 4 |
| Passes outstanding after the burst | — | 0 |

- "waits a bounded time before offering a declined/rejected run again":
on master 10 triggers offer the refused run 10 times. With the fix they
offer it once, and once the backoff passes it is offered again. The
rejected case uses the 422 "Unable to resolve responsible user" path
from the staging reproduction.
- "starts a run that becomes ready during a chain of passes when a fresh
trigger arrives": red at 3d48c4bed2, green now.
- "stops chaining passes once every refused run was tried": 60 refused
runs and one trigger cause exactly 3 passes, then no more.
- "starts runnable work queued behind more refused person runs than one
pass tries": 30 refused person runs ahead of 3 runnable runs, with free
slots. The runnable runs start after one trigger, in at most 3 passes.
At 68e64a2de9 the same 24 person runs were offered on every pass and the
runnable runs never started.
- `server/src/services/agent-start-lock.test.ts`: one pass for a burst
of 50 triggers plus one follow-up, no second pass after 45 s, a hung
pass stopped at its deadline without overlap, no self-wait, settle after
release, and errors passed to the caller.
- `node server/scripts/typecheck-layers.mjs`: pass.
- Vitest (embedded Postgres): `heartbeat/queue.test.ts` (102),
`recovery-wake-herd.test.ts` (35),
`heartbeat-queued-run-claim-isolation.test.ts`,
`workspace-recovery-preparation-dispatch.test.ts`,
`heartbeat-restart-claim-lapse.test.ts`,
`issue-reassignment-queued-run.test.ts`,
`queued-comment-consumption.test.ts`,
`issue-queued-comment-queue.test.ts`, `chat-run-milestone-work.test.ts`.

## Risks

- Refusals of runs queued for less than an hour are not backed off. A
backoff there delayed successors that wait for in-process predecessor
ownership, which can end without a task change (two CI tests failed with
it). Per trigger, such a run costs at most one claim attempt.
- A refused run that has waited more than an hour now waits up to 60 s
(a person's run 2 s) before an event-driven pass offers it again, unless
its task changes. The sweep and the redrive ignore the backoff.
- A successor may start while the run it replaces is still `running`.
The limit check leaves that one predecessor out of the count.
- Claims of an agent that is at its limit now fail inside the
transaction. A run that `executeRun` claims directly (for example a
native resume) stays queued when the agent is full, and the next pass
starts it.
- A pass that passes its 60 s deadline can still finish one in-flight
claim next to the follow-up pass. The limit check in the transaction
keeps the `running` count within the limit.
- No schema or migration change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:55:38 -06:00
aa4a06653c fix(native): name the recorded turn failure when a native run fails (#21235)
Failed native runs now report the runner's recorded turn-failure code instead of the generic unclassified fallback, and the Sentry run-failure fingerprint includes runner:<code>.

Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:22:55 -08:00
DottaandPaperclip 6ad4529c5a fix(native): preserve pack proof across interrupted reattachment (#21246)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server can connect again to a live Runner after a restart.
> - Each new controller claim changes provider-pack admission to
pending.
> - The claim kept the previous verified proof only in local memory.
> - If recovery failed before its callback, the next claim lost that
proof.
> - This pull request saves a small reference to the verified proof with
the claim.
> - The next claim can check the same receipt without weakening
admission checks.

## Linked Issues or Issue Description

**What happened?**
A restart recovery attempt timed out before its admission callback. The
next attempt refused the saved pack with `source_state_pending`. It then
failed with `native_provider_artifacts_admission_unverified`.

**Expected behavior**
The next attempt can reuse the previous verified proof for the same
receipt, session, Runner, and lease. Current admission must remain
pending until its normal callback succeeds.

**Steps to reproduce**
1. Save verified provider-pack admission for a run.
2. Claim that run again. Commit the pending admission state.
3. Interrupt recovery before the session admission callback.
4. Claim the same run again and reconnect to the same Runner.
5. Observe that the old code refuses the saved receipt.

Related work: #20682 and #20850 address lock waits. #21085 adds refusal
logs. #21161 addresses command waits after restart. This change
addresses lost admission proof across interrupted claims. The original
transport timeout remains a separate investigation.

## What Changed

- The admission module prepares the claim patch and its recovery source.
- The claim saves a versioned reference with the source run, receipt
digest, and previous verified state. It adds no copy of provider files
or credentials.
- A later pending claim can use that reference only if the receipt is
unchanged. Normal company, agent, session, Runner, and lease checks
still apply.
- Missing proof, unknown state, a changed receipt, or a different Runner
stays refused. The callback clears the reference when it records its
decision.
- The executor uses this helper under its existing claim lock. Existing
artifact-retention and ownership checks stay in place.
- The execution guide states the previous-release behavior. Old verified
records seed the reference. Old pending records without a reference
remain refused. Older readers ignore the field and keep their existing
refusal behavior. No migration is needed.

## Verification

- The new interrupted-claim regression failed before the fix: admission
returned false instead of true.
- Focused admission, PostgreSQL fencing, and executor lease tests passed
after the rebase: 61 tests on commit c8123cc35d.
- The database case saves two interrupted claims, reads their persisted
state, checks a different Runner is refused, and publishes verified
admission for the original Runner.
- Negative cases cover missing proof, unknown state, a changed receipt,
a different source run, a different Runner, and a different lease.
Existing company and claim fences also pass.
- `pnpm verify` returned exit 2 because the execution guide requires
broader CI. Required CI must cover the final commit.
- Local server typecheck was killed with SIGKILL, including its
single-thread retry. This is an environment gap. Required CI must
provide typecheck and build results.
- Repository preflight installed dependencies and built shared packages.
It could not finish Runner preparation because cargo is absent.
- A live OpenCode restart workflow and recovery of the original task are
not verified. No task replay or deployment was performed.

## Risks

- The added profile field carries historical evidence. It does not mark
the current claim verified. Its use is limited to an unchanged receipt
with the normal identity checks.
- Existing pending or unknown incidents without this evidence cannot
gain proof from this fix. They keep their normal recovery path.
- No provider attempt limit, approval gate, workspace hold, or company
boundary changes.

## Model Used

OpenAI GPT-6, through Codex, with code search, tool use, and local test
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:56:57 -06:00
DottaandClaude Opus 5.5 89e6575418 test(native-runtime): cover released session tool authority and add a catalog memory benchmark (#21440)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs get their tools from a session tool authority,
which holds the run's assigned app (MCP) tool catalog
> - #21415 releases a finished session's authority, so the server heap
no longer grows with each finished session
> - Two behaviors of that fix had no test: a released session fails
closed, and the release when a follow-up run retires its warm
predecessor and then fails before it admits its own authority
> - The memory saving was measured once, by hand, with no script in the
repository
> - This pull request adds the two tests and a benchmark script. It
changes no product code
> - The benefit is that a later change cannot silently bring back the
leak or let a released session reach its tools

## Linked Issues or Issue Description

Refs #21415 (the fix these tests cover). Refs #21430 (closed duplicate;
a reviewer comment there named the follow-up failure path).

**What is missing?**

#21415 tests that finished sessions return the authority map to its
starting size. No test checks that a released session refuses late tool
calls. No test checks the follow-up path, where a follow-up run retires
the previous warm session (so the release at forget time is skipped
while the follow-up owns the scope) and then fails before it admits its
own authority.

**Expected behavior**

A late tool call or completion feedback for a released session fails
with `native_session_tool_authority_unavailable`. The next run of the
same scope gets a new authority with only its own assignments. When a
follow-up fails after it retires its predecessor, the authority map is
back at its starting size after that run settles.

**Paperclip version or commit**

master `edb6945ff3` (after #21415, `85f4a91e70`).

## What Changed

- Test "refuses a released session's late tool call and completion
feedback and admits the next run with only its own assignments".
- Test "releases a retired warm predecessor's tool authority when the
follow-up fails before admitting its own". A follow-up of the same warm
scope with changed configuration retires the predecessor. Then its
assigned tool listing fails. The test asserts that the predecessor was
closed, that the follow-up failed with that error, and that
`sessionToolAuthorityCount()` is back at its starting value.
- `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: retained
bytes of one assigned tool catalog per session, and schema copies per
`definitions()` call.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "released
native session tool authority|retired warm predecessor"`: 2 passed.
- Red check: with the release at settle removed from
`native-session-executor.ts`, both tests fail (`expected 1 to be +0`,
and the late tool call is not refused).
- Retained bytes per finished per-turn session were measured through
`executePaperclipNativeSession` with the real `createAssignedMcpTools`
(80 tools, 2 KB schemas, 100 sessions, forced GC, `heapUsed` delta). On
`85f4a91e70^` (before #21415): 604,566 bytes. On `85f4a91e70`: 19,794
bytes (test-harness noise), and the authority map stayed at its starting
size.
- `cd server && node --expose-gc --import tsx
scripts/benchmark-assigned-mcp-tools-memory.ts --tools 80 --schema-bytes
2048 --sessions 200 --turns 20` reports about 546 to 567 KB retained per
assigned tool catalog. Each `definitions()` call makes 80 new schema
copies (about 296 KB), and none are kept.
- Full `native-session-executor.test.ts` locally, with a load average
near 78 from other work on the machine: 1,483 passed and 5 failed. All 5
failures were timeouts in tests this change does not touch (the slowest
took 163 s). CI gives the clean full-suite result.

## Risks

- Low risk. Tests and one benchmark script only. No product code
changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:17:25 -06:00
DottaandClaude Opus 5.5 df1b12ad8e fix(runner): advertise flat tool schemas and stop repeated runner-side input rejections (#21398)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools. The runner shows each tool to
the model with an input schema and checks every call against that schema
before it runs
> - Some tool schemas had a root `oneOf`, `anyOf`, `allOf` or
`if`/`then`. `handoff_case` had no top-level `properties` at all
> - Some model hosts convert such a schema into an empty argument list.
The model then calls the tool with no arguments, and the runner rejects
the call as "Invalid tool input"
> - The repeated-failure guard from #21360 does not see these
rejections, because they happen in the runner before the call reaches
the control plane. The model sent the same rejected call again and again
> - This pull request gives every agent-facing tool a plain object
schema with named properties, checks the old cross-field rules with
named errors, and adds a runner-side guard for identical "Invalid tool
input" rejections
> - The benefit is that models on these hosts can see and send the
arguments, and a dropped argument ends in a clear correction or a stop,
not a long loop

## Linked Issues or Issue Description

**What happened?**
In a chat eval cohort with OpenCode and
`openrouter/deepseek/deepseek-v4.1-flash-1017`, the hosts Wafer and
DigitalOcean served most model calls. In the case handoff case,
`handoff_case` arrived with no arguments on all 10 calls across two
cases. The runner rejected each one with "must have required property
idempotencyKey, case, toAgentId, ...". The model's reasoning said that
the tool showed an empty property list. The model fell back to
`call_api`, but the reply waiter had already expired, so the case
failed. In the preserve-draft case, `request_human_input` was rejected 9
times in a row with the same root `then` error.

**Expected behavior**
Every tool that an agent sees has a plain object schema with named
properties, so every host can show its arguments. When the same call is
rejected twice with the same arguments, the model gets a correction that
lists the expected properties. On the third identical call, the model is
told to change the arguments or stop.

**Paperclip version or commit**
`master` at f7ddab8e94.

**Steps to reproduce**
1. Run an OpenCode agent with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, where OpenRouter routes
to Wafer or DigitalOcean.
2. Ask the agent to hand a case to another agent with `handoff_case`.
3. See `handoff_case` arrive with no arguments and get rejected on every
call.

## What Changed

- Add `providerToolSchemaShapeIssues`
(`packages/paperclip-runner/src/contracts/provider-tool-schema-shape.ts`).
It reports a root that is not `type: "object"`, has no `properties`, or
uses `oneOf`, `anyOf`, `allOf`, `not`, `if`, `then`, `else`,
`dependent*` or `$ref`.
- Add inventory tests that run this check on every agent-facing tool:
the live semantic catalog, the semantic action catalog, the Codex tool
specs, `paperclip_finish` and `paperclip_block` (runner), and the
server-only tool definitions (server). Before this change, the runner
test listed 16 entries for 8 tools: `handoff_case`,
`request_human_input`, `read_case`, `update_case`, `create_case`,
`read_workflow`, `update_workflow` and `paperclip_finish`.
- Flatten these 8 schemas. `handoff_case` is now one object with `to`
(optional, `"done"`), and `case` is one object with optional `id` and
`create`. Descriptions on the properties state which fields each variant
needs.
- Add `toolInputVariantError`
(`packages/paperclip-runner/src/contracts/tool-input-variants.ts`). It
checks the former cross-field rules and returns an error that names what
is wrong and how to send the call correctly. For example: `Invalid tool
input for handoff_case: An agent handoff requires toAgentId, phase. To
finish an existing case instead, send to: "done" with
expectedHandoffVersion and expectedIssueStatusVersion.`
- The server runs this check first in
`PaperclipRunnerToolAuthority.execute` and returns `422`. The #21360
guard therefore counts these errors too. Assigned MCP tools and
connector tools are not checked. The runner semantic dispatchers run the
same check after schema validation. The runner MCP bridges run it for
`paperclip_finish`, which never reaches the control plane.
- Add `RepeatedToolInputRejectionGuard`
(`packages/paperclip-runner/src/drivers/repeated-tool-input-rejection.ts`)
and use it in the OpenCode MCP bridge and the runner tool bridge, where
runner-side validation happens. The key is the tool, the canonical
arguments and the error text, separated per MCP session. The second
identical rejection adds the arguments that arrived and the expected
required and optional properties. The third and later identical calls
say that the call was not accepted and that the model must change the
arguments or stop and report the blocker. State is in memory, per
bridge, and capped at 64 entries.
- Regenerate `semantic-tool-contracts.json`,
`semantic-action-catalog.json`, `protocol/manifest.json` and the seeded
eval fixture.
- Add `spec/fixtures/pre-flat-tool-input-schemas.json`: the 8 strict
schemas from the previous release, for the upgrade test.

## Verification

- `cd packages/paperclip-runner && npx vitest run src/contracts
src/catalog src/protocol-actions src/semantic-tools
src/drivers/opencode/mcp-bridge.test.ts
src/drivers/runner-tool-bridge.test.ts`: all pass.
- `cd server && npx vitest run
src/services/native-runtime/tool-input-variant-admission.test.ts
src/services/native-runtime/repeated-tool-failure-guard.test.ts`: 9
passed.
- Red/green:
- The runner inventory test fails on `master` and lists the 16 entries.
It passes after the change.
- The guard test replays the recorded empty `handoff_case` call 4 times
through the real OpenCode bridge. Without the guard, the second call has
no correction and the test fails. With the guard, the second call lists
`required idempotencyKey, case, instructions; optional to, toAgentId,
phase, ...`, and the third says the call was not accepted.
- Negative controls: changed arguments get a fresh error, and a valid
call still runs once.
- Upgrade safety: `tool-input-variants.test.ts` compiles the previous
release's strict schemas from the fixture. For a corpus of valid and
invalid inputs per tool (27 for `handoff_case`), it checks that "flat
schema accepts and `toolInputVariantError` is null" gives exactly the
same result as the old schema. Each tool has accepted and rejected
inputs.
- Generated outputs: `check:semantic-contracts`,
`check:capability-contract`, `check:protocol-coverage`,
`check:capability-inventory`, `check:protocol-manifest` and the replay
golden check pass.
- Rust tests that read the generated contract: see the CI `runner`
checks.
- Paid narrow run: see the comment below.

## Risks

- Previous-release story: tool schemas are not saved run identity. The
OpenCode bridge lists tools again when a session resumes. A durable Rust
run that is already attached keeps the tool catalog it saved, because
re-attaching the same run does not replace the catalog. Every input the
old strict schemas accepted is still accepted, and every input they
rejected is still rejected, so saved calls, receipts and replays carry
forward. No session is reset, and no profile version or provider
`commandDigest` changes. The pack `contractDigest` changes because
`generated/` changed, as it does for any tool contract change.
- A rejected variant now arrives later for the semantic tools: the
server returns `422` instead of the runner bridge rejecting it. The
error text names the problem.
- For `paperclip_finish` on Codex, the advertised schema no longer has
the root conditional. The Rust bridge accepts a `done` result with a
continuation object, and canonical result validation still rejects it,
as before.
- Latency: the advertised schemas of the 8 tools are 1,876 bytes smaller
(23,785 to 21,909), about 470 fewer prompt tokens when all are listed.
Compiling their validators in the bridge takes 164 ms instead of 214 ms
(median of 15). The guard adds one SHA-256 of the canonical arguments
for each rejected call only. Nothing is added before turn start.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:08:58 -06:00
DottaandClaude Opus 5.5 85f4a91e70 fix(native): release a finished session's tool authority so the server heap does not grow per run (#21415)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts each agent run in a Runner session. The
server keeps a tool authority for each session scope, so the Runner can
call Paperclip tools and assigned app tools.
> - The server stored that authority in a map that lives as long as the
process. Only an aborted start removed an entry.
> - Each finished session kept its authority and the run's assigned
app-tool catalog. A large app catalog (hundreds of tools with schemas)
is several megabytes per run.
> - Under a busy queue with many short runs, the server heap grew until
the process ran out of memory. Each crash stopped every run that was in
progress.
> - This pull request releases a scope's authority when nothing can use
it again.
> - The benefit is that the server heap stays flat however many runs
finish, so the server does not crash and stop in-progress work.

## Linked Issues or Issue Description

**What happened?**
A hosted server ran out of JavaScript heap ("Ineffective mark-compacts
near heap limit") about 50 minutes after it started, and again after the
next start. The last GC line showed about 3.3 GB still live after a full
mark-compact. Each crash stopped every run in progress. The server
started about 300 native runs per hour, and most of them failed quickly.
Each run start loaded a large assigned app-tool catalog (the server
logged the "over the compile budget" warning with hundreds of tool ids
for each start).

**Expected behavior**
Memory that a run uses is released when the run's session ends. Heap
size does not grow with the number of finished runs.

**Steps to reproduce**
1. Run `executePaperclipNativeSession` with `useRunnerd: true` and a
`per_turn` session, for many distinct sessions, each with an assigned
MCP catalog.
2. Let each run succeed or fail.
3. Read `sessionToolAuthorityEpochs.size` and the post-GC heap. Before
this change, both grow by one entry and about 3 MB for each finished
run.

**Paperclip version or commit**
`master` (and the hosted build that crashed).

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: new
`releaseIdleSessionToolAuthority(scope)`. It removes and revokes the
scope's tool-authority epoch when no execution, authority
initialization, settle, warm owner, or warm close holds the scope. It is
called:
- at the end of `executePaperclipNativeSession` (after the scope
reservation and the settle mark are released),
  - after a warm session's idle-timeout close,
- after a warm session's close in `closeIdleWarmNativeSessions` (restart
and sleep sweeps).
- A warm session that stays retained keeps its authority, so the next
run in that session still drains the previous run's accounting events as
before.
- `forgetWarmNativeSession` also calls it, so a warm session that a new
session identity supersedes (`retireSupersededWarmNativeSessions`)
releases its authority too (review finding).
- New `sessionToolAuthorityCount()` export, for the regression tests.

## Verification

Red/green regression test in
`server/src/services/native-runtime/native-session-executor.test.ts`:
"releases each finished session's tool authority and assigned tool
catalog (succeeded|failed)". It runs five runnerd `per_turn` sessions
with distinct session ids and an assigned MCP catalog, then checks that
the authority map is back at its starting size.
- Red (fix disabled): `expected 10 to be 5` (one entry stays for each
finished run).
- Green (fix): both cases pass.

Second red/green test, "releases a superseded warm session's tool
authority and keeps only the retained session's": two warm runnerd runs,
where the second has a new session identity and retires the first. Red
(no release in `forgetWarmNativeSession`): `expected 2 to be 1`. Green:
one entry for the retained warm session, and zero after the warm sweep
closes it.

Heap measurement, same harness, 200 runs with distinct sessions, each
with a 300-tool assigned catalog, provider failing fast, mock call
history cleared after each run, post-GC `heapUsed` delta:

| Finished runs | Entries before | Heap delta before | Entries after |
Heap delta after |
| --- | --- | --- | --- | --- |
| 50 | 50 | 150.5 MB | 0 | 16.2 MB |
| 100 | 100 | 300.6 MB | 0 | 16.8 MB |
| 150 | 150 | 452.8 MB | 0 | 17.3 MB |
| 200 | 200 | 605.3 MB | 0 | 18.0 MB |

Before: about 3.0 MB retained for each finished run, linear, never
released. After: flat.

Commands:
- `cd server && pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts -t "releases
each finished session"`
- `cd server && pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`
- `cd server && pnpm exec tsc --noEmit`

## Risks

- A late Runner callback for a session that already ended now gets "tool
authority unavailable" instead of reaching the old, revoked authority. A
finished session has no Runner left, so no supported path makes that
call. A warm session that stays retained keeps its authority.
- A same-run restart recovery in the same process starts with no
previous epoch for an ended session, so it does not drain a previous
run's accounting events. That only applies when the earlier session
already ended.
- No schema change. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs, heap measurement).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:00:09 -06:00
DottaandClaude Opus 5.5 f7ddab8e94 fix(runner): give agent task and workspace reads the workspace page href (#21384)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People ask a chat agent where the files of a task are.
> - The answer must link the page of the workspace that holds the task's
files, when that page exists.
> - The agent reads the task with `get_task`. The task's
`currentExecutionWorkspace` had no `href`.
> - The chat directive tells the agent that a workspace without an
`href` has no page. So the agent said the page did not exist, although
isolated workspaces were on.
> - This pull request adds the workspace page `href` to the agent's task
and workspace reads, as `get_workspace` already does.
> - The benefit is that the agent links the real workspace page and does
not make a false claim.

## Linked Issues or Issue Description

**What happened?**
A person asked the CEO chat agent: "Where are the files for the website
task actually being worked on?" The agent named the task workspace and
linked the file. It did not link the workspace page. It also said "the
workspace has no separate page on this instance". The instance had
isolated workspaces on, so the page existed at
`/<prefix>/execution-workspaces/<id>`. The chat eval
`chat-features.runner-opencode.local.workspaces.locate-files` (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`) failed its
`bound-workspace-page-link` check in four cohorts and in one more paid
run for this reason.

**Expected behavior**
The agent reads the task, gets the workspace page link from the tool
result, and links that page in its reply.

**Steps to reproduce**

1. Turn on isolated workspaces (`enableIsolatedWorkspaces`).
2. Let an agent complete a projectless task, so the task has a bound
execution workspace.
3. In a different chat, ask the CEO where the files of that task are.
4. The agent calls `get_task` for the task.
`data.currentExecutionWorkspace` has no `href`. The agent reply names
the workspace and says it has no page.

**Additional context**
`get_workspace` and `list_workspaces` add the page `href` only for the
current task. The agent reads another task with `get_task` (`GET
/api/issues/{id}`) or with `call_api`. Those results did not get the
`href`. The run transcript shows the agent's reasoning: "The task API
includes currentExecutionWorkspace with no href", followed by the
directive rule "a workspace without an href has no page on this
instance".

## What Changed

- `server/src/services/task-workspace-tools.ts`: add
`withWorkspacePageHref` and `apiResultMayHaveWorkspace`. They add the
page `href` to the workspace in successful agent reads of `GET
/api/issues/{id}` (`currentExecutionWorkspace`), `GET
/api/issues/{id}/workspace` (`workspace`) and `GET
/api/execution-workspaces/{id}`. The function uses the same `href`
builder as `get_workspace`. It adds no `href` when the page is off, when
the read failed, or when the workspace is in a different company.
-
`server/src/services/native-runtime/paperclip-runner-tool-authority.ts`:
apply the function to the results of the task bridge tools (`get_task`)
and `call_api`. The tool authority reads the page setting only for these
three operations.
- Tests: unit tests for the function, and an integration test through
the real tool authority and API. The integration test checks `get_task`
and `call_api` with workspace pages on (the `href` is present) and off
(no `href`).

## Verification

- Red/green: with the base tool authority, the new integration test
fails (`undefined` instead of `/<prefix>/execution-workspaces/<id>` for
both reads). With the fix, it passes. The unit tests fail on the base (4
failures) and pass with the fix.
- `cd server && npx vitest run src/__tests__/chat-project-tools.test.ts
src/__tests__/task-workspace-tools.test.ts
src/__tests__/agent-conversations.test.ts`: 110 passed.
- `cd server && npx tsc --noEmit -p .`: passes.
- Latency: the change runs only when the agent calls one of the three
reads. It does not touch run boot or time to first token. Local
`get_task` timing through the tool authority, 40 calls per sample, pages
on: base p50 51 ms and 74 ms; fix p50 49 ms and 54 ms. The difference is
in the noise.
- Paid eval: one narrow run of
`chat-features.runner-opencode.local.workspaces.locate-files` on this
branch:
https://github.com/paperclipai/paperclip/actions/runs/43436040405
(result in a comment below).

## Risks

- Low risk. The change only adds an `href` field to three agent read
results. It does not change REST responses for the board UI.
- The `href` appears only while isolated workspaces are on, because the
page route exists only then. This is the same rule `get_workspace` uses.
- An agent can now link a workspace page that it read through
`get_task`. The page itself still applies normal board access checks.

## Model Used

- Claude, Anthropic, model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:16:44 -06:00
DottaandClaude Opus 5.5 8af889f612 feat(mcp-apps): view messages with consent, app context, full screen and provider tool icons (#21266)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - MCP Apps (#21119) show interactive widgets from MCP servers, such as
Circleback meeting cards, with the agent's reply in the task chat.
> - The first version deferred three parts of the MCP Apps spec. A
widget could not propose a chat message (`ui/message`). It could not
give the agent context (`ui/update-model-context`). It could not go full
screen.
> - Paperclip also ignored the `icons` that MCP servers declare on tools
(SEP-973).
> - The security review of #21119 showed what these features need from
day one: explicit consent, untrusted-data labels and strict icon
sanitizing.
> - This pull request adds the three host features and provider tool
icons, with those protections built in.
> - The benefit is that widgets can now work with the person and the
agent, not only display data.

## Linked Issues or Issue Description

Refs #21119

**Subsystem affected**
MCP Apps host (task chat widgets), tool catalog, run wake prompt.

**Problem or motivation**
The MCP Apps host refused `ui/message` (-32601), dropped
`ui/update-model-context` and kept every view inline. MCP tool `icons`
were not parsed or shown. So widgets could not ask the agent a
follow-up, the agent could not see what the person selected in a widget,
large widgets were cramped, and tool rows used only generic icons.

**Proposed solution**
Add consent-gated view messages, per-view app context delivered to the
task's next run as untrusted data, a full-screen dialog, and sanitized
provider icons. Details are under "What Changed".

**Alternatives considered**
- Send view messages with a new server route that creates the comment
itself. Rejected: comment creation has many rules (agent chat ownership,
paused conversations, queued delivery, wakeups, receipts). The widget's
Send button uses the same send path as the composer. A small route then
records which view the comment came from.
- Keep the iframe in place for full screen with CSS. Rejected: chat
containers can create containing blocks that break `position: fixed`. A
moved iframe always reloads, so the dialog hosts a new frame instead.

**Roadmap alignment**
Continues the MCP feature work after #21119, #21140 and #21219. It does
not touch the MCP Tasks work (gateway `tools/call` polling and
elicitation).

## What Changed

- **`ui/message`.** The host shows "<App> wants to send: …" with
**Send** and **Dismiss** under the widget (and inside the full-screen
dialog). Nothing is sent before the click. Send posts the text as the
viewer's own comment through the chat's normal send path
(`handleThreadAdd`). `POST
/api/issues/:issueId/mcp-app-views/:viewId/messages` then logs
`issue.mcp_app_message_sent`. It accepts only the caller's own comment
on the same task, matched by `clientRequestId`, from the last 10
minutes. One proposal per view can wait at a time. The view gets `{}`
after the send, or -32000 "Message sending denied" after Dismiss. Only
`user` text up to 8,000 characters is accepted.
- **`ui/update-model-context`.** `PUT
/api/issues/:issueId/mcp-app-views/:viewId/model-context` stores the
latest context per view in `tool_mcp_app_views.model_context` (text
blocks and `structuredContent`, 16 KB at most). Updates are applied in
order and deduped in the browser and on the server. A change logs
`issue.mcp_app_context_updated`, and an empty update clears the context.
The task's next run gets the three most recently updated contexts as an
`## App context (untrusted)` wake-prompt section. The section quotes the
data as JSON and says it is third-party data, not instructions. The read
runs in parallel with the other run preparation and never fails the run.
CLI adapters and the native runner render it through the same
`renderPaperclipWakePrompt`.
- **Who may do this.** Messages and context need a signed-in person with
`issue:comment` on the task. Agents and API keys get 403. Other tasks
and companies get 404. `message` and `updateModelContext` are advertised
in `hostCapabilities` only where the viewer can send messages.
- **Full screen.** The host advertises `availableDisplayModes:
["inline","fullscreen"]` and handles `ui/request-display-mode`. It
honors the view's declared modes and answers with the resulting mode. It
sends `host-context-changed` with `displayMode` and the container (fixed
`height`/`width` in full screen), and again when that container resizes.
A new **Open full screen** control sits next to "Widget from …". Full
screen is a Paperclip dialog that closes with Exit, Escape or the view's
own request.
- **Provider tool icons.** `icons` on MCP tool descriptors are sanitized
into the new `tool_catalog_entries.icons` column, outside the version
hash, so a new icon never quarantines a reviewed tool. Only `https:`
URLs and `data:image/*` URIs are kept, and SVG only as `data:`. A
`data:` icon is at most 32 KB, a URL at most 2,048 characters, with at
most 4 icons per tool. `GET /api/companies/:companyId/tool-icons` lists
them keyed by gateway tool name, in one cached request per chat. Task
chat tool rows and the app's actions list show them through
`ProviderToolIcon`. It renders `<img referrerPolicy="no-referrer">` at a
fixed size and never inlines markup. It checks the source again before
rendering and falls back to the taxonomy icon.
- **Schema.** Migration `0477` adds `tool_catalog_entries.icons`,
`tool_mcp_app_views.model_context` and `model_context_updated_at`, all
nullable.
- **Docs and stories.** `doc/connections/MCP-APPS.md` documents the new
behavior, and the finished items are removed from "Limits and
follow-ups". `Tasks/MCP App views` has new stories: "View asks to send a
message", "Full screen" and "Tool rows with provider icons". The fixture
view now uses `ui/message`, `ui/request-display-mode` and
`ui/update-model-context`.

## Verification

- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts
server/src/__tests__/tool-catalog-publication.test.ts
server/src/__tests__/openapi-routes.test.ts
packages/shared/src/mcp-icons.test.ts
packages/shared/src/mcp-apps.test.ts`: pass (embedded Postgres). These
tests cover:
- Context: stored, deduped, logged once, 413 when too large, cleared,
scoped to the task and company, refused for agents (403), and rendered
into the wake prompt as untrusted data.
- Messages: attributed only to the viewer's own recent comment on the
same task. A teammate's comment, an unknown request id or an old comment
gets 404. An agent gets 403.
- Icons: stored sanitized, and the version hash and status do not
change. Another company sees nothing.
- `cd ui && pnpm exec vitest run src/components/mcp-app/
src/components/ProviderToolIcon.test.tsx src/pages/apps/app-detail/`: 69
pass. These tests cover:
- The host protocol for each new method: capabilities, consent wait, one
proposal at a time, invalid input, ordered and deduped context, the size
limit, and the display-mode rules.
  - The consent card (nothing sent without Send).
  - The full-screen dialog (one frame at a time).
  - Icon rendering (no-referrer `img`, no inline SVG, fallback).
- `cd ui && pnpm typecheck` (including Storybook) and `tsc --noEmit` for
the server: clean.
- Red/green: the new route and host tests fail on `master`, where the
routes do not exist and the host answers -32601 or drops updates. The
icon descriptor test failed when the publication write was removed.
- Not run locally: the full suite, and
`packages/adapter-utils/src/server-utils.test.ts › runChildProcess ›
does not crash when the child closes stdin before a delayed write`. That
test times out on this Mac with or without this change, so it is a local
platform issue. CI covers both.

## Risks

- Migration risk is low: three nullable columns, no backfill.
- Every run on a task now does one extra indexed read, in parallel with
the other preparation. Errors are logged and the run continues without
app context.
- If the attribution request fails after the comment is sent, the
comment stays without the `mcp_app_message_sent` activity row. This is
documented.
- Switching between inline and full screen restarts the view, so state
inside the widget resets. This is documented as a limit.
- Icon names shortened with a hash by the gateway (very long names) do
not match, and those rows keep the taxonomy icon.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. One subagent (the same model) built the provider-icon
part.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:32:48 -06:00
DottaandClaude Opus 5.5 5ba26600ef fix(native-runtime): a remote Runner reattachment continues a turn the Runner can recover (#21362)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime runs agent turns through a Runner, often in a
remote sandbox, and must recover runs after a restart without help from
a person
> - When a Runner process is lost, a new Runner restores the
conversation. If the provider turn is gone, the Runner sends
`turn.failed` with `provider_turn_lost_on_restore` and `recoverable:
true`. The Runner sends this only after it proves that the old process
ended
> - The runtime can continue such a turn in the same conversation, but
only when the server sets `resumeInterruptedTurn`. The server set it
only for a local `resume_dead_runner` claim or a scheduled resume, and a
scheduled remote resume is refused. So on a remote Runner reattachment
the runtime did not continue the turn, and the run ended
> - This pull request sets `resumeInterruptedTurn` for a
`reattach_remote_runner` claim too
> - The benefit is that a remote run continues the lost turn in the same
conversation immediately, instead of failing and then restarting 3 to 6
minutes later

## Linked Issues or Issue Description

**What happened?**
On hosted staging (build `d1d71a7755`, read-only trace), 10 runs in 3
hours ended with `native_provider_terminal_failed` and the message "The
runner restored the conversation after process loss, but the previous
turn is no longer active." The Runner marked each of these events
`recoverable: true`. Example: run 646e2efb. A new Runner process started
at 08:32:04. `session.resumed` came at 08:34:49. `turn.failed
provider_turn_lost_on_restore` (recoverable) came at 08:34:51. The run
failed at 08:35:21. Example: run 11fe2cdc. `session.resumed` came at
09:23:24. `turn.failed` came at 09:23:25. The run failed at 09:23:52. In
each case, a separate recovery path continued the task only 3 to 6
minutes later.

**Expected behavior**
When the Runner marks a lost turn as recoverable after a restore on a
remote reattachment, the run continues the turn in the same
conversation.

**Steps to reproduce**
1. Start a native run on a remote sandbox Runner.
2. Lose the Runner process while a turn is active, so that a new Runner
reattaches through a `reattach_remote_runner` claim.
3. The new Runner restores the conversation and reports
`provider_turn_lost_on_restore` with `recoverable: true`.
4. See that the run ends with `native_provider_terminal_failed` and does
not continue the turn.

**Agent adapter(s) involved**
Not adapter-specific (core bug). Seen with Codex on a Daytona remote
sandbox.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`:
`resumeInterruptedTurn` is now also true for a `reattach_remote_runner`
restart-recovery claim. The runtime still continues the turn only when
the Runner itself sends `turn.failed provider_turn_lost_on_restore` with
`recoverable: true` for the checkpointed active turn.
- `doc/execution-semantics.md`: one paragraph describes this behaviour,
and says that a lost turn that still ends the run keeps
`native_provider_terminal_failed`.
- Tests:
- `native-scheduled-resume-continuation.integration.test.ts`: "continues
a lost turn on a same-run remote Runner reattachment" (2 cases: a plain
reattachment and a `replaceUnansweredRunner` reattachment).
- New `native-recoverable-lost-turn.integration.test.ts`: a guard
through `executePaperclipNativeSession`. The runtime saves a
failed-terminal checkpoint and then raises the lost turn (recoverable
and not recoverable). The run ends as `native_provider_terminal_failed`,
and the hold cause stays one that the replacement lane continues.
- An earlier revision of this PR also mapped a recoverable lost turn to
`native_session_interrupted`. Review showed that this does not make the
run retryable, and the hold it writes is not picked up by any automatic
continuation. That hunk is removed.

## Verification

- Red without the product change: the two integration cases fail with
`passed` = `[false]` (expected `[true]`).
- Green with the change:
- `cd server && npx vitest run
src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts
src/services/native-runtime/native-recoverable-lost-turn.integration.test.ts`
— 14 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
— passed (with the scheduled-resume file: 224 passed, 14 skipped,
pre-existing skips).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts` — 1469
passed, 7 skipped.
  - `cd server && npx tsc --noEmit` — exit 0.
- Staging evidence (hosted staging, build `d1d71a7755`, read-only
trace): 10 runs in 3 hours ended with `native_provider_terminal_failed`
although the Runner marked the lost turn `recoverable: true`. Run
646e2efb: new Runner pid 08:32:04, `session.resumed` 08:34:49,
`turn.failed provider_turn_lost_on_restore` (recoverable) 08:34:51, run
failed 08:35:21. Run 11fe2cdc: `session.resumed` 09:23:24, `turn.failed`
09:23:25, run failed 09:23:52. The task continued only 3 to 6 minutes
later through a separate recovery path.

## Risks

- This change has not run live on staging or on a Daytona sandbox yet.
The evidence is the staging trace and the local tests.
- This PR does not introduce or widen the gap where a recoverable lost
turn still ends the run. On master, that run already ends as
`native_provider_terminal_failed`, and the replacement lane continues it
after its decision (3 to 6 minutes on staging). This PR removes the
run-ending case whenever the Runner reports the turn recoverable for the
checkpointed active turn on a remote reattachment. Where the run still
ends (for example, when the continuation itself is lost), the behaviour
is unchanged, and the guard test
`native-recoverable-lost-turn.integration.test.ts` proves that it stays
`native_provider_terminal_failed`.
- The remaining case is tracked with the owner of the recovery path:
recovery evidence will accept `provider_turn_lost_on_restore` /
`recoverable` directly, after their ownership-deadline PR #21364. An
earlier hunk here tried to shorten that case. An integration test
through the real path showed that it would create a board-owned
ambiguous hold that no automatic pass continues, so the hunk is removed.
- This change does not fix why the turn is lost. In staging, a retry
replaced a live Runner. A separate fix for that is in progress.
- The runtime continues a turn only when the Runner sends the
recoverable lost-turn signal. The Runner sends it only after it proves
that the old process ended. So no provider turn runs twice.
- No Runner image change. No schema change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edit, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:09:20 -06:00
DottaandClaude Opus 5.5 eb54831d28 fix(native): relaunch a remote Runner that exited for a fresh bootstrap (#21374)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs often use a remote Runner in a sandbox. After a restart,
a deploy or a same-run retry, the server reattaches to that Runner or
starts a new Runner over its saved state
> - A Runner that no controller reaches before its connection lease
expires stops its provider, saves its state as `recoverable_failure`
with `lease_expired_requires_bootstrap`, and exits. The Runner protocol
says that only a new process with a fresh bootstrap ticket can
reconnect, and that the new process continues from the same cursor
> - Reattachment started a new Runner over saved state only for a
settled `suspended` exit (or an exit before `session.open`). So it
refused this exit, and the run ended with
`runner_remote_process_identity_unavailable`
(`native_continuation_requires_reconciliation`)
> - This pull request lets reattachment launch a Runner over the exact
saved state of a Runner that exited to ask for a fresh bootstrap
> - The benefit is that the run continues by itself after a long
controller gap, with no Retry and no manual sandbox repair

## Linked Issues or Issue Description

Refs #21329, #21362, #20853

**What happened?**
On hosted staging (build `2b69cdfcd6`, read-only trace), Codex runs on
remote Daytona Runners ended with
`native_continuation_requires_reconciliation` and the error
`runner_remote_process_identity_unavailable`, about 5 per hour, after
#21329 was deployed. All five sample runs (c6fabe9a, 31d54b02, 995b63df,
dd595934, 50c6e6e6) ended the same way. The last
`native.remote_runner_recovery_failed` event recorded
`process_verification` / `process_not_running`, `runnerLifecycle:
recoverable_failure`, `runnerRecoverableFailure:
lease_expired_requires_bootstrap`, and the Runner diagnostic "connection
lease expired; a fresh bootstrap is required". Since 10:30 UTC, 8 of the
15 runs that ended with this error had this exact exit.

Timeline of run c6fabe9a:
- 10:46 to 10:54: `runner.session.startup` took 476 s in a loaded
sandbox.
- 10:54:02: the controller sent `turn.start`. At 10:54:47 the command
timed out: "PRP command turn.start timed out (runner connected; pending
ahead 0; last runner event seq 2 91s ago)". The run resumed in the same
run and the turn then ran (tools completed from 11:14).
- 11:39: a graceful server shutdown suspended the controller. At 11:43
the next controller started the reattachment, but it did not reconnect
to the Runner until the server restarted again at 11:58. In that gap the
Runner's lease expired and the Runner exited.
- 11:58:53: the reattachment proved the process gone and refused the
`recoverable_failure` state. The run failed at 11:59.

Why `turn.start` timed out while the Runner was connected: the Runner
answers `turn.start` only after Codex answers its `turn/start` JSON-RPC
request. While it waits, the Runner buffers Codex notifications and
commits no events, so the controller sees a quiet connected Runner with
nothing pending ahead. The controller's bound for the `turn.start`
command result is a fixed 30 s. #21329 changed only the later wait for
`turn/started`. In a sandbox that needed 8 minutes to start, Codex
answered later than 30 s. That timeout is recoverable, and the run did
recover from it. This pull request does not change it (see Risks). The
failure that ended the runs is the refused reattachment.

**Expected behavior**
When a remote Runner exits because its lease expired (or its reconnect
grace or deadline ended) and its saved state is intact, reattachment
starts a new Runner from that state and the run continues.

**Steps to reproduce**
1. Start a native Codex run on a remote sandbox Runner.
2. Keep every controller from reaching the Runner for longer than its
connection lease (on staging: a deploy that stalled the reattachment for
about 15 minutes).
3. The Runner saves `recoverable_failure` /
`lease_expired_requires_bootstrap` and exits.
4. The next reattachment fails with
`runner_remote_process_identity_unavailable`, and the run ends.

**Agent adapter(s) involved**
`paperclip_runner` with Codex on a Daytona remote sandbox. The rule is
not provider-specific.

## What Changed

- `remote-runner-recovery-diagnostics.ts`: new
`exitedRunnerRequestedFreshBootstrap` (diagnostic) and
`runnerStateRequestsFreshBootstrap` (saved state). They accept lifecycle
`recoverable_failure` with `lease_expired_requires_bootstrap`,
`transport_reconnect_grace_exceeded` or
`transport_reconnect_deadline_exceeded`. These are the three codes the
Rust Runner records after `shutdown_preserving_cleanup`. The state check
also requires no pending terminal delivery and no warm transition.
- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): when
the exact Runner process is proven gone and its diagnostic names one of
these exits, the server reads the saved state again. If the state has
the same six identity fields and still names that exit, the reattachment
launches a Runner from it instead of failing. The outbox does not need
to be empty and a turn may be active: the new Runner replays from the
same cursor. The attempt bound is the one the reattachment already
admits (3, or 4 for an unanswered-Runner replacement claim). Other exits
are unchanged.
- `native-session-executor.ts` (host preparation in
`createRunnerdBackend`): a resumed sandbox for a reattachment that did
not adopt a process re-reads the Runner state. When it still has the
same six identity fields and names the fresh-bootstrap exit, preparation
keeps it in place, as for an adopted live Runner. It no longer fails
with `runner_harness_state_mismatch: resumed_not_suspended` (found by
Greptile).
- `doc/execution-semantics.md`: one paragraph describes the rule and the
staging evidence.

## Verification

- Red on `origin/master`: the 5 new "launches a new Runner over the
saved state after ..." cases failed (the reattachment threw
`runner_remote_process_identity_unavailable`). The 7 new fail-closed
cases passed.
- Green: `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "remote
controller restart adoption"` passes 118 tests.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
1712 passed, 21 skipped, 1 failed. The failure ("verifies a live warm
owner before refreshing run authority", warm-session supervision, not
touched here) passes when run alone. It is a timing failure under
full-file load.
- Launch path: "keeps the saved mid-turn state of a remote Runner that
exited for lease_expired_requires_bootstrap when the reattachment
prepares its sandbox" runs `createRunnerdBackend` and
`prepareExternalRunnerState` with a mid-turn Codex checkpoint. Red
before the host-preparation change (`resumed_not_suspended`), green
after. An unknown recoverable failure stays fail-closed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`
on the final revision: 1494 passed, 7 skipped.
- `cd server && node scripts/typecheck-layers.mjs` (includes test
programs): passes.
- Fail-closed cases covered: an unknown recoverable failure, a warm
transition, a pending terminal delivery, a state that changed after the
exit, a different Runner instance, attempt 4 on an ordinary claim, and
attempt 5 on a replacement claim.
- Staging read (read-only): the 5 sample runs and 69 runs with this
error in the last 9 hours were classified from their
`native.remote_runner_recovery_failed` events.

## Risks

- Medium. Reattachment now starts a new Runner over a saved state that
can be mid-turn. The Runner itself stops its provider before it saves
this state. The new Runner restores the session. It continues the turn,
or it reports the lost turn as recoverable only after it proves that the
old provider process ended. A recoverable lost turn on a remote
reattachment ends the run as `native_provider_terminal_failed` until
#21362 merges; the replacement lane then continues the task. That is
still better than the current reconciliation hold.
- The attempt bound does not grow. A Runner that exits this way again on
a later reattachment is bounded by the same provider attempt budget.
- This does not change the 30 s `turn.start` command bound or the deploy
gaps that let the lease expire. Those are separate follow-ups. 57 older
runs with this error had a different exit (`ready`, not settled). This
pull request keeps that case fail-closed; #20853 covers the
settled-`ready` part.
- No schema, API, UI or Runner image change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edit, code execution, read-only browser
reads of staging run data).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:53:38 -06:00
DottaandClaude Opus 5.5 57b787d3d2 fix(retry): continue from recorded work when saved files cannot be recovered (#21363)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run finishes but copying its files out of the sandbox fails,
Paperclip holds the task and tries to recover the saved files. When
recovery gives up, the task offers Retry.
> - Some saved files can never be recovered: the sandbox that held them
was destroyed, or Paperclip cannot prove which runs used a shared
sandbox. Each Retry ran the same check again, failed again, and parked
the Retry in a wait that never ends.
> - The task told the person to press Retry again or to wait for a
Paperclip update. That breaks the rule that recovery must work and that
a Retry that is shown must work.
> - This pull request makes Retry continue the task in a fresh session
and a new workspace from the work that is recorded, when the saved files
are provably unrecoverable or cannot be proven safe. The task says
clearly that the unsaved files were not recovered.
> - The benefit is that an offered Retry starts a run, and nobody waits
for a fix that another click cannot bring.

## Linked Issues or Issue Description

Refs #21315, #21295, #21267

**What happened?**

On hosted staging, two blocked tasks offered Retry (`canRetry: true`).
Each Retry was accepted, and within about 3 seconds it was parked in a
wait. No run started. Every later Retry did the same.

- Task A: the hold cause was
`native_workspace_sync_out_retry_exhausted`. Recovery ended with
`source_unavailable` (`use_alternative_strategy`). The sandbox that held
the files had a signed absence proof and a `destroyed` termination
receipt. Retry was parked with `saved_work_recovery_failed`: "Saving the
agent's files failed. Retry to try again".
- Task B: the same hold cause. Recovery ended with
`allocation_ownership_ambiguous` (`service_required`), because the
sandbox was a reusable sandbox that other runs also used. Each Retry
opened a new recovery generation, ran the same check, and was parked
with `saved_work_recovery_refused`: "Retry runs the same check again and
succeeds only after Paperclip is updated."
- In both tasks, the agent's turn had finished and its result was
accepted. Its work was recorded outside the sandbox: a plan document and
screenshot attachments for task A, and a pushed branch with an open pull
request and an implementation document for task B.

Root cause: admission had a branch for an accepted result whose files
were not yet saved. When the recovery budget was spent, the branch
always returned a `saved_work_recovery_*` wait. It did not check whether
the files could ever be recovered, so the fresh-workspace path that
admission already has was never reached. The automatic retained-source
fallback did not help either, because it handles only the remote
workspace v1 reference, and these runs use the native workspace sync v3
reference. Also, Retry opened a new recovery generation for an ownership
refusal, which only repeated the same check.

**Expected behavior**

When the saved files are provably unrecoverable or cannot be proven
safe, Retry continues the task in a fresh session and a new workspace
from the recorded work. The old files are never applied and the source
sandbox is not changed. No provider turn is replayed. The task says that
the unsaved files could not be recovered. The Retry offer and admission
agree.

**Steps to reproduce**

1. Let a native run finish with an accepted result, and let its
workspace copy-back fail (`native_workspace_sync_out_retry_exhausted`).
2. Let saved-work recovery end with `source_unavailable` (sandbox
destroyed) or with `allocation_ownership_ambiguous` (shared reusable
sandbox).
3. Press Retry.
4. Observe that the Retry receipt waits with
`saved_work_recovery_failed` or `saved_work_recovery_refused`, and no
run starts. Press Retry again and observe the same result.

**Paperclip version or commit**

Staging on `origin/master` after #21315 (`bbafa78fe2`).

**Deployment mode**

Hosted cloud (staging), remote Daytona sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (native runtime runs).

## What Changed

- `savedFilesUnrecoverable` and `unrecoverableSavedFilesFailure`
(`system-recovery-state.ts`) are the one rule for saved files that no
recovery can apply. The rule is true for a spent recovery whose failure
is a permanent source refusal: `source_unavailable`,
`allocation_ownership_ambiguous` or `historical_source_missing`. It is
false for `source_binding_invalid`, because a new recovery generation
can still verify that source. It is false for `archive_unsafe` and
`snapshot_verification_failed`, because a fresh attempt still imports
earlier saved snapshots and would fail before the agent starts. It is
also false after a fresh attempt is recorded.
- Admission (`explicit-native-continuation.ts`) no longer parks a Retry
of an accepted result in `saved_work_recovery_*` when every live
recovery of the run is unrecoverable. The Retry takes the existing
fresh-workspace path (`fresh_isolated_workspace`). That path keeps all
stop proofs, leaves the source sandbox and workspace untouched, and
starts a fresh session.
- When such a Retry starts, the task gets one system comment. The
comment says that Paperclip could not recover the unsaved files, why,
and that the task continues from its recorded work (comments, documents,
attachments and pushed commits). The settled hold records the same text.
- Retry (`task-retry.ts`) binds to an unrecoverable recovery generation.
It does not open a new generation that repeats the same check.
- The blocker text for unrecoverable files says that Retry continues the
task in a new workspace from its recorded work. It no longer says that
Retry succeeds only after an update.
- `canAdoptNativeSourceRetention` and the permanent source-refusal lists
moved into `system-recovery-state.ts`, so that admission, Retry and
messages share them. The old modules re-export them.

## Verification

- Red/green: `cd server && npx vitest run
src/services/unrecoverable-saved-work-retry.test.ts`. The tests fail on
`origin/master` sources (or on the earlier head for the snapshot case)
and pass with this change. They are built from the two staging shapes
(destroyed sandbox with `source_unavailable`, and a shared reusable
sandbox with `allocation_ownership_ambiguous`):
- offers a Retry that continues the task (blocker and task view agree,
and the text names the fresh workspace)
- starts a fresh run from the recorded work: the Retry receipt has no
wait and a run, the strategy is `fresh_isolated_workspace`, the hold is
settled with the fresh attempt, one system comment explains the lost
files with no raw IDs, the source lease is unchanged, and no new
recovery generation opens
- keeps a Retry out of a fresh attempt that would import a succeeded
snapshot whose archive is unsafe
- `workspace-recovery-warm-handoff.test.ts` › "wakes physical
preservation from HTTP Retry before passive history without replaying
the provider" failed on an earlier head of this PR, which also treated
`source_binding_invalid` as unrecoverable. It passes unchanged now.
- Updated tests that pinned the old repeat-the-check behavior:
`workspace-recovery-preparation-gates`, `task-retry`,
`system-recovery-state`, `workspace-recovery-retry-copyback`.
- Related suites pass: `system-recovery-state`,
`workspace-recovery-preparation-gates`,
`workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`,
`task-retry`, `saved-work-retry-eligibility`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
`workspace-recovery-admission`, `task-interruption-query`,
`native-workspace-sync-retention`.
- `cd server && node scripts/typecheck-layers.mjs` passes (source and
test layers).

## Risks

- Medium-low. A Retry of an unrecoverable hold now starts a new agent
turn in a new workspace instead of waiting. The turn starts only through
the existing fresh-workspace admission, which keeps every stop and
cleanup proof.
- Unsaved files of the old run are not applied. That was already true;
the change is that the task continues and says so.
- `source_binding_invalid`, `snapshot_verification_failed` and
`archive_unsafe` keep today's path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:23:23 -06:00
DottaandClaude Opus 5.5 5504151eae fix(recovery): restart a remote bootstrap again after it waited for a busy agent directory (#21370)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each interrupted native run to restart
recovery. Recovery decides if the run can continue safely.
> - For a remote sandbox run that reached no provider work, recovery
restarts the bootstrap on the same run (`bootstrap_incomplete`).
> - The restarted executor can meet a busy agent directory while it
prepares. It then defers the run for a few seconds and asks restart
recovery to look again.
> - That deferral saves the coordinator as `retryable_failure` while the
run stays `running`. No remote proof accepted this shape, so recovery
blocked the run with `ambiguous_remote_recovery_evidence`.
> - This pull request lets restart recovery restart the bootstrap again
when the previous claim was a bootstrap restart and nothing reached
execution since that claim.
> - The benefit is that a deploy plus agent-directory contention no
longer ends the task as failed. No provider turn is replayed.

## Linked Issues or Issue Description

**What happened?**

On staging, native runs of a Codex agent on Daytona failed about 3 times
an hour with `native_restart_recovery_blocked` and error
`ambiguous_remote_recovery_evidence`. The same failure was seen on the
two days before. I read 12 failed runs. 11 of them have the same event
sequence:

1. The first executor prepares the remote workspace. A deploy restarts
the server.
2. Restart recovery waits for the old controller lease, then records
"Restarting an incomplete native runner bootstrap on the same heartbeat
run" (`bootstrap_incomplete`, controller generation 2).
3. The new executor starts the run again and records "Another run of
this agent is still using its files" (`agent_directory_operation_busy`).
4. About 15 seconds later, restart recovery records "Native restart
recovery blocked ambiguous or conflicting ownership"
(`ambiguous_remote_recovery_evidence`). The run fails.

The busy-directory deferral writes `phase: retryable_failure`,
`failureCode: agent_directory_operation_busy`, `recoveryMode:
bootstrap_retry`, no lease owner, and keeps the run `running`. Every
remote proof in restart recovery needs either `phase: observed` with a
lease owner, or a `failed` run with a specific saved failure code. So
none matched. With no checkpoint and no provider evidence, the
classifier returned `ambiguous_remote_recovery_evidence`. The other
result fields on the failed run (workspace restore failure,
fresh-continuation supersede) were written after this block. They are
consequences, not the cause.

**Expected behavior**

The deferral happens while the executor prepares, before it requests a
runner. The previous recovery claim already proved that the run reached
no provider work. If nothing reached execution after that claim,
recovery must restart the bootstrap again without manual Retry. If a
runner start, runner identity, provider event or checkpoint appears
after the claim, recovery must keep the run held and not replay provider
work.

**Steps to reproduce**

1. Start a native remote-sandbox run and stop the server before the
runner receives a command.
2. Let restart recovery claim the run as `bootstrap_incomplete`.
3. Make the agent directory lock busy when the new executor prepares
instructions, so it defers with `agent_directory_operation_busy`.
4. Let the scheduled restart recovery run at the retry time. The run is
blocked with `ambiguous_remote_recovery_evidence`.

The new integration test reproduces these steps with the real recovery
and deferral functions.

**Paperclip version or commit**

Staging Core `70be453c38`. Fix based on `e23549cd9c`.

**Deployment mode**

Hosted staging (cloud workspace), remote Daytona sandbox, frequent
auto-deploys (every 20 to 45 minutes).

**Agent adapter(s) involved**

`paperclip_runner` with the `codex_app_server` backend on a Daytona
sandbox.

## What Changed

- Restart recovery accepts a new remote proof,
`remote_bootstrap_restart_deferred_before_launch`. It applies when all
of these are true:
- The most recent recovery claim was `bootstrap_incomplete`, with a
recorded event watermark.
- The coordinator is `retryable_failure` with `failureCode:
agent_directory_operation_busy`, `recoveryMode: bootstrap_retry`, no
lease owner, and no successor run. The run is still `running`.
- No `native.process_start_requested`,
`native.process_identity_recorded`, `adapter.invoke`, tool start or
runner-sourced event exists after the claim watermark. No process
identity was ever recorded for the run.
- No provider event exists. A checkpoint, if present, is only the
continued session with no turn, result or request.
- The workspace is a prepared remote workspace with no durable
controller checkpoint and no staged attachment.
- The run then goes through the same resumption admission as the other
settled remote retries (agent invokable, task not paused, dependencies
ready, no competing run).
- Added red/green integration tests built from the staging shape: the
run restarts again (generation 3), and a runner identity, a provider
event, or a checkpoint saved during the wait keeps it held.

## Verification

- Red: on the unchanged source, the new test returns `{ kind: "blocked",
reason: "ambiguous_remote_recovery_evidence" }`, the same error as
staging.
- Green: `npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/native-controller-recovery.integration.test.ts`
passes (251 passed, 14 skipped).
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The change only adds a narrow proof. It requires that
nothing reached execution since a claim that already proved no provider
work. Any runner start, runner identity, provider event or new turn
keeps the run held, so a provider turn is never replayed.
- The restart counts against the existing provider attempt budget and
the no-progress deadline, so a loop of deferrals stays bounded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
code editing, browser read-only API fetches) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Refs #20829

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:12:35 -06:00
DottaandClaude Fable 5.2 18fb6ee571 fix(recovery): continue a native terminal failure whose provider is never proven stopped (#21364)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs that end in a terminal provider failure go through
automatic recovery: an exact replacement first, then a fresh
continuation from saved work
> - The exact replacement waits for proof that the previous provider
stopped. Without that proof it writes neither a successor nor a denial
> - That wait has no end. If the old sandbox or Runner can never be
proven stopped (for example after a server swap), the recovery resolver
keeps waiting for the replacement, and the task sits with no run and no
reason a person can see
> - This pull request bounds the wait. After 5 minutes on the database
clock from the run's terminal time, the replacement pass records a
denial, and the existing fallback continues the task in a fresh run in
an isolated workspace
> - The benefit is that these tasks continue by themselves instead of
stalling forever, and exact replacement still wins when proof arrives in
time

## Linked Issues or Issue Description

**What happened?**
A native run that failed with a terminal provider failure stayed stuck
when its previous provider could not be proven stopped. The replacement
pass skipped the run on every pass. It recorded no successor and no
denial, and it set no deadline. The recovery resolver waits for one of
those two records, so it never continued the task. The replacement queue
also did not wake again on its own, because a completed scan reports
nothing pending.

**Expected behavior**
Exact replacement gets a bounded first chance. After the bound, the task
continues in a fresh run that cannot use the old writer's workspace.

**Steps to reproduce**
1. Create a native run in `terminal_failure` with code
`native_provider_terminal_failed` and a finished time more than 5
minutes ago.
2. Record a process on the run that this server cannot prove stopped
(for example a PID recorded by another server).
3. Run the replacement pass and then the recovery resolver.
4. Before this change: nothing happens, on every pass. After this
change: the pass records a denial and the resolver records one fresh
continuation with isolation `fresh_isolated_workspace`.

**Paperclip version or commit**
`master` at 2b69cdfcd6.

## What Changed

- `native-safe-replacement.ts`: an unverified-ownership decision now
waits only until 5 minutes after the run's terminal time (`finished_at`,
else the finalization's last update), compared with the database
`now()`. After that, the pass records `replacementDenied` and the
existing watchdog hold, with a plain-language next action. The expiry is
checked again inside the locked transaction, so two servers cannot both
act, and a successor that a concurrent pass committed first still wins.
- The deadline applies only to `native_provider_terminal_failed` runs
whose finalization has no lease owner. Other failure codes and
person-owned causes are unchanged.
- The bound reuses `RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 minutes) from
`recovery-continuation-pacing.ts`, as `NATIVE_OWNERSHIP_PROOF_BOUND_MS`.
- The wake has no time window. The wake query returns the deadline of
every marked run that is still undecided (no successor, no denial, no
lease owner), with the same eligibility as the denial. When that
deadline is past, the queue asks for a pass at once. That pass always
removes the run from the query:
  - It denies the run, which ends the wait.
  - Or it starts the exact replacement, when ownership was proven.
- Or it finds the run legitimately undecidable and clears the marker.
This happens when the history has unresolved interactions, or when the
task is not eligible for a successor (for example in backlog). The
existing task and interaction notifications wake the queue again later.
- More than 25 overdue runs: the scan cursor requests the next pass at
once.
- A lost lock or any other error: the pass throws, and the coordinator
retries after its existing `retryMs`
(`EXECUTION_RECONCILIATION_INTERVAL_MS`, 15 seconds). That is the same
backoff as every replacement-decision error today.
- An environment lease that is never released no longer blocks the
decision after the deadline. Before the deadline, the run waits as
before and records the marker. After it, an unreleased lease counts as
"provider not proven stopped", so the pass denies the run and never
starts an exact replacement next to a live lease. The fallback then
treats the unreleased lease as pending cleanup and picks
`fresh_isolated_workspace`. Admission (`heartbeat/queue.ts`, the
wake-queue adapter) does not block a run that a fresh continuation
superseded, so the new run does not wait for or touch the old lease. The
environment orphan-settlement modules are not changed.
- When the pass first sees unverified ownership before the deadline, it
writes `ownershipUnverifiedAt` once into the failure detail. It does not
change `updated_at`. This marker only limits which runs schedule a wake.
The deadline still counts from the terminal time.
- `execution-control-work.ts`: when no replacement is pending, the
replacement queue schedules its next pass at the earliest future
deadline of a marked run. This is a single timer, not a polling loop.
Rows already past their deadline are handled by the pass that runs at
startup, with or without the marker.
- No changes to `execution-recovery-resolution.ts` or
`fresh-continuation-fallback.ts`. The fallback already selects
`fresh_isolated_workspace` when its own owner proof cannot show that the
old local writer ended, `remote_epoch` for a remote workspace, and waits
(bounded) only for a process proven alive on this host.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-safe-replacement.test.ts`: new cases:
- Past the deadline (6 minutes, and 2 days for a row stuck before this
release): denial recorded on the first pass, no exact successor, then
exactly one fresh continuation with `fresh_isolated_workspace` after two
resolver passes.
- Before the deadline: no denial, no hold, no continuation, and the
replacement schedule wakes at terminal time + 5 minutes.
- Ownership proven before the deadline: the exact replacement is
scheduled and no denial is written.
- Red before the fix: 3 of the 4 new cases failed (no denial written; no
wake scheduled). The exact-replacement case passed, as expected, because
it guards existing behavior. Green after: 93 of 93 passed in
`native-safe-replacement.test.ts` and `execution-control-work.test.ts`.
- `tsc --noEmit` for `server`: clean.
- Review fixes, red before and green after:
- A deadline that passed 10 minutes before the schedule was asked: the
queue requests an immediate pass, which denies the run and continues it
isolated.
- An overdue run with an unreleased environment lease: denied, no exact
successor, continued with `fresh_isolated_workspace`.
- An overdue run the pass cannot decide (task in backlog): one wake,
then no second wake.
- 26 overdue runs: 25 decided on the first pass, all 26 on the second,
then no wake. This one also passed before, because the existing scan
cursor covers it. It stays as a guard.
  - On the earlier head, the first three failed.
- Green after all commits: 97 of 97 passed in
`native-safe-replacement.test.ts` and `execution-control-work.test.ts`.
- `pnpm verify` selected 2624 of 3444 test files, because
`execution-control-work.ts` is widely imported. I stopped it on a
heavily loaded machine (load average about 20). Its partial output
showed only unrelated attention-test timeouts near 30 seconds. Required
CI supplies the broad coverage.

## Risks

- After 5 minutes, a run whose old provider is truly still alive on
another server is continued in a new isolated workspace. The old writer
cannot write to it, and its files are not applied. A process proven
alive on this server still makes the fallback wait for its own bound
first.
- Deploy burst: rows stuck before this release are denied on the first
pass, at most 25 per pass (existing scan limit). Continuations then go
through the existing pacing: 6 per task per hour, 5 per company per
pass, one per agent per pass, and the agent's concurrency limit.
- No schema or API change. No change before `turn.start` for healthy
runs.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 06:03:53 -06:00
DottaandClaude Opus 5.5 a090a6a932 fix(runner): stop a model from resending the same rejected tool call (#21360)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools such as `call_api`, and the tool
layer refuses an invalid call before it sends anything
> - Some model hosts drop optional tool arguments. A `PATCH` call then
arrives without `body` every time, even when the model says it added it
> - The refusal already names the missing argument, but nothing stops
the model from resending the same call. In one recorded run the agent
sent the same rejected call 12 times in 80 seconds and never finished
the task
> - This pull request makes the tool layer recognise a repeated
identical validation failure, correct it once with the exact arguments
that arrived, and then stop running that exact call for the rest of the
run
> - The benefit is that a dropped argument ends in a clear correction or
a reported blocker, not a long loop that burns time and tokens

## Linked Issues or Issue Description

**What happened?**
An agent created a task and then tried to update it with `call_api`
`PATCH /api/issues/{id}`. Every call arrived with `operationId`,
`pathParams`, `query` and `responseText`, but no `body`. The tool layer
refused each call with "requires body" and showed the shape to resend.
The model's reasoning said that the body was missing, but each new call
arrived without `body` again. The agent sent the same call 12 times and
was still running when the test ended.

**Expected behavior**
After the same call fails twice in the same way, the agent should get
one clear correction that says what arrived and what is missing.
Paperclip should then refuse to run that exact call again, so the agent
changes the call or reports the blocker.

**Paperclip version or commit**
`master` at ff971149c2.

**Steps to reproduce**
1. Run an agent on a model host that drops optional tool arguments.
2. Ask the agent to create a task and then update it.
3. See the same body-less `PATCH` call refused again and again until the
run times out.

## What Changed

- Add `RepeatedToolFailureGuard`
(`server/src/services/native-runtime/repeated-tool-failure-guard.ts`).
It keys each failure by tool name and a hash of the canonical arguments.
- Count only failures that must repeat for identical arguments:
`400`/`422` refusals and schema errors. Other errors, such as `503`,
pass through unchanged and the call still runs.
- On the second identical failure, keep the original error text and add
the names of the arguments that arrived, a note that a missing argument
did not arrive, and that Paperclip will not run this exact call again.
- On the third and later identical calls, do not run the tool. Return
the original error with an instruction to change the arguments or stop
and report the blocker.
- Any change to the arguments runs normally. A success clears the record
for that call.
- Run identical calls that arrive in parallel one after another, so each
sees the failures before it. Different calls never wait on each other.
- Wire the guard into the per-run `SessionAuthorityEpoch.execute`,
inside the tool trace, so blocked attempts are still recorded. State is
in memory, per run, and capped at 64 entries.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/services/native-runtime/native-tool-trace.test.ts`: 11 passed (6 in
the new file).
- Red/green: with the guard removed from
`SessionAuthorityEpoch.execute`, the recorded-run test fails (the second
failure has no correction and every call runs). With the guard in place,
it passes.
- The test replays the recorded body-less `PATCH` through the real
`executeRunnerApi` validation. It checks that the tool runs only twice
for five identical calls and that no HTTP request is sent.
- Parallel identical calls run at most twice, and a different call is
not held back while an identical one runs.
- Negative controls: changed arguments run again, a `503` keeps running,
and different error messages are not merged.
- `cd server && npx tsc --noEmit`: clean.

## Risks

- Low risk. The guard acts only on identical arguments with an identical
`400`/`422` or schema error, which cannot succeed on resend.
- A refusal that depends on state that changes during the run (for
example, a validation that later passes) would be blocked after two
tries with the same arguments. The agent can still send changed
arguments.
- Latency: the guard adds one SHA-256 of the canonical arguments per
tool call (about 2 µs for a 4 KB argument). It adds no wait and nothing
before turn start. Only identical calls that arrive in parallel wait for
each other.
- No persisted state changes. After a restart, the record starts empty.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:00:26 -06:00
DottaandClaude Opus 5.5 f0679ee0c7 fix(recovery): bound automatic attempts per task, one deadline per interruption, and batch the invariant report's reads (#21347)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a server swap or a failure interrupts an agent's task, recovery
continues the task automatically: the assigned-work sweep, the
scheduler's bounded retries and native session restart recovery.
> - Release-swap testing on staging found four gaps in that recovery.
The assigned-work report took 60–81 s. One lasting outage was retried by
two owners under two separate caps. A server with a fast clock could
take a session lease that another server still held. Each server
generation started its own 15-minute recovery wait, so the waits chained
for hours.
> - Each gap either slows recovery down or lets a task loop or stall
without any limit.
> - This pull request batches the report's reads, gives each task one
shared hourly cap for automatic attempts, uses the database clock to
judge a lease, and gives each interruption one deadline across server
generations.
> - The benefit is that recovery stays fast and bounded. A lasting
outage gets at most six automatic attempts an hour. A run whose provider
makes no progress ends after 15 minutes, and the fallback then continues
it in isolation.

## Linked Issues or Issue Description

**What happened?**
1. On staging, the assigned-work invariant report took 60–81 s with more
than 1,000 open agent tasks. For each task with nothing live or queued,
the report read the tree pause gate in sequence: one round trip, plus
one per ancestor while a pause hold existed. It also read the company's
agent list once per assignee.
2. A lasting repository catalog outage was retried by two owners. Each
sweep continuation started a fresh run. Each failure of that run got two
more bounded transient retries from the scheduler. One failure could
start up to 18 runs an hour.
3. `claimNativeSessionResumptions` compared the lease expiry with the
claiming server's own clock. A server whose clock ran fast could take a
lease that another server still held.
4. Each server generation's restart-recovery claim started its own
15-minute wait for the provider to recover. Swaps came every 10–14
minutes, so the waits chained for up to 12 generations and none of them
expired.

**Expected behavior**
The report reads pause holds in a fixed number of statements. One task
gets at most one hourly cap of automatic attempts across all owners. The
database clock judges a lease. An interruption with no provider progress
ends after one 15-minute deadline, whatever the number of server
generations.

**Steps to reproduce**
1. Seed 1,500 open tasks with 300 held roots and 400 queued wakes, then
run the invariant report. Before this change it used 2,279 statements.
2. Make the repository catalog unavailable for longer than an hour on an
assigned task. Count the runs.
3. Let a server whose clock is ten minutes fast claim a native session
resumption whose lease is still live.
4. Replace the server every 7 minutes while a native run's provider
makes no progress.

## What Changed

- `issue-tree-control.ts`: a new batched pause-hold read. For a set of
tasks it uses at most two statements: one to find any active hold, then
one recursive ancestor read. It has the same depth bound and cycle guard
as the per-issue gate. `assigned-work-sweep.ts` uses it, and reads the
org chart once per batch to judge each assignee.
- New `recovery/task-automatic-attempts.ts`: one per-task ledger of
automatic attempts in the last hour (sweep continuations plus the
scheduler's `transient_failure` retries). The sweep counts both for its
cap and its spacing (`assigned-work-sweep.ts`). The scheduler's
transient retry lane (`heartbeat/retries.ts`) reads the same ledger.
Once the sweep has continued a task in the hour, a transient retry that
would exceed the cap of six is reported as exhausted. If the sweep never
continued a task, the scheduler keeps its own chain bound.
- `native-finalization-reconciler.ts` (`claimNativeSessionResumptions`):
the compare-and-set also requires the other owner's lease to have lapsed
by `clock_timestamp()`.
- `native-restart-recovery.ts`: each restart-recovery claim records
`interruptionStartedAt` in its recovery-history entry. A later claim
keeps that value while the interrupted attempt shows no provider
progress, and resets it when there is progress. A claim made 15 minutes
or more after the interruption began, with no progress, ends the run as
no longer resumable. This uses the existing terminal path (run failed,
slots and task lock freed), so the fresh-continuation fallback continues
the task in isolation. The bound is
`NATIVE_INTERRUPTION_NO_PROGRESS_DEADLINE_MS` in
`fresh-continuation-fallback.ts`, set equal to the fallback's
exact-recovery cap. Coordinators saved by older releases have no start
time, so the first claim by this code starts the clock. The normal run
path is not changed.
- Review fixes:
- A graceful drain no longer counts as "no progress". The claim always
reads provider progress. The drain only decides whether the resume
spends an attempt, so a run that progressed before a deploy is never
ended by the deadline.
- The transient lane checks the hourly cap again inside the task-locked
transaction, after it reuses an existing successor and before it inserts
a new one. The sweep already reads the ledger under the same lock.
- A same-run resume claim that the database clock rejects (another
owner's lease is still live) no longer pushes that owner's
`nextAttemptAt` out.
- Code comments that the original commits added are removed
(CONTRIBUTING). Merged with current master (includes #21336):
`assigned-work-sweep.ts` keeps master's `BLOCK_ATTRIBUTION_WINDOW_MS`
import and adds the ledger import.

## Verification

Red-to-green tests, each red on the parent commit and green with the
fix:
- `server/src/__tests__/assigned-work-sweep-round-trips.test.ts` "reads
pause holds and assignees once per batch on 1,500 open tasks…": 1,500
tasks, 300 held roots plus 300 children, 400 queued wakes, 20 agents.
Red: 1,100 per-task gate reads and 2,279 statements. Green: 0 and 239.
Report verdict counts are asserted to be the same before and after.
- Same file, "the batched pause read gives the per-issue gate's answer:
deep chains, cycles, released holds, other companies": checks that the
batched read gives the same result as the per-issue gate.
- `server/src/__tests__/assigned-work-invariant.test.ts` "counts the
scheduler's transient retries of the same task against the one
continuation cap": red (repaired), green (deferred_cap).
- `server/src/services/heartbeat/retries.test.ts` "a lasting outage gets
at most the task's hourly cap of automatic attempts across the sweep and
the transient lane".
- `server/src/__tests__/native-session-resumption.test.ts` "judges
another owner's lease by the database clock, not a server clock that
runs fast".
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
"resumes a run that progressed for 20 minutes before a graceful drain
suspended it" (red before: the deadline ended it) and "still ends a run
with no provider progress at the deadline when a graceful drain
suspended it".
- `server/src/services/heartbeat/retries.test.ts` "re-checks the task's
hourly cap under the task lock when the sweep records an attempt while
the retry is being prepared" (red before: attempt seven was scheduled).
- `server/src/__tests__/native-session-resumption.test.ts`: the
clock-skew test also asserts that `nextAttemptAt` is not pushed out (red
before).
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
"ends a run whose generations show no progress at the deadline from the
first interruption, not per generation", "restarts the clock when a
generation's resume made provider progress", "starts the clock at the
first claim for a coordinator saved before the deadline existed".

Local runs on this branch (based on master 566843820c):
- `cd server && pnpm exec vitest run <file>`, one file at a time:
  - `src/__tests__/assigned-work-sweep-round-trips.test.ts`: 2 passed
  - `src/__tests__/assigned-work-invariant.test.ts`: 31 passed
  - `src/services/heartbeat/retries.test.ts`: 30 passed
  - `src/__tests__/native-session-resumption.test.ts`: 20 passed
-
`src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
217 passed, 14 skipped
- `src/__tests__/heartbeat-native-restart-suspension.test.ts`: 115
passed
- `cd server && pnpm run typecheck` (CI's layered typecheck, including
test files): clean.

- `cd server && pnpm exec tsc --noEmit`: clean.
- Timing note: local round trips take about 0.2 ms, so local
milliseconds do not show the gain (736 ms became 303–583 ms on a loaded
machine). At staging's ~50 ms per round trip, the ~2,040 sequential
statements removed are about 100 s.
- No existing test expectation was changed.

## Risks

- After this change, a task the sweep continued in the last hour can get
fewer transient retries. This is intended: one task gets at most six
automatic attempts an hour. A person's Retry is not counted in this
ledger.
- The interruption deadline ends a native run that shows no provider
progress for 15 minutes across server swaps. The fallback then continues
it in a fresh isolated workspace, so the task is not left for a person
to fix. A run that shows progress resets the clock.
- Coordinators saved by older releases start the clock at their first
claim by this code, so an upgrade does not expire any run immediately.
- No migration. No schema change. The new history field is inside
existing JSON.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 05:10:28 -06:00
DottaandClaude Opus 5.5 43aabb7080 fix(runner): keep configured secret values out of OpenCode output, storage and model requests (#21330)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can configure a secret that Paperclip delivers to an
agent's commands as an environment variable
> - The value must stay out of what the agent runtime stores and what it
sends to the model
> - With the OpenCode runtime, a command that printed the variable put
the value in OpenCode's tool output. OpenCode stored that output in its
session database and sent it to the model. Paperclip's own run log
already redacted it
> - Paperclip did not tell the runner which configured values are
secrets, so the runner could not protect them without also hiding
ordinary settings
> - This pull request sends secret names (never values) to the runner,
filters exact secret values before OpenCode captures any tool output,
redacts them from model requests, and tells the agent how to check a
secret without printing it
> - The benefit is that a configured secret stays out of model context
and out of runtime storage, even when an agent prints it

## Linked Issues or Issue Description

**What happened?**
The `chat-features.runner-opencode.local.secrets.personal-secret` eval
failed because the persisted-state scan found a configured secret value
in OpenCode's session database. The person entered the value through the
normal secret entry form. Nobody pasted it in chat. The agent ran a
shell command that printed the environment variable it was given.
Paperclip's run log redacted the value. OpenCode stored the raw output
in its database and included it in the next model request.

**Expected behavior**
A configured secret must never reach model context or runtime
persistence. Ordinary configured values must not be redacted.

**Steps to reproduce**
Configure a secret environment variable for an OpenCode agent. Have the
agent run a command that prints the variable. Then search the OpenCode
session data and the model requests for the value.

## What Changed

- **Secret names cross to the runner.**
`configuredEnvironmentProjection` takes the controller's secret key
names and adds `PAPERCLIP_SECRET_ENV_KEYS`. This marker holds only the
names of configured entries that came from secrets. It travels with the
configured projection across the TypeScript and Rust runner boundaries
(`configured-environment.ts`, `configured_environment.rs`). The server
passes `secretKeys` in `runtime-execution.ts`.
- **Command output is filtered before OpenCode reads it.** For a launch
that has configured secrets, the runner sets OpenCode's `shell` to a
small POSIX wrapper (`drivers/opencode/output-custody.ts`). The wrapper
runs the shell OpenCode would use anyway. It sends stdout and stderr
through an `awk` filter that replaces exact secret values. OpenCode
never receives the raw bytes, so its streamed previews, its database and
its model requests stay clean. The wrapper keeps the exit status. It
does not wait for background processes. The values come from the
environment that OpenCode already holds and are never written to disk.
- **Other tool results are rewritten before they are saved.** A local
OpenCode plugin rewrites completed non-shell results (file reads,
searches, MCP results) in `tool.execute.after`. OpenCode runs that hook
before it stores or returns the result. The wrapper refuses to run any
command until the plugin has started for the current launch (per-launch
nonce). So no command can write a secret to a file that a later read
returns without filtering.
- **Model requests are redacted as a second layer.** The OpenCode
provider proxy removes exact secret values from the string values of
each parsed model request. Field names and other literals are never
changed. A quick byte scan for each value, raw and JSON-escaped, sends
the original bytes unchanged when no value is present.
- **The agent gets guidance.** Native task prompts list the configured
secret names. They say to check a secret with `[ -n "$VAR" ]` and never
to print, echo, log or write a secret value.
- **No added startup delay.** The wrapper is content-addressed under the
runtime directory and reused across launches, so it is not rewritten on
every launch. OpenCode's plugin-dependency install is marked as already
done, so loading the plugin does not add wait time to the first tool
call.
- **Only exact values are redacted.** Command output is filtered line by
line, so each line of a multi-line value that has at least 4 characters
is redacted too. If a configured secret has no line of 4 or more
characters, the launch is refused with
`opencode_output_custody_unavailable` instead of running unprotected.

## Verification

-
`packages/paperclip-runner/src/drivers/opencode/output-custody.test.ts`
(10 tests): the wrapper filters stdout, stderr and a partial last line;
keeps exit codes (7, and 143 for SIGTERM); filters each line of a
multi-line value; refuses to run before plugin readiness or with an old
nonce; returns at once when a background process keeps running; leaves
ordinary values unchanged. The plugin rewrites nested results.
Model-request redaction covers raw and JSON-escaped values and never
changes field names or literals (a secret equal to `true` leaves
`"stream": true` intact). Secrets the filter cannot match are refused.
No value is written to disk.
- `opencode-server-driver.test.ts`: one saved session launches three
times. First with no secret marker (older controller), then with the
marker, then with no marker again (rollback). The first launch is
unchanged, the second applies custody, and the third launch still
starts. This test fails without the driver change.
- `configured-environment.test.ts`: the marker names only secret
entries, crosses the runner spawn and OpenCode boundaries, reads as "no
secrets" when absent, and is refused when malformed. Rust:
`secret_name_marker_crosses_only_with_its_projection`.
- `server/src/__tests__/configured-secret-guidance.test.ts`: guidance
content, and no guidance when there are no secrets.
- Real OpenCode 1.18.34 qualification (`PAPERCLIP_OPENCODE_QUALIFY=1
node --test scripts/qualify-opencode-runtime.test.mjs`). A fake model
asks for a command that prints the secret and writes it to a file, then
a read of that file. Unprotected baseline: the value is in the session
database and in model requests (red). With custody: it is in neither,
and the exit status is kept (green).
- **Latency** (local, macOS, real OpenCode, 8 runs each): boot 629 ms vs
697 ms baseline; first model request after the prompt 678 ms vs 698 ms
(no regression); first command 139 ms vs 191 ms; each later command +9
ms (36 vs 27 ms). Proxy cost per request when no secret is present (the
normal case): 33 µs at 50 KiB, 164 µs at 300 KiB, 0.55 ms at 1 MiB, 2.3
ms at 4 MiB. When a secret is present: 91 µs, 0.48 ms, 1.6 ms and 6.8
ms. The scan runs only for launches that have configured secrets, and it
adds nothing to OpenCode's own startup.
- A narrow paid run of
`chat-features.runner-opencode.local.secrets.personal-secret` on this
branch is linked in a comment.

## Risks

- **Previous-release story.** The marker is new and optional. A runner
that receives no marker (older controller) launches exactly as before.
An older runner drops the marker at its environment boundary. The change
saves no new run or session state. OpenCode configuration is regenerated
on every launch, so sessions saved before this change resume under
custody (covered by the cutover test). Warm-process reuse keys include
the marker, so a warm process from before the deploy is not reused for a
run that has secrets.
- **Commands go through the wrapper.** Output arrives line by line.
Output written after the shell exits is dropped, which matches how
OpenCode stops reading at exit. If `awk`, `mkfifo` or `rm` is missing,
the launch is refused with `opencode_output_custody_unavailable` and
does not run unprotected.
- **Scope.** Only exact values are redacted. An agent that encodes a
value (for example base64) is out of scope. The model-request layer
applies when Paperclip routes the model through its proxy (OpenRouter
and Paperclip connections).

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:23:56 -06:00
DottaandClaude Opus 5.5 0986c3eb3a fix(native): let a same-run reattachment take over its own retained remote Runner cleanup (#21328)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 keeps an unreachable remote Runner alive when the failed
attempt closes, so that the same-run reattachment can adopt it
> - The same close also leaves the session in the in-process cleanup
quarantine. The quarantine refuses every new session of the task, and
that includes the reattachment that the Runner was kept for
> - Close retries cannot settle it. After the first close fails, the
Runner transport returns the same rejected close for every retry
> - On staging, this was the largest failure class: 14 Codex runs in one
hour failed with `native_session_cleanup_quarantined`, while only 5 runs
succeeded. Each task then needed Retry
> - This pull request lets the same run's reattachment to that Runner,
in the same remote sandbox, take over its own retained cleanup
> - The benefit is that the task continues on its own, with no spent
attempt, no failed run and no Retry

## Linked Issues or Issue Description

Refs #20923, #21031

**What happened?**
A remote native run (Codex on Daytona) failed a turn with a recoverable
error, for example `runnerd did not report the provider turn identity`
or `native_finalization_missing: session returned no semantic result`.
The close then logged `runner did not prove durable suspension before
checkpoint` and `remote runner left alive unsettled for same-run
reattachment`. Each later same-run reattachment was refused at
admission:
- When the close error was retryable, admission waited out the grace and
failed with `native_session_cleanup_quarantined: prior session cleanup
remains incomplete`. That spent a same-run attempt. After the attempt
budget, the run failed. All close retries replayed the same failed close
within milliseconds (`runner transport close requested: native session
quarantined cleanup recovery`), so they could never settle the
quarantine.
- When the close error was `native_session_close_unrecoverable`
(operator-required), the next attempt failed at once with
`native_session_cleanup_quarantined: the previous session is still
shutting down; Paperclip finishes the cleanup automatically and the
saved work is kept.` That code is a permanent failure.

"Paperclip finishes the cleanup automatically" did not happen in the
process. Only a server restart (a deploy) removed the in-memory
quarantine, and by then the run had failed and the task waited for
Retry. Staging restarts every 20 to 45 minutes, so attempts often end
mid-cleanup and meet this gate.

**Expected behavior**
The same run's reattachment adopts the Runner that its earlier attempt
kept for it. It does not wait for a close that cannot settle, and it
does not spend an attempt. The task continues without Retry. Other runs
of the task and other sandboxes stay gated.

**Steps to reproduce**
1. Start a remote native run, and let its turn fail with a recoverable
transport error while the Runner cannot prove suspension.
2. The close keeps the Runner alive and quarantines the session.
3. The server schedules the same-run reattachment 30 s later.
4. The reattachment fails with `native_session_cleanup_quarantined`, and
the task needs Retry.

**Paperclip version or commit**
Deployed Core `c730872974` on staging. Reproduced on `master` at
`49f0030f06`.

**Deployment mode**
Hosted cloud (staging), Daytona remote sandboxes.

**Agent adapter(s) involved**
`paperclip_runner` native runtime with the Codex app server backend.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
`executeNativeSession` takes a new option `retainedRunnerReattachment`.
With a remote cleanup scope, admission retires quarantine entries that
belong to the same run in the same remote scope. Removal stops later
close retries. A close attempt that is still in flight stays an
admission owner until it settles, so the old and new controllers do not
overlap. Other runs, other sandboxes and local backends keep the gate.
- `server/src/services/native-runtime/native-session-executor.ts`: sets
`retainedRunnerReattachment` only for a `reattach_remote_runner`
recovery of the same run.
- `doc/execution-semantics.md`: records the takeover.
- Tests:
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: four
new cases. An operator-required close with a retained Runner. A
retryable close whose retries replay one failure. A close that a deploy
interrupts while it is in flight (the reattachment waits for it, then
takes over). Controls: another run, a run without the takeover option
and a local backend stay refused.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
asserts that the option is set for a same-run remote reattachment and is
not set for a fresh sandbox run.

## Verification

- Red on `master` (runtime source restored): 3 of the 4 new runtime
cases fail with exactly the two staging errors (`the previous session is
still shutting down…` and `prior session cleanup remains incomplete`).
The control case passes.
- Red on `master` (executor source restored): the 3 executor assertions
fail.
- Green with this change:
- `cd packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts` (294 passed)
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "exact
Stop cleanup|fresh sandbox|retirement"` (46 passed)
- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/completed-turn-cleanup-hold-settle.test.ts` (61 passed)
- `npx tsc --noEmit` in `packages/paperclip-runner` and `server` passes.
- Staging evidence: runs `7b7ded99` and `27c487ef` show the sequence
above in their events and logs.

## Risks

- The takeover applies only when the server recovers the same run with
`reattach_remote_runner` in the same remote cleanup scope. This is the
case that #20923 keeps the Runner alive for. After a deploy the
quarantine is already gone and the same reattachment runs today, so the
takeover gives the in-process retry the same behavior as a restart.
- A close attempt that is in flight is never overlapped. The
reattachment waits for it within the existing admission grace.
- The reattachment still verifies Runner ownership and its controller
generation before it adopts the Runner. If the Runner is gone, the
existing relaunch or reconciliation path handles it.
- No persisted shape changes. Saved state from the previous release is
not affected because the quarantine lives only in process memory.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:15:40 -06:00
DottaandClaude Opus 5.5 22c9300dac fix(recovery): a run that fails before the agent is asked anything continues by itself on a fresh computer (#21297)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox opens a provider session and then
waits at a launch gate before the first prompt goes to the provider
> - On remote OpenCode sandboxes the gate saves the provider launch
files; when that save fails, no prompt was ever sent
> - Today that failure ends as `native_session_interrupted` with
ambiguous recovery evidence, so the task goes blocked and the user must
press Retry
> - Nothing was sent to the provider, so nothing can be duplicated: the
system can safely continue by itself
> - This pull request types that failure, proves it from the Runner turn
ledger at the failure point, and continues the task on a fresh sandbox
in the existing interrupted-preparation lane
> - The benefit is that a setup failure before the first prompt no
longer needs a human, and the user sees plain words instead of internal
codes

## Linked Issues or Issue Description

Refs #21287
Refs #21304

**What happened?**
On a remote OpenCode sandbox, the provider launch-files gate failed
before the first prompt (for example
`native_provider_artifacts_transfer_timeout`). The run ended blocked
with a Retry banner. The run error showed internal codes and raw byte
counts to the user.

**Expected behavior**
A failure that happens before any prompt reaches the provider continues
automatically on a fresh sandbox. The task shows as working while the
continuation is pending. The visible text is plain and has no internal
codes or raw numbers.

**Steps to reproduce**
1. Use an OpenCode agent on a Daytona sandbox environment with
controller persistence.
2. Arm the crash point `native.provider_artifacts.before_capture` in
fail mode (or let the launch-file upload time out).
3. Start a task. The first run fails at the gate before any prompt.
4. On master the task goes blocked with Retry. With this change a
successor starts on a new sandbox and the task completes.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- New typed cause `native_failed_before_provider_turn`
(`server/src/services/native-runtime/failed-before-provider-turn.ts`).
The executor raises it only when the launch gate fails, the run was not
cancelled, and the Runner turn ledger proves that no turn command was
issued. The proof is taken at the failure point, before the session
closes. In all other cases the original error is rethrown and today's
path stays.
- The executor quarantines the failed run's session root, so the
successor starts in a new root.
- `heartbeat/predecessor-retirement-fallback.ts` accepts the new cause:
it records the proof on `runnerProfileJson.failedBeforeProviderTurn`,
marks the run's reusable sandbox leases for replacement, and schedules
the successor in the existing `interrupted_preparation` lane (same cap
of 12 and the same backoff).
- `remote-workspace-unchanged-proof.ts` accepts the runner process,
runner command and pre-turn events only when that proof record is
present. It still refuses any turn evidence, an unsettled save by
another run on the same workspace, and setup writes. When the proof
refuses, the run's epoch is abandoned unapplied: the files are kept and
recorded, and the next run restores the last committed save into a fresh
root.
- When the proof refuses and the epoch is abandoned, the replacement
mark stays. The next acquisition replaces the sandbox. #21304's
retention hold makes `retireReplacedReusableSandboxLease` park the old
lease as `retained` instead of `pending_cleanup`, so the task continues
on a fresh sandbox and the sandbox that holds the abandoned files is
kept. When the proof holds (only the Runner's own session folder was
written), the old sandbox is replaced and cleaned up as usual.
- `task-interruption-presentation.ts` shows a pending
`interrupted_preparation` retry as `working` with no Retry until it is
due (plus the dispatch grace).
- The visible run error is now plain: "Setting up the computer took too
long, so the task is continuing on a fresh one." (for a setup timeout)
or "The computer could not be set up before the agent started, so the
task is continuing on a fresh one." The technical cause code stays in
`runnerProfileJson.failedBeforeProviderTurn.technicalCause`, the server
log and the run log.
- New crash point `native.provider_artifacts.before_capture` (fail mode
throws the typed transfer timeout at the gate).
`tests/runner-e2e/source-sdk-public-inputs.json` is refreshed for the
changed registry file.
- `tests/restart-chaos/main.ts`: `RESTART_CHAOS_EXTERNAL_CRASH_CONTROL`
lets an outside driver arm fail-mode crash points without a restart
fault.

## Verification

Local (targeted):
- `server/src/__tests__/heartbeat-native-predispatch-retry.test.ts`: 8/8
pass. New cases: fresh remote sandbox and fresh plugin-task sandbox
continuation (now also asserts the plain visible text with no
identifier-looking tokens, and the technical cause in the proof record),
the ambiguous hold when the ledger cannot prove no turn, and backoff
plus the lane cap.
- `server/src/services/remote-workspace-unchanged-release.test.ts`:
27/27 pass. New cases: hand-on when the ledger proves no turn; refusal
on a recorded turn, an unsettled predecessor save, and setup writes. Two
more cases follow the marked lease through the next acquisition's
replacement decision (`readReplaceBeforeReuseMarker` +
`reusableSandboxReplacementPlan` + `retireReplacedReusableSandboxLease`)
and then through `sweepPendingCleanupLeases`. "replaces the marked
sandbox when the failed run provably left its files unchanged": the old
lease goes to `pending_cleanup` and the sweep destroys it. "continues on
a fresh sandbox and parks the one that holds abandoned files, which
cleanup leaves alone": the old lease is `retained` with the
abandoned-files reason and is held by the receipt, the sweep does not
destroy it, and the file is still there.
- Red without the product change (product files reset to the base, tests
kept): 8 failed, 25 passed of 33. All 8 failures are the new cases. The
visible-text assertion alone was red on the previous commit: the error
read "The computer could not be prepared before the agent started
(native_provider_artifacts_transfer_timeout: the upload of 536870912
bytes did not finish within 120 s). The task continues on a fresh
computer."
- `task-interruption-presentation.test.ts`,
`native-provider-artifact-gate.test.ts`,
`crash-point-call-sites.test.ts`: 79/79 pass.
`packages/shared/src/crash-points.test.ts`: 16/16.
`tests/restart-chaos/crash-points.test.ts`: 12/12.
- `tsc --noEmit` clean for `server`, `packages/shared` and
`tests/restart-chaos`.

Daytona (local servers, Daytona sandboxes, image `sha256:787c3e9c…`,
these commits rebased onto a298ca7fbc, gate failures forced with the
crash point):
- F1, wave `w-20261113T042929-s2301`: 4 OpenCode tasks, one forced
failure on each first run. 4/4 failed runs recorded cause
`native_failed_before_provider_turn`. Each successor (retry reason
`interrupted_preparation`) was created 0.0–0.1 s after the failure and
started 1–3 s later, each on a NEW sandbox. Successor first agent event
at 32.8–56.6 s. 4/4 tasks done hands-off. 0 Retry banners. 0 blocked. No
operator wording. No reconciliation codes.
- F3, wave `w-20261113T043934-s2303`: 1 task, 3 consecutive forced
failures. Retries started 7 s, 35 s and 57 s after creation (backoff).
The 4th run succeeded. 0 Retry banners while pending. Not proven: that
retries 2 and 3 in F3 got fresh sandboxes (proven in F1 only).
- F2 (what bootstrap writes before the first turn): the whole sandbox
workspace was sampled every 5 s on all 4 failed runs, right after the
workspace was ready and at the gate failure. Bootstrap wrote nothing to
project files. All 14 new paths are under
`<workspace>/.paperclip-runtime/paperclip-runner/sessions/<session-sha>/`
(runner state, identity and lineage, `runtime-context.json`, the
instructions copy `AGENTS.md`, OpenCode config, data and log). The only
deletion was the staged agent-files `AGENTS.md` (or a
`.paperclip-upload-<uuid>` temp file). The successor never reuses them:
new session root, new workspace generation, restored from the snapshot.
3 of the files are byte-identical when recreated; the rest are
run-specific and recreated by the successor's own bootstrap. The project
workspace is not a git checkout there. Raw manifests are kept with the
PR author.
- The Daytona runs used the error text before the plain-wording commit.
The wording change is covered by the unit test above.

## Risks

- Scope: this applies only where the gate exists: remote OpenCode
sandboxes with controller persistence. Local runs cannot hit it.
- The proof is the Runner's in-memory turn ledger, read at the failure
point. In unit tests it is mocked. If a turn may have been sent, today's
ambiguous hold stays (tested).
- At the lane cap (12) the run stops with a working Retry instead of
waiting longer. A later change can move this onto the shared hourly
limit.
- A direct "is the workspace dirty" check by the Runner is not added,
because it needs a new Runner command. The unchanged proof instead
relies on the recorded evidence and refuses on any turn evidence,
unsettled saves or setup writes.
- Whether a replacement mark gives a fresh sandbox depends on
`reusableSandboxReplacementPlan`. When it refuses, the sandbox is
resumed, but the session root and workspace root are still fresh.
- Refused case (abandoned files): the task continues on a fresh sandbox,
in a fresh session root and a fresh workspace root. The old files stay
on the parked sandbox, recorded in the abandoned-epoch receipt and
unapplied. That sandbox is kept under #21304's hold until the files are
recovered, discarded, or the retention period ends; this PR does not
change that hold.
- The changes in `heartbeat/` and `remote-workspace-unchanged-proof.ts`
were cleared with their owner under these conditions: evidence-scoped,
latest committed epoch only, files kept when unproven, negatives tested.
- `server/src/services/fresh-continuation-fallback.ts` is deliberately
untouched.
- Upgrade safety: runs saved by an older build have no proof record, so
the unchanged proof keeps its previous behavior for them.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use (shell, file edits, test runs). Earlier commits in this branch
came from Claude sessions in the same fix loop.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:48:51 -06:00
Nicky LeachandClaude Opus 5.5 41998d551c feat(plugins): send the claimant's credential on plugin task resume (#21335)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugin environment drivers run agent tasks on remote computers, and
a warm follow-up run resumes a parked Runner through the `resume` task
operation
> - `resume` carries only the durable identity and a bootstrap ticket,
which is enough while the parked Runner process is still alive
> - A provider sometimes has to start the Runner again to resume the
session: after it moved the parked session to a new sandbox, or after
the computer rebooted
> - A Claude Runner cannot start without its credential, and today only
the portable handoff path (`handoffCredential`) delivers one
> - This pull request lets a provider opt in to receiving the claimant
run's harness credential on `resume`
> - The benefit is that a provider can continue a parked session on a
new computer without Core orchestrating a handoff

## Linked Issues or Issue Description

Refs #21289
Related: #21305

**What existing behavior does this improve?**
Warm resume for plugin task providers that restart the Runner to resume
a session.

**Subsystem affected**
Plugin SDK environment task schema, the server's plugin Runner task
owner, and the plugin environment task docs.

**Current behavior**
`submit` can carry the harness credential (`credential`), but `resume`
cannot. A provider that must start the Runner again on resume has no
credential for a Claude task.

**Proposed behavior**
A provider that returns `resumeCredentialVersion: 1` from `describe`
receives `credential` on `resume` whenever Core would send it on
`submit`, resolved for the claimant run. Other providers see no change.

**Reason and benefit**
Providers can restart a Claude Runner on resume without the Core-run
handoff protocol.

**Breaking changes**
None. The field is opt-in: the SDK worker parses operations strictly, so
Core sends it only to providers that advertise the capability.

## What Changed

- `resume` gains an optional `credential`, with the same schema and
validation as `submit`'s.
- `describe` gains an optional `resumeCredentialVersion: 1` capability.
- The plugin Runner task owner adds the credential on `resume` when the
provider advertised the capability and Core resolved a credential, using
the same resolution as `submit` (`preparePluginRunnerCredentials`).
After a warm rebind, the claimant run's credential is sent.
- `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` documents the field and its
handling rules.
- The runner-e2e SDK public-input pin is updated.

`handoffCredential` on the session handoff path is unchanged.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts` (68 passed): `resume` accepts and
rejects the same 11 credential cases as `submit`; the capability is
optional and rejects other versions; a resume credential reaches the
plugin through the serialized worker RPC.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts` (151 passed): credential sent with
the capability and a resolved credential; omitted without a credential
or without the capability; the rebound claimant's credential is sent;
the credential appears in no metadata write and not in the sanitized
error, even when the worker error echoes it.
- `native-session-executor.test.ts -t "uses a task-only"` (5 passed):
the executor forwards the capability to the owner.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts` (46 passed).
- `tsc --noEmit` for `packages/plugins/sdk` and `server`.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- No server-side provider check on resume, unlike `submit`: the lease
has no reliable harness at resume time. Protection relies on the owner,
whose rebind refuses a changed harness, account or credential identity.
- The credential is the claimant run's current one, which can be newer
than the one that started the parked Runner. That is intended.
- A provider must advertise the capability only after its SDK pin
includes this schema; roll Core out first.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:40:46 +00:00
DottaandClaude Opus 5.5 7e7452e88f fix(native): keep a reattached Runner alive when its provider recovery deadline ends (#21321)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs on remote sandbox Runners. A same-run reattachment
adopts a live Runner after a failed attempt or a server deploy.
> - On staging, a remote Codex Runner is often gone when the next
attempt reattaches (`process_verification` / `process_not_running`). The
task then continues in a fresh session and loses its context.
> - The deploy does not stop the Runner. The logs show `controller
authority detached for restart; durable runner left alive`, and the new
controller adopts the same Runner after the deploy.
> - Our own cleanup stops it. The adopted Runner answers its recovery
reads slowly while its in-flight turn runs. The 900 s provider recovery
deadline then aborts the recovery, and the abort closes the transport
with `turn.stop` and `runner.suspend` and signals the process.
> - This pull request makes the recovery deadline detach from the
Runner, as a recovery read timeout already does. The server marks the
failure as an unanswered adopted Runner, so the next attempt replaces
the Runner from its saved journal and resumes the same provider thread.
> - The benefit is that a deadline no longer kills a live Runner and no
longer forces a fresh session. The task keeps its session context.

## Linked Issues or Issue Description

Refs #21314, #21272, #21161, #21023

**What happened?**
Codex runs on Daytona remote Runners failed often on staging. In three
sampled chains (attempt 2 of each), the same-run reattachment adopted
the live Runner, authenticated in under 1 s, and then waited the full
deadline: `native session provider recovery timed out after 900000ms`
(909 s, 930 s and 915 s). The deadline cleanup sent `turn.stop` and
`runner.suspend` (`provider turn stop timed out before runner
suspension`). The next attempt found the Runner gone 25–180 s later
(`process_not_running`, `runnerLifecycle: ready`, empty stderr) and
continued in a fresh session. During the 900 s wait, the in-flight turn
was still running Paperclip tools (`tool.execute` spans at 04:43 in a
recovery that started at 04:29).

**Expected behavior**
A remote Runner survives a recovery deadline and a control-plane deploy.
The next attempt continues the same provider session. A Stop still stops
the Runner.

**Steps to reproduce**
1. Start a Codex run on a remote Runner and let a turn start.
2. Fail the attempt so that its close keeps the Runner (for example, a
turn start that outlives its bound).
3. Let the same-run reattachment adopt the Runner, and make the Runner
stop answering recovery commands.
4. Let the provider recovery deadline expire.
5. See that the Runner process is gone and the next attempt reports
`process_not_running`.

**Paperclip version or commit**
master at b555087f0f (after #21314).

**Deployment mode**
Hosted staging (cloud), remote Daytona sandbox Runners, deploys every
20–45 minutes.

**Agent adapter(s) involved**
Native Codex (`paperclip_runner`, Codex app-server through
`paperclip-runnerd`).

## What Changed

- Add `NativeProviderRecoveryDeadlineError` (same message as before) and
use it for the provider recovery deadline in the native session runtime.
- In the Codex driver, a recovery aborted by this deadline detaches the
controller (no `turn.stop`, no suspend, no signal). A Stop and every
other abort still close and stop the Runner.
- On the server, `nativeRecoveryCommandUnanswered` recognizes the
deadline. The failure records `adoptedRunnerUnanswered`, so the next
claim carries `replaceUnansweredRunner`. That attempt fences the old
Runner lineage, launches a fresh Runner from the saved journal and
resumes the same provider thread (the #21272 path). It does not adopt
the slow Runner again for another 900 s.
- Update `doc/execution-semantics.md`.

## Verification

- Real `paperclip-runnerd` + fake Codex
(`runnerd-codex-transport-remote-retention.test.ts`): a retained Runner
is adopted, authenticates, then stops answering. A deadline abort leaves
it alive with no pending `turn.stop`/`runner.suspend`/`runner.shutdown`.
A Stop abort still stops it.
- Red/green: without the fix, the real-runnerd deadline test fails
(`expected 'gone' to be 'alive'`), the driver unit test fails (detach
called 0 times), and the server classification test fails. With the fix,
all pass.
- Measurements on real runnerd: the deadline cleanup took 10.0 s and
killed the Runner before. Now it returns in 0 ms and the Runner stays
alive. The replacement over the kept Runner's journal resumes the same
provider thread in about 2.2 s with one `thread/start` and one
`turn/start` (no replay).
- `npx vitest run
src/live/runnerd-codex-transport-remote-retention.test.ts
src/drivers/codex/codex-app-server-driver.recovery.test.ts
src/native-session-runtime.test.ts` (368 passed).
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "never
answered its recovery read"` (passed).
- `tsc --noEmit` for `packages/paperclip-runner` and `server` (clean).

## Risks

- A deadline now leaves a live Runner behind. The next attempt replaces
it through the launch fence, which fails closed when it cannot prove the
old lineage gone. This is the same risk as the existing recovery read
timeout detach.
- When the provider attempt budget is already spent, the kept Runner is
not replaced by this run. This also matches the existing timeout detach.
- Not in this PR: the upstream trigger. The first attempt abandons a
slow turn start (`runnerd did not report the provider turn identity` or
`turn.start timed out`) while Codex later runs that turn. Two sampled
chains then stopped the Runner on
`protocol_failure:tool_binding_mismatch` / `thread_binding_mismatch`
after a deploy reattachment. These need a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:30:22 -06:00
Nicky LeachandClaude Fable 5.2 314b893678 fix(native-runtime): deliver assigned gateway tools to every native harness through the PRP relay (#21274)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can have app connections. Paperclip gives the agent the
tools of those apps through an MCP gateway, with its approval and audit
rules.
> - Native harnesses reach the gateway in two ways today. In the relay,
the server keeps the gateway token and offers the tools over the
authenticated Runner channel. In the direct way, the server gives the
gateway URL and a bearer token to the harness.
> - Which way a harness uses comes from a list of special cases that
grew one harness at a time. Local Codex and all ACPX agents use the
direct way. The others use the relay.
> - The direct way puts a run-scoped token in a provider process. A
process that the agent starts can read it. A provider process that
serves more than one run can hold a revoked token.
> - This pull request replaces the list with one rule. Each native
harness gets its assigned gateway tools through the relay.
> - The benefit is one behavior for all native harnesses, and no gateway
token in a provider process.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #13942 (the relay), #17149 (ACPX accepts relayed tools) and #19912
(the design for turn-scoped ACPX MCP authority).

**What existing behavior does this improve?**

How a native harness reaches the tools of its assigned MCP gateway.

**Subsystem affected**

Native runtime in the server (`native-session-executor.ts`,
`native-session-resume.ts`). One Runner test file. One Rust test.

**Current behavior**

- `relayAssignedMcp` is true for plugin tasks, Cursor Cloud, OpenCode,
Codex on a remote target, and Dot.
- For local Codex and for each ACPX agent (Claude, Codex, Cursor, Pi,
Kimi, Grok, Copilot, Antigravity), Core puts
`PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` and
`PAPERCLIP_NATIVE_MCP_TOKEN` in the Runner environment. The ACPX host
gives them to the provider as an HTTP MCP server. The local Codex home
gets them in `config.toml`.
- The token is run-scoped. Core revokes it when the run ends.

**Proposed behavior**

- A native run that has an assigned gateway uses the relay. The
condition names no provider.
- Core builds the gateway tools on the control plane, offers them with
the other Runner tools, dispatches each call over PRP, and authorizes
each call again through the gateway.
- Core removes the three `PAPERCLIP_NATIVE_MCP_*` variables from each
Runner environment.
- The tool contract fingerprint has the relay marker for each target,
not only for remote targets.

**Reason and benefit**

- All native harnesses have the same behavior.
- No provider process holds a gateway token.
- A sandbox does not need to reach the gateway URL of the host.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
- The list of special cases is gone. Core validates the binding and
builds the relayed tools when a binding is present.
- Core does not list the gateway for a review run, Grok Bot or Muse. The
tool authority never offered app tools to those. A gateway error can
therefore not fail those runs.
- OpenCode keeps its `on_demand` catalog. That option depends on the
provider, not on the relay.
  - Core removes the three variables from each Runner environment.
- `server/src/services/native-runtime/native-session-resume.ts`: the
relay marker in the tool contract fingerprint is unconditional.
- `doc/execution-semantics.md`: the text about the tool contract
fingerprint now states the rule, the tool names and the cutover.
- App tool schemas. New module
`packages/paperclip-runner/src/drivers/tool-input-schema.ts` compiles
each schema by itself, in its declared dialect (draft-07, 2019-09 or
2020-12), with no shared `$id` registry. Core uses it when it builds the
relayed tools (`assigned-mcp-tools.ts`). Core does not offer a tool
whose schema has an unsupported dialect or does not compile, and writes
a log entry with the tool name and the reason. The run does not fail.
The ACPX tool bridge and the OpenCode MCP bridge use the same function.
The tools of Paperclip itself stay strict.
- Large arguments. The ACPX sidecar sends the arguments of a tool call
to runnerd with no display limit. The limit is the tool-call admission
limit of runnerd, 256 KiB. A larger call fails with a tool error that
gives the limit. The sidecar does not dispatch it.
- Tests in `native-session-executor.test.ts`,
`native-session-resume.test.ts`, `runtime-context-materializer.test.ts`
and `acpx_provider_backend.rs`.

Not changed in this pull request: the direct code path. It is still in
the Runner, but it receives no binding. #21311 deletes it.

## Behavior that each harness sees

| Harness | Before | After |
|---|---|---|
| Codex on a remote target, OpenCode, Cursor Cloud, Dot, plugin tasks |
Relay | No change |
| Local Codex | Direct: a `paperclip-assigned` server in `config.toml` |
Relay. App tools are dynamic tools named `app_<name>_<digest>` |
| ACPX agents | Direct: a second HTTP MCP server | Relay. App tools are
on the `paperclip` bridge as `mcp__paperclip__app_<name>_<digest>` |
| ACPX with a preconfigured home | No gateway | No change |
| Grok Bot, Muse | Got the variables and ignored them | Do not get the
variables |

- A relayed tool name has a maximum of 64 characters. With the
`mcp__paperclip__` prefix it has a maximum of 80.
- Permission behavior keeps the same intent. For Claude in the
`approve-reads` and `approve-paperclip` modes, app tools ask for
approval, as before. For the other ACPX agents the permission decision
does not use the server name.
- In a planning run or an ask run, the relay offers only tools with read
risk. The direct gateway offered all tools. This is stricter.
- When the catalog is too large, each harness gets the two compact tools
`paperclip_search_assigned_tools` and `paperclip_call_assigned_tool`.
- The four helper tools for resources and prompts
(`paperclip_list_resources`, `paperclip_read_resource`,
`paperclip_list_prompts`, `paperclip_get_prompt`) are not in the relay.
No harness loses them: a run token permits only `tools/list` and
`tools/call` (`run-preparation.ts`), so these four tools already fail
with `gateway_token_action_denied` on the direct path.

## State saved by the previous release

- **Local sessions of each provider start again one time.** The tool
contract fingerprint changes for local targets. The next run sees a
different fingerprint, starts a new provider session with the full task
context in the same workspace, and keeps the old saved state. This is
the existing rotation for a tool contract change. Codex needs it,
because Codex cannot add tools to a thread that it resumes. For the
local providers that already used the relay, the restart is not
necessary. It is accepted so that the rule names no provider.
- **Remote sessions continue.** Their fingerprint does not change. For
ACPX, the attach of the next run replaces the saved tool set.
- **A kept Codex home.** Paperclip writes `config.toml` again at each
start and recovery. The old gateway server and its token are removed.
- **A deploy lands during a run, and the Runner survives.** The
controller attaches to the same Runner for the same run. The old
provider keeps its direct gateway connection. The controller detach
keeps the gateway token of that run, and Paperclip revokes it when the
run ends.
- **A deploy lands during a run, and the Runner is dead.** The Runner
starts again for the same run with its saved tool set and no direct
gateway. The assigned app tools are not available for the remainder of
that one run. The next run gets them through the relay. This applies to
each harness that moves to the relay. It has no test.
- Paperclip refuses no saved state and discards none.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`: `1508 passed
| 7 skipped`, 1 failed. The failed test is "archives a damaged prior
epoch before completing guarded same-task replacement with real runnerd"
(`ENOENT … authority-epochs`). It fails in the same way on `master` at
53608bd1d6 and with the product change removed, on macOS.
- `cd server && npx tsc --noEmit -p tsconfig.json`: no errors.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/runtime-context-materializer.test.ts`: `18 passed`. `npx tsc
--noEmit -p tsconfig.test.json`: no errors.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes. `cargo test -p paperclip-runner-core --lib
acpx_provider_backend`: `50 passed`.

- `npx vitest run src/drivers/tool-input-schema.test.ts
src/drivers/runner-tool-bridge.schemas.test.ts
src/drivers/runner-tool-bridge.test.ts
src/cli/acpx-runtime-sidecar.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: `21 passed (21)` files, `377 passed` tests.
`pnpm run typecheck:typescript`: no errors.
- `cargo test -p paperclip-runner-core --test acpx_event_payload --test
tool_input_schema_fixtures`: 10 and 3 passed.
- Schema tests use one shared fixture
(`tool-input-schema-fixtures.json`) for the TypeScript compiler, the
tool bridge and the Rust `begin_call`: a draft-07 schema, a second
schema with the same `$id`, 2019-09, 2020-12, draft-04, an unknown
dialect, a `$ref` with no target, and an invalid keyword. Before the fix
the bridge failed with `no schema with key or ref
"http://json-schema.org/draft-07/schema#"`.
- A tool schema whose `$id` is the URI of a meta-schema cannot damage
the shared compiler. Before this fix, `removeSchema` deleted that
meta-schema, and later schemas failed to compile for each run in the
process. A test with 7 reserved ids failed before the fix and passes
now.
- Core limits the compile work of each run start. A schema over 512 KiB
is not compiled, and one listing compiles a maximum of 768 KiB of
schemas. These are the limits of the Runner. A tool over a limit is
served only through the search and call tools. Core keeps the outcome
for each schema digest (1,024 entries) and yields to the event loop
before each compile. Four tests failed before this change.
- A sidecar test sends a tool call with 200 KiB of arguments and
receives them byte for byte, refuses a call with 300 KiB with the named
error, and keeps the replacement of a lone surrogate.

Tests, each seen to fail without the product change:

- "keeps assigned MCP credentials on the control plane for %s" now runs
for remote Codex, local Codex, OpenCode and ACPX Claude, Pi, Kimi,
Copilot and Cursor. It checks that Core builds the relay, offers the
tool, dispatches a call, and gives no `PAPERCLIP_NATIVE_MCP_*` variable
to the Runner. Without the change, the five new harness cases fail.
- "relays the assigned gateway when a deploy lands mid-run…" runs for
Claude and Pi.
- The fingerprint tests keep the old local fingerprint as a retained
value and check that the new one is different. The remote fingerprints
for Codex and OpenCode do not change.
- "drops a retained direct gateway server when the relayed launch has no
binding": a Codex home with the old `config.toml` entry loses that entry
at the next start and at a recovery of the same run.
-
`next_run_attach_offers_relayed_gateway_tools_to_a_direct_gateway_checkpoint`
runs for Claude and Pi.

Not run: a live run with a real provider and a real app connection for
each harness. Not run: `pnpm verify` and the full suite. CI supplies
that coverage.

## Risks

- **An ACPX app call has a limit of 256 KiB for its arguments.** The
direct gateway accepted a request body of 10 MB. Codex and OpenCode
permit 480 KiB. To get nearer to 10 MB, runnerd needs a larger admission
limit for tool calls or a protocol that sends arguments in parts. That
is not in this pull request.
- **A tool with a draft-04 or draft-06 schema is not offered.** Ajv 8
does not support draft-04, and the TypeScript and Rust validators must
agree. On the direct path the provider got such a tool.
- **Core compiles app schemas on its event loop.** Before this pull
request Core did not compile them. With the limits above, one valid
schema near 512 KiB can hold the loop for about 0.65 s, and one listing
for about 1.2 s in total (measured on a developer machine). An app that
changes its schemas at each listing gets no help from the cache. A
worker thread with a timeout would remove this; it is not in this pull
request.
- **Tool names change for local Codex and all ACPX agents.** A saved
agent instruction that names a gateway tool directly does not match the
new name.
- **Each local native session starts again one time after the deploy.**
The provider conversation of that session does not continue. The task
context and the workspace stay.
- **A run that is in flight at deploy, with a dead Runner, has no app
tools until it ends.**
- **Names of 80 characters are not tested live** with Kimi, Copilot,
Grok, Antigravity and Codex through ACPX. Pi already shortens a name
longer than 64 characters.
- **Managed Claude and AWS AgentCore now receive relayed app tools.**
This is not tested live.
- **A gateway token expires after 1 hour.** The relay keeps the token on
the controller for the whole run, as OpenCode and remote Codex do today.
A longer run loses its app tools on both paths. This pull request does
not change that.
- **The gateway URL must have the path `/mcp/gateways/<id>`.** With a
path prefix in the API base URL, a run with an assigned gateway fails
with `assigned native MCP gateway path is invalid`. This was true for
the relayed harnesses before. It is now true for all.
- No test covers the rule that Core does not list the gateway for Grok
Bot and Muse.

## Follow-up

The direct path is now unreachable. #21311 deletes it. That pull request
is in the same stack, above the pull requests that change the same ACPX
host files.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff and ran the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated each harness
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 00:16:58 -08:00
Nicky LeachandClaude Opus 5.5 2483cb46c2 feat(plugins): deliver configured environment to plugin Runner tasks (#21280)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent gets environment variables from its execution environment,
its own configuration, its project, and its routine. Secrets are
resolved for each run.
> - A plugin environment can run the agent as a native Runner task on
its own host, through the environment task contract.
> - That contract has no field for the agent's variables. Runs on a
plugin environment get only the variables that the plugin sets itself.
> - This pull request adds a versioned contract field for the values,
sends them on a session handoff, lets Codex commands inherit the
variables in an external sandbox, and lets a plugin protect the
variables that it sets itself.
> - The benefit is that plain and secret variables work on plugin
environments in the same way as on local environments.

## Linked Issues or Issue Description

Refs #21233. That pull request by @lucktastic finds the same gap and
proposes the same capability and field names
(`configuredEnvironmentVersion: 1`, `configuredEnvironment`). This pull
request uses the same names, so providers implement one contract. It
differs in these places:

- Codex commands get the variables through an external-sandbox denylist,
not through key names sent at `prepare`. A retained Runner does not need
a new preparation when a key is added or removed.
- A session handoff `resume` gets the values, because it starts a new
Runner on another computer. Value changes on the same computer need no
new field: Core's process reuse key includes the values, so Core starts
a new task.
- A driver can declare `protectedEnvironmentKeys`. Paperclip rejects or
warns when one is saved.
- The run log names skipped variables only for an older provider.

Thank you to @lucktastic for the diagnosis and the contract shape.

**What happened?**
An agent has environment variables, plain and secret. Its runs on a
plugin task environment do not get them. `PAPERCLIP_CONFIGURED_ENV_KEYS`
in the run lists only the variables that the plugin sets.

**Expected behavior**
Each run gets the agent's configured variables.

**Steps to reproduce**
1. Set a plain and a secret environment variable on an agent.
2. Assign the agent to a plugin environment that provides Runner tasks.
3. Start a run. Read the names in `PAPERCLIP_CONFIGURED_ENV_KEYS` in a
command that the agent runs.
4. The agent's names are not in the list.

## What Changed

- **SDK contract** (`packages/plugins/sdk/src/environment-tasks.ts`):
- A provider advertises `configuredEnvironmentVersion: 1` on `describe`.
- `submit` gets `configuredEnvironment`. `resume` gets it only for a
session handoff.
- Names, count, and sizes have the same limits as the Runner projection.
- **Server**:
- `createRunnerdBackendWithinSessionClaim` gives the run's configured
projection to the plugin task owner. The projection has no provider
credentials and no generated runtime keys.
- `createPluginRunnerTaskOwner` sends the fields only to a provider with
the capability. An older provider gets no new field, and the run log
names the variables that were not set. The log never prints values.
- **Codex** (`codex-security-config.ts`):
- An externally sandboxed launch uses `inherit="all"`,
`ignore_default_excludes=true`, and an `exclude` list of Codex's own
credentials, instead of an `include_only` list fixed at preparation
time.
- A test reads the Runner's Codex environment keys, and fails if a new
key is not classified.
  - Host launches keep the bounded allowlist.
- **runnerd** (`codex_provider.rs`): `runtime_notification_codex_args`
still grants the notification files to read-only profiles when no
allowlist is present. Plan mode in an external sandbox needs these
grants.
- **Protected keys**:
- An environment driver can declare `protectedEnvironmentKeys` in its
manifest.
- Paperclip rejects a protected key on an agent or environment that uses
the driver. The 422 has code `protected_environment_variable`.
- On a project, a routine, or another agent, Paperclip saves the key.
The response has a `configurationWarnings` notice that names the
environments. The project and routine pages show the notice as a toast.
The agent pages already show configuration warnings.
- **Docs**: `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` describes the
provider obligations.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the reviewed
SDK source pin.

## Verification

- `packages/plugins/sdk`: `vitest run tests/environment-tasks.test.ts`
(38 pass). Accepts bounded values. Rejects bad names, NUL bytes, too
many variables, and oversized values.
- `server`: `vitest run
src/services/native-runtime/plugin-runner-task.test.ts` (95 pass).
Values go only to a provider with the capability, never in `prepare`.
`resume` sends values only for a handoff.
- `server`: `vitest run src/services/protected-environment-keys.test.ts
src/__tests__/project-routes-env.test.ts` (10 pass). Rejection, notices,
and the project route response.
- `server`: all route suites that import the project, routine,
environment, or agent routes (45 files, 1689 pass).
- `packages/shared`: `vitest run src/validators/plugin.test.ts` (64
pass).
- `packages/paperclip-runner`: `vitest run src/drivers/codex
src/configured-environment.test.ts
src/live/runner-provider-launch.test.ts` (444 pass).
- `packages/paperclip-runner/runner`: `cargo test -p
paperclip-runner-core --lib runtime_notification` (9 pass, 2 ignored as
before).
- `pnpm typecheck` in `packages/shared`, `packages/plugins/sdk`,
`packages/paperclip-runner`, `server`, and `ui`.
- Not run locally: `pnpm verify --full` and browser suites. Required CI
covers the final revision.

## Risks

- **Existing providers:** low risk. Core sends no new field to a
provider without the capability. Their runs behave as before, with one
added log line when variables were skipped.
- **Codex in an external sandbox:** commands now see every variable in
the Runner environment except Codex's own credentials. The execution
environment builds that environment from an allowlist. If the Runner
gives Codex a new credential, the drift test fails until the key is
classified.
- **Saving env:** new rejections happen only when an installed driver
declares `protectedEnvironmentKeys`. No driver declares them today.
- **Providers:** they must keep the values transient and never save or
log them. The doc lists these obligations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:11:23 +00:00
DottaandClaude Opus 5.5 86f9555695 fix(native): give a Runner whose first unanswered read came on the last attempt one replacement attempt (#21322)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner, and a same-run retry
reattaches that Runner after an interruption
> - #21272 made the next attempt replace a Runner that left its recovery
read unanswered, instead of adopting it again
> - On staging, three runs after #21272 went live still failed: their
first unanswered read came on the third and last attempt, so no next
attempt existed
> - This pull request gives such a run one more attempt, used only for
that replacement
> - The benefit is that the replacement from #21272 can run when a
restart used up the middle attempt

## Linked Issues or Issue Description

Refs #21272, Refs #21161

**What happened?**

After #21272 (497ccba8f1) went live on staging at 05:34Z, three Codex
runs still failed with `native_session_recovery_failed:
HarnessCapabilityUnavailableError: read is unavailable: Error:
provider_transport_failed: PRP command session.snapshot timed out
(runner connected; pending ahead 0; last runner event seq … 55–84s
ago)`. The run events of all three show the same sequence:

1. Attempt 1 fails at 06:24 with `runnerd did not report the provider
turn identity`.
2. Attempt 2 starts at about 06:25. The 06:24–06:25 server restart ends
it with `local_runtime_api_listener_unavailable`. This attempt sends no
recovery read.
3. Attempt 3 adopts the same Runner. Its first recovery read gets no
answer. Attempt 3 is the last of three, so the run ends.

`adoptedRunnerUnanswered` was recorded on attempt 3, but no next attempt
read it. The replacement from #21272 never ran.

**Expected behavior**

A Runner whose first unanswered read comes on the last attempt is still
replaced once by a fresh Runner from the saved journal. The provider
turn is not sent again. The run does not end only because a restart used
the middle attempt.

**Steps to reproduce**

1. Start a Codex run in a remote Daytona Runner. Let attempt 1 fail at
turn start, so the Runner stays alive.
2. Restart the server during attempt 2, so that attempt fails without a
recovery read.
3. Let the Runner leave its recovery read unanswered on attempt 3.
4. See the run end with `session.snapshot timed out` and no replacement.

**Paperclip version or commit**

Staging with #21272 (497ccba8f1). Fixed on `master` after `bbafa78fe2`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`).

## What Changed

- `nativeUnansweredRunnerReplacementDisposition` (executor): when
attempt 3 fails because its adopted live Runner left a recovery read
unanswered, and that attempt was not itself a replacement, the failure
stays `retryable_failure` (`native_session_interrupted`, retry in 30 s)
instead of `native_session_retry_exhausted`.
- `claimNativeRestartRecoveries`: a failed remote retry at attempt 3
with `adoptedRunnerUnanswered` is not treated as an exhausted budget.
The claim is `reattach_remote_runner` at provider attempt 4 with
`replaceUnansweredRunner`.
- `nativeAdmittedProviderAttemptLimit` (executor admission gate): a
`reattach_remote_runner` claim with `replaceUnansweredRunner` is
admitted up to provider attempt 4. Every other claim keeps the limit of
3.
- `verifyRemoteRunnerOrSettledExit`: the replacement of a live Runner is
allowed up to attempt 4. An exited Runner keeps the fail-closed path.
- Bound: the extra attempt is a replacement (`replaceUnansweredRunner`),
so it never gets another one. Attempt 4 is exhausted by the existing
rule.
- `doc/execution-semantics.md` describes the extra attempt.

## Verification

- End to end (embedded Postgres):
`native-remote-retry-admission.test.ts` › "executes the one replacement
attempt after the last attempt's first unanswered recovery read, and no
further one". It seeds attempt 3 with the staging failure, claims it,
and executes the claim through `executePaperclipNativeSession`. The
provider is reached once, the run then ends at attempt 4, and no further
claim is made. With the old `nextAttempt > 3` gate this test fails
before the provider is reached.
- Red on `master`, green with the fix (7 tests, including "admits
provider attempt 4 at execution only for the replacement claim"):
- `native-runner-restart-recovery.integration.test.ts` › "gives a Runner
whose first unanswered read came on the last attempt one replacement
attempt: true" (embedded Postgres; the coordinator is seeded at attempt
3 with the staging message). On `master` no reattach claim was made.
- `native-session-executor.test.ts` › "bounds the replacement after …"
(4 cases: the last attempt's first unanswered read is retryable; a
replacement that also went unanswered, another failure, and attempt 4
stay terminal) and "replaces the live Runner on the extra replacement
attempt".
- Focused suites: `native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation`: 1737 passed, 21 skipped.
- `pnpm typecheck` in `server` passes.
- Not run locally: the container-gated remote Runner suites and the full
suite (CI covers it). Why the remote Runner stops answering is a
separate investigation and is not changed here.

## Risks

- Low to medium. This adds at most one provider attempt per run, and
only after an unanswered recovery read on a live adopted Runner. The
attempt replaces the Runner through the launch fence, which proves the
old Runner and its lineage gone or fails closed. No provider turn is
sent again.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events),
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:27:35 -06:00
Nicky LeachandClaude Fable 5.2 23c2343015 feat(server): read a retained provider pack from its Runner image (#21308)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner executes agent work in a sandbox. It needs a provider
pack of agent tools, and the server uploads that pack when the sandbox
does not have it
> - A retained session keeps the pack it started with. When that pack is
not the server's own, the server downloads the pack's transport archives
from a registry (#19779)
> - Today the server looks for them in the application image tagged
`sha-<commit that built the pack>`
> - `master` now publishes the Runner one time for each set of inputs,
as `ghcr.io/paperclipai/paperclip-runner`. A pack from that image names
the commit that published the image, and few commits get an application
image, so that tag often does not exist
> - This pull request reads such a pack from its Runner image, and keeps
the old source for every other pack and for a Runner image that cannot
be read
> - The benefit is that a retained session can still get its pack when
the pack came out of the published Runner image

## Linked Issues or Issue Description

Refs #19779 (the retained pack source), #21228 (the pack field
`runnerInputsDigest`), and #21204, #21229, #21230 (the Runner image). No
issue exists for this change.

**Problem or motivation**

`retained-provider-pack-oci.ts` reads
`ghcr.io/paperclipai/paperclip:sha-<runnerSourceRevision>`. It needs
that image to exist, to carry the pack, and to have one layer with a
known build-history line. For a pack from a published Runner image,
`runnerSourceRevision` is the commit that published the image.
`docker.yml` writes `sha-<commit>` only when its `master` run for that
commit completes, and a newer push can replace a waiting run. The
restart-chaos harness has the same assumption in another place: it
requires the pack of a sandbox image to be built by the commit in the
image's `org.opencontainers.image.revision` label.

**Proposed solution**

A pack that names its Runner image in `payload.runnerInputsDigest` is
read from `ghcr.io/paperclipai/paperclip-runner:inputs-<digest>`. A
sandbox image can name the commit that built its Runner in a label of
its own. No application image carries a pack with that field yet, and no
image carries that label, so behaviour today does not change.

**Alternatives considered**

- Keep the application image as the only source and require an
application image for every Runner publication. Publication would then
depend on `docker.yml`, which serves one ref at a time.
- Take a fixed number of layers ("the image has four, take the last
two"). A server in the field cannot be changed when the image gets
another layer. The server now uses the rule that holds for any number of
layers: the full image is the slim image plus transport layers.
- Fall back to the application image for every failure of the Runner
image. A Runner image that exists and does not match the pack is a
fault, and a second source would hide it.

**Roadmap alignment**

This is build and runtime infrastructure. It adds no product feature and
does not overlap an item in `ROADMAP.md`.

This is part 3 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image (this
pull request)
4. #21310: a registry build cache for the Runner image build

This pull request uses no code from parts 1 and 2. It is stacked on them
because all four parts edit `doc/runner-image-contract.md`.

## What Changed

- `server/src/services/native-runtime/retained-provider-pack-oci.ts`,
for a pack whose `payload.runnerInputsDigest` is 64 hexadecimal
characters:
- The server reads the full index `inputs-<digest>` of the Runner image.
The index must carry `io.paperclip.runner.inputs-digest` equal to the
pack's value, `org.opencontainers.image.revision` equal to the pack's
`runnerSourceRevision`, and a valid digest in
`io.paperclip.runner.slim-digest`.
- It reads the slim index by that digest, never by the slim tag, and
checks the bytes it gets against the digest.
- For the pack's platform, the layer digests of the slim image must be
the first layer digests of the full image, in the same order. The layers
after them are the transport layers. There must be at least one and at
most eight, and their total size has the existing limit.
- The server downloads only those layers. It checks each against its
descriptor and refuses every entry that is not a transport archive, the
receipt or the `provider-pack.archives` directory. A file may not be in
two layers.
- When the Runner image cannot be read, the server reads the application
image as before. That covers a registry error status for the token, for
either index or for either manifest: for example 404 for a tag or a slim
index that does not exist. Nothing is written before that point.
- A Runner image that can be read and does not match fails with
`runner_remote_provider_source_invalid`. The server does not try the
application image then.
- The same file, for every other pack: the old path, unchanged in what
it checks. Its code moved into functions that both paths use (`request`,
`pullToken`, `platformImage`, `receiveTransportLayer`).
- The downloaded layer file is removed as soon as it is unpacked.
Before, the importer removed it after its checks, so
`retained-provider-pack-source.ts` loses that one line.
- The cache record gets an optional `runnerImage` field with the inputs
digest and the digests of the transport layers. `transportLayerDigest`
is the last of them and `transportLayerBytes` their total. An older
server ignores the new field.
- `tests/restart-chaos/daytona-image.ts`: when the image has a non-empty
label `io.paperclip.runner.source-revision`, the pack must be built by
that commit. Otherwise the pack must be built by the commit in
`org.opencontainers.image.revision`, as before. The receipt field
`providerPackRunnerSourceRevision` records the pack's commit.
- `doc/runner-image-contract.md` has a new section, "Reading the
transport archives alone". `tests/restart-chaos/README.md` describes the
label.

One choice needs a reviewer's eye. When the full index can be read and
the slim index cannot (a registry error status), the server falls back
to the application image; it does not fail. Without the slim manifest
the server cannot tell which layers are transport layers, so it has not
accepted any byte of the Runner image, and the old source has its own
complete checks. A slim index that is read and has other bytes than its
digest still fails.

## Verification

Linux (`node:24.22.3-bookworm`, GNU tar 1.34, zstd 1.5.4, the worktree
mounted read-only):

- In `server/`: `retained-provider-pack-source.test.ts` (34 tests, 15 of
them new), `provider-pack-archive.test.ts`,
`remote-provider-pack-verification.test.ts`,
`remote-provider-pack.test.ts`: 127 tests pass in 4 files.
- `node node_modules/vitest/vitest.mjs run --config
tests/restart-chaos/vitest.config.ts
tests/restart-chaos/daytona-image.test.ts`: 25 tests pass, 5 of them
new.

macOS (Node 26.12.0):

- `daytona-image.test.ts`: 25 tests pass.
- `retained-provider-pack-source.test.ts`: the 15 new tests pass. Six
older tests fail, with and without this change. They run the real
uploader, which needs GNU tar, and this host has bsdtar.
- `tsc` on the two changed server files and the test, and on the two
restart-chaos files, with the settings of their test projects: no error.

The new cases:

- Read from the Runner image with 6 registry requests (two indexes, two
manifests, two layers), none to the application image and none for the
slim tag.
- Take every layer after the slim image's: one transport layer, and
three.
- Fall back when the tag does not exist, when the token is refused, and
when the slim index cannot be reached.
- Refuse, with no request to the application image: an index for other
inputs, an index for another revision, no slim annotation, a slim
annotation that is a tag, a slim index with other bytes than its digest,
a slim image that is not a prefix of the full image, a slim image equal
to the full image, nine transport layers, and a transport layer that
holds a pack file.
- Four deliberate faults in the reader each make a test fail: the slim
index bytes are not checked, no prefix check, the slim annotation is not
validated, and no bounds on the number of transport layers.

A real run, in the same Linux container (`linux/arm64`), with no
credential:

- The script took `provider-pack/provider-pack.json` of `linux/arm64`
from the published image
`ghcr.io/paperclipai/paperclip-runner:inputs-0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`.
Pack digest
`sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f`,
`runnerSourceRevision` `25e9a9c2e9277f3be8649be8c809cc81bde6f0c5`.
- `resolveProviderPackUploadSource`, with another pack as the server's
own, returned `{ archiveOnly: true }` after 23 seconds. That function
calls `receiveRetainedProviderPack` and then the importer's checks: the
manifest inside the overlay archive is the selected manifest, every
archive matches the receipt, and the content of every file archive
matches the digest of its file.
- The server code made 9 requests: one anonymous token request, the full
index, the `arm64` manifest, the slim index `sha256:f63ea436…4f3c` by
digest, its `arm64` manifest, and two layers, each with one redirect. No
request went to the application image.
- The cache record:

  ```json
  {
    "schema": "paperclip.retained-provider-pack-source.v1",
    "sourceRevision": "25e9a9c2e9277f3be8649be8c809cc81bde6f0c5",
"providerPackDigest":
"sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f",
"imageIndexDigest":
"sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162",
"imageManifestDigest":
"sha256:73dbe864fbab6208616f0aa6f1610a7ece4ef22f68a73d30901cbaea60139589",
"imageConfigDigest":
"sha256:740e48463d5caf0bfead4b923cdbd8b220acb93d058419c3db9d00c7f1831d65",
"transportLayerDigest":
"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf",
    "transportLayerBytes": 426602696,
    "runnerImage": {
"inputsDigest":
"0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca",
      "transportLayerDigests": [

"sha256:6a340088ca2cfa85f10f0385d9c8f5cdccb22c3a2bbb2062620b1d90616ba5b5",

"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf"
      ]
    },
"receiptSha256":
"2afdd26645845369d9e851b1e6e2884eb6c3884ea7c4866a1eeb2efb9fb230d7",
    "verifiedTarPaths": true
  }
  ```

- `uploadVerifiedProviderPack` then installed the pack from that source
through a local stand-in for the sandbox, with no further registry
request, in 4 seconds. The installed tree is 1,249,060,841 bytes. Its
`provider-pack.json` is the selected manifest, all 7 artifacts of the
manifest have the digest the manifest names, and the pack's own Node
prints `v24.22.3`.

Never run:

- The same real run for `linux/amd64`, whose transport layers are about
1 GB.
- A session in a real sandbox. The real run used a local stand-in for
the sandbox.
- The fallback against the real registry. Fixtures cover it.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The old path was moved into shared functions. Its checks are the same,
and its 19 existing tests pass on Linux, but a review must read that
code as changed.
- The layer file of the old path is now removed before the importer's
checks, not after them. The importer does not read it.
- The reader depends on two rules of the image contract: the full index
names its slim index, and the full image starts with the slim image's
layers. A Runner image that breaks either rule is refused, with no
fallback.
- The fallback covers a registry error status only. A network error ends
the import, as it does today.
- The label `io.paperclip.runner.source-revision` is new and no image
carries it yet. The change to the sandbox image that sets it must use
this name.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:02:19 -08:00
DottaandClaude Opus 5.5 0477f622d6 fix(recovery): keep the sandbox that holds abandoned unsaved files out of replacement and cleanup (#21304)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in remote sandboxes, and the workspace files live in
those sandboxes until a save
> - When a run cannot resume exactly, the fresh continuation fallback
(#21135) abandons the old workspace epoch and keeps its unsaved files
"recorded and unapplied" on the old sandbox
> - The abandonment clears the dirty flag, the writer and the location,
so the workspace looks clean. No code reads the abandonment receipt
> - A later failed resume can then replace the sandbox, and cleanup can
destroy it. That sandbox holds the only copy of the unsaved files
> - This pull request adds one shared hold. The task continues on a new
sandbox, and the old sandbox is parked until its files are saved,
discarded, or kept for 7 days
> - The benefit is that the fallback keeps its promise. Abandoned files
are not lost

## Linked Issues or Issue Description

Refs #21135
Refs #21297

**What happened?**
After the fresh continuation fallback abandoned a workspace epoch, the
receipt in `remoteWorkspaceAbandonedEpochs` was write-only.
`reusableSandboxReplacementIsSafe` saw a clean workspace with a
committed save and allowed replacement.
`retireReplacedReusableSandboxLease` then moved the old lease to
`pending_cleanup`, and the cleanup sweep destroyed the sandbox with the
unsaved files. `nativeWorkspaceExportPendingForLease` (the check that
the sweep, the retained orphan settlement and the scoped destroy paths
use) also answered "nothing pending".

**What did you expect?**
A sandbox that holds abandoned, unsaved files stays until the files are
saved, a person discards them, or a stated retention period ends.

**Steps to reproduce**
1. Abandon a remote workspace epoch whose location names sandbox L.
2. Let the next run fail to resume L.
3. The replacement decision says safe, L goes to `pending_cleanup`, and
the sweep destroys it.

## What Changed

- New `server/src/services/remote-workspace-abandoned-files-hold.ts`
holds the single predicate for "this lease holds abandoned files". It is
one SQL fragment (`abandonedWorkspaceFilesHeldSql`) plus a TypeScript
wrapper (`abandonedWorkspaceFilesHeldOnLease`). A lease is held while a
workspace in its company has an abandoned-epoch receipt whose
`location.providerLeaseId` equals the lease's provider lease id, and
that receipt has no `released` object. A receipt written by the current
release (no new fields) counts as unreleased.
- Replace and park. A failed resume of a held sandbox still lets the run
replace it, so the task continues at once on a new sandbox restored from
the last committed save. `retireReplacedReusableSandboxLease` parks the
old lease as `retained` with the `abandonedWorkspaceFilesParked` marker
(failure reason `abandoned_workspace_files_held`) instead of
`pending_cleanup`. `reusableLeaseCanBeResumed` and
`reusableLeaseCanBeCleanedUp` skip a parked lease, so later runs never
select it again. The check is a metadata read with no query, so the run
path does no extra work.
- Every path that can destroy a sandbox now honours the hold:
- `nativeWorkspaceExportPendingForLease` answers pending for a held
lease. This covers the pending cleanup sweep (defer), the orphaned
active lease sweep, the retained orphan settlement `heldBy`,
`destroyClaimedSandboxLease` (terminal workspace reaper
`destroyRemoteWorkspaceLease`, `destroyReusableSandboxLeases` and the
environment delete path, and run lease release with `destroy`), the
`destroyRunLease` dispatcher, and the plugin sandbox teardown retry.
- The run lease settlement (`environment_run_lease_active`) is the one
caller that ignores the hold (`{ destructive: false }`). It never
destroys a sandbox. It stops the sandbox and keeps it, or it moves a
lease without a sandbox to `pending_cleanup`, where the sweep parks a
held lease. If it waited for the hold, a dead run's lease would stay
`active` and block the retry for the whole retention period.
- `destroyReusableSandboxLease` parks the lease and returns without a
provider call. This covers obsolete reusable lease cleanup, the destroy
after an expired or mismatched resume, release with `expired`, and the
sandbox driver's `destroyRunLease`.
- The `retryPendingSandboxTeardown` dispatcher refuses a destructive
teardown of a held lease. Stop-only cleanup still runs, because a stop
keeps the filesystem.
- The retained orphan listing (`alwaysHeldSql`) uses the same SQL
fragment, so held orphans are not polled.
- Release. `releaseAbandonedWorkspaceFiles` marks a receipt `released`
with a reason (`recovered`, `discarded`, `retention_expired`) and an
actor. It moves the parked leases for that sandbox to `pending_cleanup`
when no other unreleased receipt holds them, and it writes a
`workspace.abandoned_files_released` activity entry on the task in plain
language. No export path for abandoned epochs exists today. The
`recovered` reason is the hook for one; this pull request adds no new
export mechanism.
- Retention expiry. New receipts carry `retainUntil` (7 days). An old
receipt uses `abandonedAt` plus 7 days. `nextEnvironmentCleanupAt`
includes the earliest retention end of any unreleased receipt. The
shared cleanup worker therefore wakes at that real deadline, and a
restart rebuilds it from the database. At the deadline the sweep runs
`expireAbandonedWorkspaceFileHolds`. This releases each due receipt as
an attributed system decision (actor `workspace-retention`) and logs the
held receipts and parked leases for each company. Each expiry always
writes a task entry: "Unsaved files from an interrupted run were kept
for N days, and their retention period has ended." The entry adds "They
are scheduled for removal." only when a parked lease was queued for
cleanup. A separate entry, "... kept for N days and have now been
removed.", is written only after the provider confirms the delete. Each
receipt is released in its own transaction. A failed receipt is logged
and skipped, and it is retried after a backoff (5 minutes, doubling, at
most 6 hours). The other receipts are still processed. The entries
reference a run or a task only if it still exists. If the task was
deleted, the decision goes to the server log only.
- Upgrade. A `pending_cleanup` lease (not in flight, not stop-only) that
an unreleased receipt holds goes back to `retained` and is parked by the
sweep. A sandbox that a release before this change already destroyed
cannot be recovered. Its receipt ends at its retention time like any
other receipt.
- Receipt trim (review P1). `abandonUnverifiedRemoteWorkspaceEpoch`
trims only released receipts and keeps the last 20 of them. It never
drops an unreleased receipt. A new abandonment of the same sandbox and
root replaces the unreleased receipt for that root. So a workspace holds
at most one unreleased receipt for each distinct sandbox and root
abandoned in the last 7 days.

## Verification

- `server/src/__tests__/abandoned-workspace-files-hold.test.ts` (19
tests). It follows the lease through the real acquisition path with a
failed resume, the real `abandonUnverifiedRemoteWorkspaceEpoch`, the
real `sweepPendingCleanupLeases`, and `settleRetainedOrphanSandbox`:
- `gives the next run a new sandbox at once and parks the held one,
which cleanup and later runs leave alone`
- `pulls a held lease already queued for cleanup back to retained, and
leaves it on the provider`
- `treats a receipt written before release fields existed as unreleased`
  - `records a retention expiry on the receipt`
- `hands the parked lease to normal cleanup once the files are
recovered` / `... discarded`
- `removes the files only after the retention period, and says removed
only after the provider confirms it`
  - `does not report removal when the provider delete fails`
- `records the end of retention without scheduling removal when another
receipt still holds the sandbox`
- `records the end of retention when it expires before the lease was
ever parked`
- `wakes the cleanup worker at the receipt's retention end, also after a
restart, and then goes quiet`
- `releases a receipt whose run was deleted and records it without the
run reference`
- `releases a receipt whose task was deleted without writing a task
entry`
- `keeps expiring later receipts when one fails, and retries the failed
one after a backoff`
- `keeps a held sandbox when the retained orphan settlement reviews it`
- `lists the sandbox for the retained orphan settlement again once the
receipt is released`
- `keeps holding the first sandbox after many later abandonments on
another sandbox`
- `replaces and cleans the sandbox as before when the receipt names a
different sandbox`
- `replaces and cleans the sandbox as before when no epoch was
abandoned`
- Red on the base revision (62d673cc3f): the park, upgrade, old-receipt,
expiry, many-abandonments and scheduling tests fail. The controls pass.
- Green: 19 of 19. Also
`environment-runtime-reusable-resume-replacement.test.ts` 17/17,
`environment-retained-orphan-sandbox.test.ts` 32/32,
`heartbeat-pending-cleanup-sweep.test.ts` 51/51,
`fresh-continuation-fallback.test.ts` 58/58.
- `tsc --noEmit -p server/tsconfig.json` and a tsc pass over the new
test file: clean. `environment-cleanup-work.test.ts` +
`heartbeat-pending-cleanup-sweep.test.ts`: 80/80.

- No existing test expectation was changed. Three CI failures on bb43f52
came from this change. Two came from the hold blocking the run lease
settlement: `ownership-reconciliation-report.test.ts > releases the
lease and frees the workspace for the retry once a preparation-only
epoch ends` and `remote-workspace-save-concurrency.test.ts > lets a
retry prepare the workspace again after its first writer was killed
during preparation`. The third, `native-session-executor.test.ts >
restores a warm backup into a created lease only when a settled restore
hold retained the stopped source`, came from a mocked database that has
no `execute`; the predicate now runs through `select`. All three pass
locally now. So do `heartbeat-orphaned-active-lease-sweep`,
`environment-run-lease-ownership`, `native-workspace-warm-reuse` and
`remote-workspaces.integration`.

## Risks

- Cost of a parked sandbox. Parking makes no provider call and changes
only the lease row. A sandbox that could not be resumed is not running.
A reusable sandbox that was released is already stopped and kept. In
both cases the cost is stopped storage for up to 7 days. One case is
different: if a destructive path finds a held warm sandbox that is still
running (`retained` after a keep-running release), the sandbox stays
running until the receipt is released. The expiry log line counts parked
leases for each company, so this cost can be seen.
- Run lease release for an `ephemeral` lease calls the provider's
release. Some providers delete the sandbox on that call. A held sandbox
normally has a reusable lease, so this pull request does not gate that
call.
- No schema change. The receipt is JSON in
`execution_workspaces.metadata`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:00:20 -06:00
DottaandClaude Opus 5.5 bcadbde449 feat(runner): record the OpenRouter host that served each OpenCode model call (#21298)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agents can use models through OpenRouter, and the Runner
proxies those model calls
> - OpenRouter routes one model across many hosts, and these hosts use
different quantizations (fp4, mxfp4, fp8)
> - Chat evaluations with identical prompts behave very differently
between runs, but the run log does not record which host served a call
> - The run usage shows `provider: "deepseek"` and `biller:
"openrouter"`, and `providerRequestId` is always `null`
> - This pull request records the serving host and the OpenRouter
generation ID for each model call in the run log
> - The benefit is that we can compare evaluation cohorts by host, and
the cost event names a real OpenRouter generation

## Linked Issues or Issue Description

**Subsystem affected**
Runner OpenCode provider proxy, runnerd usage normalization, and native
run usage accounting.

**Problem or motivation**
OpenRouter's default routing sends a request to one of many hosts. Each
host can use a different quantization. Chat evaluation results change a
lot between runs with identical prompts. Paperclip does not record which
host served a call, so we cannot compare results by host.
`usageJson.providerRequestId` is always `null` for OpenCode runs.

**Proposed solution**
Every OpenRouter completion response has a top-level `provider` field
and an `id` (the generation ID). Each streamed chunk has them too. The
Runner proxy reads them from the first chunk after it forwards the
bytes. The Runner attaches them to the usage of that call. The values go
to the run log, and the run's first generation ID goes to
`usageJson.providerRequestId`.

**Alternatives considered**
A new PRP event type such as `provider.upstream.served` would need a new
event enum value. An older control plane rejects an unknown event type.
A `harness.diagnostic` event shows in the task chat as a diagnostic. The
usage payload already flows to every consumer, and older consumers
ignore extra fields. Thus this change uses the usage payload.

**Additional context**
Run-log change only (AGENTS.md section 7). The data stays in
`heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch
`packages/shared/src/telemetry/` or OpenTelemetry.

## What Changed

- New `createUpstreamServedTap`
(`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`).
It is a pass-through stream. It pushes each chunk downstream first, then
it reads `provider` and `id` from the first SSE `data:` event (or from a
JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of
a JSON body. It skips encoded bodies and unknown content types. Any
parse problem or recorder error stops the scan and leaves the response
unchanged.
- `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The
OpenCode session keeps the most recent served call. Each
`item.completed` usage payload and `session.usage()` now carry
`upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint
does not include these fields, so the number of usage events does not
change.
- The OpenCode app-server proxy adds the two fields to
`thread/tokenUsage/updated`. Rust `usage.reported` now fills its
existing `providerRequestId` field and adds an optional
`upstreamProvider`. The TypeScript runnerd transport keeps both fields
when it rehydrates the usage notification. `usage.schema.json` documents
`upstreamProvider` (the protocol manifest and schema bundle are
regenerated).
- Server native turn accounting keeps the first generation ID the run
reports. The usage checkpoint and the native adapter result carry it as
`providerRequestId`, so `usageJson.providerRequestId` and the final cost
event name it.
- `doc/run-log-events.md` documents the fields in a new "OpenRouter
serving host" section.

## Verification

- Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's
stream through unchanged and records the host that served it on the
turn's usage". The test sends a streamed fixture with a comment line, a
host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks
that the bytes are identical and that the turn's usage names `Novita`
and the generation ID. The test fails without the proxy wiring.
- `upstream-served.test.ts`: streamed fixture, chunk split inside a
multibyte character, non-streamed JSON, malformed chunks, a truncated
JSON body, an encoded body, the 64 KiB limit, and a recorder that
throws.
- `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts`
(rehydration), Rust
`provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`,
and server `native-usage-accounting.test.ts` (first generation ID stays
stable across turns).
- Local: OpenCode driver, OpenCode proxy, and runnerd transport suites
(31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes
(one timing test failed once under load and passed alone). Runner and
server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks
pass.
- Latency: a local benchmark proxied a 400-chunk SSE stream 200 times
with the tap and 200 times without it. The median time to first byte was
292 µs with the tap and 259 µs without it (+33 µs, which is the extra
stream stage). The total stream time changed from 491.4 ms to 492.5 ms.
The tap spends about 5.5 µs of CPU to inspect the first chunk, after it
has forwarded that chunk. OpenRouter time to first token is hundreds of
milliseconds.
- Paid evidence: one single-case run of
`chat-features.runner-opencode.local.skills.fork-read-only` on this
branch. The result is in a comment on this pull request.

## Risks

- Low risk. All new fields are optional and additive. An older control
plane ignores the extra usage fields. `usage.schema.json` is not bound
by the event schema, and no validator enforces it on `usage.reported`.
- Saved state from the previous release: this change adds no persisted
run, session, checkpoint, or launch state. A saved session resumes
unchanged. A run that is in flight during a deploy reports usage without
the new fields until its Runner is new. `usageJson.providerRequestId`
then stays `null`, as before.
- `providerRequestId` on the run and its cost event is the run's first
generation ID, not each call's ID. The value is stable, so a late usage
receipt still matches the posted charge. Per-call IDs and hosts are in
the run log.
- If OpenCode starts a new model call before it reports the usage of the
previous call, a usage event can name the newer call. OpenCode reports a
message's usage when its step ends, so this is not expected.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code as an agent with tool use (shell, file edits, test runs). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:15:53 -06:00
DottaandClaude Opus 5.5 c730872974 fix(recovery): a chat turn waiting on the person is not stranded work (#21302)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In a chat, the agent's turn ends and the conversation waits for the
person's next message. The stranded-work sweep must leave that wait
alone
> - The sweep recognizes a finished chat turn by
`finalizationReasonCode: "conversation_turn_finished"` on the run record
> - Run completion re-reads the run through the 64 KiB read projection.
When the stored result is larger, that projection drops the finalization
fields, and completion wrote the reduced copy back
> - The chat turn then lost its finalization reason. Its reply was not
presented, and the sweep queued
`issue.productive_terminal_continuation_recovery` 30 s later. That is an
extra run while the agent waits on a person
> - This pull request keeps the full stored result through run
completion, and makes the sweep read the committed decision from its
durable rows
> - The benefit is that a turn waiting on a person is never treated as
stranded work

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `app-permissions.new-quarantined-action`, the
CEO's chat turn yielded with a `response_wake` continuation and waited
for a person to allow an app action. The stored run result was 113,937
bytes. Its saved record had the read projection's `truncated`,
`truncationReason: oversized_result_json` and `originalSizeBytes` fields
and no `finalizationReasonCode` or `decisionId`. The run's presentation
decision was `interaction_owns_conversation_response`, so no reply
comment was written. 30 s later the sweep queued
`issue.productive_terminal_continuation_recovery`, and the case failed
its run limit.

**Steps to reproduce**
1. Finish a native chat turn that yields `response_wake`, so
finalization records `conversation_turn_finished`.
2. Let the stored run result be larger than 64 KiB when run completion
reads it.
3. Run completion writes back the reduced copy without
`finalizationReasonCode`.
4. Run the stranded-work sweep. It queues a productive continuation run.

**Expected behavior**
The run record keeps its finalization fields, and the sweep leaves the
waiting chat turn alone. No new run starts until the person replies.

## What Changed

- `server/src/services/heartbeat/run-completion.ts`: run completion
reads the run with `unsafeFullResultJson` before it merges the native
result. It no longer writes the read projection's reduced copy back.
- `server/src/services/native-runtime/native-response-wake-wait.ts`:
`hasCommittedNativeConversationTurnWait` checks the run's committed
finalization decision (`conversation_turn_finished`) through
`native_run_finalizations` and `status_decisions` for the conversation's
own agent.
- `server/src/services/recovery/service.ts`: the sweep's
finished-chat-turn skip also uses that durable check, so runs already
saved with the reduced copy are recognized.
- Tests: `run-completion.test.ts` keeps the finalization record when the
stored result is larger than the read projection.
`heartbeat-process-recovery.test.ts` finalizes a real chat turn with the
eval's yielded result, saves the reduced run record, and expects two
sweeps to start no run.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts -t "larger than the read
projection"` — fails on master (the saved result is the truncated copy)
and passes here.
- `cd server && npx vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "chat turn that
yielded"` — fails on master (`continuationRequeued: 1`) and passes here.
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts
src/__tests__/heartbeat-process-recovery.test.ts` (614 passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- Run completion now reads the full stored result for one run. This is
one row; the read projection still serves every API read.
- The sweep check adds one indexed query for chat issues whose latest
run succeeded and whose run record lacks the reason.
- Previous-release saved state: no persisted shape changes. Runs already
saved with the reduced copy are covered by the durable decision check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:11:30 -06:00
Nicky LeachandClaude Opus 5.5 e65a3095a7 fix(runner): quiesce the Core-dialed route once the Runner commits its suspend (#21245)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Core runs agent sessions through the Paperclip Runner. With provider
ingress, Core dials the Runner over a PRP WebSocket.
> - When Core closes a session, it asks the Runner to suspend. The
Runner suspends, reports the result, and exits.
> - Core's ingress connection did not know that this exit was expected.
It treated the drop as a transport loss and redialed the Runner. Each
redial got a 404 until the close released the route.
> - The release (park) path already quiesces the connection before the
Runner goes away. The suspend path did not.
> - This pull request makes the suspend close quiesce the route in the
same way, once the Runner commits its suspend.
> - The benefit is that a clean suspend no longer looks like a lost
Runner. A drop before the suspend is committed still redials and
recovers as before.

## Linked Issues or Issue Description

**What happened?**
After a provider-ingress session closed cleanly, Core redialed the
Runner's ingress endpoint several times and got 404 each time. The
Runner state was `suspended`, `runner.drain` and `runner.suspend` had
both completed, and the Runner had exited with code 0. The redials
followed the outbound reconnect backoff (about 250, 500, 1000 and 2000
ms) and stopped when the registration release closed the outbound.

**Expected behavior**
After Core commits to `runner.suspend` and the Runner reports it
completed, the Runner's exit is expected. Core must not redial or
classify the drop as a transport loss. The release path already works
this way: it calls `outbound.quiesce()` before it releases the task.

**Steps to reproduce**
1. Open a session through a Core-dialed route (plugin provider ingress
or the `listen_ws` driver).
2. Close the session. Core drains, queues `runner.suspend`, and the
Runner suspends and exits.
3. See that the outbound reconnect loop redials the gone Runner until
the registration release closes it.

**Paperclip version or commit**
`master` at b4ca19f3a8.

**Agent adapter(s) involved**
Not adapter-specific (core bug). It affects any Core-dialed Runner
route: plugin provider ingress and the `listen_ws` driver.

**Additional context**
The redials do not fail the session. The plugin outbound uses a 7-minute
recovery grace, and `#failTransport` ignores failures after
`#closeAndSettle` sets `#closed`. The new test confirms this: without
the fix, the close still settles, but the route redials. This PR removes
the misleading redial. It does not change how a run's own failure is
classified.

## What Changed

- `RunnerExecutionDriver.controlPlaneRegistration` results can now
include an optional `quiesce()`. Core calls it when it has committed the
Runner's exit. The route then stops liveness and reconnect, and it keeps
the established receive/ACK path.
- `DurablePrpControlPlane.onRunnerExitCommitted(listener)` is new. The
authority calls its listeners after it saves a completed
`runner.suspend` or `runner.shutdown` result and before it sends the
acknowledgement. The Runner exits when it gets that acknowledgement, so
a route quiesced here never sees the exit as a loss.
- `awaitRunnerSuspensionBarrier` calls a new `onExitCommitted` callback
once, when its suspend command is `completed`. It also calls the
callback when it reuses a completed receipt and the durable state is
still `suspended`. A rejected, failed or unanswered suspend never calls
it.
- `DurablePrpCodexTransport` registers each control-plane route through
one `#registerControlPlane` helper. The helper records each route's
`quiesce` until that route is released. The suspend close subscribes to
`onRunnerExitCommitted` while its barrier runs and quiesces all of these
routes from that notification. The barrier's own `onExitCommitted`
covers a completed receipt that was acknowledged before this close.
Quiescing clears the tracked routes, so it runs once per close. The
helper copies property descriptors, so getters such as the `listen_ws`
driver's lazy `failure` stay live.
- The plugin task registration (`plugin-runner-task.ts`) and the
`listen_ws` driver registration (`native-runner-execution-driver.ts`)
implement `quiesce` with `outbound?.quiesce()`. This is the same call
that the release path makes.

## Verification

- New transport test with a real runnerd:
`packages/paperclip-runner/src/live/runnerd-codex-transport-suspend-route.test.ts`.
The test route acts like a Core-dialed provider ingress: on a socket
drop that it was not told to expect, it records a redial into a 404.
Before the fix the events were `connected, redial: 404, release`. After
the fix they are `connected, quiesce after runner.suspend completed,
release`. The close settles, and the durable Runner state is
`suspended`. A second case delays the durable-state read by 1.5 s, so
the Runner exits while the barrier still reads. That case fails without
the authority hook and passes with it.
- New barrier unit test in `runnerd-codex-transport.test.ts`. It shows
that the exit is committed only after the retried suspend completes,
never on a rejected or unanswered suspend, and on a reused receipt only
when the state is `suspended`.
- New server tests for the plugin task and `listen_ws` registrations.
`quiesce` reaches the active outbound and does not close it. Release
still closes it.
- Existing tests pass. These include the outbound release-path quiesce
test, the half-open and lease-expiry failure tests (a drop before the
commit still fails after its grace), and the plugin release/resume
facade tests.
- `pnpm verify` exited 2 (the change touches files outside the resolved
test graphs), so I ran focused suites. Locally: 8 runner transport files
and the 4 control-plane test files, 667 tests
(`runnerd-codex-transport*.test.ts` for suspension, release/resume,
remote retention, settlement, warm attachment and attachment recovery,
plus the new suspend-route test), and 5 server files, 148 tests (both
plugin Runner integration tests, the plugin task, the execution driver
and the outbound). Required CI supplies the broad coverage.
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Low risk. `quiesce` is optional. Registrations that do not supply it
behave as before.
- The quiesce happens only after the Runner reports a completed suspend.
A socket drop before that point still redials and recovers.
- The quiesced route keeps its receive/ACK path, so the Runner's last
frames before exit are still acknowledged.
- No persisted state or protocol change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:19:24 -08:00
DottaandClaude Opus 5.5 8a0f95849d perf(native-runtime): start the first turn without re-copying launch files, and never fail a run on a slow copy (#21287)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode in a remote sandbox must have a durable,
verified copy of the provider launch files before their first turn (the
rule from #19622)
> - Today each new run copies these files out of the sandbox, uploads
them and verifies them again, even when the company already has a ready
artifact with the same files
> - The whole copy runs under one five-minute clock. When eight runs
start together on small sandboxes, the copies compete, and slow copies
fail the run before its first turn
> - This makes cold starts slow (p50 273 s from run start to turn
submitted) and makes half of the first runs fail in a burst
> - This pull request reuses a ready artifact with the same digests,
shares one in-flight copy per company and manifest, and bounds each
network step instead of the whole copy
> - The benefit is that first turns start in about one minute on a cold
burst and in about 20 s when the artifact is already ready, and a slow
copy no longer fails the run

## Linked Issues or Issue Description

Refs #19622

**What happened?**
When 8 OpenCode runs started together on 1 vCPU Daytona sandboxes, 4 of
8 first runs failed before their first turn. Each run copied, uploaded
and verified the same provider launch files. The single 300 s clock over
the whole copy expired. Run start to turn submitted was p50 273 s and
p95 314 s.

**Expected behavior**
A run starts its first turn without a new copy when a ready, verified
artifact with the same files exists. A slow copy does not fail the run.

**Steps to reproduce**
1. Use a company with remote OpenCode agents on Daytona sandboxes with
durable storage.
2. Start 8 runs at the same time on new sandboxes.
3. See the time before the first turn and the failed runs.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- `retainReadyNativeProviderArtifacts` (`native-provider-artifacts.ts`):
reuse a ready, verified company artifact with the same paths and SHA-256
digests through a lookup only. No file is copied, uploaded or verified
again. The lookup (`readyNativeProviderArtifactCandidates`) matches
target, paths and digests in Postgres with jsonb containment on the
company's ready rows (existing partial index), newest first, limit 4. It
does not load or hash other retained versions. No migration.
- `shareNativeProviderArtifactCapture`
(`native-provider-artifact-capture.ts`): one in-flight capture per
company and manifest digest per server. Other runs wait for it, then
reuse its result.
- The single 300 s clock over the whole capture is replaced by a bound
for each network step (upload, verifying download): max(120 s, size at
256 KiB/s). A timeout has the typed cause
`native_provider_artifacts_transfer_timeout`. The capture makes up to 3
attempts from the local copy.
- Stop now interrupts the capture and the wait for a shared capture
(`ActiveNativeSession.admissionAbort`).
- `doc/opencode-cold-recovery.md` describes the new behavior.
- Tests: new `native-provider-artifact-gate.test.ts` (7 tests), 5 new
tests in `native-harness-backup-storage.integration.test.ts`, and new
executor variants `opencode-new-durable-slow-capture` and
`opencode-new-durable-ready` in `native-session-executor.test.ts`.

## Verification

Local (this branch, on master 9b613562b7):

- `npx vitest run
src/services/native-runtime/native-provider-artifact-gate.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
3 files, 1511 passed, 7 skipped.
- `npx tsc --noEmit -p server`: pass.
- Lookup scale: the integration test "loads only the ready artifact
whose launch files match, however many other versions the company keeps"
adds 200 ready non-matching versions. It returns only the matching
artifact. Without the SQL filter it fails (4 rows returned).
- Red/green: with master's `native-session-executor.ts` and the new
tests, `opencode-new-durable-slow-capture` fails with `TimeoutError: The
operation was aborted due to timeout`, and `opencode-new-durable-ready`
fails because the ready artifact is not looked up. With this change,
both pass.

Daytona (same commit content, cherry-picked onto the baseline commit
84a1ee3e11 with #21262; image `sha256:9461cc75…`; 1 vCPU sandboxes;
OpenCode):

| Wave | Shape | First runs OK | Run start → turn submitted | Other |
|---|---|---|---|---|
| Baseline `w-20261113T011027-s1702` | 8 runs started together | 4/8 (4
failed) | p50 273 s / p95 314 s | "only a getter" TypeError ×8,
`bootstrap_not_proven_incomplete` ×8 |
| Fix `w-20261113T031045-s1702` | same seed and shape | 8/8 | p50 66.0 s
/ p95 85.7 s | 8/8 tasks hands-off, 0 Retry banners; first agent event
p50 72.9 s / p95 90.0 s; exactly 1 upload, 7 runs waited on it (~65 s
shared capture); all failure codes 0 |
| Fix `w-20261113T034033-s1702` | 16 tasks, cap 8 | 16/16 | p50 19.1 s /
p95 24.3 s | second batch (artifact already ready, cold sandboxes) gate
time 24, 25, 26, 27, 31, 31, 45, 119 ms; first agent event p50 25.9 s |

## Risks

- The rule from #19622 still holds: a durable, verified artifact exists
before the first turn. Reuse only accepts a ready, verified artifact of
the same company with the same paths and digests.
- The first run per company and manifest per server still pays the full
transfer.
- Two servers can still upload duplicate artifacts for the same company
and manifest. Sharing is per server.
- No test drives Stop through the executor during a capture. Signal
handling is unit-tested only.
- A capture that fails all 3 attempts still ends blocked with a working
Retry. A follow-up PR (pre-turn failures continue automatically) removes
that.
- The documented five-minute deadline per capture is replaced by bounds
per network step. A very large artifact on a slow link can take longer
than five minutes in total, but each step still has a bound.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Default context
window; extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:48:55 -06:00
DottaandClaude Opus 5.5 0c04ff76eb fix(native-runtime): a follow-up that arrives as a warm session closes continues the task (#21279)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent session on a remote sandbox stays warm after a run,
and an idle timeout closes it after about 5 minutes
> - When a person's follow-up comment arrived just as that idle close
began, the new run failed with `native_session_supervisor_busy`
> - The idle close of the finished owner could also fail its final
checkpoint with `native_runner_checkpoint_owner_lost`, because the task
already pointed at the queued follow-up run
> - Then the retry was held for workspace repair and the task stayed
blocked; the environment could not be deleted (409) because its warm
runner could not be stopped
> - This pull request makes the follow-up wait for the idle close that
is in flight, and lets the finished owner save its final checkpoint
while its successor still waits in the queue
> - The benefit is that a follow-up that lands at the warm session's
idle close continues the task with no person involved

## Linked Issues or Issue Description

Refs #18011 (the checkpoint ownership fence this change extends).

**What happened?**

A follow-up comment arrived 295–340 s after the previous run ended,
while the warm session's idle close was in progress. The new run failed
with `native_session_supervisor_busy`. In another case, the idle close
failed its final checkpoint with `native_runner_checkpoint_owner_lost`,
the next run failed
`native_instruction_materialization_handoff_unavailable`, and the retry
was cancelled with `execution_reconciliation_required`. The task stayed
blocked. A hosted instance showed the same class as "previous run's
session on this sandbox could not be retired".

**Expected behavior**

The follow-up run waits for the idle close (inside the existing bounded
release wait) and then starts. The finished owner saves its final
checkpoint and closes. The task completes hands-off.

**Steps to reproduce**

1. Run an opencode agent on a Daytona sandbox with a warm session (idle
timeout 300 s).
2. Let a run finish.
3. Post a comment on the task 298–306 s after the run ended.

**Paperclip version or commit**

master `444dc18d98`.

**Deployment mode**

Local server with Daytona remote sandboxes (restart-chaos harness, light
lane). The same class was seen on a hosted staging instance.

## What Changed

- `claimWarmNativeInstructionCopy` now also waits when an idle close is
in flight for the session scope (`closingWarmNativeSessions`). The wait
uses the same deadline as the existing release wait
(`warmSessionReleaseWait`), so it adds no new wait and nothing before
`turn.start` on the normal path.
- `validateNativeRunnerCheckpointOwner` accepts the finished owner's
final checkpoint (accepted result, no new work) when
`issues.executionRunId` points at a successor run that is `queued` or
`scheduled_retry` for the same company and task and has no
`native_run_finalizations` row.
- Tests: a server test for a follow-up that arrives while the idle
timeout closes the warm session; four new scenarios (`queued_successor`,
`scheduled_retry_successor` × `runner.suspend`/`turn.stop`) and four new
negatives (`queued_wrong_issue`, `queued_new_work` × both commands) in
the early-retirement checkpoint matrix.
- Restart-chaos harness:
`RESTART_CHAOS_MIX_FOLLOW_UP_AFTER_END_MS=<min>-<max>` and
`RESTART_CHAOS_MIX_FOLLOW_UP_COUNT` post the staging-mix follow-up a set
delay after the task's last run ended, so a cell can aim the follow-up
at the warm idle close. The restart-chaos README describes both
variables. A failed post is retried with the same `clientRequestId`, the
grade check `after_end_follow_ups_served` needs an agent reply after
each post and the ledger marker, and the task budget includes the
planned waits.
- Harness: the wave database cleanup
(`tests/runner-e2e/controller-database.ts`) no longer waits forever when
a group signal (for example `timeout` or Ctrl-C) already stopped the
embedded Postgres. Before, `embedded-postgres` `stop()` waited for an
exit event that had already fired, and an interrupted wave hung after
the Daytona R6 sweep.

## Verification

- Fix 1, red without the product hunk: `native-session-executor.test.ts
-t "idle timeout is closing"` fails with `promise rejected "Error:
native_session_supervisor_busy"`. Green with the fix: 1 passed.
- Fix 2, red without the product hunk:
`native-runner-restart-recovery.integration.test.ts` fails 4 of 208
(`queued_successor` and `scheduled_retry_successor` ×
`runner.suspend`/`turn.stop`) with
`native_runner_checkpoint_owner_lost`. Green with the fix: 208 passed,
14 skipped. The two new negative cases pass in both states.
- Harness hang fix, red without the fix: the new
`controller-database.test.ts` case ("finishes cleanup when the database
already stopped because a group signal reached it") times out on cleanup
(`expected 'hung' to be 'cleaned'`). Green with the fix: 5 passed, 1
skipped.
- After the rebase onto master `7154030ef6`, both server test files
together: 1639 passed, 21 skipped.
- Restart-chaos harness tests (`staging-mix`, `staging-mix-retry`,
`observer`): 75 passed, including a test for the after-end retry, the
`after_end_follow_ups_served` grade check and the added budget. `tsc
--noEmit` passes for `server`, `tests/runner-e2e` and
`tests/restart-chaos`.
- `pnpm verify` exits 2 because `tests/restart-chaos/observer.ts` is a
global CI input. Required CI supplies the broad coverage.
- Daytona cell on this branch (4 opencode tasks, warm sandboxes, 1 vCPU,
follow-up comment posted 300.8–308.2 s after each task's last run ended,
image built for master `444dc18d98`): 4/4 tasks done hands-off, 8/8 runs
succeeded, 0 Retry banners, 0 `owner_lost` / `supervisor_busy` / "could
not be retired" / `execution_reconciliation_required` lines in the
server log, environment delete clean, warm-shutdown grade pass, 0
sandboxes left. The harness exited by itself after 1076 s.
- Same cell before the fix (master, 8 tasks): 1/8 tasks blocked
(`native_session_supervisor_busy` →
`native_runner_checkpoint_owner_lost` →
`execution_reconciliation_required`). With fix 1 only: 1/8 still blocked
(`owner_lost` on the idle close). With both fixes: 8/8 done and 0 failed
runs (cut by the 20-minute cap before cleanup), then 4/4 above.
- The cell grade `T:completion_once` fails on all 4 tasks by design: the
timed follow-up comment arrives after the task is done, so the agent
posts a second completion comment.

## Risks

- Fix 1 can delay a follow-up by up to the existing warm release wait
(30 s) while an idle close finishes. Before this change, that run
failed. If the close fails or the wait expires, the claim returns null
and the run uses a fresh instruction copy, as before.
- Fix 2 widens the checkpoint ownership fence from #18011. It does not
reopen what #18011 closed:
- The exemption applies only to the finished owner's final save: the
checkpoint must hold an accepted result and no new work, so no provider
turn can be granted through it.
- The successor must still be unclaimed (`queued` or `scheduled_retry`)
and must have no native coordinator (`native_run_finalizations`) row. No
controller has started it, so nothing can have consumed or depended on
the predecessor's checkpoint. Once the successor is claimed, the
existing preparing/claimed rules from #18011 apply unchanged.
  - The successor must belong to the same company and the same task.
- Cases that must still reject, with tests in the same matrix:
- `queued_wrong_issue`: a queued successor for a different task is
rejected with `native_runner_checkpoint_owner_lost` and the stored
checkpoint does not change.
- `queued_new_work`: a `turn.start` (new work) while the successor is
queued is rejected with `native_runner_checkpoint_owner_lost` and the
stored checkpoint does not change.
- All earlier rejections stay: unclaimed native coordinator,
dispatching, expired or foreign preparation, foreign native preparation,
wrong task, new work, unaccepted result, replaced controller.
- Not changed: an owner whose close fails still blocks environment
delete (409). That is a separate product decision.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Earlier commits on
this branch were written by the same model in an earlier session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:42:50 -06:00
DottaandClaude Opus 5.5 497ccba8f1 fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner that the server
controls over the durable PRP channel
> - When a run is interrupted, a same-run retry reattaches the live
Runner and reads its session (`session.snapshot`) before it continues
> - On staging, the reattached Runner authenticated and restored its
session, and then did not answer the read
> - The recovery kept that Runner, so each later attempt adopted the
same Runner and failed the same way until the attempt budget ended the
run
> - This pull request makes the next attempt replace that Runner with a
fresh Runner over its saved journal, and makes a failed Codex history
read say what failed
> - The benefit is that an interrupted Codex run continues without a
manual Retry, and its provider turn is not sent again

## Linked Issues or Issue Description

Refs #21161, Refs #21053, Refs #20853

**What happened?**

A Codex agent on Daytona ran recovery runs (wake
`issue_recovery_action_restored`). In one hour, 11 of about 35 finished
runs failed. All failed the same way:

- `native_session_recovery_failed: HarnessCapabilityUnavailableError:
read is unavailable: Error: runnerd exited while waiting for
session.snapshot` (8 runs)
- `native_session_recovery_failed: HarnessReconciliationError:
codex_history_read_failed: thread/turns/list requires supported
paginated history; Error: provider_transport_failed: PRP command
session.snapshot timed out (runner connected; pending ahead 0; last
runner event seq 338 76s ago)` (3 runs)

The run log of each run shows one sequence:

1. Attempt 1 fails with `runnerd did not report the provider turn
identity`. The close leaves the Runner alive and unsettled for the
same-run reattachment.
2. Attempt 2 adopts that Runner (same PID). The Runner authenticates.
The log shows `runnerd restored its durable PRP session and provider
thread`. Then the driver's recovery read (`thread/read` or
`thread/turns/list`, which both send `session.snapshot`) gets no answer.
The Runner still answers WebSocket pings, but it sends no command result
and almost no events (`pending ahead 0`, last event 76 s to 391 s
earlier). The recovery treats this as transient and detaches. The Runner
stays alive.
3. Attempt 3 adopts the same Runner again and fails the same way. In
some runs a protocol fault (`thread_binding_mismatch`) closes the
transport during the read, so the message is `runnerd exited while
waiting for session.snapshot`.
4. The provider attempt budget ends the run.

#21161 did not help. Its wait extension applies only while a queued
answer is being committed. Here the Runner sent no answer at all.

The `paginated history` text was also wrong. `codex-history.ts` called
every failed history read unsupported, also a transport timeout.

**Expected behavior**

A Runner that left a recovery read unanswered is not adopted again. The
next attempt starts a fresh Runner from the saved journal, resumes the
provider session from the saved checkpoint, and does not send the
provider turn again. A failed history read reports the real failure. A
Codex build that refuses paginated history falls back to one complete
thread read.

**Steps to reproduce**

1. Start a Codex run in a remote (Daytona) Runner.
2. Let the first attempt fail at turn start, so that the close keeps the
Runner alive.
3. Let the reattached Runner stop answering commands after it
authenticates (for example, stop the Runner process after its first
answer).
4. See `PRP command session.snapshot timed out (runner connected;
pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3,
with the same Runner PID.

**Paperclip version or commit**

Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on
`master` after `53608bd1d6`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `nativeRecoveryCommandUnanswered` (executor) finds a
`native_session_recovery_failed` whose Runner did not answer a recovery
command (`PRP command <type> timed out` or `runnerd exited while waiting
for <type>`). Identity, authority, integrity and incompatibility
refusals do not match.
- When such a failure ends a `reattach_remote_runner` attempt on a
sandbox target, the failure detail records `adoptedRunnerUnanswered:
true`.
- `claimNativeRestartRecoveries` copies it to the next claim as
`replaceUnansweredRunner: true`, in the same way as
`exitedBeforeSessionOpen`.
- `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps
all identity checks. When the old Runner is still alive, it returns no
adopted process, and the attempt launches a fresh Runner from the saved
journal. The existing launch fence stops the old `paperclip-runnerd` on
that state folder and its whole lineage, and fails the launch closed
when it cannot prove them gone. A Runner that already exited mid-turn
keeps the existing fail-closed path, because its provider or tool
processes can survive where the fence cannot find them. The task then
continues in a fresh session from the saved work. The attempt budget is
unchanged.
- `codex-history.ts`: a history read that fails without a provider
answer is now `codex_history_read_failed: <method> read failed;
<error>`. A provider refusal of the method (`-32601` or `-32600`) on the
first page falls back to one `thread/read` with `includeTurns: true`.
That fallback fails closed when the thread differs, or when turns, items
or turn identities are missing. A refusal on a later page still fails
closed.
- `doc/execution-semantics.md` describes both rules. The launch-fence
code comment now names the replacement case.

### Changed existing test

`codex-history.test.ts` "does not turn an interrupted page or
unsupported method into idle history" used a plain `Error("method not
found")` on page 2 and expected the `supported paginated history` text.
A plain error is now a failed read, not an unsupported method. The test
now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2
and still expects a fail-closed rejection. A new test covers the
failed-read message.

## Verification

- Red before the fix, green after:
- `server/src/services/native-runtime/native-session-executor.test.ts` ›
"a same-run reattachment whose adopted Runner never answered its
recovery read": the replacement and staging-message tests failed on the
old code (the Runner was adopted again, the three staging messages were
not recognized). Two further tests ("never resumes beside an exited
unanswered Runner's lineage") failed on the first revision of this PR,
which relaunched beside an exited mid-turn Runner, and pass now. All 15
pass.
- `native-runner-restart-recovery.integration.test.ts` › "asks the
same-run reattachment to replace a Runner that left its recovery read
unanswered" (embedded Postgres): the claim had no
`replaceUnansweredRunner` on the old code. Now passes.
- `codex-history.test.ts`: 7 new tests failed on the old code. All 16
pass.
- Real `paperclip-runnerd` and fake Codex:
`runnerd-codex-transport-remote-retention.test.ts` › "continues the same
provider thread from a fresh Runner over the journal of a retained
Runner that stopped answering, without replaying its turn". It runs the
staging sequence: turn started, close keeps the stopped Runner, a second
controller adopts it, the Runner stops answering and the controller
detaches, the Runner and provider are killed (the launch fence), and a
fresh Runner over the same journal returns the same thread. The fake
Codex call log shows one `thread/start` and one `turn/start`.
- Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus
the remote-retention, adoption and cold-restore transport suites (442
tests). Two suites timed out under parallel load and pass alone. Server:
`native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation` (1714 passed, 21 skipped).
- `pnpm typecheck` in `server` (all layers) and `tsc` for
`paperclip-runner` source and tests pass.
- Not run locally: the container-gated remote Runner suites
(`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the
full suite.

## Risks

- Medium. The replacement relaunches a Runner over a journal that is not
settled, but only while the old Runner is alive. Safety comes from three
facts: the launch fence stops the old Runner and its lineage with proof
or fails the launch; the Runner restores its own durable journal (as
local `resume_dead_runner` does); and the real-runnerd test shows that
the turn is not sent again. An exited Runner is never relaunched beside
its possible survivors.
- If the previous close left a `runner.suspend` pending, a fresh Runner
can execute it and exit. The attempt then fails as it does today. The
usual detach path after an unanswered read sends no suspend.
- The replacement uses the existing attempt budget. It does not add
attempts.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events
and logs), extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:06:24 -06:00