Commit Graph
853 Commits
Author SHA1 Message Date
Devin FoleyandPaperclip ad320aeb71 fix(native): wait for a dated provider limit reset and keep it out of Sentry (#19775)
Keep native quota cooldowns tied to valid dated reset messages. Parse explicit years and strict month names while preserving already-expired dates. Keep typed native quota and rate-limit failures on their tasks in unexpected Sentry reporting mode. Preserve setup, process, malformed and all-mode reporting.

Verified current-head CI, Greptile 5/5 at Medium impact, local parser/report red-green tests and PostgreSQL integration coverage.

Co-Authored-By: Devin Foley <devin@devinfoley.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-12 17:06:22 -08:00
Nicky LeachandClaude Fable 5.2 2dcfdabc79 feat(runner): let a provider pack record the Runner image it was built in (#21228)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - A provider pack has a manifest, `provider-pack.json`. It records the
commit the pack was built from as `payload.runnerSourceRevision`
> - The next pull requests build the Runner alone as an image that has
an identity of its own: a digest of the Runner's inputs
> - A pack from that image must be able to say which image it came from.
The commit does not say that, because many commits have the same inputs
> - This pull request adds one optional field to the pack manifest for
that digest
> - The benefit is that a reader of a pack can find the exact image that
built it. Packs that are built today do not change

## Linked Issues or Issue Description

Refs #21204 (the Runner image, part 2 of this stack). No issue exists
for this change.

**Problem or motivation**

The provider pack manifest records a source revision. It has no place
for the identity of a Runner image. The Runner image in #21204 is named
by a digest of its inputs, and one image serves every commit that has
those inputs. Without a field in the manifest, a pack cannot name the
image that built it.

**Proposed solution**

Add the optional field `payload.runnerInputsDigest`. The pack builder
writes it only when the environment variable
`PAPERCLIP_RUNNER_INPUTS_DIGEST` is set. The value must be 64 lowercase
hexadecimal characters. The pack builder fails for another value.
Nothing sets the variable in this pull request.

**Alternatives considered**

- Put the digest in a file beside the pack. Then the pack digest does
not cover it, and a copy of the pack loses it.
- Reuse `runnerSourceRevision`. That field is a commit, and code
compares it with other commits.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 1 of 4. Each part is on top of the one before it:

1. `runner/pack-inputs-digest`: an optional field in the provider pack
manifest (this pull request)
2. `runner/runner-image-build` (#21204): build and check the Runner
image, with no publication
3. `runner/runner-image-layers`: layers by change rate, and the slim
variant
4. `runner/runner-image-publish`: publish both variants from `master`

## What Changed

- `packages/paperclip-runner/scripts/build-provider-pack.mjs`: reads
`PAPERCLIP_RUNNER_INPUTS_DIGEST`. When it is set, the script checks its
form and writes `payload.runnerInputsDigest` after
`runnerSourceRevision`. When it is not set, the manifest is the same as
before, byte for byte.
-
`server/src/services/native-runtime/remote-provider-pack-verification.ts`:
the manifest type has the optional field `runnerInputsDigest`. No check
reads it.
-
`server/src/services/native-runtime/remote-provider-pack-verification.test.ts`:
one test. A sealed manifest with the field passes verification. A
manifest whose field was changed after it was sealed fails, because the
pack digest covers the field.

The change is 20 added lines in 3 files.

## Verification

- `cd server && node node_modules/vitest/vitest.mjs run
src/services/native-runtime/remote-provider-pack-verification.test.ts`:
19 tests pass.
- Part 2 builds a pack with the variable set and a pack without it, on
`linux/arm64`. The first has the field and the second does not.

Never run:

- The full test suite and the full typecheck. CI must supply them.

## Risks

- Low risk. The field is optional and additive, and nothing sets the
variable yet.
- The pack digest covers the payload. A pack that has the field
therefore has another digest than the same pack without it. No pack has
the field until part 2 builds one.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 16:58:14 -08:00
DottaandClaude Opus 5.5 79334f58ab test(runtime): stop one slow plugin Runner release/resume case from failing the cases after it (#21213)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The plugin Runner release/resume integration test proves that a
released Runner can be resumed by the next run
> - The test failed on unrelated PR #21171 in the "Verify Paperclip
Runner (vitest native server)" lane
> - One slow case caused 5 more cases to fail, and the PR retry could
not run
> - #21199 fixed the same class of failure in the plugin Runner eviction
test
> - This pull request applies the same isolation to the release/resume
test, and also isolates its shared spies
> - The benefit is that one slow case no longer fails the cases after it

## Linked Issues or Issue Description

Refs #21199

**What happened?**

`server/src/services/native-runtime/plugin-runner-release-resume.integration.test.ts`
failed on unrelated PR #21171. "completes codex runs through a retained
cached process" timed out at 60s. Then 5 more cases failed, some at 60s
and some in under 3s with `RangeError: Maximum call stack size
exceeded`. Every retry failed on `duplicate key value violates unique
constraint "issues_identifier_idx"`.

Root cause: every case changed process-wide state and restored it in its
own `finally` block. That state was the default TLS trust,
`https.globalAgent`, a `vi.spyOn` on
`PaperclipRunnerToolAuthority.prototype.definitions`, and a `vi.spyOn`
on `inspectRemoteWarmSessionRecovery`. Vitest continues to the next case
when a case times out, but the timed-out function continues to run.
While it ran, the next case read the timed-out case's spy as the
"original" `definitions`. When the timed-out case restored its spies,
the two spies called each other, and the stack overflowed. Its late
`finally` also removed the next case's TLS trust, so that case stayed in
`connecting` with no wire frames until its own 60s timeout. It also
closed the next case's warm sessions with
`closeIdleWarmNativeSessionsForRestart()`. The retry could not help,
because the issue identifier came only from the case name.

**Expected behavior**
A case that times out fails alone. Later cases and the PR retry run
normally.

**Steps to reproduce**
1. Make the `codex cached` case sleep 62s once, after its first run.
2. From `server/`, run `npx vitest run
src/services/native-runtime/plugin-runner-release-resume.integration.test.ts
-t "codex runs through a retained (cached|adopted|adopted restart)
process"`.
3. Before this change, 3 of 3 cases fail, with `Maximum call stack size
exceeded`. With `--retry=1`, 3 of 3 still fail.
4. After this change, only the slow case fails. With `--retry=1`, 3 of 3
pass.

**Paperclip version or commit**
origin/master at the base of this branch (after #21199)

**Deployment mode**
CI and local test only (embedded Postgres, real `paperclip-runnerd`)

**Agent adapter(s) involved**
paperclip_runner (codex and claude ACPX harnesses through the plugin
task driver)

## What Changed

- Create one fixture certificate in `beforeAll`. Set the TLS trust and
`https.globalAgent` once for the file, and restore them in `afterAll`.
- Read the original `PaperclipRunnerToolAuthority.prototype.definitions`
once at module load, before any case spies on it.
- Move the per-case teardown into an `afterEach` that is registered
before the database fixture. The teardown closes idle warm sessions, the
host control, and the TLS proxy, and it restores both spies. Each case
binds its own teardown holder before its first await.
- Add a random suffix to the issue identifier of each attempt, so the PR
retry can run the case again.
- Increase the idle timeout of the adopted cases from 5s to 10s. The
product limits a release to `min(idleTimeoutMs, 30s)`, and #21199
measured a loaded release that took more than 4.8s.
- Probe the Runner port on `0.0.0.0`, the address that the Runner binds.

No product code changed.

## Verification

- Injection test (one case sleeps 62s): see the steps above. Before: 3
of 3 failed, also with `--retry=1`. After: 1 of 3 failed without retry,
and 3 of 3 passed with `--retry=1`.
- Before/after loops of the full file (14 cases), run in parallel on the
same loaded macOS host: see the loop result comment on this PR.
- `tsc --noEmit -p server/tsconfig.json` reports no errors in the
changed file.

## Risks

Low risk. Only the test changed. The adopted (non-restart) cases take
about 5s more each. The file creates one certificate instead of 14. A
case that hangs inside Core execution can still leave a busy warm
session in module state.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with
tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:19:06 -06:00
DottaandClaude Opus 5.5 57aeda190b test(runtime): stop one slow plugin Runner eviction case from failing the cases after it (#21199)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The plugin Runner eviction integration test proves that a retained
Runner continues after the host evicts it
> - The test failed on many unrelated pull requests in the "Verify
Paperclip Runner (vitest native server)" lane
> - One slow case caused other cases to fail, and the PR retry could not
run
> - This pull request isolates each case from a case that timed out, and
gives the retry a fresh issue
> - The benefit is that one slow case no longer fails the cases after it

## Linked Issues or Issue Description

**What happened?**

`server/src/services/native-runtime/plugin-runner-eviction.integration.test.ts`
failed in CI on at least 5 unrelated branches in 48 hours, most recently
on #21149. The errors were `Test timed out in 60000ms` and `Timed out
waiting for automatic release A`. Each retry then failed at once with
`duplicate key value violates unique constraint
"issues_identifier_idx"`. The failed cases logged a Runner in
`connecting` with no wire frames.

Root cause: every case changed process-wide state. It set the default
TLS trust and `https.globalAgent` for its own certificate, and it
restored them in its own `finally` block. Vitest continues to the next
case when a case times out, but the timed-out function continues to run.
Its late `finally` then removed the TLS trust of the next case and
closed the warm sessions of the next case with
`closeIdleWarmNativeSessionsForRestart()`. The next case could not open
its Runner connection, waited for the 60s startup deadline, and also
timed out. That case then did the same to the case after it. In CI run
43266976387, one timeout caused 3 more failures. Locally, one timeout
caused 5 more failures.

Two more problems made this worse:
- The issue identifier came only from the case name. A retry of the same
case therefore failed on the unique identifier, so the PR retry never
ran the test again.
- Detached cases used a 5s idle timeout. The product limits a release to
`min(idleTimeoutMs, 30s)`, so these cases had 5s for the drain and the
park. Under load, one park took more than 4.8s, and the release expired.

**Expected behavior**
A case that times out fails alone. Later cases and the PR retry run
normally.

**Steps to reproduce**
1. Make one case slower than 60s. For example, add a 62s sleep after
release A in `codex cached retired`.
2. Run `npx vitest run
src/services/native-runtime/plugin-runner-eviction.integration.test.ts
-t "codex with a (cached|detached) Core owner"` from `server/`.
3. Before this change, 3 of 4 cases fail: the slow case, and 2 cases
that it disturbs. With `--retry=1`, the slow case also fails on the
duplicate identifier.
4. After this change, only the slow case fails. With `--retry=1`, all 4
cases pass.

**Paperclip version or commit**
master at ef6d347f79, rebased on the current origin/master

**Deployment mode**
CI and local test only (embedded Postgres, real `paperclip-runnerd`)

**Agent adapter(s) involved**
paperclip_runner (codex and claude ACPX harnesses through the plugin
task driver)

## What Changed

- Create one fixture certificate in `beforeAll`. Set the TLS trust and
`https.globalAgent` once for the file, and restore them in `afterAll`. A
case no longer changes process-wide trust.
- Move the per-case teardown (close idle warm sessions, close the host
controls, close the TLS proxy) into an `afterEach`. It is registered
before the database fixture, so it runs before the database clients
close, also when a case times out.
- Add a random suffix to the issue identifier of each attempt, so the PR
retry can run the case again.
- Increase the detached idle timeout from 5s to 10s, and wait 200ms past
it as before. This gives the release the time that it needs under load.
- Probe the Runner port on `0.0.0.0`, the address that the Runner binds.

No product code changed.

## Verification

All loops ran on a shared, loaded macOS host (load average 15 to 20 on
18 cores), plus 2 CPU busy loops. Each iteration ran the 12 cases that
match `with a (after inspection|missing state|reused lease) Core owner`.

- Before: 20 iterations, 2 failed (10%). In iteration 10, `claude reused
lease retired` timed out at 60s, and 5 later cases failed on
`readNativeSessionIdleWork() === "none"` because of the case that timed
out. In iteration 17, `claude after inspection retired` failed with
`Timed out waiting for automatic release A`.
- After: see the loop result comment on this PR.
- Injection test (one case sleeps 62s): before, 3 of 4 cases failed, and
with `--retry=1` the retry failed on the duplicate identifier. After,
only the slow case failed, and with `--retry=1` all 4 passed.
- Full file after the change: 40 of 40 passed in 320s locally (335s
before, because the file now creates one certificate instead of 40).
- `tsc --noEmit -p server/tsconfig.json` reports no errors in the
changed file.

## Risks

Low risk. Only the test changed. The detached cases take about 5s more
each (12 cases, about 60s in all), which is partly offset by creating
one certificate per file. A case that hangs inside Core execution can
still leave a busy warm session in module state. This change does not
prevent that. It removes the causes of the cascade that this PR found.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with
tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:18:01 -06:00
DottaandClaude Opus 5.5 de675d924a fix(heartbeat): stop a comment the closing run already read from reopening its finished task (#21175)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can add a comment to a task while the agent's run on that
task is still working
> - The comment must reach the agent once: in the running turn, or in
one later turn
> - In a restart-chaos wave, a comment that the running turn already
read reopened the finished task, and the agent did the completion turn a
second time 15 minutes later
> - This pull request makes a read by the running turn count as
delivery, and holds the comment behind the running run so the run's own
release settles it
> - The benefit is no reopened done task, no repeated completion turn,
and an unread comment that is delivered when its run finishes, not after
a slot wait and a stale cancellation

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261112T181948-s7101`, task t005 (OpenCode,
native Runner), during a two-server overlap deploy (K15).
- 18:26:56 run A started on the old server. 18:26:55 traffic switched to
the new server.
- 18:27:03.953 a board user posted a follow-up comment. The new server
handled it. Run A was not in that process's `activeRunExecutions`, so
`filterZombieCoalesceTarget` treated it as a zombie. Admission returned
`proceed`, and the comment got its own queued run B (18:27:04.252). The
queued run is not part of the release drain.
- 18:33:32 run A called `get_task_history`, received the comment,
applied it, and finished the task (`done`, 18:34:20).
- Run B waited for an agent slot until 18:41:49 and was cancelled as
`continuation_task_ownership_changed`.
- `deferUndeliveredCommentsOfStaleRun` then saved the comment again. Its
"unread" test is only "a run started before the comment and finished
after it". `deliveredQueuedCommentIds` had no record of run A's read.
The release drain reopened the task (done to todo, 18:41:51), and run C
(18:49:22) posted a second "Task complete" summary. The harness grade
`done_comment_once` failed.

**Expected behavior**
- A comment that the running turn received is consumed. No queued work
outlives that run, and the task stays done.
- A comment that the running turn did not receive is delivered once,
through the normal comment-on-done-task path, when that run finishes.
- A stale continuation is never re-issued for a comment that a completed
run already read.

**Steps to reproduce**
1. Start a native run on a task.
2. Post a person's comment while another server instance executes the
run (or seed a queued comment run behind it).
3. Let the run read `get_task_history`, then finish the task.
4. Before this change, the queued comment run is cancelled as stale, the
task reopens, and the agent runs a second completion turn.

**Paperclip version or commit**
`master` at ef6d347f79.

## What Changed

- `server/src/services/queued-comment-delivery.ts`: new run-log event
`task_history.read` and `recordTaskHistoryRead`.
`deliveredQueuedCommentIds` also counts a comment as delivered when a
succeeded run of the assignee on that task read the comment's current
revision (`historyReadDeliveredSql`). All existing consumers use this
one check: the release drain liveness check, the stale-continuation save
(`saveUndeliveredUserComments`), the ownership reconciler, and
first-turn queued input.
- `paperclip-runner-tool-authority.ts` and
`runner-semantic-authority.ts`: `get_task_history` records the returned
comment IDs and `updatedAt` values. The tool output does not change. A
failed record is logged and leaves the comment owed, so it is delivered
again.
- `server/src/modules/wake-queue/application/use-cases.ts`: a comment
follow-up for the same agent's `running` run is held as a deferred wake
even when this process does not track that run. Before, a run executed
by another server instance was treated as a zombie, and the comment got
its own queued run that bypassed the release drain. Coalescing into an
untracked run is still refused.
- `doc/run-log-events.md`: documents `task_history.read`.
- Regression tests in
`server/src/__tests__/heartbeat-held-comment-wake.test.ts`.

Intended behaviour chosen: in t005 run A received the comment mid-turn
through its own `get_task_history` call (run events at 18:33:32 show the
comment ID in the tool result, and the result says it added the
follow-up line), so (a) applied. The fix makes that read the durable
proof of delivery. A comment that no run read still follows the existing
outbox: it is held behind the run and delivered once at the run's
release, reopening the done task as a person's comment does. Envelope
task history of another message's turn still does not count (unchanged
doctrine and test).

## Verification

- RED on master (`ef6d347f79`), `npx vitest run
src/__tests__/heartbeat-held-comment-wake.test.ts -t "mid-turn|another
server|held message"`: 3 failed, 1 passed.
- "consumes a message the closing run read mid-turn, so its stale queued
run reopens nothing": delivered 1, expected 0.
- "holds a message behind a run another server is executing instead of
queueing a run of its own": a queued run was created.
- "settles a held message as soon as its run closes the task (read
mid-turn: true)": delivered 1, expected 0.
- The unread variant (delivered once, no cancelled stale run) passes on
both, as intended.
- GREEN with this change: all 4 pass.
- Review follow-up (f43e79c3): a history read counts as delivery only
for a succeeded run on the read revision. Tests cover an edit after the
read (the stale queued run delivers the comment once) and reads by an
edited or failed run (the comment stays owed). Full file: 37/37 pass.
- Security-scan follow-up (239f0597): the delivery check probes each run
binding (`native_issue_id`, `context_snapshot->>'issueId'`) through its
own existing index with `offset 0`. It reads only the assignee's
succeeded runs of the task with `finished_at >= ` the comment's current
revision. It does not use `started_at`, because the t005 reader started
before the comment. It stops at the first matching `task_history.read`
event of each run through `heartbeat_run_events_company_run_idx`. No
migration is needed.
- Seeded plan (local PostgreSQL: 200 tasks, 4,000 runs, 1.2M run events,
one comment read by one run), `EXPLAIN (ANALYZE, BUFFERS)`:
- Before: 1.202 ms, 6,097 shared buffers. It scanned every event of all
21 runs of the task.
- After: 0.030 ms, 23 shared buffers. It reads one index-matched run and
one event. The full `deliveredQueuedCommentIds` call takes 1.55 ms.
- New case: a succeeded run's read is not delivery when the comment was
edited after that run finished. Full held-comment file: 38/38 pass.
- `muse-control.test.ts` compared the whole run row after a
`get_task_history` call. It now allows that read's own run-log append
(`nextEventSeq` +1, `updatedAt`), and every other column must be
unchanged. That CI failure (General tests server 2/12) came from this
PR. "e2e shard (5/8)" failed because the self-hosted runner lost
communication; that is not related to this PR.
- Runner and Muse suites (`dot-runner`, `paperclip-control-plane-port`,
`runner-prp-coordinator`, `historical-task-questions`,
`paperclip-runner-tool-authority`, `muse-native-follow-up`,
`muse-control`): 509/509 pass. Server `tsc` is clean.
- Neighbouring suites: `heartbeat-held-comment-wake`,
`src/modules/wake-queue/**`, `execution-continuation`,
`paperclip-runner-tool-authority`, `first-turn-queued-input`,
`heartbeat-comment-wake-batching`, `ownership-reconciliation-report`,
`heartbeat-zombie-guard`: 555/556 passed in one loaded run. The 2
timeouts (one batching test, one reconciler `afterAll` hook) pass when
run alone (1/1, 44/44).
- `server` `tsc --noEmit`: clean.
- The restart-chaos harness already grades this class
(`done_comment_once` in staging-mix). No new cell is necessary.

## Risks

- Product choice: a comment counts as delivered when a succeeded run
read it through `get_task_history`, even if the model ignored it. This
is the same trust that applies to a comment delivered as wake input. The
alternative fails worse: it reopens a done task and repeats a completed
turn. An edit after the read, or a reader run that fails, leaves it
owed.
- Legacy (non-native) adapters that read comments through the REST API
do not write the record. Their behavior does not change.
- A comment follow-up behind a truly dead running run now waits as a
deferred wake until recovery releases that run. Before, its queued run
also could not start until the lock was released.
- Each `get_task_history` call appends one small run-log row. This is
not in the path before `turn.start`.
- Upgrade-safe: continuations queued by older builds still go through
the dispatch stale check, and that path now reads the same delivery
record.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:31:11 -06:00
DottaandClaude Opus 5.5 f92f91f0fb fix(native): finish a parent claim when its child results arrived before the run started (#21189)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task gets a wake when one of its child tasks completes. The
wake carries the child results.
> - When the parent already has a run, the child result wake waits
(`deferred_issue_execution`) behind that run.
> - At finalization, a parent completion claim stays open
(`native_child_completion_pending`) while such a wake is still queued,
claimed or deferred. This guard protects child results that arrive while
the parent works.
> - The guard did not check when the child result arrived. A child
result saved before the parent run started was already visible to that
run, but it still blocked the claim.
> - So the task stayed `in_progress`, the deferred wake started a second
run, and the agent repeated its completion summary.
> - This pull request counts a child result as undelivered only when it
was saved after the finishing run started.
> - The benefit is that a parent that already read its child results
finishes in one turn, and child results that arrive during the run still
get a fresh turn.

## Linked Issues or Issue Description

Refs #15372 (added the pending child-completion guard). Refs #20935
(same family: a self-wake after a settled run).

**What happened?**

Restart-chaos wave `w-20261112T181948-s7101`, task t001 (OpenCode,
native runner, staging-mix parent with two subtasks). Both subtasks were
done before parent run `8006a205` started at 18:45:29. The run read
them, posted its final comment and claimed `done` at 18:46:15.860. A
child-completion wake for the parent had been deferred since about 18:42
behind that run. The finalizer found that wake and committed
`native_child_completion_pending`, so the task stayed `in_progress`. 63
ms later the release promoted the deferred wake into run `9282de34`
(`invocationSource: automation`, `wakeCommentId: null`). Its
`paperclip.native-continuation.v1` turn carried the subtask results that
the first run had already used, plus the agent's own two comments
(resume delta since the base run). That run posted a second "complete"
summary and only then moved the task to `done`.

Root cause: `hasPendingNativeChildCompletion()`
(`server/src/services/native-runtime/native-child-completion-delivery.ts`)
treated every queued, claimed or deferred child-completion wake for the
issue as unread, except a wake bound to the finishing run itself. It did
not compare the wake's save time with the run's start.

**Expected behavior**

A child result that was saved before the finishing run started is not
new input for that run. The completion claim completes the task. The
deferred wake is cancelled at release as a wake on a terminal task. A
child result saved during the run still keeps the task open and gets a
fresh parent turn.

**Steps to reproduce**

1. Create a parent task with a done child. Save a deferred
`issue_children_completed` wake for the parent.
2. Start a native parent run after that wake was saved. Let it claim
`done`.
3. Finalize the run. On `master` the decision is
`native_child_completion_pending` and the task stays `in_progress`; the
deferred wake then starts a second run.

## What Changed

- `native-child-completion-delivery.ts`:
`hasPendingNativeChildCompletion()` counts a child-completion wake only
when its `createdAt` is at or after the finishing run's `startedAt`. A
run without a recorded start keeps the old behavior: every such wake
stays pending. The finalizer, the status-commit recheck and status
reassessment all use this function.
- `heartbeat-process-recovery.test.ts`: a regression test finalizes a
real native done claim with a deferred child-completion wake saved
before (task becomes `done`) or after (task stays `in_progress` as
`native_child_completion_pending`) the run started. It then releases the
issue execution through the wake queue. In the `before` case the
deferred wake is cancelled and there is one run total. In the `after`
case the wake is promoted into a fresh run that carries the child
results.
- `doc/architecture/native-status-arbitration.md`: describes when a
child result counts as undelivered.

## Verification

- RED on `master` (`ef6d347f79`): `heartbeat-process-recovery.test.ts`
"treats a child result saved before the parent run started…" fails with
decision `reasonCode: "native_child_completion_pending"`, `toStatus:
"in_progress"`. This is the same decision as the wave. The "after" case
passes on `master`.
- GREEN: both cases pass with the fix.
- Neighbouring suites pass: `native-status-arbiter-corpus` (includes the
queued / claimed / deferred / consumed child-delivery cases from
#15372), `heartbeat-dependency-scheduling` (revised child completion
delivered after a running parent), `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, and `src/services/native-runtime`.
- `pnpm --filter @paperclipai/server typecheck` passes.
- The regression test runs finalization and release end to end. The
cancel-at-release step itself is existing behavior, also covered by
`modules/wake-queue/application/use-cases.test.ts` ("cancels stale
assignee continuations before claiming promotion on %s tasks", `done`
case).

## Risks

- A child result saved before the run started no longer blocks the
claim. The child status was already `done` when the run started, so the
run could read it. If the clocks of two server instances differ, a child
result saved during the run could look older than the run start; the
window is the clock skew.
- A re-dispatched native status wake gets a new row with a later
`createdAt`. Such a row stays pending, which is the old behavior.
- No migration. Saved wakes from older builds work unchanged, because
the check reads only existing columns.
- Scope: this change does not alter the comment-wake promotion path or
the resume-delta builder.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:19:32 -06:00
DottaandClaude Opus 5.5 82c6810107 fix(runtime): post the agent's blocker reply when a retried finalization blocks the task (#21169)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime finalizes an agent's run, applies the agent's
status decision to the task, and posts the agent's reply as a task
comment
> - When the first workspace copy-back fails, or the run reconnects, a
background retry commits the result instead of the live heartbeat
> - That retry path applies the "blocked" status but never posts the
agent's reply, so the task turns blocked with no visible reason
> - The retry path already shares one recovery step for replies, but
that step accepted only "done", "needs review" and conversation replies
> - This pull request lets the shared recovery step present a blocked
task's reply, so the retry path and the normal path converge on one
presentation step
> - The benefit is that a blocked task always tells the user why,
exactly once

## Linked Issues or Issue Description

**What happened?**

An agent reports that it is blocked. The task moves to "blocked". No
comment from the agent explains why. This happens when the run finishes
through a retried workspace copy-back or a reconnect loop instead of the
live heartbeat. The run succeeds and its result has the agent's blocker
summary, but the run has no presentation decision and the task has no
reply. On a staging company, 58 of the 150 most recent agent-blocked
tasks for one agent had no agent comment.

**Expected behavior**

Every finalization path that applies an agent's status decision also
presents the agent's reply. A blocked task always shows the agent's
explanation once. The status decision does not post a second copy.

**Steps to reproduce**

1. Run a native agent on a task. The agent reports `blocked` with a
summary and a task-wide blocker.
2. Make the first workspace copy-back fail
(`native_workspace_sync_out_failed`), so the heartbeat leaves
finalization to the retry.
3. Let the retry finish the copy-back and commit the result.
4. The task is blocked and the run succeeded, but the task has no agent
comment.

**Paperclip version or commit**

`master` at b8207c4ea9

**Deployment mode**

Hosted (staging cloud). The defect is in server finalization, so it
applies to all deployment modes that use the native runtime.

**Agent adapter(s) involved**

Native runtime (`paperclip_runner`), seen with the Codex-based agent.
The defect is adapter-independent.

## What Changed

- The response presentation recovery step
(`recoverCommittedNativeChatResponse`) now presents a non-conversation
task's reply when the agent reported `blocked` and the task is still
blocked.
- For a blocked reply, recovery uses the agent's final message when one
exists. Otherwise it uses the accepted result summary, the same as the
live completion path.
- A blocked reply on a channel task (an external chat run) uses the same
rule as live run completion. Recovery resolves the chat publication
authorization, prefers the final response, and posts with that
authorization. The worker does not refuse these runs, so it does not
record a false publication failure.
- The presentation work query now finds these runs, so the background
worker also presents them and the bounded publication retry applies to
them.
- A blocker reply is not presented when the task is no longer blocked or
when a later status decision was applied. This keeps old blockers from
posting stale comments.
- Presentation stays idempotent: the step records
`presentationDecision`, reuses a reply that the run already posted, and
never posts twice.
- Existing affected tasks need no separate backfill. The presentation
worker loads all pending candidates at startup, so each task that is
still blocked gets its saved reply once after deploy.

## Verification

- Red/green test:
`server/src/services/native-runtime/native-blocked-reply-recovery.test.ts`.
It completes a blocked native run, fails the first copy-back with
`native_workspace_sync_out_failed`, then commits the retry with
`finalizeNativeRun`.
- Red on `master`: the retry-path tests fail (no comment is posted, and
the worker finds no candidate). The channel task test also fails on the
first PR revision (5eb8cdab95).
- Green with this change: 4/4 pass. The tests check that the blocker
summary is posted once, that the final reply is preferred, that a
repeated retry and a repeated worker check do not double-post, and that
a stale blocker is not posted.
- `cd server && npx vitest run
src/services/native-runtime/native-blocked-reply-recovery.test.ts
src/services/native-runtime/native-response-publication-retry.test.ts
src/services/native-runtime/agent-chat-response-recovery.test.ts
src/__tests__/idle-native-finalization.test.ts
src/services/presentation-ownership-read.test.ts
src/services/chat-monitor-continuation.test.ts` passes (172 tests).
- `cd server && npx tsc --noEmit -p .` passes.

## Risks

- After deploy, tasks that are still blocked from earlier affected runs
get the agent's saved reply once. This is the intended backfill. Tasks
that moved on are skipped.
- The candidate query has one more branch with a `NOT EXISTS` on
`status_decisions`, keyed by the existing `(company_id, issue_id,
decision_version)` unique index. The cost is low.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, browser reads of staging API).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 14:30:32 -06:00
Nicky LeachandPaperclip 7f69ef8b44 Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path

> - Paperclip lets plugins own managed Git workspaces.
> - A portable Runner task can move to a different physical allocation.
> - The checkout must be present on the allocation that launches the
Runner.
> - Workspace preparation and finalization need exact placement when
they retry.
> - This change checks the placement at Runner admission and records it
before finalization.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Workspace preparation and finalization for portable Runner tasks.

**Current behavior**
A task replacement can change the allocation after Git preparation. A
lost finalization reply can leave retry without the original placement.

**Proposed behavior**
Prepare the checkout on the actual allocation before admission. Record
finalization placement so an exact retry can use it.

Portable ownership and controller recovery are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066).

## What Changed

- Reserve the managed workspace before acquiring the portable task
lease.
- Pass the task placement to the workspace provider. Reuse the described
task while placement stays the same and prepare again before admitting a
replacement.
- Record the initial preparation's run, agent, generation, and optional
task placement before dispatch. Inspection and retries on the same
Launcher replay the same request; VM and sandbox preparation use the
same recovery path.
- Keep provider request identity separate from Git credential authority:
reservation reconciliation uses the existing project-consent path in a
host-only scope bound to the plugin and workspace. Foreground
preparation still requires an active run; consent, repository, and
revocation checks are unchanged.
- Acquire the current Launcher before retrying preparation. If placement
changes, use ordinary preparation on the replacement so the provider can
settle the prior writer and recover the retained checkout. No additional
provider discovery call is added.
- Select finalization placement before publishing the operation record,
then dispatch and recover using that saved choice. Retain the Launcher
during bounded finalization retries.
- Preserve interrupted pre-upgrade finalization requests without
placement, including their original operation IDs and provider routing.
No migration or new durable mode flag is required.

## Verification

- Final revision `74ea3e6712`: server typecheck and exact SDK source-pin
verification (538 inputs) passed.
- All 185 tests passed across the full plugin worker manager, project
Git authorization, and plugin workspace integration suites (including 56
workspace tests). Independent verification reran the three lost-reply
cases: inspection, same-Launcher retry, and replacement after deleting
the original Launcher. The tests assert the correct request identity,
durable intent, and clearing that intent after success.
- Before the final retry-accounting fix, all 323 tests passed across
workspace admission, runtime execution, native workspace finalization,
plugin workspace integration, and terminal cleanup retry. Regressions
exercise crashes after operation insertion but before dispatch in both
finalization paths, exact replay after lease release, and retry after
ambiguous placement clears without publishing an incomplete operation.
Pre-upgrade interrupted requests retain their operation IDs despite an
available portable lease.
- Final retry-accounting fix: all 84 native workspace finalizer,
failure, and supersession tests passed. A real missing-plugin regression
first failed on the prior commit, then verified healthy-run progress,
three-attempt exhaustion, terminal cleanup, and absence of a misleading
workspace operation. Independent verification also passed the legacy and
crash-before-dispatch cases.
- Worker-process tests cover both preparation RPC methods: the provider
still receives the original run and agent, maintenance receives
project-scoped credential authority, and foreground or mismatched-scope
requests remain denied. Database tests confirm a stopped run cannot
authorize foreground Git access while valid project consent can
authorize maintenance.
- Existing metadata protection coverage proves callers cannot overwrite
the private preparation intent or read it through workspace responses.
- Two existing Launcher Git tests passed: retry preserves dirty files
and the original base commit without fetching, and retry resumes the
saved checkout intent while rejecting a changed pending branch.
- Implementation was delegated to GPT-6.1 Sol and independently
reviewed. The GitHub stack's automatic rebase preserved the original
workspace-placement patch.

- [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706)
passed on `74ea3e6712`, including server and workspace tests, Runner,
end-to-end tests, typecheck, build, and canary dry run. Ordinary
Greptile and
[Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388)
both scored `74ea3e6712` 5/5, with no unresolved findings.

## Risks

- Live Daytona end-to-end qualification is pending. The checkout record
persists on the mounted volume; replacement preparation relies on the
provider confirming the previous writer is settled before using that
retained checkout. Local tests cover Core request routing and Launcher's
durable checkout recovery separately.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 12:12:41 -08:00
DottaandClaude Opus 5.5 4363a3115a fix(recovery): continue any unfinished run in a fresh run from saved work (#21135)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs are interrupted when a server restarts, a deploy swaps
servers, a controller lease is lost, or a sandbox Runner cannot be
reached again
> - Exact recovery (resume the same session, replace, retry) refuses in
many ways, and each refusal leaves the task blocked or idle for a person
> - On staging, 577 of 1,000 runs in 11 hours failed or were
interrupted, across 25 error codes. About 900 earlier PRs each fixed one
refusal code, and the rate did not converge
> - This pull request adds one universal fallback: when an unfinished
run cannot be resumed exactly, for any reason that is not the person's,
the task continues in a fresh run from its last saved workspace and its
conversation
> - The benefit is that a new failure code next week falls back without
code changes, and tasks stop waiting on a person for infrastructure
faults

## Linked Issues or Issue Description

**What happened?**
After a server restart or deploy swap, many runs end with codes such as
`native_continuation_requires_reconciliation`
(`runner_remote_process_identity_unavailable`), `controller_lease_lost`,
`process_lost`, `server_shutdown_interrupted` and
`native_session_interrupted` (including `runner_harness_state_mismatch:
resumed_not_suspended` / `resumed_unavailable`). The disposition worker
then settles the recovery hold as "blocked, no replay". The task stays
Blocked until a person presses Retry or reconciles the run.

**Expected behavior**
A task waits on a person only for reasons that are the person's: an
approval, a question, a credential to share, a budget stop, a provider
usage limit, or a Stop they asked for. For every other failure, the
system starts a new run for the same task from the last verified saved
workspace, without a person.

**Steps to reproduce**
1. Start a task on a sandbox agent.
2. Restart or swap the server while the run is in progress, so the new
server cannot re-reach the Runner.
3. See that the run fails with
`native_continuation_requires_reconciliation` and the task stays Blocked
with no successor run.

**Paperclip version or commit**
`master` at 9e910ff9ae.

## What Changed

- New module `server/src/services/fresh-continuation-fallback.ts`. One
entry point, `ensureFreshContinuation` / `continueFreshInTransaction`,
takes a run that ended without finishing and guarantees one of:
something already follows it (a newer run, a linked replacement, an
operator decision), exact recovery is still under way (checked again
later), the cause is the person's, or one fresh continuation is
recorded.
- Classification, not an allowlist: `PERSON_OWNED_FAILURE_CODES` plus
task-level waits (pending approval, pending question, paused or
unavailable agent, paused company, review in progress, a Stop that a
person asked for, a Blocked status that another decision owns). Every
other code, including unknown future codes, falls back.
- Idempotent per source run. The task row lock serializes callers on
every server; the recorded `freshContinuation.sourceRunId` makes later
calls a no-op.
- Isolation, never waiting on proof. A local process still running on
this host is settled by its own owner first. If the old local writer is
proven gone, the task workspace is kept. Otherwise the continuation gets
a new isolated workspace (`fresh_isolated_workspace`), so two runs never
write one directory. The decision records the stop proof honestly
(`providerStopProof: isolated`).
- Claim admission no longer waits on the cleanup of a predecessor that a
fresh continuation replaced (`heartbeat/queue.ts`, predecessor loop).
Without this, the successor stayed queued behind the unreachable lease,
which is the staging failure itself.
- Only reconciliation holds that this decision owns are settled.
Workspace restore holds that workspace recovery still owns, and resolved
history, are untouched.
- `continuation_user_authorization_missing` is the system's failure (a
Retry's authorization was not carried forward), not the person's.
- Safety: the old run keeps its fencing. When it still owns its remote
workspace epoch, the epoch is abandoned
(`abandonUnverifiedRemoteWorkspaceEpoch`), so the old writer can never
publish and the next run restores the last committed save into a new
root on a new allocation. The unverified files stay in the abandoned
root, unapplied. The native coordinator records `replacementDenied:
fresh_continuation`, so no exact replay also starts.
- The replaced run loses its write authority. The fallback records
`freshContinuationSupersededAt` on the old run. The agent-JWT middleware
(`loadAgentRunAuthentication` reads one more field in its existing row
read) and `assertAgentRunWriteAllowed` refuse every write of the old
token with `agent_run_superseded`. Before this change, a failed or
interrupted run's token could still write. So an agent that is still
alive in an unreachable sandbox cannot act on the task next to its
continuation.
- The opening context (`recoveryOutcomes` in the execution continuation)
says that the previous attempt was cut off and lists the actions with
unverified outcomes, so the agent checks before it repeats a side
effect. No copy asks a person to inspect or repair a sandbox, workspace
or run. The hold text written at run end no longer says "Inspect the
original failure and reconcile".
- Delivery reuses the existing path: the settled hold gets an
`executionReconciliation` decision and `continuationDelivery: pending`,
and `deliverReconciledExecutions` starts the run (fresh session,
`retryOfRunId` = source).
- Budget per task, not per restart: at most 6 fresh continuations per
task per rolling hour (the server-restart lane's cap). At the cap the
hold stays with the system and is checked again when the window has
room. The failure allowance is never charged.
- Call sites:
- `settleUnrecoverableExecutions` (execution-recovery-resolution.ts):
every hold whose exact recovery is ruled out now goes through the
fallback before it would block the task. Interrupted preparations keep
their own replay lane first.
- `continueHeldExecutions` (same worker): picks up holds saved by older
releases (settled-but-blocked no-replay holds, finished or exhausted
saved-work recoveries) for the task's latest unfinished run.
- Wake admission (heartbeat/queue.ts): an ordinary system continuation
wake (sweep, monitor, timer) that a hold would defer now calls the
fallback in the same transaction and becomes the fresh continuation
itself. This is the path the idle-task sweep uses. A person's Retry,
message or queued send keeps its exact admission; the disposition worker
settles the hold, and the saved input then continues.
- `executionControlSchedule.disposition` wakes the worker for held tasks
and for the next due recheck.

### Human gate versus system uncertainty

One documented rule (`isHumanGate` plus `personOwnedTaskWait`) decides
what the fallback never touches. The rule goes by cause, never by owner
type:
- Human gates: a pending approval, a pending question, a user-owned
hold, an owner-gone attestation, a recorded explicit continuation, an
operator reconciliation, a person's Stop.
- System uncertainty: a no-replay record ("outcome unknown after an
interruption"), a hold that is `board`-owned only by default, an
unverifiable controller process space or host.

For uncertainty the fallback starts a fresh run, never a replay. The
interrupted turn is not re-sent. No tool call or write is re-executed.
Governed actions go through their approval gate again. The earlier
`automaticRecovery` decision is kept on the hold as
`automaticRecoveryHistory`, beside the attributed
`freshContinuationSettlement` and an activity entry.

### Restart recovery refused for an unverifiable controller

The case is `controller_process_space_unverified`, its remote variant,
or `controller_host_unrecorded` / `controller_host_unprovable`. Its
lease has lapsed; a live, renewing lease is still waited for (server
overlap). After `NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS` (2 minutes) the
run ends (failed, `native_restart_recovery_blocked`). That frees the
task's execution lock and the agent's slot, and a system-owned hold lets
the fallback continue the task. Recording the stops is optional. The "a
board operator must record both stops" text is replaced.

Board/operator attention sites in native restart recovery:
- `surfaceNativeControllerNamespaceAttention`: copy replaced; the run
now ends after the bounded wait.
- `requestNativeControllerStopConfirmation`
(native-controller-host-recovery.ts): copy says confirmation is
optional; the same bounded end applies.
- The blocked restart-recovery hold (`nextAction` "Inspect the preserved
checkpoint and reconcile…"): copy replaced; the fallback settles the
hold by cause.
- `owner-gone-attestation.ts` (generic resource records from older
releases): unchanged. It is not a run-recovery path, and it is listed as
a follow-up.

### Every wait and its bound

| Wait | Owner that rechecks | Worst-case total before the fresh
continuation |
| --- | --- | --- |
| `exact_recovery_pending` (execution settling, native finalizer,
controller lease, interrupted-preparation lane, saved-work recovery,
restore hold, retained-source fallback, live local process) |
Disposition worker, woken at each deadline; rechecks at 1, 2, 3 and 4
minutes, then every 4 | About 10 minutes from the first pending check.
Up to 15 while the owner shows progress (a new event, output, lease
renewal or coordinator update since the last check). |
| Saved-work (workspace) recovery that is `scheduled`,
`awaiting_evidence`, or `running` without progress | Held-run scan, now
selected and scheduled like the other holds | About 10 minutes; 15 at
most |
| Saved-work recovery that is `running` and advancing (revision,
attempt, state or hold update since the last check) | Held-run scan |
Its own 30-minute budget |
| Unverifiable lapsed controller in restart recovery | Restart recovery
(`retryAt`) | 2 minutes, then the run ends and the fallback continues it
within one disposition pass |
| Hourly cap (`backoff`) | Disposition worker, woken at the deadline |
Until the rolling hour frees a slot, at most 1 hour. This is the
deliberate restart-lane bound, not a stuck state. |
| Person-owned or dependency wait | The person's own action wakes the
task; rechecked hourly | Not bounded: it is the person's |

Restart-recovery classifications (`claimNativeRestartRecoveries`, which
`recoverNativeRunsAfterRestart` calls):

| Classification | Who rechecks, when | Bound |
| --- | --- | --- |
| Claims: `reattach_existing_runner`, `reattach_remote_runner`,
`resume_dead_runner`, `bootstrap_incomplete` | The claiming server
resumes exactly | Exact resume; no fallback |
| `already_finalized` | Nobody; it is terminal | None needed |
| `blocked` | The run is terminal and its hold goes to the fallback
(disposition worker) | Fallback bounds above |
| `awaiting_evidence` / `current_controller_preparing` | This process's
own executor | Ends with that executor |
| `awaiting_evidence` / `recovery_rows_busy` | Restart dispatch after 1
second | Self-resolving: the row lock clears |
| `awaiting_evidence` / a live lease (`controller_lease_held`,
`controller_lease_grace`, `controller_still_alive`) | Restart dispatch
at the lease-renew interval | Never ended while the old controller
renews its lease |
| `awaiting_evidence` / unverifiable lapsed controller (process space or
host) | Restart dispatch every 2 minutes | Ends 2 minutes after the
lease lapsed |
| `awaiting_evidence` / any other reason (startup cleanup, finalizer or
cleanup owner, recovery failure, routine busy, future reasons) | Restart
dispatch at the owner's `retryAt`, or within 2 minutes when it has none
| Ends 2 minutes after the lease lapsed, or after 10 minutes, whichever
comes first. It never ends while a lease is live. |

Ending a run sets it failed (`native_restart_recovery_blocked`),
releases the task lock and the agent's slot, and records a system-owned
hold. The fallback then continues the task once, isolated.

No wait needs longer. Saved-work recovery and retained-source checks
keep their own attempt budgets. After 10 to 15 minutes the fallback
continues anyway in a new root, without the unverified files, which stay
recorded and unapplied.

### When the fallback preempts saved-work recovery

The unverified files are not destroyed. They stay where recovery found
them: in the abandoned remote epoch's root, or in the old local
workspace that the task is no longer bound to. They are recorded and
never applied. Settling the hold records the decision on it, so the
retained-workspace worker stops scheduling it (its query requires no
`executionReconciliation` unless recovery already succeeded). If an
attempt already in flight finishes later, it cannot publish into the
fresh run's workspace. For a remote workspace, the epoch it targets is
abandoned and the writer is cleared. For a local one, the task now
points at a new workspace. Whenever a wait is preempted, the
continuation is isolated: a fenced remote epoch or a new local
workspace.

### What people see

The task's interruption presentation shows "working" while the primary
fresh-continuation record is pending and has not started (coalesced
holds never count, and they follow the primary's final state), and a
timed wait with no Retry while a check is scheduled. The Retry banner
appears only when the system has given up or the cause is the person's.
Holds settled together with the primary record are `coalesced`; they no
longer show "closed or reassigned". A pending continuation is never a
second run: if a person's Retry or saved input started a run after it
was recorded, delivery links to that run instead. A task that finished
before delivery gets no run.

### Readers of what the fallback writes

- Recheck deadline (`freshContinuationCheck`):
- `unresolvedExecutionDispositions` and
`heldExecutionsAwaitingFreshContinuation` read it to skip work that is
not due.
- `nextFreshContinuationCheckAt` and
`executionControlSchedule.disposition` read it to schedule the worker.
- Tests: the cap deadline is scheduled, and a cancelled task goes quiet.
- Isolation decision:
- The task's workspace binding is cleared. Every path that starts the
next run (system continuation, saved input, Retry, sweep wake) resolves
the workspace from that binding at run start (heartbeat.ts
`issueContext.executionWorkspaceId`).
- `freshContinuationIsolation` in the run context is read by
`readFreshWorkspaceReadiness` for the shared-directory case.
- `freshContinuationSupersededAt` on the old run is read by the auth
middleware, `assertAgentRunWriteAllowed` and the claim gate.

## Verification

- `pnpm vitest run src/services/fresh-continuation-fallback.test.ts`
(server): every test passes (it covers the cases below).
- The continuation starts (leaves the queue) while the unreachable old
lease is `active` or `pending_cleanup`. This was red before the
claim-admission fix.
- Isolation: a writer on another host gets a new isolated workspace; a
process alive on this host waits for its owner; a proven-gone writer
keeps the workspace.
  - Owned holds only: a restore hold and resolved history are untouched.
  - The recheck schedule goes quiet when the task is cancelled.
- `continuation_user_authorization_missing` continues under the task's
responsible user.
- One test per top-five staging code (native and legacy hold shapes):
the task gets exactly one fresh successor run from saved state.
- `runner_harness_state_mismatch` (`resumed_not_suspended`,
`resumed_unavailable`) falls back.
  - An unknown code falls back.
- Two concurrent callers plus the disposition worker create one
successor.
- Old-saved/new-resumed: a settled-but-blocked hold from the previous
release, with its coordinator in `awaiting_runner_reattach`, continues
on the first pass.
- An ordinary system continuation wake is admitted as the fresh
continuation.
- Fencing: after the continuation is recorded, the old run's token is
refused for writes (`agent_run_superseded`) and the continuation's token
is not. The old run's remote workspace epoch is abandoned: the writer is
cleared, the location and provider root are cleared, and the epoch
advances. So the next preparation restores the last save into a new root
(`prepareRemoteWorkspace`: `sameAllocation` is false when the location
is null, server/src/services/remote-workspaces.ts:437-470).
- The opening context names the cut-off run and an unverified tool
action.
- Person-owned causes (pending approval, credential not shared, budget
stop) are not continued.
  - The hourly cap backs off without blocking, then continues.
- Red/green: with the wiring reverted to `master`, 9 of these tests fail
(the task stays Blocked with no successor).
- Affected suites pass locally: `execution-recovery-dead-controller`,
`completed-turn-cleanup-hold-settle`, `execution-control-work`,
`heartbeat-process-recovery`, `issue-recovery-actions`,
`native-safe-replacement`, `native-failed-workspace-recovery`,
`task-retry`, `runner-lost-mid-turn-continuation`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
wake-queue `postgres`, `agent-conversations`, `native-startup-retry`,
`governed-wait-runner-loss`, `native-preclaim-continuation`.
- `pnpm verify --explain` selects the full suite (the wake queue is
shared), so CI supplies full coverage.

Existing expectations changed because they asserted the "wait for a
person" behavior this PR replaces:
- `issue-monitor-scheduler.test.ts`: "explains held monitor dispatch
without replaying uncertain execution" is now "continues a held task in
a fresh run when its monitor is due". A due monitor on a task held by a
`process_lost` hold now starts one fresh run instead of deferring.
- `agent-run-authentication.test.ts`: the auth row also reads
`freshContinuationSupersededAt` (still under 512 bytes), and a
superseded run is refused.
- `assigned-work-invariant.test.ts` (#21139): "with the real wake queue,
a hold that defers the continuation leaves a violation…" is now "…a hold
reached by the sweep's wake becomes one pending fresh continuation". The
sweep's wake now yields one pending fresh continuation, and the next
sweep adds no attempt.
- `heartbeat-native-restart-suspension.test.ts` and
`native-runner-restart-recovery.integration.test.ts`: awaiting-evidence
dispositions now carry a `retryAt`, because every wait is rechecked. A
lapsed unverifiable controller no longer waits forever.
- `native-remote-retry-admission.test.ts`: the blocked-hold text is now
"This run stopped before it finished. Your saved work is kept." The old
text asked people to reconcile.

Holdless legacy runs: in the 1,000 staging runs, 0 tasks end on
`controller_lease_lost` or `process_lost` (every one has a later run).
12 end on `server_shutdown_interrupted` with no retry scheduled (attempt
0). On staging now, 9 of those 12 sit `in_progress` with no run and no
hold, 1 is done and 2 have a live run. Those 9 belong to the idle-task
sweep (#21139).

## Risks

- Speed: nothing is added to the normal path before `turn.start`. The
fallback runs only on the disposition worker after exact recovery is
ruled out, or at wake admission when a hold already defers the wake. The
auth middleware reads one more JSON field in the row read it already
does.

- Behavior shift: tasks that today stay Blocked for a person after an
infrastructure failure now continue automatically. A side effect with an
unknown outcome is not replayed; the new run is told about it and must
check first.
- After deploy, the disposition worker continues tasks already blocked
by these codes, at most 25 per pass and at most 6 per task per hour. No
migration. No manual Retry is needed.
- The old executor is not proven stopped before the successor starts.
Fencing is: its API writes are refused, its workspace epoch is
abandoned, the native `replacementDenied` marker is set, and the
successor has a separate run identity.

Known gaps (follow-ups):
- No quarantine attachment: unverified files stay unapplied in the
abandoned workspace root (recorded on the workspace in
`remoteWorkspaceAbandonedEpochs`). They are not read into a task
attachment, because the sandbox is usually unreachable in these
failures.
- Agent Chat conversations are excluded at wake admission. They keep
their owner-bound continuation.
- Under a `reuse_by_environment` lease policy the new allocation can be
on the same provider sandbox. It always uses a new root and a new epoch,
and the old run cannot publish or write.
- Tasks left `in_progress` with no run and no hold (for example after
`server_shutdown_interrupted` with no retry scheduled) belong to the
idle-task sweep. That sweep's wake reaches this fallback when a hold
exists.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:34:34 -06:00
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00
DottaandClaude Opus 5.5 b8207c4ea9 fix(runner): evict finished API snapshots instead of blocking reads at the company quota (#21133)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs read the Paperclip API through `call_api`. Large
responses are saved as "API snapshot" assets so the agent can read them
in pages.
> - Each company has a 20 GiB snapshot quota. Before a large read, the
server reserves 1 GiB of worst-case space against that quota.
> - Snapshot assets were never removed. Reservations from interrupted
captures (crash, restart, ambiguous upload) stayed at 1 GiB each until
"an operator reconciles" them. Nothing did that.
> - After a morning of high retry volume, the quota filled. Every large
read then failed with `api_response_company_storage_limit`, and agents
blocked their own tasks and asked an operator to free storage.
> - This pull request makes company admission reclaim space by itself:
it releases stale reservations of finished runs and evicts the oldest
snapshots of finished runs, oldest first, only as much as the new read
needs.
> - The benefit is that a read never fails because old snapshots filled
the quota, and no person must clear storage by hand.

## Linked Issues or Issue Description

**What happened?**

On staging, about 97 tasks of one agent had successful runs but ended in
`blocked`. In each run, `GET` calls through `call_api` for large records
(for example heartbeat runs and their events) failed with
`api_response_company_storage_limit`: "Company API snapshot storage
quota is exhausted. Read an existing asset or ask the operator to remove
old snapshots or increase the quota." The agent then reported a
task-wide blocker owned by "the instance storage operator". At least 15
tasks had a final comment such as "Blocked: company API snapshot storage
is exhausted". The error message itself told the agent that an operator
must act.

**Expected behavior**

A read must not fail because old snapshots filled the company quota. The
server must reclaim the space automatically. Only snapshots that active
runs still hold, or that users attached to issues, cases or logos, may
stay.

**Steps to reproduce**

1. Set `PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES` to 2 GiB.
2. Store a 1 GiB `runner-api` snapshot asset from a finished run, or
leave two 1 GiB unattached reservations from finished runs that are
older than 20 minutes.
3. In a running native run, call `call_api` for a response larger than
the inline limit.
4. Before this change, the call fails with
`api_response_company_storage_limit`. After this change, the call
succeeds and the oldest finished snapshot is evicted.

**Paperclip version or commit**

Staging deploy `704b3f4d63`. Fix is based on `origin/master` at
`8bdd53c11f`.

**Deployment mode**

Hosted (staging cloud), native runtime with Daytona sandboxes.

**Agent adapter(s) involved**

Native Paperclip runner (Codex app server, `gpt-6.1-sol`). The defect is
in the server-side `call_api` capture admission, so it affects every
native adapter.

## What Changed

- Added
`server/src/services/native-runtime/runner-api-capture-retention.ts`. It
measures company snapshot usage and reclaims space under the existing
per-company advisory lock.
- Reclaim step 1: release unattached reservations older than twice the
capture deadline (20 minutes) whose run is missing or not `queued`,
`running` or `scheduled_retry`.
- Reclaim step 2: evict snapshot assets oldest first until the new
reservation fits. A snapshot is eligible only when it is older than 1
hour, every run that captured it has finished (succeeded, interrupted,
failed, cancelled or timed out), and no issue attachment, case
attachment or company logo links to it. Each eviction writes an
`asset.evicted` activity-log row with the asset id and the owning run
id.
- A capture upload that finishes after its reservation was reclaimed is
rejected in the publish transaction. The same transaction records the
object for deletion, and the bounded cleanup removes it. The upload has
a 10-minute abort signal.
- Failed deletes are logged per company and counted with the
`runner_api_capture_delete_failed` metric. When cleanup keeps failing,
the read error says so in plain words.
- Eviction credits deletes that are already pending before it chooses
more snapshots. Eviction is durable. In the same transaction that
deletes an asset row, it inserts a reservation row with
`pending_delete_object_key`, so the bytes stay counted. Admission then
deletes the stored objects in pages, under a 30-second abort signal, and
removes each row only after its delete succeeds. A failed or interrupted
delete is retried at the next over-quota admission. No background poller
is added.
- Added the `pending_delete_object_key` column and an index on
`runner_api_response_reservations.asset_id` (migration 0472).
- `#reserveApiResponseCapture` now runs reclaim only when the quota
would be exceeded, so reads below the quota do no extra work. After
marking evictions it finishes the deletes and retries admission once.
- When no space can be reclaimed, the error message now names what holds
the space (runs still working, files attached to tasks, deletes in
progress). It no longer tells the agent to ask an operator.
- Replaced the test that asserted old snapshots block reads with tests
for eviction, multi-page eviction, durable delete retry, a bounded
stalled delete, stale-reservation release, and protection of active and
attached snapshots.

## Verification

- Red: the three new tests in `runner-api.integration.test.ts` fail on
`origin/master` (two get `api_response_company_storage_limit`, one gets
the old "ask the operator" message).
- Green: `cd server && npx vitest run
src/services/native-runtime/runner-api.integration.test.ts` passes 70/70
with the fix.
- The multi-page test failed first with
`api_response_company_storage_limit` because pending deletes ran in one
200-row batch. The paged delete loop fixed it.
- `cd server && npx tsc --noEmit -p .` passes.

## Risks

- An agent that reads a saved snapshot of a finished run after it was
evicted gets a not-found response. It can read the API again. Eviction
happens only under quota pressure, oldest first.
- A released stale reservation can leave an unknown storage object from
an ambiguous upload. These bytes were never linked to an asset, so they
were already unreachable. This is the same storage the old code kept
counted forever.
- Eviction runs inside the company capture lock. The candidate query is
bounded to pages of 200 rows and uses the new `asset_id` index.
- Migration 0472 adds one nullable column (`ADD COLUMN IF NOT EXISTS`)
and builds one index with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It
is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. It is additive and
safe for old and new servers during a rolling deploy.
- During a storage outage, an over-quota read still fails until deletes
succeed. The bytes stay counted, so the quota is never exceeded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser reads of staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:30:38 -06:00
DottaandClaude Opus 5.5 90ae3c4337 fix(native): process a superseded export's late stop receipt and stop its sandbox when none arrives (#21143)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run can be retired as superseded while its workspace export
fails. Its files stay unsaved, and the run stays the writer of the
task's workspace.
> - #21136 records those files for the automatic workspace recovery.
That record needs the sandbox's stop-and-retain receipt.
> - The native run projection worker checked for that receipt for one
hour after the run ended. After that, it dropped the run. Nothing woke
it when the receipt arrived later.
> - If the receipt came late or never, the run kept the task's workspace
fenced. Every queued run on the task waited and then failed setup.
Nobody owned the problem.
> - This pull request makes the worker process a late receipt, stop and
keep the sandbox itself when no receipt arrives, and never drop the run
silently.
> - The benefit is that the task's queue moves again by itself, and two
writers never use one workspace.

## Linked Issues or Issue Description

Refs #21136 (review thread on
`native-superseded-workspace-export.ts:84`)

**What happened?**
`preserveSupersededWorkspaceExport` returned `recheckAt: null` when the
sandbox stop receipt had not arrived within `finishedAt + 1h`. The
projection worker then deleted its entry. A later stop receipt
(`settleStopOnlyCleanup`) woke only the workspace recovery queue, not
the projection queue. `preserveNativeFailedWorkspace` requires
`hasConfirmedSandboxStopAndRetain`, so no hold was recorded. The run
stayed the dirty writer, and queued runs kept failing setup.

**Expected behavior**
A late receipt is processed at once. When no receipt arrives, Paperclip
stops and keeps the sandbox itself and waits for the provider's
confirmation. The workspace stays held while the sandbox may still run.
When the provider confirms that the sandbox is gone, the task continues
from its last save. When the stop stays unconfirmed for hours, the task
shows one plain notice, and the check continues.

**Steps to reproduce**
1. Retire a native run as superseded after its workspace export fails
(the #21136 shape).
2. Let the run's sandbox stop not settle (for example, the server that
owned the release is replaced).
3. Wait more than one hour after the run ended.
4. See that the projection worker drops the run, and that a later stop
receipt records nothing. The workspace stays held.

## What Changed

- `sandbox-stop-and-retain.ts`: a confirmed stop-only receipt now also
notifies `DELIVERY_QUEUES.nativeRunProjection` (scoped to the company
and run). New `prepareUnconfirmedSourceSandboxStop` puts a
stop-and-retain intent on the source lease from its current status. It
skips a destroyed sandbox, a stop already in progress, and a cleanup in
flight.
- `native-superseded-workspace-export.ts`:
- Within one hour of the run ending, the worker checks every minute
(unchanged).
- After that hour, it requests the stop-and-retain itself. The existing
pending-cleanup sweep dispatches the stop and retries it until the
provider confirms. The worker checks again every 15 minutes, and the
receipt wakes it at once. The workspace stays held.
  - A sandbox that another lease holds is never stopped.
- When the provider confirms that the sandbox was destroyed, the files
cannot be saved. In one transaction (task, run, then workspace locks),
the worker sets aside only that run's workspace epoch, wakes queued-run
dispatch, and records `issue.previous_run_files_lost` on the task.
- When the stop is still unconfirmed six hours after the run ended (or
the sandbox is held by another lease), the task gets one
`issue.previous_run_files_waiting` activity entry. The worker keeps
checking. It never drops the run.
- A stop-only dispatch now settles when the provider confirms that the
exact sandbox is already gone. Before, the runtime accepted only a
`stopped` reply and threw on `destroyed`, so a gone sandbox kept the
stop pending forever.
- `sandbox-stop-and-retain.ts`: new `acceptedStopOnlyReceipt` accepts
`stopped`, or `destroyed` for a stop-and-retain intent (a native
workspace export resume still needs `stopped`). `settleStopOnlyCleanup`
stores a `destroyed` receipt as `released` / `success` /
`workspace_source_unavailable`, removes the stop intent, never writes a
retain receipt, and wakes the projection and workspace recovery queues.
- `environment-runtime.ts`: the plugin `environmentStopLease` and
built-in `stopLease` checks use `acceptedStopOnlyReceipt`.
- `workspace-recovery.ts`: when its own stop settles as `destroyed`, the
source is unavailable (the existing alternative-strategy path) instead
of a binding failure.
- Daytona plugin: stop of a sandbox that Daytona reports as not found
returns `destroyed` (destroy already did this) instead of throwing. boxd
already returns `destroyed`. Other sandbox plugins have no stop-only
method, so nothing changes for them.
- New shared `SANDBOX_ALLOCATION_HOLDER_STATUSES` (`active`, `retained`,
`pending_cleanup`). The stop-only cleanup ownership check and the
worker's "held by another lease" guard both use it, so a retained warm
sandbox is never stopped and the two checks cannot drift.
- `ui/src/lib/activity-format.ts`: plain labels for the two task
activity entries. They do not ask a person to fix anything.
- `doc/execution-semantics.md`: document the rule.
- Not changed: the shared owner prover still refuses to treat a stopped
instance on another host as proof that its child processes stopped.

## Verification

- `server/src/services/superseded-workspace-export-recovery.test.ts` (4
new tests, red/green):
- "checks again every minute while the run's own release may still stop
its sandbox": the recheck branch. Nothing is recorded or stopped, and
the workspace keeps its writer.
- "stops and keeps the sandbox itself when no stop receipt arrived, and
records the files once the late receipt lands": after the hour, the
lease gets a stop-only intent. A second pass changes nothing. The late
receipt wakes the projection queue. The next check records the hold, and
the system recovery frees the workspace. Fails on master (`expected {
…(19) } to match object { status: 'pending_cleanup' }`).
- "names the unconfirmed stop on the task once, and keeps checking, when
the provider never confirms it": the expiry branch. One notice, the
check continues, no hold, and the workspace stays held. Fails on master.
- "continues from the last save when the provider confirms the sandbox
is gone": epoch 1 is set aside, the workspace is free, the task says so,
and the result is not applied. Fails on master.
- Review-round tests (all 5 fail on the first head `a208bfa58f`, pass
now):
- `environment-runtime.test.ts` "settles a plugin stop whose provider
reports the sandbox gone (boxd, Daytona not found) through the cleanup
sweep": provider dispatch through `releaseRunLeases` and
`sweepPendingCleanupLeases`. The lease settles `released` with a
`destroyed` receipt and no retain receipt.
- `environment-runtime.test.ts` "settles a built-in stop whose provider
reports the sandbox gone, without claiming retained files": the built-in
`stopLease` path. No destroy call.
- `superseded-workspace-export-recovery.test.ts` "continues from the
last save when Paperclip's own stop finds the sandbox already gone": the
worker's stop request gets a `destroyed` reply, the projection queue
wakes, and the workspace is freed at epoch 2.
- `superseded-workspace-export-recovery.test.ts` "never stops a sandbox
a retained warm lease still holds, says so once and keeps checking": the
source lease is unchanged, a recheck is returned, and `sandbox_in_use`
is recorded.
- Daytona `plugin.test.ts` "explicit stop confirms a sandbox Daytona no
longer has as destroyed".
- `environment-runtime.test.ts` (207) and Daytona `plugin.test.ts` (421)
pass.
- Related suites pass: `native-run-projection-work`,
`retained-native-session-cleanup-work`, `workspace-recovery-stop-proof`,
`native-failed-workspace-recovery`, `legacy-workspace-restore-recovery`,
`environment-retained-orphan-sandbox`,
`workspace-recovery-retry-copyback`, `workspace-recovery-fallback` (398
tests), and `ui/src/lib/activity-format.test.ts`.
- `tsc --noEmit` for `server`.

## Risks

- Low to medium risk. The new behavior acts only on an ended superseded
run that failed in its workspace export, still holds a dirty workspace,
and has no retained-file hold.
- The worker now stops an ended run's sandbox one hour after the run
ended if no stop receipt arrived. The stop uses the existing stop-only
contract and keeps the sandbox. A sandbox held by another lease is never
stopped.
- A destroyed sandbox's unsaved files are not recoverable. The task
continues from its last save and says so.
- If a provider never confirms a stop, the workspace stays held (by
design: no second writer). The task shows one notice, and Paperclip
keeps retrying.
- Shared cleanup code now treats a confirmed `destroyed` stop-only reply
as settled. Every caller was checked: the cleanup sweep and run release
count it as settled; the workspace recovery treats the source as
unavailable; the fallback treats it as stopped (nothing can run there).
No caller reads it as retained files, because no retain receipt is
written. A native workspace export resume still requires `stopped`.
- Each extra delivery notification per settled stop-only cleanup is
cheap. The projection worker's scoped load finds nothing for runs that
do not match.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:54:47 -06:00
DottaandClaude Opus 5.5 3fb35fcf13 fix(native): restart an interrupted bootstrap whose runner connected before the deploy (#21120)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. A deploy stops the server and a new server recovers
each run.
> - When the server stops during a run's bootstrap, recovery restarts
that bootstrap on the same run (`bootstrap_incomplete`). The claim is
only granted when the controller checkpoint in Postgres proves that no
provider turn was sent.
> - Host preparation then reads the sandbox Runner state. A Runner that
already authenticated is `ready`, not `suspended`. The restart is
allowed only with a stop proof for that Runner.
> - The stop proof was taken in two places only: from a restored local
controller root, or for a Runner that never connected. A new server
after a deploy can have neither, so the run failed with
`runner_harness_state_mismatch: resumed_not_suspended` and needed a
manual Retry.
> - This pull request lets host preparation take the same Postgres
checkpoint proof that granted the claim, stop that exact Runner, and
restart the bootstrap.
> - The benefit is that a deploy during a bootstrap no longer fails the
run. No provider turn can be replayed, because the proof shows that none
was sent.

## Linked Issues or Issue Description

**What happened?**
On staging, three Codex runs on Daytona failed with
`native_session_interrupted` and `runner_ingress_unavailable: runnerd
process failed: runner_harness_state_mismatch: resumed_not_suspended`
(12:48Z twice and 13:19Z on 2026-11-12). In each run the Runner
authenticated 1 to 4 seconds before a deploy stopped the server
("controller authority detached for restart; durable runner left
alive"). The new server chose "Restarting an incomplete native runner
bootstrap on the same heartbeat run". Its host preparation found the
sandbox Runner `ready`, had no stop proof, and failed the run. The run
log has no "Stopped the interrupted runner" line, so the earlier stop
step did not run on the new server.

**Expected behavior**
The run restarts its bootstrap on the same sandbox and continues without
a manual Retry. A provider turn that may have run is never replayed.

**Steps to reproduce**
1. Start a Codex run on a Daytona sandbox.
2. Deploy the server after the Runner authenticates and before the first
turn starts.
3. Let the new server recover the run with a `bootstrap_incomplete`
claim on a server that has no local controller root for the run.
4. Host preparation fails with `runner_harness_state_mismatch:
resumed_not_suspended`.

**Paperclip version or commit**
Staging at 1d153f62b6. The code is the same on master at 704b3f4d63.

**Deployment mode**
Hosted staging (paperclip.staging.paperclip.app), remote sandbox
execution on Daytona.

**Agent adapter(s) involved**
Paperclip Runner with Codex (`codex_app_server`).

## What Changed

- `native-runner-checkpoint.ts`:
`nativeRunnerCheckpointRemoteBootstrapEvidence` returns the bootstrap
proof together with the runner instance that the checkpoint names.
`nativeRunnerCheckpointRemoteBootstrapProof` now wraps it and behaves as
before.
- `native-session-executor.ts`:
- State migration runs before this server registers its own controller
for the run. Under a `bootstrap_incomplete` claim for a remote sandbox,
it now reads the interrupted controller's checkpoint proof there
(`captureClaimTimeRemoteBootstrapProof`). A later read is not used,
because the new controller rewrites the checkpoint with its own
bootstrap commands.
- Host preparation: when this run's sandbox Runner is not suspended, it
runs the same-run restart proof. The proof accepts a Runner that never
connected (as before), or the claim-time proof for the exact runner
instance in the sandbox state. It then stops that exact Runner by its
identity marker and records the stop proof. A backup, a different run
id, another runner instance, or a failed stop still fails closed.
- `native-session-executor.test.ts`: new
`interrupted_bootstrap_connected_checkpoint_proven` case. The Runner is
`ready` and connected, and no stop proof exists yet. It fails before the
fix with `resumed_not_suspended` and passes after it.
`interrupted_bootstrap_unproven` still fails closed.
- `native-bootstrap-claim-time-proof.integration.test.ts` (embedded
Postgres, no mocks of the proof): an interrupted controller that had
connected registers its checkpoint, the claim-time proof is read, and
then a replacement controller registers its own checkpoint. A live read
after that returns `bootstrap_incomplete` even when the interrupted
controller had sent `turn.start`. The claim-time proof stays false for
that case and true for the bootstrap-only case, and it refuses another
runner instance.

## Verification

- Red: with the source change reverted, `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session"` fails on the new case.
- Green: the same command passes (65 tests after merging master).
- `pnpm exec vitest run
server/src/services/native-runtime/native-bootstrap-claim-time-proof.integration.test.ts`:
2 passed.
- `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts
server/src/services/native-runtime/native-restart-recovery.test.ts`:
1474 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.
- Merged `origin/master` (including #21134) without conflicts and reran
the tests above.

## Risks

- Low. The new path runs only under a `bootstrap_incomplete` claim for
the same run id, with no failover backup, and only after the Postgres
controller checkpoint proves that no provider turn was sent. It stops a
Runner only when the identity marker and process fingerprint match,
which is the same check the reattach path uses.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21134 (the never-launched branch of the same helper), #21079
(stops a leftover Runner at launch time; this change acts earlier, in
host preparation).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:38:06 -06:00
DottaandClaude Fable 5.2 2174f2dfa5 perf(server): stop re-reading and re-decrypting unchanged checkpoint headers on every upgrade readiness probe (#21130)
Cloud repeats the upgrade readiness probe. Every probe loaded, decrypted and parsed the checkpoint header of each active native run and of each retained task session's last run, even when nothing had changed.

The statement that reads a run now also reads its header's row revision (xmin, ctid, updated_at, stored size) and returns the material only for a revision the process has not inspected. A bounded memo in the checkpoint owner keeps the four identifiers the answer compares. It reuses an answer only for the same binding, row revision and secrets key, loads the key again for each answer it serves, never remembers a header that failed to decrypt, and forgets a header the latest probe did not inspect.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 10:45:44 -06:00
DottaandClaude Opus 5.5 d5bd38468d fix(runner): stop a leftover Runner on the same state folder before a relaunch (#21079)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. The Runner keeps its saved state in one folder per
session and listens on one port in the sandbox.
> - During an automatic upgrade, the old server can launch a Runner a
few seconds before it stops. That Runner stays alive in the sandbox and
keeps the state folder and the port.
> - The new server then restarts the run on the same sandbox. Each new
Runner fails at once with "Address already in use"
(`runner_ingress_bind_conflict`). Each launch also overwrites the
process marker that pointed at the old Runner, so the next check cannot
find the old Runner to stop it. The run restarted about 12 times in 19
minutes and then stopped, and the task stayed blocked.
> - Two Runner processes must never use the same saved state at the same
time.
> - This pull request makes each Runner launch stop any live Runner that
still names the same state folder, before the new Runner starts.
> - The benefit is that a restart after an upgrade gets the port and
continues the run, instead of looping on the bind conflict.

## Linked Issues or Issue Description

**What happened?**
On staging, a Runner launched by the old server just before an upgrade
kept the Runner port. Every relaunch by the new server exited within
seconds with "runner_ingress_bind_conflict: failed to bind listener:
Address already in use (os error 98)". The run looped for about 19
minutes and then failed with `native_runner_process_exited`. The task
was blocked and needed a person.

**Expected behavior**
The relaunch stops the leftover Runner of the same session, gets the
port, and continues the run.

**Steps to reproduce**
1. Launch a Runner for a run in a sandbox.
2. Stop the server before the Runner is adopted (an upgrade). The Runner
stays alive.
3. Let the new server restart the run's bootstrap. The new Runner cannot
bind the port, and the next restart cannot see the old Runner because
the marker now names the dead new process.

## What Changed

- `native-session-executor.ts`: the detached Runner launch script now
runs a short fence before it writes the process marker.
- It finds processes whose argv[0] is `paperclip-runnerd`, that are not
a `reap-lineage` command, and whose `--state-dir` is followed by this
exact folder. Runners on another folder, a folder named by a different
flag, and other programs are not touched.
- It records the stale Runner's lineage by PID **and start time**: its
descendants, and members of a process group or session that a recorded
process leads while that leader is alive (provider tools start their own
sessions). It signals a PID only while its start time still matches, so
a reused PID is never signalled, and it never signals a whole process
group.
- TERM, up to 5 s wait, then KILL, up to 2 s wait. Then it runs
`<runnerd> reap-lineage --state-dir … --runner-pid …` for the stopped
Runner, which stops the tools its lineage journal recorded that have
already left the process tree.
- It fails closed: a survivor (`runner_stale_state_holder_alive`) or a
reap that leaves a recorded process alive or gives no report
(`runner_stale_lineage_unproven`) makes the launch exit 75 **before**
the marker is written, so the existing marker is kept. The reason goes
to the process stderr log only when that path is a plain file (same rule
as the Runner stderr log). The whole stop is bounded at about 14 s,
under the 20 s marker wait.
- Two reap refusals let the launch continue with a
`runner_stale_lineage_partial` warning: `journal_unavailable` (the
Runner kept no journal; its visible lineage is already gone) and
`unattributed_orphan` (reported only after every recorded process is
gone, for orphans the reaper can neither claim nor signal) **only when
every orphan it names is proven to have another owner**: this launch's
own detached shell, a `paperclip-runnerd` serving a different
`--state-dir`, or that Runner's stderr trimmer. Any other orphan could
be an old tool that escaped between samples, so the launch fails closed.
- When `/proc` is not readable, or `grep` is missing, the fence does
nothing (the old behavior).
- `native-session-executor.test.ts`: Linux tests (a stale Runner whose
tool runs in its own session, plus four look-alikes that must survive;
fail-closed with the marker kept, with a plain and a symlinked log; reap
refusals for orphans owned elsewhere, an unknown orphan, and no journal;
a normal launch starts no extra node process) and a cross-platform check
that the embedded script parses and has no shell-special characters.

**When a launch happens vs. a reattach.** The launch is reached only
after the server chose not to adopt a Runner. A same-run reattachment
(`reattach_remote_runner`) first verifies the marker's Runner: if it is
alive it is adopted and nothing launches; a transient verification
failure (for example the read timeout that #21053 keeps the Runner alive
for) throws and never reaches a launch; only a Runner proven not running
falls through to a fresh launch. Warm recovery likewise throws instead
of launching when revalidation fails. Every launch overwrites the
marker, so after a launch no reattachment could adopt a leftover Runner
anyway, and its port would make the new Runner fail with
`runner_ingress_bind_conflict`. Stopping it therefore never removes a
Runner that the same run could still re-adopt.

## Verification

- Linux (Debian bookworm container, node 24): the Linux tests pass on
this head. The same tests against the previous head (8df6a9099e) fail:
it killed a `paperclip-runnerd reap-lineage --state-dir <same folder>`
command, and it launched although the stale lineage was not proven gone.
On master there is no fence, so the stale Runner stays alive.
- Boot time: a normal launch took 19–35 ms for the whole launch script
in the container with the fence; node is not started by the fence
(asserted by a test). With a stale holder, the stop and launch took
about 160 ms.
- macOS: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts` passes
(1412 passed, 4 skipped: the Linux-only tests).
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- A launch now stops an older Runner on the same state folder. See "When
a launch happens vs. a reattach" above: this only happens when that
Runner is already impossible to adopt and would block the new one.
- A normal launch pays one `grep` over `/proc` plus reading argv[0] of
the processes that name the folder (its own subshell). No node start, no
extra round trip, no wait.
- If `paperclip-runnerd reap-lineage` cannot verify the stale lineage,
the launch fails closed instead of starting a second Runner next to old
tools.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:34:22 -06:00
DottaandClaude Opus 5.5 df89950eb6 fix(native): save or set aside a superseded run's unsaved workspace so its task queue moves (#21136)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result. Then a finalizer exports the
run's workspace files and applies the result to the task.
> - When a newer run already decided the task, the finalizer retires the
older result as superseded. Since #21106, this also ends the run.
> - If that retirement interrupts a failed workspace export, the run's
files stay unsaved. The run also stays the writer of the task's
workspace. Nothing records those files for recovery, so nothing ever
releases the workspace.
> - Every run queued on the task then waits behind a save that never
settles. After one hour, each queued run fails setup instead.
> - This pull request records those files for the existing automatic
workspace recovery. Recovery saves them from the retained sandbox, or,
when every bounded attempt fails, continues the task from its last save
with the files attached.
> - The benefit is that the task's queue moves again by itself, and the
run's unsaved work is saved or kept findable. The superseded result is
never applied.

## Linked Issues or Issue Description

Refs #21106

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. Its workspace export failed once with
`remote_workspace_archive_fs_eacces`. The finalizer then retired it as
`native_finalization_superseded`, because another run had already
decided the task. #21106 ended that run at 14:59:17Z (`failed`). The
task still could not move. The next queued run was cancelled with
`workspace_busy` ("ended, but its workspace save has not settled"). Its
retry re-armed every minute. Four more runs stayed queued. The task
showed no recovery action.

**Expected behavior**
The ended run's unsaved files are saved from its retained sandbox, and
the workspace is released. If they cannot be saved within the bounded
retries, the task continues from its last committed save, and the files
are attached to the task. The queued runs proceed in both cases. The
superseded result is never applied.

**Steps to reproduce**
1. Accept a native result for a run that writes a remote workspace.
2. Let the workspace export fail once, while a newer run has already
committed the task's decision.
3. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is `failed`.
4. See that the workspace still names the run as its dirty writer, and
that no retained-file recovery action exists.
5. Queue another run on the task. It defers with `workspace_busy`, and
after one hour it fails setup with "The workspace still has an
unfinished writer or save operation."

**Root cause**
`recordRetryableFailure` (`native-run-finalizer.ts`) retires a
superseded finalizer without preserving the export.
`preserveNativeFailedWorkspace` records a retained-file hold for an
accepted result only when the failure code is
`native_workspace_sync_out_retry_exhausted`. A
`native_finalization_superseded` coordinator never qualifies.
`readRemoteWorkspaceWriterHold` keeps an ended dirty writer as the
holder until its save settles. Without a hold, no worker ever settles
it.

**Paperclip version or commit**
Staging Core with #21106 (`81d2e4d8d9`). Master `ef2b53b675` has the
same behavior.

## What Changed

- `native-superseded-workspace-export.ts` (new): identifies a superseded
finalizer that failed in the workspace export. It selects ended runs
that still hold their workspace with unsaved files and no retained-file
hold. It records the hold through `preserveNativeFailedWorkspace`. It
checks again only while a just-ended run's sandbox stop receipt is still
arriving (at most one hour).
- `native-failed-workspace-recovery.ts`: `preserveNativeFailedWorkspace`
treats a superseded workspace export like an exhausted copy-back, with
the same proofs (run ended, no live controller, accepted result,
confirmed sandbox stop-and-retain, source still bound to the workspace).
It marks the hold `supersededWorkspaceExport`.
- `native-run-projection-work.ts`: the native run projection worker also
selects these runs. It loads them at startup and wakes when a finalizer
changes. Rows written by older releases are therefore settled after
deploy without a person.
- `workspace-recovery-fallback-discovery.ts` and
`workspace-recovery-fallback.ts`: a marked hold whose bounded re-export
is spent (at most 5 retries in 30 minutes) is eligible for the existing
continue-from-last-save fallback. That fallback sets aside only this
run's workspace epoch and keeps its location on the workspace record. It
attaches the readable files to the task unapplied. The task message says
Paperclip "could not save" the previous run's unsaved files.
- `preserved-result-finalization.ts`: a superseded result is never
resumed for application, even after its files are saved.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/services/superseded-workspace-export-recovery.test.ts`
(new). The first two tests fail on master and pass on this branch:
- "records the files for automatic recovery, saves them from the
retained sandbox and frees the workspace". It uses the staging row
shape. The projection worker records the hold, and a second pass changes
nothing. The workspace waits for that save. The system recovery saves
`saved.txt` into a committed snapshot. The workspace becomes free. The
finalizer, the result and the task's decision do not change, and the
result is not resumed. On master the worker selects nothing (`expected 0
to be 1`).
- "continues from the last save with the files attached when every
re-export fails". After an exhausted re-export budget, the hold is
resolved `restored`. The files are attached (`quarantine.state =
captured`). The workspace moves to epoch 2 with no writer, and the
abandoned epoch keeps its lease location. The message has no request for
a person to fix anything.
- "leaves a superseded result alone when it never failed in the
workspace export".
- Related suites pass: `native-run-projection-work`,
`native-finalization-recovery`, `workspace-recovery-fallback`,
`native-failed-workspace-recovery`, `workspace-recovery-retry-copyback`.
- `tsc --noEmit` for `server`.

## Risks

- Low to medium risk. The change acts only on runs whose finalizer is
already terminal and superseded, that failed in the workspace export,
and that still own a dirty workspace. All existing preservation proofs
still apply.
- The saved files of a superseded run become the workspace's next saved
version. The run's result (the task decision) is still never applied.
- When the export can never succeed, the task continues from its last
save. Files that cannot be read are not attached. The workspace record
keeps the set-aside epoch's location. Saving the readable part of a
partly unreadable workspace is not in scope.
- Runs that already failed setup before this deploys are not restarted
by this change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:20:54 -06:00
DottaandClaude Opus 5.5 fb8b61bc3e fix(native): restart a bootstrap whose closing server staged files but never launched its Runner (#21134)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. A release swap stops the old server and a new server
recovers each run.
> - When the old server stops during a run's bootstrap, the new server
restarts that bootstrap on the same run (`bootstrap_incomplete`). The
claim is granted only when the Postgres controller checkpoint proves
that no provider turn was sent.
> - The old server can stop after it staged launch files (Codex home,
config) into the sandbox session folder, but before it launched the
Runner. Its detach fence stops it there.
> - The new server then finds no Runner state in the sandbox. A fresh
bootstrap must create the session folder itself, and the folder already
exists, so the run failed with `runner_harness_state_mismatch:
resumed_unavailable`. The failure is an ambiguous session, so the task
waits for a person.
> - This pull request lets host preparation prove that no Runner was
ever launched in that folder, move the folder aside, and restart the
bootstrap.
> - The benefit is that a release swap during this part of a bootstrap
no longer blocks the task. No provider work can be lost or replayed,
because no Runner ever ran.

## Linked Issues or Issue Description

Refs #21120 (same release-swap class: the Runner was launched and
connected before the old server stopped; that pull request stops it in
host preparation. This pull request covers the case where no Runner was
launched.)
Refs #21079 (launch-time fence for a stale Runner on the same state
folder.)

**What happened?**
On hosted staging, during a release swap on 2026-11-12, a Codex run on
Daytona failed with `native_session_interrupted` and
`runner_ingress_unavailable: runnerd process failed:
runner_harness_state_mismatch: resumed_unavailable`. The old server was
in `harness_state.launch_material` when its shutdown drain detached the
run. It recorded no Runner process identity. The new server chose
"Restarting an incomplete native runner bootstrap on the same heartbeat
run". Its host preparation found the sandbox resumed and no Runner
state. Then the claim of the session folder failed, because the old
server had already created that folder. The run needed a person.

**Expected behavior**
The run restarts its bootstrap on the same sandbox and continues without
a person. A provider turn is never replayed and two Runners never share
the run.

**Steps to reproduce**
1. Start a Codex run on a Daytona sandbox.
2. Stop the server after the bootstrap stages launch files into the
sandbox session folder and before it launches the Runner.
3. Let a new server recover the run with a `bootstrap_incomplete` claim.
The new server has no local controller root for the run.
4. Host preparation fails with `runner_harness_state_mismatch:
resumed_unavailable`.

**Paperclip version or commit**
Staging release `704b3f4d63`. The code is the same on master at
`ef2b53b675`.

**Deployment mode**
Hosted staging, remote sandbox execution on Daytona.

**Agent adapter(s) involved**
Paperclip Runner with Codex (`codex_app_server`).

## What Changed

- `native-session-executor.ts`: under a same-run `bootstrap_incomplete`
claim with no failover backup, when host preparation finds no Runner
state, a new check asks the sandbox in one command whether the session
folder exists and holds no Runner trace. A Runner trace is the live
`runner/runner-state.json`, the `runner/runner-process.identity` launch
marker (the launch script writes it before the Runner starts), an
`authority-epochs` archive, or any other `runner-state.json` or launch
marker anywhere under the folder. If the folder holds no Runner trace,
no Runner was launched there, so host preparation records the restart
proof (`absent_confirmed`). It then uses the existing path: move the
folder aside (it is kept for inspection), claim a new folder, and
bootstrap again. If the folder is missing, the existing fresh-claim path
runs unchanged. If any Runner trace exists, the run stays fail-closed as
before. An archived session is therefore never discarded.
- `native-session-executor.test.ts`: new
`interrupted_bootstrap_unlaunched` cases. Resumed and created sandboxes
restart the bootstrap. A launch marker, an archived authority epoch,
Runner state elsewhere in the folder, or a failover backup keeps the run
fail-closed with `resumed_unavailable`.

## Verification

- Red: with the source change reverted, `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session" --maxWorkers=2` fails on
both `interrupted_bootstrap_unlaunched` restart cases with
`runner_harness_state_mismatch: resumed_unavailable`, which is the
staging error.
- Red: without the archived-state guard, the `_archived` and
`_nested_state` cases fail (the folder is moved aside). With the guard
they pass.
- Green: the same command passes (64 tests).
- `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts
--maxWorkers=2`: full file, 1422 passed before the guard commit. The
focused 64 pass on the final revision.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.
- `git merge-tree` against #21120's head (`0d18f18928`) merges without
conflict. The new check runs only when no Runner state exists, so
#21120's new call site (it runs only when a Runner state exists) gets
the same results as before.

## Risks

- Low. The new path runs only in host preparation, under a
`bootstrap_incomplete` claim for the same run id, with no failover
backup. That claim is granted only after the previous controller's lease
has ended. With no Runner state or launch marker anywhere in the folder,
no Runner and no provider session existed there. The old folder is moved
aside, not deleted.
- A controller that is still launching at the same moment would have to
hold a lease that the claim already took over. The detach fence stops
such a launch, and #21079 adds a launch-time fence for a leftover Runner
on the same state folder.
- No added wait before a normal turn starts: the check runs only on this
failure path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser for read-only run evidence, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:06:55 -06:00
DottaandClaude Fable 5.2 5b698b68cf perf(run-events): append a native event in one transaction of two statements (#21131)
One appended native run event cost eight statements on the database
connection (BEGIN, SAVEPOINT, run lock, replay probe, sequence update,
insert, count/max read, COMMIT). The query builder built five of them
again for every event, and each event created two transaction objects.

A native source event now takes the run lock and then runs one statement
that finds a replayed event or reserves the sequence and inserts, and
reads the retained count and maximum for the acknowledgement. The lock
stays a statement of its own so the append reads a snapshot taken after
the lock was granted. The control-plane port passes its controller check
to the append instead of opening an outer transaction, so there is no
SAVEPOINT. The hot statements are fixed SQL text.

Events that wake a queue still register the wake before the write and
only when the event is new. Order, sequence numbers and durability do
not change: every event still commits before its append returns.

Local benchmark (server/scripts/benchmark-run-event-append.ts), one
native event: 8.3 -> 4.6 statements, 589 -> 222 KB allocated,
1.09 -> 0.53 ms CPU, 28 -> 11.5 garbage collections per 1,000 events.
One ordinary event: 105 -> 27 KB allocated.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 09:58:31 -06:00
DottaandClaude Opus 5.5 9e910ff9ae fix(chat): keep raw IDs and internal codes out of chat replies (#21081)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and the agent replies in plain
text
> - In a full chat-eval cohort, 47 of 158 passing replies showed raw
UUIDs, error codes, field keys, tool names or server paths
> - Users must not see internal jargon or raw IDs, so these replies are
a product defect, even when the facts are correct
> - Agents copy these terms from API error messages and saved fields;
the chat directive did not tell them to translate
> - This pull request fixes the sources: the chat directive, and API
denial messages that put IDs and policy codes in the human message
> - The benefit is plain replies for every model, while machine codes
stay available in separate fields

## Linked Issues or Issue Description

**What happened?**

Chat replies quote internal names and raw IDs. Examples: run, handoff,
revision, interaction and trigger UUIDs in prose;
`native_session_retry_exhausted`, `board_access_required`,
`review_policy_denied`, `local-board`, `enableCases`; a raw user ID
inside the inbox denial message.

**Expected behavior**

Replies name records the way the person sees them (task identifier,
title or name, with a link) and explain codes in plain words. IDs stay
only in link targets, in commands the person must run, or when the
person asks for an ID.

**Steps to reproduce**

Run the chat-features OpenCode cohort (for example `cases.close-case`,
`routines.webhook-trigger`, `inbox-search.archive-without-grant`) and
read the agent replies.

**Paperclip version or commit**

`master` at f30005440b.

**Agent adapter(s) involved**

Not adapter-specific (core bug). Seen with the Paperclip runner and
OpenCode.

**Deployment mode**

Local dev server (embedded PGlite) and the CI chat-eval runner.

**Installation method**

Source checkout (`pnpm dev`).

## What Changed

- `AGENT_CHAT_DIRECTIVE`: add one rule. Name records by their
identifier, title or name and link them. Do not write raw IDs, internal
codes, field keys, tool names, process names or server paths, even in
inline code. The rule has exceptions for link targets, commands the
person runs, and IDs the person asks for. The rejection rule now asks
for the platform's reason "in plain words".
- Authorization denials (inbox management, membership, responsible user)
no longer put user or agent IDs in the explanation. The `reason` and
`code` values do not change.
- Review-policy denials explain the rule in plain words. The
`review_policy_denied` code and the `policy` and `allowedActor` fields
do not change.

## Verification

- Red/green: `server/src/__tests__/agent-conversations.test.ts`
(directive rule, and it reaches the chat task prompt),
`server/src/__tests__/authorization-service.test.ts` (inbox denials
carry no IDs), `server/src/__tests__/issue-review-policy.test.ts` and
`server/src/services/issue-thread-interaction-resolution.test.ts` (plain
review messages). Each new assertion failed before the fix.
- `pnpm verify` and the server typecheck pass locally.
- A paid OpenCode run on 8 affected chat cases is in progress; results
will be added here.

## Risks

- Low risk. Only human-readable messages and prompt text change. No
client matches the old message text; codes and machine fields stay the
same.
- The directive grows by one paragraph. A model can still copy an ID;
the rule reduces this but cannot prevent it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:37:07 -06:00
DottaandClaude Opus 5.5 81d2e4d8d9 fix(native): end a run whose finalizer a newer run superseded, so it stops holding its task (#21106)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result, then a finalizer exports the
workspace and applies the result to the task.
> - When a newer run already committed the task's decision, the
finalizer retires the older result as `native_finalization_superseded`.
> - That retirement set the finalizer terminal, but it did not end the
run. The run stayed `running` and held the task and every run queued
behind it, with no way out except a person.
> - A stuck run must never hold a task and its queue forever.
> - This pull request ends the run when its finalizer is retired as
superseded, and it ends runs that an older release left in that state.
> - The benefit is that the task continues with its queued runs by
itself.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. The workspace export then failed
(`remote_workspace_archive_fs_eacces`). The task's last decision came
from another run that was created 42 seconds later but ran earlier. The
finalizer judged this run superseded: it set the finalizer to
`terminal_failure` with `native_finalization_superseded`. The run stayed
`running` with "Recovery needed". Over one hour later it still held the
task, and five queued runs waited behind it. A server upgrade at 12:02Z
did not change it.

**Expected behavior**
A run whose finalizer is retired ends. The task keeps the newer run's
decision, the retired result is never applied, and the queued runs
proceed.

**Steps to reproduce**
1. Accept a native result for a run.
2. Commit a status decision on the same task from a run that was created
later.
3. Let the older run's finalization fail once, so the finalizer records
the failure.
4. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is still `running`.

**Root cause**
`recordRetryableFailure`
(`server/src/services/native-runtime/native-run-finalizer.ts`) projects
a terminal run status only when `exhausted && !supersededByNewerRun`. A
superseded retirement therefore never ended the run. No other path ends
a run whose finalizer is already `terminal_failure`: the reconciler
skips `terminal_failure` rows.

**Paperclip version or commit**
Staging Core `3eca7a6112`. The behavior is the same on master
`482063aeca`.

## What Changed

- `native-run-finalizer.ts`: a superseded retirement now ends the run
with the same status mapping as an exhausted finalization (workspace
failure: `failed`; otherwise the provider's terminal state). It does not
change the task, does not apply the result, and does not keep the
workspace export lease. The run's error says in plain words that a newer
run already decided the task.
- `native-run-finalizer.ts`: add `endSupersededNativeFinalizationRuns`.
It ends a native run that is still `running` while its finalizer is
`terminal_failure` / `native_finalization_superseded`. This covers rows
that an older release wrote. It locks in the finalizer's order
(coordinator, task, run), and re-checks the row under the lock.
- `native-run-projection-work.ts`: the registered native run projection
worker (it loads at startup, pages through all candidates, and wakes on
finalization changes) also selects such runs and ends them with
`endSupersededNativeFinalizationRuns`.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/__tests__/native-finalization-recovery.test.ts`, red on
master and green on this branch:
- "retires an older failed finalizer when a newer run already committed
the issue": the retired run now ends (`succeeded`, `finishedAt` set).
Master: the run stays `running`.
- "ends a run an earlier release left running after retiring its
finalizer as superseded": the shape of the staging row (workspace export
failed, superseded finalizer, run `running`). The repair ends it as
`failed` with the plain message. The task, its decision and the
finalizer stay unchanged, and no status decision is written for the run.
A second pass changes nothing.
- `server/src/__tests__/native-run-projection-work.test.ts`, "ends, at
startup, a run an earlier release left running after retiring its
finalizer as superseded": the registered worker ends the staging-shaped
row at startup and settles its claimed wake. Red without the worker
change.
- Related suites pass: `native-finalization-recovery`,
`native-run-projection-work`, `native-finalization-reconciler-timing`,
`native-reconciliation-timing`, `native-workspace-finalizer`,
`native-workspace-finalizer-failure`, `native-run-finalizer-telemetry`.
- `tsc --noEmit` for `server`.

## Risks

- Low risk. The change only ends runs whose finalizer is already
terminal and superseded. It never applies a result and never changes the
task.
- A superseded run whose workspace export failed now releases its
sandbox lease through the normal settlement instead of waiting forever.
Its result was not applied, because a newer run decided the task.
- Not changed here: the superseded check compares run creation times. A
run created earlier but executed later is still judged superseded. This
PR makes that verdict end the run; it does not change the verdict.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:07:18 -06:00
DottaandClaude Opus 5.5 0a2f618cae fix(native): leave a run another live server is preparing alone, and re-adopt a run a shutdown handed off (#21107)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade replaces the server. The old and the new server serve
together for a few minutes, then the old server gets SIGTERM and hands
its native runs to the next controller.
> - #21046 taught the legacy reaper to leave a run with its owner while
the owner's server process is proven alive. Native runs have a similar
gap.
> - Runtime selection writes a native run's coordinator with no
controller and no lease, and stops the run lease. The executor claims
the coordinator about 20 seconds later. In that window the other
server's sweep sees no owner and marks the run "ownership could not be
verified".
> - When the old server then hands the run off before any Runner
checkpoint exists, no sweep looks at it again. Only the next server boot
does, 15 to 40 minutes later. The task and its queued runs wait.
> - This pull request leaves such a run with its live preparing server,
and lets the serving server re-adopt a handed-off run through the
ordinary restart claim.
> - The benefit is that a run started during an upgrade continues within
one sweep, with no false "blocked" mark and no wait for another upgrade.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, an upgrade started a new server at
12:19:57Z. The old server started a native run and selected its runtime
at 12:20:32Z. At 12:20:35Z the run logged "Native execution ownership
could not be verified; automatic recovery is blocked"
(`observed_owner_unverified`). The old server's executor claimed the run
at 12:20:51Z. The old server got SIGTERM and suspended the run at
12:20:54Z (`awaiting_runner_reattach`). After that the run showed
"Confirming execution" for 29 minutes, and two queued runs waited behind
it. At 12:49:45Z the next upgrade's startup recovery restarted its
bootstrap, and the run succeeded at 12:58Z.

**Expected behavior**
- A server does not mark a run unverified, or recover it, while another
live server is preparing it.
- A run that a graceful shutdown hands to the next controller is
re-adopted by the serving server at once, not at the next boot.

**Steps to reproduce**
1. Seed a native run whose coordinator is `observed` with no controller
and no lease, and whose `controllerProcess` names another server with a
live controller instance lease.
2. Run the recovery sweep on this server. See that the run gets
`native_execution_ownership_unverified`.
3. Claim the coordinator for the other server, then write what its
graceful shutdown writes (`restart_suspended`,
`awaiting_runner_reattach`, lapsed lease) and stop its instance lease.
Give the run a predecessor's Runner PID.
4. Run the sweep again. See that nothing claims the run.

**Root cause**
- `recoverActiveExecutions`
(`server/src/services/heartbeat/recovery.ts`) treats an `observed`
coordinator without a live controller lease as an owner it cannot
verify. Between runtime selection and the claim, the only owner on
record is the run's `controllerProcess`, and the sweep did not check it.
- `listExpiredNativeRunnerAuthorities` (`native-runner-checkpoint.ts`)
selects runs for the restart claim only when they have a controller
checkpoint, or no process identity at all. A run handed off just after
its claim has no checkpoint, and a follow-up carries its predecessor's
Runner PID. The sweep then skips `awaiting_runner_reattach` runs, so
only a server boot looked at them again.

**Paperclip version or commit**
Staging Core `814fe5d20f` (old server) and `3eca7a6112` (new server).
The behavior is the same on master `849f109f62`.

## What Changed

- `recovery.ts`: before it lists expired native runs, the sweep checks
runs in the runtime-selection window (coordinator `observed`, no
controller, no lease, not tracked here). When the run's
`controllerProcess` names another server process that the shared owner
prover proves alive, the sweep leaves the run with that server: it does
not recover it and does not mark it unverified. It re-checks within 30
seconds. After 5 minutes without a claim
(`LEGACY_LIVE_OWNER_SETTLE_GRACE_MS`), or once that server is gone, the
existing rules apply.
- `native-owner-stamps.ts`: add `preparingControllerOwnerStamp` for that
proof.
- `native-runner-checkpoint.ts`: `listExpiredNativeRunnerAuthorities`
also selects a run that a graceful shutdown handed off and whose lease
lapsed: recovery state `awaiting_runner_reattach` or
`awaiting_evidence`, with a `restart_suspended` history entry that no
restart claim followed. It stays selected through ownership waits (the
departing server can still be alive at the first look). `recovery.ts`
schedules a 30-second re-check for an ownership wait that has no retry
of its own. The ordinary restart claim then proves the takeover and
picks the disposition, exactly as the startup pass of a new server does.
A row that an older release already marked unverified is re-adopted too:
the claim clears the mark.
- `doc/execution-semantics.md`: document both rules.
- No schema change. Nothing changes on the path that starts a run or a
turn: the new proof runs only in the recovery sweep, and only for runs
in the window.

## Verification

- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`,
block "a run the old server started during an upgrade overlap". Red on
master, green on this branch:
- "leaves the run with its live preparing server, then re-adopts it
after that server hands it off": two concurrent sweeps leave the run
`running` with no error and do not touch the coordinator. After the old
server claims, hands the run off and stops, one sweep of the new server
re-adopts it and reaches provider dispatch. Master: the run is marked
`native_execution_ownership_unverified`, and after the handoff nothing
claims it.
- "keeps a handed-off run eligible while the departing server is still
alive, and re-adopts it once it exits": the first sweep waits
(`awaiting_evidence`), the sweep after the departing process exits
re-adopts the run. Red with a filter that matches only the last history
entry.
- "re-adopts a handed-off run that an older release already marked
ownership-unverified": the stuck staging shape. Master: nothing claims
it.
- Related suites pass: `heartbeat-native-restart-suspension` (all 109),
`services/heartbeat/recovery.test.ts` (all 103).
- `tsc --noEmit` for `server`.

## Risks

- A run that another live server prepares, but never claims, now stays
`running` for up to 5 minutes before the existing rules apply. That is
the same bound #21046 uses.
- The guard trusts the shared owner prover. It reads "alive" only from a
live controller instance lease or a live PID in the same PID space.
- More handed-off runs reach the restart claim earlier. The claim keeps
all its fencing and evidence checks; this change only makes the serving
server ask the same question a new boot asks.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:06:57 -06:00
DottaandClaude Opus 5.5 67403baaa4 fix(native): an agent termination Stop with no turn result stays a cancellation (#21093)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person terminates an agent, Paperclip stops the agent's
running native turn
> - Two paths can record that Stop: the terminate route ("Cancelled
because the agent was terminated ...") and the lifecycle worker ("Agent
lifecycle stop requested"). Both are system-initiated and expected
> - When the stopped turn ends without a semantic result, the native
executor keeps the run cancelled only for a person's Stop, a
reassignment, or a governance pause. A termination Stop fell through, so
the run ended `failed` with `native_session_interrupted`
> - Whether the run ended cancelled or failed depended on which path won
the race
> - This pull request treats an acknowledged termination Stop like a
person's Stop in the executor and in the lease-release terminalization
> - The benefit is that a terminated agent's interrupted run ends
cancelled, whichever path records the Stop first

## Linked Issues or Issue Description

Refs #20186 (the same treatment for a governance pause Stop)

**What happened?**
A person chose Terminate for an agent while its native turn ran. The
lifecycle Stop recorded an acknowledged native cancellation (initiator
system). The turn ended without a semantic result. The executor then
wrote the run as `failed` with `native_finalization_missing: session
returned no semantic result`.

**Expected behavior**
The run ends `cancelled`. A termination is an intentional Stop, not a
failure to retry.

**Steps to reproduce**
Terminate an agent while its native turn runs, so that the Stop
interrupts the turn before it records a result. Before this change, the
run became `failed` when the lifecycle worker recorded the Stop first.

## What Changed

- `governance-pause-stop.ts`: adds `AGENT_TERMINATED_STOP_REASON`,
`COMPANY_ARCHIVED_STOP_REASON` and `isAgentLifecycleStopRecord`. The
predicate accepts a system-initiated, expected control-plane
cancellation whose reason is a lifecycle, termination or company-archive
Stop.
- `acknowledged-native-stop.ts`: adds
`hasAcknowledgedNativeLifecycleStopIntent`, which requires that record
and an acknowledged, dispatched native Stop receipt.
- `native-session-executor.ts`: the no-semantic-result branch accepts
that intent, so it settles the coordinator as `native_retry_cancelled`
and raises `NativeCancellationPendingRecoveryError`, as for a person's
Stop.
- `heartbeat/run-lifecycle.ts`: lease-release terminalization keeps such
a run `cancelled`.
- The terminate route and the lifecycle driver use the shared reason
constants. The reason text does not change.

## Verification

- `native-session-executor.test.ts`: a new case for each reason
(lifecycle Stop, terminate route, company archive). Each was red before
the change and is green now. The earlier control case now uses a system
Stop that is neither a pause nor a termination, and it still takes the
retry path. Full file: 1414 passed.
- `heartbeat-run-lease-release-terminalization.test.ts`: a new case for
a lifecycle Stop when teardown wins the finalizer race. It was red
before the change (`interrupted`) and is green now.
- Product evidence: a paid chat-eval run of `agents.retire-agent` on a
branch that combines this change with three related fixes. The result is
in a PR comment.

## Risks

- A termination or company-archive Stop that interrupts a turn no longer
schedules an exact resume of that turn. The agent is leaving service, so
no resume can run.
- Other system Stops keep their current behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:19:21 -06:00
DottaandClaude Opus 5.5 3eca7a6112 fix(runner): log which check refuses a saved provider-pack admission on reattach (#21085)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode turns run in a remote Runner. When a server restarts, the
new server reconnects to the Runner and reuses the provider pack
admission that the run saved earlier.
> - On staging, a run reconnected cleanly after an upgrade but then
failed with `native_provider_artifacts_admission_unverified`. Its saved
admission state stayed "pending".
> - The reuse step refuses a saved admission for several reasons. It
returns only false and logs nothing, so we cannot tell which check
refused it.
> - This pull request adds diagnostics only. It names the refused check
in the run log and the server log. It also logs when the admission stays
unverified for a session that needs it.
> - The benefit is that the next upgrade audit shows the real cause, so
the next fix can target it.

## Linked Issues or Issue Description

**What happened?**
After an automatic staging upgrade, a run reconnected to its live Runner
without a new process. The next checkpoint failed with
`native_provider_artifacts_admission_unverified`, which used one of the
run's 3 attempts. The run log had no line that explained why the saved
admission was not reused.

**Expected behavior**
When a saved admission is not reused, the run log and the server log
name the check that refused it.

**Steps to reproduce**
1. Start an OpenCode run on a remote sandbox.
2. Restart the server, so the run reconnects to its Runner.
3. If the saved admission is refused, the run fails later with
`native_provider_artifacts_admission_unverified` and no reason in the
logs.

## What Changed

- `native-provider-pack-admission.ts`:
`rememberRecoveredNativeProviderPack` accepts an optional `onRefused`
callback. It is called with one reason: `source_state_pending`,
`source_state_unknown`, `source_state_missing`, `receipt_missing`,
`runner_instance_differs`, `environment_lease_differs`,
`receipt_invalid`, or a session identity reason. The result of every
check stays the same. An error in the callback is ignored.
- `native-session-executor.ts`: the reattach path logs the reason to the
run log and the server log. When `recordNativeProviderPackAdmission`
returns false for a session that requires admission, it logs that too.
- `native-provider-pack-admission.test.ts`: each refusal names its
reason, and a reused admission reports nothing.

## Verification

- `npx vitest run
src/services/native-runtime/native-provider-pack-admission.test.ts`
passes 46/46. The new reason tests fail on the old code because
`onRefused` is never called.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts` passes
1412/1412.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. This change adds diagnostics only, and admission decisions
do not change. The new log lines contain only a fixed reason code and
run ids. They do not contain secrets or file paths.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:38:23 -06:00
DottaandClaude Opus 5.5 2e593bf3cd fix(native): drop a repeated identity-record check and prove an unsettled predecessor stays an ordinary refusal (#21086)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21001 sends a re-wake to a fresh sandbox when live adoption of a
crashed predecessor's warm Runner is refused, and lets an inherited
Runner identity pass the "workspace unchanged" proof
> - The reviewer found two gaps on #21001 that were not answered before
it merged
> - The unchanged proof checked for a `native.process_identity_recorded`
event twice, and no test proved that an unsettled predecessor stays an
ordinary refusal
> - This pull request removes the repeated check and adds that proof
> - The benefit is simpler proof code and a test that stops the
fresh-sandbox fallback from widening to unsettled predecessors

## Linked Issues or Issue Description

Refs #21001, #20863

**What happened?**
Two review warnings on #21001:
1. `carriesOnlyInheritedRunnerIdentity` in
`remote-workspace-unchanged-proof.ts` looked for the run's own
`native.process_identity_recorded` event.
`executionStartedEventPredicate()`, later in the same proof, already
refuses a run with that event (`execution_recorded`).
2. The existing denial cases in
`native-controller-recovery.integration.test.ts` use `toThrow(denied)`.
That also matches the wrapped
`NativePredecessorRetirementUnprovenError`, so a regression that wraps
an unsettled predecessor's refusal would pass.

**Expected behavior**
The proof has one check per fact, and the tests fail when an unsettled
predecessor goes to the fresh-sandbox fallback.

**Steps to reproduce**
1. Read `proveRemoteWorkspaceUnchanged` and
`executionStartedEventPredicate`.
2. Change `remote-warm-session-recovery.ts` to wrap every refusal; the
old denial cases still pass.

## What Changed

- `server/src/services/remote-workspace-unchanged-proof.ts`: delete
`carriesOnlyInheritedRunnerIdentity` and its `PROCESS_IDENTITY_RECORDED`
import; call `carriesInheritedRunnerIdentity(tx, run)` directly. A run
with its own identity record is still refused, now by
`execution_recorded`.
-
`server/src/services/native-runtime/native-controller-recovery.integration.test.ts`:
- The denial cases for an unsettled predecessor (other company, other
execution, finalization lease held, run still `running`) now assert the
error is not a `NativePredecessorRetirementUnprovenError`.
- The pending-command case after the predecessor is `succeeded` and
`committed` now asserts it is one (the #21001 behavior).
- The crash-restart case adds a predecessor run that is `failed` or
`running`: an unsettled journal is refused with
`warm_recovery_refused=owner_unsettled`, not wrapped.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts
src/services/remote-workspace-unchanged-release.test.ts`: 50 passed. The
three inherited-identity cases from #21001 return the same results
(released / refused / refused).
- `cd server && npx tsc --noEmit -p tsconfig.json`: passes.

## Risks

- Low risk. No behavior change: the removed check is covered by
`executionStartedEventPredicate()`. The rest is tests.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:35:15 -06:00
DottaandClaude Opus 5.5 5820d43197 fix(runner): relaunch a settled OpenCode Runner with its admitted artifact paths (#21076)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agent turns run in a remote Runner in a Daytona sandbox.
The Runner saves its state, so a retry of the same run can start a new
Runner from that saved state.
> - The first launch of a run starts OpenCode from the admitted provider
files under one folder in the sandbox. The new Runner replays that first
launch and checks that its own launch profile names the same files.
> - When a retry reconnects to a Runner that already stopped, Paperclip
starts a new Runner. On that path, Paperclip did not restore the
admitted provider files and gave the new Runner the provider pack folder
instead.
> - The new Runner then refused the replayed launch ("does not match the
runner-owned qualified profile"). That used the last retry, and the task
stayed blocked.
> - This pull request restores the admitted provider files and gives the
new Runner the same folder as the first launch.
> - The benefit is that a retry after a server upgrade continues the
same run instead of blocking the task.

## Linked Issues or Issue Description

**What happened?**
On staging, after an automatic upgrade, a run used all 3 attempts. The
last attempt started a new Runner from its saved state. It failed in the
drain step with "OpenCode run.prepare launch does not match the
runner-owned qualified profile". The task was blocked and needed a
person.

**Expected behavior**
The new Runner gets the same OpenCode launch files as the first launch
of the run, and the run continues.

**Steps to reproduce**
1. Start an OpenCode run on a remote sandbox with durable provider
artifacts.
2. Let the server restart, so the retry reconnects with
`reattach_remote_runner`.
3. The Runner is found settled, so a new Runner starts from the saved
state. Its launch profile points at the provider pack folder, not at the
admitted artifact folder.

## What Changed

- `native-session-executor.ts`: when a reconnect relaunches a settled
Runner and the run has admitted provider artifacts, restore those
artifacts and prepare the same artifact folder as the first launch.
Other reconnects (live adoption) do not change. Normal turn starts do
not change.
- `native-session-executor.test.ts`: the existing settled-relaunch case
now also covers a run with admitted provider artifacts. It expects the
restore and the admitted folder in the launch profile.

## Verification

- Red: without the fix, the new test case fails
(`restoreNativeProviderArtifacts` is not called, and the launch uses the
provider pack folder).
- Green: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-opencode-artifacts.test.ts` passes
1417/1417.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The change applies only when a reconnect must relaunch a
settled OpenCode Runner and the run has admitted provider artifacts.
- That relaunch now downloads the admitted artifacts from storage, as a
cold continuation already does. This adds time only on that recovery
path, not on a normal turn start.
- The artifact folder check (same account and same digest) still
applies, so a different sandbox account still fails closed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:24:17 -06:00
DottaandClaude Opus 5.5 559551e977 fix(native): wait for a busy agent directory while retiring a predecessor (#20991)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a reused warm sandbox first retires the previous
run's warm session, and that retirement collects the old session's agent
files
> - On a server without a persistent volume, agent-file collection takes
a per-agent Postgres lock, so runs of one agent collect one at a time
> - #20863 sends a run whose predecessor cannot be retired to a fresh
sandbox. It classified every retirement error as unprovable, including a
busy lock held by another run of the same agent
> - So ordinary lock contention failed the run, dropped a healthy
sandbox and rested the task
> - This pull request makes retirement wait for a busy directory lock in
the same run, and keeps the fresh-sandbox fallback for retirements that
stay unprovable after the lock is obtained
> - The benefit is that busy agents no longer fail follow-up runs on
their warm sandboxes

## Linked Issues or Issue Description

**What happened?**
A no-fault restart-chaos wave with 8 concurrent runs of one Claude agent
on Daytona, with a volume-free server, failed one monitor-due follow-up
after 6 seconds. The error was `adapter_failed`, cause
`native_predecessor_retirement_unproven`: "The previous run's session on
this sandbox could not be retired (Agent directory operation is already
active)". The run also logged `INSTRUCTION_COLLECTION_UNAVAILABLE`. The
stack shows the error came from `claimWarmNativeInstructionCopy` during
instruction preparation. The previous session had closed; only its
agent-file collection met the per-agent lock that another run held.

**Expected behavior**
A busy agent-directory lock is transient. The run waits for it and then
retires the predecessor on the same sandbox. Only a retirement that
stays unprovable after the lock is obtained takes the fresh-sandbox
fallback.

**Steps to reproduce**
1. Run several warm native tasks of one agent at once on Daytona with a
volume-free server.
2. Re-wake one task while another run of the agent collects its files.
3. The follow-up fails with `native_predecessor_retirement_unproven` and
its sandbox is dropped.

Refs #20863, #20829, #20957.

## What Changed

- `retireWarmPredecessor` (native executor): a busy agent-directory lock
(`AgentDirectoryOperationBusyError`) is waited out in the same run,
polling every second for up to five minutes
(`warmRetirementDirectoryWait`). The usual holder is another run's short
collection, so retirement then finishes on the same sandbox with the
copy this run adopted. The run is not deferred: deferring after adoption
would release the adopted copy and could skip saving the predecessor's
edits.
- Past that budget the run fails with its own error,
`native_predecessor_retirement_directory_busy`, not as an unprovable
retirement. Before failing it files the adopted copy's retirement
receipt, so the copy is kept as `pending_collection` and run cleanup
does not mark it `unavailable`. The owner is not marked unretired and
stays registered for its sandbox, so a later run retires it there.
- An idle-timeout close that meets the lock retries after the
agent-directory retry delay instead of leaving the session registered
with no retry.
- Closing idle sessions for environment delete (#20957) reports such an
owner as `busy`, not `failed`.

## Verification

- RED on master, then GREEN:
- `native-session-executor.test.ts -t "directory lock"`: two runs of one
agent contend on the lock while a follow-up retires the predecessor. On
master the claim rejects with `NativePredecessorRetirementUnprovenError`
(`Agent directory operation is already active`). Now a short-lived lock
is waited out and the predecessor is retired and collected on the same
sandbox; a lock that outlasts the budget fails with
`native_predecessor_retirement_directory_busy`, environment delete
reports `busy: 1`, a fresh-sandbox attempt does not forget the owner,
and once the lock is free a later run retires it.
- `native-session-executor.test.ts -t "idle close whose file
collection"`: on master the idle close never retries; now it retries and
closes.
- Existing suites pass: native executor, agent-directory busy heartbeat,
native pre-dispatch retry, native controller recovery and plugin runner
eviction. Server `tsc --noEmit` passes.

## Risks

- Under contention a follow-up can wait up to five minutes before its
first turn. The normal path adds no wait.
- A lock held for more than five minutes fails the run visibly instead
of dropping the sandbox.
- An idle owner whose collection stays busy keeps retrying at the 5 to
15 second agent-directory delay until the lock frees, a controller
restart, or the next admission.
- No persisted state changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:08:15 -06:00
DottaandClaude Opus 5.5 533f135780 fix(native): send a re-wake whose crashed predecessor's warm Runner cannot be adopted to a fresh sandbox instead of failing it (#21001)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a warm remote sandbox leaves its Runner `ready` in
the sandbox. The next run of the task adopts that Runner live, after it
proves that the previous owner finished
> - When the server is killed after a run parked on a question, the warm
owner lived only in the killed server's memory. The Runner in the
sandbox was never suspended
> - A re-wake on the replacement server resumed that sandbox and tried
live adoption. A proof failed, and the new run failed with
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate` and needed a Retry, although
it did no provider work
> - #20863 already sends such a run to a fresh sandbox when the
predecessor's Runner cannot be retired, but the live-adoption path of a
resumed sandbox (`remote-warm-session-recovery.ts`) was not connected to
it, and the failed run's inherited Runner identity blocked the
"workspace unchanged" proof that the fallback needs
> - This pull request connects the two: a refusal of live adoption after
the predecessor is durably terminal with an accepted result goes to the
#20863 fresh-sandbox path, and an inherited Runner identity no longer
counts as a process of the follow-up
> - The benefit is that a re-wake seconds after a crash restart
continues without a Retry

## Linked Issues or Issue Description

Refs #20863, #20844, #20928, #20979, #20953, #20972

**What happened?**
Restart-chaos wave `w-20261112T012841-s7000` (master `4b24352712`,
volume-free mode, Daytona warm, Codex, 10 tasks, K1 and K15 with a fresh
home and overlap), task t08 (W2Q):
- Run `58f57bf1` asked its question and succeeded at 01:29:56. Its warm
owner stayed in the server's memory.
- K1 killed the server at 01:30:06. The replacement was ready at
01:30:15.
- A board comment woke the task at 01:30:17. Run `8f812df6` started at
01:30:19 and resumed the same sandbox (`run.startup.reuse`
`verified_reuse`).
- The first live-adoption inspection passed. The second check
(`revalidate`, in backend setup, before the Runner attach) refused at
01:30:21 (stack: `remote-warm-session-recovery.ts:217` from
`revalidate`). The run failed `native_session_interrupted` with
`INSTRUCTION_COLLECTION_UNAVAILABLE`.
- The failed run kept its full save because the unchanged-files proof
refused with `process_recorded`: the run carried its predecessor's
Runner process identity. Retry (`6968dc0f`) worked.

**Expected behavior**
The predecessor finished with an accepted result before the kill, and
the new run did no provider work. The task must continue without a
Retry.

**Steps to reproduce**
1. Finish a warm remote run that parks on a question.
2. Kill the server (SIGKILL) and start a replacement.
3. Wake the task within seconds. The new run resumes the warm sandbox.
4. Any live-adoption proof fails, and the run fails terminally with
`prior_owner_terminal_state_indeterminate`.

**Which proof failed**
The old refusal had no reason. The wave evidence proves that it was
`revalidate`, after a passed inspection, and not the SIGTERM of the next
K15 fault (that server logged "shutdown signal received" only at
01:31:09). `revalidate` checks the predecessor's coordinator row, the
current run's coordinator row and lease, and the task. The evidence
cannot tell which one changed in the first seconds after the restart.
This pull request names the failed proof
(`warm_recovery_refused=<reason>`), so the next occurrence identifies
it. The fix covers the class: every refusal that concerns the
predecessor or the sandbox.

## What Changed

- `server/src/services/native-runtime/remote-warm-session-recovery.ts`:
- Each refusal names its proof from a closed list:
`identity_incomplete`, `target_not_resumed_sandbox`, `owner_unsettled`,
`execution_mismatch`, `checkpoint_unsettled`, `journal_unsettled`,
`controller_not_stopped`, `prior_lease_mismatch`, `owner_changed`,
`current_lease_changed`, `current_run_not_admitted`,
`current_owner_changed`.
- After the predecessor's run is `succeeded` and its coordinator is
`committed` with a result and no lease, a refusal about the predecessor,
its journal or the sandbox throws
`NativePredecessorRetirementUnprovenError` (predecessor run, sandbox).
The existing #20863 path then marks the sandbox `replaceBeforeReuse` and
re-queues the run in the uncharged `interrupted_preparation` lane. The
fresh sandbox is used only when the workspace save is clean and the
harness backup is verified (unchanged #20863 safety rules); otherwise
the sandbox is resumed as before.
- A refusal because the new run is no longer admitted (stopped,
reassigned, paused, company inactive) or no longer owned by this
controller stays an ordinary failure, so a fallback cannot become a path
around Stop or reassignment.
- `server/src/services/remote-workspace-unchanged-proof.ts`: a run whose
recorded Runner process identity is exactly a settled predecessor's
identity of the same session (`carriesInheritedRunnerIdentity`, from
#20979), with no `native.process_identity_recorded` event of its own, is
no longer refused with `process_recorded`. A provider session of its own
(`sessionIdAfter`) and every other check still refuse.
- `doc/execution-semantics.md`: document both rules.

## Verification

- Red on `origin/master` (`8818402e7a`):
- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts
-t "crash restart"`: `AssertionError: expected Error:
runner_state_identity_mismatch: pr… to be an instance of
NativePredecessorRetirementUnprovenError`.
- `npx vitest run
src/services/remote-workspace-unchanged-release.test.ts -t "follow-up
that carries"`: `expected false to be true` for the inherited-identity
case; the own-identity and own-record cases passed (still refused).
- Green with the fix: `native-controller-recovery.integration`,
`remote-workspace-unchanged-release`, `native-recovered-owner-followup`
and `heartbeat-native-predispatch-retry` pass (149 tests). Also run:
`native-session-executor.test`, `explicit-native-continuation.test`,
`task-workspace-binding.test`,
`native-runner-restart-recovery.integration.test` (see the PR comment
for the result). `tsc --noEmit` for the server passes.
- The new integration case is the staging shape: a predecessor that
parked on a question (unanswered `semantic_tool.input` before its
terminal), a replacement controller, a resumed sandbox. It checks the
journal refusal, a predecessor row taken by another worker between
inspection and revalidation, a changed current lease, and that a
cancelled current run is not sent to the fallback.
- Chaos: the before/after rerun of seed 7000 is in a PR comment.

## Risks

- The fallback starts the next attempt on a fresh sandbox instead of
reusing the warm Runner. That costs one cold start, only after live
adoption is refused.
- The unchanged-files proof accepts an inherited Runner identity. The
identity must match a settled predecessor of the same session exactly
(PID, group and start time), and the run must have no own identity
record, no execution event, no runner checkpoint, no workspace operation
and no result. These are the same facts #20979 uses for restart
recovery.
- Error text of this refusal gains `; warm_recovery_refused=<reason>`.
Callers match the `runner_state_identity_mismatch: ` prefix only.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:08:03 -06:00
DottaandClaude Opus 5.5 814fe5d20f fix(runner): issue the remote bootstrap ticket at dispatch and retry unauthenticated starts (#21073)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox start `runnerd` in the sandbox, and
Paperclip connects to it through provider ingress and authenticates it
with a one-use bootstrap ticket
> - The ticket lives for 60 seconds, but Paperclip issued it before the
remote launcher staged its files in the sandbox
> - Under load, staging took 60 to 81 seconds, so `runnerd` presented an
expired ticket and every connection was closed. The run failed with
`runner_ingress_auth_failed`, and the task stopped behind a manual Retry
> - This pull request issues a fresh ticket right before dispatch,
retries these failures automatically when the controller checkpoint
proves no runner ever authenticated, and adds spans that attribute the
slow startup window
> - The benefit is that a slow but healthy start authenticates, a failed
start recovers on its own, and the remaining startup delay becomes
measurable

## Linked Issues or Issue Description

**What happened?**

On a hosted instance, 9 runs of one Codex agent on Daytona failed in
about 70 minutes with `native_session_interrupted` and the message
`runner_ingress_auth_failed: Runner ingress did not complete PRP
authentication before the deadline.` In the same window only 6 runs
succeeded company-wide. The run log showed Paperclip reconnecting for 60
seconds while `runnerd` reported that its peer closed each WebSocket. In
294 remote runs with timing spans, the time between the bootstrap ticket
issue (end of `runner.transport.connect`) and the process dispatch
(start of `runner.process.launch`) decides the outcome without
exception:

- 60 s or more: 23 of 23 runs failed authentication.
- Under 60 s: 271 of 271 runs authenticated, in 0 to 1 s.

The tasks did not recover by themselves, because the failure was
classified as ambiguous state.

**Expected behavior**

A runner that starts slowly but correctly authenticates. If ingress
authentication still fails before any runner authenticated, Paperclip
retries the start automatically. The time between the start request and
the runner connection is visible in run spans.

**Steps to reproduce**

1. Run a Codex agent on a Daytona sandbox environment that uses provider
ingress.
2. Make remote staging (`runner.runtime.stage`, inside the remote
launcher) take 60 seconds or more. Many concurrent runs of the same
agent do this on staging.
3. See that `runner.prp.authenticate` fails after 60 seconds and the run
ends with `runner_ingress_auth_failed`.
4. The regression test `runnerd-bootstrap-ticket-refresh.test.ts`
reproduces this locally with a 75-second staging delay.

**Paperclip version or commit**

`4e519bd4` (deployed), reproduced on `origin/master` at `4228bcf89c`.

**Deployment mode**

Hosted (cloud), controller on Railway, sandboxes on Daytona.

**Agent adapter(s) involved**

`paperclip_runner` with the Codex app-server provider (`gpt-6.1-sol`),
remote sandbox target, provider-ingress transport.

## What Changed

- `RunnerProcessLaunchSpec` and `spawnRunner` accept
`refreshBootstrapTicket`. A restart does not inherit it.
- The runnerd Codex transport passes a refresh that issues a new
bootstrap ticket and flushes the controller checkpoint. It does not do
this for a warm-transition launch.
- The remote sandbox launcher calls the refresh after `ensureArtifact`
and immediately before the launch RPC, and puts the new ticket in the
launch environment.
- `inspectUnauthenticatedBootstrapCheckpoint` proves from a controller
checkpoint that no runner authenticated: no connections, no leases, no
used tickets, no events, and only pending commands.
- `nativeSessionFailureSourceCode` classifies the outbound ingress
startup failures as `runner_ingress_startup_failed`.
`nativeProviderRecoveryEvidence` treats it as pre-session
(`bootstrap_retry`) only when that proof holds for the exact run
identity and there is no provider session or provider event. Restart
recovery and failed-run retry accept the new code with the same rules as
similar startup codes.
- Remote startup now records `runner.control_plane.open`,
`runner.launch.prepare`, `runner.checkpoint.register` and
`stage.asset.codex_skills.verify`. The 6 to 11 minute gap between
`native.process_start_requested` and `runner.transport.connect` had no
spans before.

## Verification

- Red/green:
`packages/paperclip-runner/src/live/runnerd-bootstrap-ticket-refresh.test.ts`
starts a real `runnerd` with a 75-second staging delay. Without the fix
it fails with `runner never connected`. With the fix it opens the
session.
- Red/green: `native-session-executor.test.ts` "launches runnerd with a
bootstrap ticket issued after remote staging". Without the fix the
launch RPC gets the stale ticket.
- Red/green: `native-controller-recovery.integration.test.ts` "retries a
fresh remote start whose launched runner never authenticated through
provider ingress" (embedded Postgres). Without the fix the failure is
`native_session_interrupted` and `ambiguous_state`.
- `durable-prp-control-plane.test.ts`: the new checkpoint proof, its
negative cases, and the restart rule for the refresh.
- `native-run-trace.test.ts`: remote startup spans are recorded without
duplicate launcher spans.
- Focused suites pass: runner package 5 files and 235 tests; server 9
files and 1,607 tests, including `native-session-executor`,
`native-runner-execution-driver`, `remote-launch-assets`,
`native-restart-recovery`, `runner-prp-outbound` and the recovery
integration tests.
- `tsc --noEmit` passes for `packages/paperclip-runner` and `server`.

## Risks

- A launch now issues two bootstrap tickets. The first one is never sent
to `runnerd` and expires in 60 seconds. Bootstrap counters already allow
more than one issued credential.
- The new retry path needs a checkpoint proof that no runner
authenticated, so a run that did provider work cannot be retried this
way. The existing attempt budget still bounds the retries.
- This change does not remove the 6 to 11 minute controller-side delay
before `runner.transport.connect`. The new spans make it attributable
for a follow-up change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, code search) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:02:30 -06:00
DottaandClaude Opus 5.5 f30005440b fix(recovery): reconnect a live remote Runner with its admitted skills instead of re-downloading them (#21064)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner sandbox. A server upgrade
detaches each running turn, and the new server must reconnect to its
Runner (restart recovery).
> - A reconnect runs the full turn preparation before it contacts the
Runner. On a new server, skill preparation downloads every saved skill
version again, under one lock for the whole company.
> - After an upgrade, many reconnects wait behind each other's
downloads. The Runner stops answering after about 2 minutes, so a late
reconnect loses the turn.
> - This pull request lets a reconnect to the same run's live remote
Runner use the exact saved skill versions, without downloading them and
without the lock.
> - The benefit is that skill preparation no longer adds to the
reconnect delay, and it does not grow with the number of reconnects.
Normal turn starts do not change.

## Linked Issues or Issue Description

**What happened?**
After an automatic staging upgrade, runs that the new server reached
late failed. The measured time from detach to reconnect was 114–152 s
for the failed runs. Turn preparation took 30–130 s, and skill
preparation alone took 46 s on one run.

**Expected behavior**
A reconnect to a live Runner reaches it quickly. Its speed must not
depend on how many other runs prepare skills at the same time.

**Steps to reproduce**
1. Run a remote Runner turn for each of many tasks in one company.
2. Upgrade the server while those turns run, so the new server starts
with an empty skill cache.
3. See that each reconnect waits in
`heartbeat.prepare.configuration_skills` for the company skill cache
lock and downloads the skills again.

**Paperclip version or commit**
master at 4228bcf89c

**Additional context**
Root cause: `withRuntimeSkillEntries` holds the company-wide skill cache
lock (a Postgres advisory lock) while it materializes each saved skill
version. A new server has an empty skill cache, so each reconnect
downloads its skills again, one run at a time. The Runner already has
these files, and the reconnect path does not read them.

## What Changed

- `company-skills.ts`: new `admittedVersionsOnly` option. It still
checks that each saved version exists and that the agent can read the
skill. It does not materialize the files or take the company skill cache
lock.
- `heartbeat/live-runner-adoption.ts`: `adoptsLiveRemoteRunner` is true
only for a `reattach_remote_runner` claim for this run, with the claim's
own lease owner, on a native run.
- `heartbeat/execution-configuration.ts`: uses the option only when all
of these are true: the reconnect gate is true, the run has its saved
native input with a runtime context, and the run has its saved skill
versions. In that case it times the phase as
`configuration_skills_adopted` and adds one lifecycle run event
(`skillPreparation: "reused_admitted"`).
- `heartbeat.ts`: passes the gate to execution configuration.
- `native-runtime/runtime-context.ts` and `native-session-executor.ts`:
on the same reconnect to a remote Runner filesystem, the Runner
controller describes the Runner's skills by their admitted digests. It
rehydrates the pinned skill files (which takes the same lock) only if it
must launch a Runner process and stage them. All other starts rehydrate
as before.
- `doc/execution-semantics.md`: describes the new behavior.

Not in this PR (by design): a separate "dispatch-only" reconnect path,
and a new startup scheduling lane. The native executor needs about 40
values from preparation (runner environment, MCP token, managed AI
credentials, the reattached execution target, run output). Connector and
instruction preparation also feed the session fingerprint. That cut is
not mechanical, so this PR removes the largest measured single cost
only.

Other `heartbeat.prepare.*` costs that stay on the reconnect path (use
post-upgrade timings to choose the next cut):
- `configuration_secrets`, `configuration_connector_assignments`,
`configuration_connection_instructions`,
`configuration_connector_skills`/`_delivery`,
`configuration_agent_identity`, `configuration_ai_runtime`
- `environment_selection`, `workspace` (retained lease resume and
realization), `session`, `task_context`, `run_identity`
- `github_environment`, `github_launchers`, `run_output`,
`runtime_services`, `instructions`, `runtime_selection` (including
`runtime_selection.skill_assets` and `runtime_selection.workspace_sync`)
- The wait from detach to "run started" (queue and controller-preparing
waits) is outside `heartbeat.prepare.*`.

## Verification

- New test `reconnects to a live remote Runner with its admitted skills
without re-downloading them or waiting on the cache`
(`server/src/__tests__/heartbeat-execution-configuration.test.ts`):
- It admits a run, deletes its skill cache (like a new server), and
holds the company cache lock.
- The reconnect preparation finishes while the lock is held. It returns
the same key, runtime name and version. The session fingerprint is
identical. The cache is not created again.
- Normal preparation of the same saved selection still waits for the
lock and materializes the files ("normal start unchanged").
- Red on master source: `A live Runner reattach must not queue behind
skill downloads: expected null not to be null`. Green with this change.
- New test in `managed-skill-content.test.ts`: with the cache lock held
and an empty cache, `prepareRuntimeContextSource` for a reconnect
returns at once with the admitted context. Its staging step and a normal
start still wait for the lock and then rehydrate the same digests.
- New `server/src/services/heartbeat/live-runner-adoption.test.ts`: the
gate is false for a normal start, a scheduled resume, a dead-Runner
resume, an incomplete bootstrap, another run's claim, a different lease
owner, and a legacy run.
- `npx vitest run
src/__tests__/heartbeat-execution-configuration.test.ts
src/__tests__/skill-version-cache.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/heartbeat/live-runner-adoption.test.ts`: 289 passed, 14
skipped. After the controller change: restart-recovery, diskless
restart, runtime skill preservation, managed skill content, session
executor and execution configuration suites: 1686 passed, 14 skipped.
- `tsc --noEmit -p server/tsconfig.json`: pass.
- `pnpm verify` asks for broader coverage (exit 2). Required CI supplies
it.

After deploy, a reconnect run shows
`heartbeat.prepare.configuration_skills_adopted` and the
`reused_admitted` event, and no `skills.cache.acquire` or
`skills.version.materialize` phases.

## Risks

- Low. The option applies only to a same-run reconnect of a retained
remote Runner that has a saved runtime context and saved skill versions.
Every other start keeps the current path.
- Access revocation and deleted versions still fail the reconnect, as
before. A failure to download the files on this server no longer fails a
live reconnect, because the Runner already holds them.
- Nothing new is saved, so rows from older releases, two live servers,
and a deploy during a reconnect behave as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code search, editing, shell, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:58:01 -06:00
DottaandClaude Fable 5.2 fd9985b571 fix(native): say what a Runner command waited behind when it times out (#21056)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes talk to a Runner through durable PRP
commands. The controller waits for each command's result
> - After a server replacement, the new controller reattached to live
Runners. For runs it reached later than about 115 s after detach,
`session.snapshot` got no answer within 30 s
> - The failure said only `provider_transport_failed: PRP command
session.snapshot timed out`. The run log could not tell whether the
Runner was disconnected, whether the command waited behind older
commands, or when the Runner last sent an event
> - This pull request adds that evidence to the timeout message. It
names types, counts and durations only, never payloads
> - The benefit is that the next upgrade's failures explain the stall,
so the runner-side cause can be fixed

## Linked Issues or Issue Description

Refs #21053, #21052, #21039

**What happened?**
At the 07:10Z staging swap, 9 remote native runs that the new controller
reached 114–152 s after detach failed. Each recovery read ended with
`PRP command session.snapshot timed out`. The transport had
authenticated, and in some runs the same Runner process was verified.
The Runner journal recorded no diagnostic other than "WebSocket peer
closed the connection". Runs reached within 39–116 s succeeded.

**What did you expect to happen?**
A command timeout should say what the command waited behind, so the
cause can be found from the run log.

**Steps to reproduce**
1. Start a remote native run and replace the server while a provider
turn is active.
2. Let the new controller reach the Runner more than about two minutes
after detach.
3. Read the run log. The failure says only "timed out".

## What Changed

- `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`:
`waitForRunnerCommand` accepts an optional `describeStall`. When a
command times out, the message ends with a bounded description, for
example `(runner connected; pending ahead 1 [semantic_tool.result]
oldest 188s; last runner event seq 152 181s ago)`. The description
covers whether a Runner is connected or how long it has been
disconnected, the pending commands ahead of this one (types and count,
up to 6 types, and the age of the oldest), and the last committed Runner
event's sequence and age. A failing or empty description keeps the plain
message.
- The transport passes this description for every command it waits on.
The last event's age comes from the PRP payload's `emittedAt`.
- `server/src/services/native-runtime/native-session-executor.ts`: the
startup `session.open` timeout classifier accepts the optional
description, so a startup timeout stays `provider_session_open_timeout`
and keeps its bounded same-run retry. A regression case covers each
startup code with the suffix.
- Tests: the exact message for the staging shape, the plain message when
there is nothing to describe or the description throws, and a
disconnected Runner with no events.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/live/runnerd-codex-transport.test.ts -t "a Runner command that gets
no answer"` passes 3 of 3. These tests fail on `origin/master`, which
has no `describeRunnerCommandStall` export and no description in the
message.
- The full `runnerd-codex-transport.test.ts` file passes 307 of 308. The
one failure (`stops a lost runnerd's provider and escaped tool job from
its lineage journal`) is a process-timing test that passes alone and
does not touch this code.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes.
- Message matchers: the only matcher anchored on the end of the timeout
text was the startup `session.open` classifier. It now accepts the
suffix. `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"classifies a startup session.open timeout"` passes. Without the
classifier change it fails, because the suffix turns the timeout into
`native_session_interrupted`.

## Risks

- Low. Only the text of a timeout message changes, by appending text. No
control flow, persisted state or Runner behavior changes.
- The description holds command types, counts and durations only. It
holds no payloads, identifiers or provider text.
- No wait is added before a turn start.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, with tool use (shell, file edits,
test runs, read-only staging API reads). Model ID as attributed in the
commit trailer: Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 03:00:32 -06:00
DottaandClaude Opus 5.5 4e519bd428 fix(chat): send writes with an empty responseText and point card declines to resolve-from-comment (#21050)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents call the Paperclip API through `call_api` and find
routes with `search_api`
> - #21035 added a pre-send refusal for writes that carry
`responseText`, and new summaries for the question cancel and withdraw
routes
> - Full chat cohort 5 (OpenCode with DeepSeek v4.1 flash) showed that
both changes went too far
> - The refusal blocked a valid write that had an empty `responseText`,
and the summaries hid the route that records a person's decline
> - This pull request narrows the refusal and corrects the guidance text
> - The benefit is that valid writes go through on the first try, and
agents settle a declined card instead of leaving it pending

## Linked Issues or Issue Description

Refs #21035, #21002

**What happened?**
- inbox-search.restore-archive: DeepSeek sent `responseText: {}` with a
valid body on `POST /api/issues/{id}/inbox-archive`. The refusal from
#21035 rejected 24 POSTs in 3 minutes. The model did not drop the key,
and the run was cancelled at the 240s window. In cohort 4 the same POSTs
went through on the first try.
- questions-and-approvals.reject-and-revise: the person said "Do not
publish it; just make a draft I can review." The CEO quoted the cancel
summary ("an agent withdraws its own pending card with POST
.../withdraw"), saw that the person created the publish card, and left
the card pending. The plan decision did not settle. In cohorts 3 and 4
the CEO declined the card with `resolve-from-comment` (resolver policy
`anyone`).

**Expected behavior**
- A write with an empty or default `responseText` is sent once. A write
whose `responseText` sets a window is still refused before it is sent.
- The guidance says that withdraw is only for the agent's own obsolete
card, and that a person's explicit decline or answer goes through
`resolve-from-comment` when the card's resolver policy allows.

**Steps to reproduce**
Run the two chat cases above with OpenCode and DeepSeek v4.1 flash on
master.

## What Changed

- `call_api` (`assertRunnerApiDispatchable`): the check reads the raw
`responseText` value. `{}` or `{ offsetBytes: 0 }` on a write is dropped
and the write is sent. A `responseText` with `limitBytes` or a later
offset is still refused. `saveResponse: false` on a write was already
accepted; a test now covers it.
- OpenAPI: the cancel and withdraw summaries say that withdraw is only
for the agent's own obsolete card, and that a decline or answer on the
person's explicit instruction uses `POST .../resolve-from-comment` when
the resolver policy allows. The `resolve-from-comment` summary says it
also records a decline of a card the person created.
- `AGENT_CHAT_DIRECTIVE`: a person's explicit decline in chat answers a
pending confirmation with reject, whoever created the card. Withdraw
never records the person's decision.
- `doc/runner-api-tools.md`: describes the narrowed refusal.

## Verification

- `server/src/services/native-runtime/runner-api.test.ts`: new test
sends `POST /api/issues/{id}/inbox-archive` with a valid body and
`responseText: {}`, `{ offsetBytes: 0 }`, `saveResponse: false`, or
both. The request is sent exactly once. Red before the fix (3 of 4 cases
refused), green after. The refusal test for a `responseText` with
`limitBytes` stays.
- `server/src/__tests__/openapi-routes.test.ts`: new test for the
cancel, withdraw and resolve-from-comment summaries. Red before, green
after.
- `server/src/__tests__/agent-conversations.test.ts`: new directive test
for the decline rule. Red before, green after.
- Also passed: `runner-api.integration.test.ts`,
`native-case-tools.integration.test.ts`, server `tsc --noEmit`. SDK
public input pins: `--check` reports no change.
- Paid run (OpenCode + DeepSeek v4.1 flash) on this branch for
inbox-search.restore-archive, questions-and-approvals.reject-and-revise,
gateways-profiles.rotate-token and budgets-costs.repair-cost-recording:
4 of 4 cells passed on 3f20fb2bcc (run 43177772183); details in a PR
comment.

## Risks

- A write with a populated `responseText` is still refused, as in
#21035. Only the empty and default shapes change. A dropped empty
`responseText` changes nothing, because the server already removed it
from writes before sending.
- The directive grows by two sentences, which adds a small number of
prompt tokens per chat turn.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 02:09:14 -06:00
DottaandClaude Opus 5.5 0dfbbdeb31 fix(chat): route company creation, decline retries and question cancellation correctly (#21035)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents follow the chat directive and discover API routes with
search_api and call_api
> - Full chat cohort 4 (OpenCode with DeepSeek v4.1 flash) showed four
guidance gaps
> - The missing-name rule overrode the board-only company-creation
boundary; a repeated request after a decline did not post a new approval
card; search_api showed the board-only question cancel route as open to
agents (and withdraw as board-only); write calls carried `responseText`
beside or instead of `body`
> - This pull request fixes the guidance, the route authorization
metadata and the pre-send refusal
> - The benefit is that agents take the supported path the first time
instead of asking for the wrong input, refusing a valid retry, or
calling a route that returns 403

## Linked Issues or Issue Description

**What happened?**
- onboarding.new-company: asked to "set up a new company for my website
business", the CEO cited the missing-name rule (#21007) and asked for a
business name with a question card. Creating a company is a board
action, so the agent should send the person to the creation flow.
- app-permissions.ask-before-action: the person declined a write card
and then repeated the same request. The CEO asked the person to tell it
to resubmit instead of posting a new approval card. The gateway already
allows a fresh request after a new human message
(`isNewHumanIntentAfterToolRejection`), but the decline text said "Do
not retry the same call".
- questions-and-approvals.multi-part-question: search_api listed `POST
/api/issues/{id}/interactions/{interactionId}/cancel` as
`board_or_agent`. The route rejects agents. The agent route,
`/withdraw`, was listed as board-only although its route admits the
card's creator agent and the task's assignee.
- Several cases: write calls through call_api carried `responseText`
with no body (for example "Received arguments: operationId, pathParams,
responseText").

**Expected behavior**
The agent links the company-creation flow; it posts a fresh approval
card for a repeated request; search_api shows the true authorization of
cancel and withdraw; a write with `responseText` is refused before it is
sent, with the same guidance #21002 gave `saveResponse`.

**Steps to reproduce**
Run the four chat cases above with OpenCode and DeepSeek v4.1 flash on
master.

Refs #21002, #21007

## What Changed

- `AGENT_CHAT_DIRECTIVE`: the missing-name rule applies to work the
agent creates itself and never comes before a board-only boundary. For a
new company the agent links [Create organization](/onboarding) (no
company prefix) and repeats the business details. A repeated request
after a declined approval card is an explicit retry: make the same call
so a fresh card is posted. A clarification question card puts candidate
links in its description, because labels show as plain text.
- Decline instructions in `tool-action-delivery.ts` and
`tool-gateway.ts` say that a later request from the person for the same
action is an explicit retry.
- OpenAPI: `POST /api/issues/{id}/interactions/{interactionId}/cancel`
is board-only and its summary names the `/withdraw` route for agents.
`/withdraw` is no longer marked board-only, which matches its route.
- `call_api`: a new write that carries `responseText` is refused before
it is sent; the message says `responseText` never carries request fields
and names the allowed body fields. Older saved calls still validate for
recovery. `doc/runner-api-tools.md` is updated.

## Verification

- `server/src/__tests__/agent-conversations.test.ts`: new directive
tests for the board-only boundary order, the decline retry and card
links.
- `server/src/__tests__/openapi-routes.test.ts`: cancel is `board`,
withdraw is `board_or_agent`, the cancel summary names withdraw.
- `server/src/services/native-runtime/runner-api.test.ts`: a write with
`responseText` is refused before sending (three window shapes), the same
write without it is sent once, and saved calls still validate.
- `tool-gateway.test.ts`, `tool-gateway-service.test.ts`,
`issue-thread-interaction-routes.test.ts`,
`tool-action-replay-intent.test.ts`: passed. `server` `tsc --noEmit`:
passed.
- Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) together with the
eval PR: see the PR comment for results.

## Risks

- A model that used to send a write with `responseText` and a valid body
now gets one refusal and must resend without it. Nothing is sent before
the refusal, so there is no duplicate side effect.
- The directive grows by a few sentences, which adds a small number of
prompt tokens per chat turn.
- Agents that read `/withdraw` as board-only before now see that they
can withdraw their own card; the route already enforced who may.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:27:35 -06:00
DottaandClaude Opus 5.5 aa94939988 fix(native): continue a run whose remote Runner died mid-turn after a restart, without replaying the turn (#21039)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a server restart, recovery reattaches to that
Runner on the run's own sandbox lease
> - On staging, a deploy detached the controller and left the Runner
alive. The restart's reattachment then found the Runner process gone
with a provider turn in flight. The run failed with
`runner_remote_process_identity_unavailable` and the task waited for a
person
> - The run log said `startup reuse: environment (fresh) created`, so
the sandbox looked re-created. It was not. The reattachment resumed the
exact retained lease. The receipt reported the lease's original creation
> - `requestInterruptedNativeTurnRetry` refused every unsettled turn
(`provider_turn_unsettled`), and refused every system wake unless the
journal proved that no turn was issued. So this run got no automatic
continuation
> - This pull request continues such a run once in a fresh provider
session when the refused reattachment proved the Runner process gone. It
records that the turn's outcome is unknown and never replays the turn.
It also makes the environment receipt tell the truth
> - The benefit is that a restart that kills a Runner in the middle of a
turn no longer needs a Retry click

## Linked Issues or Issue Description

Refs #20853, #20923, #21023, #20278, #20627, #20828

**What happened?**
A staging deploy landed in the middle of a remote native Codex run on
Daytona. Shutdown detached the controller and logged "durable runner
left alive". The new server reattached through the run's original lease
(same lease row, same provider sandbox id, same directory). Its process
check then found the Runner gone: `process_verification` /
`process_not_running`, `runnerLifecycle: ready`, `runnerSettled: true`,
last diagnostic "WebSocket peer closed the connection", empty stderr.
The provider checkpoint still had an active turn. The run failed with
`runner_remote_process_identity_unavailable` /
`native_continuation_requires_reconciliation`, and the task went to
`blocked`.

**Root cause**
- `server/src/services/heartbeat/restart-interrupted-turn.ts`
(`requestInterruptedNativeTurnRetry`): an unsettled last turn always
returned `provider_turn_unsettled`, also when the refusal proved the
Runner process gone. A system wake (here `transient_failure_retry`)
continued only with `providerTurnIssued: false`.
- The sandbox was not re-created. The environment and lease listings
show one lease for the run, never replaced, and the final workspace
snapshot used the same provider sandbox. `environmentStartupReuse` read
`sandboxLeaseAcquisition: created` from the lease's first acquisition,
so the reattachment's receipt said `fresh / created`. The 46 s acquire
was the provider resume of that exact lease. The evidence available to
GET requests cannot say what stopped the Runner process (the resume
result is not kept). This pull request does not change that path.
- #21023 named this run as not covered. #20853 resumes a Runner only at
a settled, idle `ready` cut. Here the provider checkpoint has an active
turn, so #20853 stays fail-closed, as it should.

**Expected behavior**
The run continues once by itself in a fresh provider session over the
saved work. The turn whose outcome is unknown is recorded and is not
replayed.

## What Changed

- `restart-interrupted-turn.ts`: read the last
`native.remote_runner_recovery_failed` refusal once. When it proves the
Runner process gone, an unsettled last turn no longer blocks the
automatic Retry. Before it requests the Retry, the server records
`native.interrupted_turn_outcome_unknown` once (turn id, last turn
event, `replayed: false`, closed `runnerExit` fields). The Retry is the
existing one: saved-work recovery and stop proof first, then a fresh
provider session (`forceFreshSession: true`), once per chain.
- The same file: a system wake whose run is an automatic retry of owed
work (`retryOfRunId` set) also continues under the run's responsible
user once the refusal proved the Runner gone. Other system wakes (for
example a monitor) keep their policy.
- `packages/adapter-utils/src/startup-reuse.ts` and
`native-session-executor.ts`: a `reattach_remote_runner` recovery
reports its `environment` receipt as `reused / verified_reuse`. That
path only resumes the run's exact retained lease and never acquires one.
- Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`.

Previous-release story: no persisted shape changes. The rule reads the
closed `remoteRunnerRecovery` fields that the previous release already
writes; the new test uses the staging refusal payload as written. A run
that a previous release already failed this way continues on the next
completion or restart hook call only; runs already settled stay as they
are and keep their Retry. The new event is additive.

## Verification

- Red on `origin/master` (51d77648bc), with only the new tests: `cd
server && npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts` fails 4 of 6.
The three continuation cases return `{ requested: false, reason:
"provider_turn_unsettled" }`, and the monitor case returns that reason
instead of `not_user_request`.
- Green with the fix: 6 of 6. The cases cover a user's assignment
through the restart claim, the staging shape (system
`transient_failure_retry` with `retryOfRunId`), and a same-run retry.
Each one checks one recorded unknown-outcome event, an idempotent second
call, saved-work recovery first, one queued successor with
`forceFreshSession: true`, and one Retry intent. Negative cases: a
refusal that did not prove the process gone, no refusal, and a monitor
wake. Each keeps the visible Retry and records nothing.
- `cd packages/adapter-utils && npx vitest run
src/startup-reuse.test.ts`: the new receipt case fails before (`fresh /
created`) and passes after.
- Focused suites pass: `workspace-recovery-retry-copyback.test.ts`,
`governed-wait-runner-loss.test.ts`,
`heartbeat-native-remote-failure-retention.test.ts`,
`native-runner-restart-recovery.integration.test.ts` (302 passed, 14
skipped), and `native-session-executor.test.ts -t "startup
reuse|restart|reattach"` (149 passed).
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/adapter-utils typecheck` pass.
- `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global
CI input. CI supplies that coverage.
- Not run: a live Daytona restart. The change is in the server's
recovery decision; the tests use the fixture's sandbox confinement.

## Risks

- Medium. A run whose Runner died in the middle of a turn now continues
by itself. The turn's outcome is unknown, so tool work from that turn
can be partly done. The successor is a fresh session over the saved
workspace; it does not replay the turn. Saved-work recovery stops and
keeps the sandbox before the successor starts, so the old provider
cannot still run.
- Without positive proof that the Runner process is gone, an in-flight
turn keeps the visible Retry, as before.
- An automatic retry of owed work now continues after a lost Runner.
Monitor and other system wakes without `retryOfRunId` do not. The bound
is still one automatic Retry per chain.
- The environment receipt changes only for the reattachment path. No
execution decision reads it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:13:25 -06:00
DottaandClaude Opus 5.5 940d681974 fix(heartbeat): hold a queued native run while its task's previous session finishes cleanup (#21031)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner runs keep a provider session per task. When a run
stops, its session close continues in the server process
> - The runner refuses every new session of the task until that close
settles. A close that cannot prove the old runner suspended needs a
stopped-process proof before it is released
> - A reassignment stops the old assignee's run mid-turn. The new
assignee's run was dispatched one second later into that gate, and it
failed with `native_session_cleanup_quarantined`
> - Only a Retry continued the task. Recovery must work without a
person, and the needed-retry rate must be 0%
> - This pull request keeps such a run queued until the close settles or
its old owner is proven stopped
> - The benefit is that the new run starts by itself after cleanup,
without a failed run or a Retry

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `tasks.edit-properties` (and
`tasks.assignment-conflict` in an earlier cohort), the CEO reassigned a
task while the old assignee's native run was mid-turn, with a Paperclip
tool call in flight. The reassignment cancelled that run. Its session
close then failed with `native_session_close_unrecoverable: runner did
not durably suspend before checkpoint`, and the runner kept an
operator-required cleanup gate for the task. The new assignee's run had
started 1 second after the cancel. The runner's admission waited for the
in-flight close. When the close escalated, admission refused the run
with `native_session_cleanup_quarantined`. No run was queued again.

**Steps to reproduce**
1. Run a native task and stop its run while its session close cannot
settle at once.
2. Queue a run of another agent for the same task.
3. The run is dispatched into the cleanup gate and fails before it
launches.

**Expected behavior**
The run waits for the cleanup, and starts once the close settles or the
old owner is proven stopped. It never fails, and no Retry is needed.

## What Changed

- `server/src/services/native-runtime/native-cleanup-admission-hold.ts`:
`nativeCleanupHoldsAdmission` reads the task's cleanup gate. For an
operator-required gate it first tries
`retireStoppedSessionCleanupGates`, which retires the gate when the old
owner's runner and provider processes are proven stopped. This is the
same proof that Retry uses.
- `server/src/services/heartbeat/queue.ts`: `claimQueuedRun` keeps an
unstarted `paperclip_runner` run queued while its task has a settling or
unproven close. This matches the existing
`predecessor_settling_in_process` hold, which covers only the same
agent. A reassignment changes the agent.
- `server/src/services/native-runtime/native-cleanup-release-wake.ts`: a
released close now also wakes queued-run dispatch. A close that
escalates to operator-required also wakes it, so admission tries the
stopped-process proof at once. The queued-run liveness redrive covers
any later proof.
-
`server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts`:
an assembled heartbeat test. It is red on master (the run is claimed
into a pending close and into an operator-required close). It is green
here (the run stays queued, then is admitted after the close settles). A
control case shows that a run for another task is not held.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts` (3
passed; 2 red on master)
- `cd server && npx vitest run
src/services/quarantined-task-retry.test.ts
src/services/heartbeat/queue.test.ts
src/__tests__/heartbeat-workspace-busy.test.ts
src/services/queued-run-dispatch-liveness.test.ts` (180 passed)
- `cd server && npx tsc --noEmit` passes.
- Paid E2E run on a combined branch:
https://github.com/paperclipai/paperclip/actions/runs/43161593494
(results in a comment below).

## Risks

- A run whose task has an operator-required gate that is never proven
stopped now stays queued instead of failing. Before, that run failed,
and Retry was blocked by the same proof. The liveness redrive re-checks
it with back-off.
- The hold applies only to unstarted `paperclip_runner` runs of the
gated task. Other tasks and agents are not affected.
- Previous-release saved state: no persisted shape changes. The gate
lives in process memory, and runs saved by the previous release are
claimed by the same rule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:53:54 -06:00
Nicky Leach d68088465c Reuse retained Runner descriptions during startup (#21004) 2026-11-11 21:20:08 -08:00
DottaandClaude Opus 5.5 5a7378af56 fix(heartbeat): re-drive a queued run whose dispatch signal was lost after a restart (#20986)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts queued runs. The run-dispatch worker
re-runs the queue only when a write sends a "queued run dispatch"
signal.
> - A follow-up run can queue while its predecessor still holds its
sandbox lease. The claim then declines the run without a log line, and
the run stays `queued`.
> - After a controller restart, nothing sent a new signal once that
lease was settled. The run stayed `queued` forever. The issue had no
Retry, and the reconciler counted the queued run as a valid next action.
> - This pull request sends a dispatch signal from the native
finalization writes that can clear a predecessor's cleanup gate, logs
why a claim was declined, and bounds the wait. The dispatch worker
re-drives an undispatched queued run after 60 s as a backstop, and the
ownership reconciler reports and re-drives one still queued after 5
minutes.
> - The benefit is that a follow-up starts within seconds of its
predecessor settling, a lost signal delays it by about a minute instead
of forever, and a run that still cannot start is visible.

## Linked Issues or Issue Description

Refs #20909, #20935, #20953, #20970

**What happened?**

Restart-chaos wave `w-20261111T223714-s9512` (master `6154b9485e` +
#20907 + #20909, Daytona warm, Codex, W2, CR1 with a 360 s hold,
`--ephemeral-server`, seed 9512), task t02. Run `0b95b1cd` crossed the
outage with an accepted result. The next controller finalized it at
22:47:59Z. Its continuation run `1ad79418` was queued at 22:48:00Z
(`retryOfRunId` = `0b95b1cd`, wake `d967700c`, source
`native_status_decision`). The run stayed `queued` with `claimed_at =
null` until the wave ended 19 minutes later. The agent stayed `running`
with no running run. The task had no Retry. The grade was
`R1:no_rest_within_budget` and U1 `no_owner_no_action ... queued 485s`.

**Root cause**

`claimQueuedRun` (`server/src/services/heartbeat/queue.ts`,
`lockIssueExecutionClaim`) declines a queued run while
`hasPendingRunCleanup` (`server/src/services/execution-blocker.ts`) is
true for its predecessor. After the restart, the predecessor's warm
`reuse_by_environment` lease was still `active` when the continuation
queued, so the first claim was declined. The decline is correct, but
nothing owned the retry. `registerRunDispatchWork`
(`server/src/services/heartbeat/dispatch-work.ts`) has `next: async ()
=> null`: the queue is only re-run by a later dispatch signal. At wave
end the lease was `retained` with `cleanupStatus: success`, which does
not block a claim. A dispatch pass in that state starts the run
(verified locally). So no pass ran after the gate cleared. The ownership
reconciler listed the run as `queued_run` with owner `run_dispatch`
(always "alive"), and I7 treats any queued run as a next action, so
nothing reported it.

This is not the #20935 throttle. The wave base predates #20935, and a
throttled intent keeps its wake intent `queued` with a `nextAllowedAt`
deadline. Here the run row existed and its claim was never retried. On
current master #20935 parks this specific `response_wake` self-wake, but
any continuation, comment wake or Retry that queues behind a settling
predecessor after a restart has the same gap. The red test uses a
`same_agent` continuation, which master still enqueues.

**Steps to reproduce**

`server/src/__tests__/heartbeat-native-restart-suspension.test.ts`,
"starts a continuation that queued behind its predecessor's sandbox
release after a graceful restart".

## What Changed

- Event-driven dispatch. A lease release already signals queued-run
dispatch (`environmentsSvc.releaseLease`, `writeEnvironmentCleanup`,
stop-and-retain settlement). I enumerated every `environment_leases`
status writer: each one that moves a lease out of a blocking state
already signals. The writers that do not signal only acquire a lease or
set `pendingCleanupInFlight`, so they add a block and do not clear one.
The native coordinator side had no signal, so:
- `notifyNativeFinalizationWork` (the shared notifier of coordinator
commits and terminal transitions) now also signals queued-run dispatch
for the company.
- The coordinator writes that clear `leaseOwner` or end the phase
outside that notifier now signal in the same transaction:
`releaseLapsedCoordinatorOwner`, native startup cleanup,
observed-workspace preservation, `record_finalization_error` in the
interaction bridge, the wake-queue terminal recovery, and the
reconciler's ambiguous-state terminal failure.
- Not changed: three non-transactional pre-dispatch closes in
`run-completion.ts` and `agent-execution-stop.ts`. Their callers queue
or settle the successor after the write, so the successor's own enqueue
starts the queue.
- `claimQueuedRun` now logs the reason when it declines a queued run at
the issue execution claim (`predecessor_cleanup_pending`,
`predecessor_still_executing`, `issue_execution_owned_by_another_run`,
case handoff, task chat cleanup), when a predecessor is still settling
in this process, and when a temporary budget hold keeps it queued.
- New `server/src/services/queued-run-dispatch-liveness.ts`:
`listUndispatchedQueuedRuns` reads queued runs in active companies whose
ready, non-paused agent has no running run. It reads every page with a
keyset cursor until the set is exhausted, so older held runs cannot hide
a newer stranded run. `createQueuedRunRedriveSchedule` re-drives each
such run 60 s after it was created, then every 2, 4, 8 minutes, capped
at 15 minutes. Run ids that leave the set are dropped.
- `server/src/services/heartbeat/dispatch-work.ts`: the queued-run
dispatch worker's `next` now plans those re-drives. It re-reads only the
companies the pass just dispatched, keeps the deadlines of every other
company, and runs no query when the pass dispatched nothing. Only an
instance-wide pass (startup or explicit wake) reads every company. It
adds each due company to the dispatch scopes and returns the next due
time.
- `packages/db`: new partial index `heartbeat_runs_queued_created_idx
(company_id, created_at, id) WHERE status = 'queued'` (migration
`0470_heartbeat_runs_queued_created_index.sql`, `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`, registered as a top-level migration in
`client.ts`). It returns `null` (no deadline, no SQL until a signal)
when no run is undispatched or scheduling is suppressed. Every re-drive
goes through the normal `startNextQueuedRunForAgent` and
`claimQueuedRun` gates, so it never bypasses a hold, budget, pause or
cleanup gate.
- `server/src/services/heartbeat.ts`: wires `listUndispatchedQueuedRuns`
into the worker (two lines).
- `server/src/services/recovery/ownership-invariant-report.ts`: new
check. A queued run still undispatched 5 minutes after creation is an I7
violation (`kind: queued_run`, `reason: queued_run_not_dispatched`),
written to the activity log once. A younger one sets the report's next
due time.
-
`server/src/services/native-runtime/native-finalization-reconciler.ts`:
`applyOwnershipReportBackstops` sends a queued-run dispatch signal once,
for each company with a newly found violation. Repeat passes do not
signal again, so a held run does not wake the report every 30 s.
- `tests/restart-chaos/README.md`: records the wave and the fix in the
CR results table. The existing CR1 cell (#20909) and the enforced U1
check already cover this class in the harness.

## Verification

- RED on master (`23909077ad`):
- `heartbeat-native-restart-suspension.test.ts` "starts a continuation
that queued behind its predecessor's sandbox release after a graceful
restart": the continuation is `queued` behind the predecessor's `active`
warm lease. The run-dispatch worker is registered, the lease then
settles to `retained`/`success` with no dispatch signal, and the run
stays queued: `AssertionError: expected null not to be null` after 75 s.
- `ownership-reconciliation-report.test.ts` "reports and re-drives a
queued run that no dispatch started while its agent had nothing
running": `expected [] to deeply equal [ ObjectContaining{…} ]` (no
violation for a run queued 19 minutes).
- `ownership-reconciliation-report.test.ts` "signals queued-run dispatch
when finalization work settles or a finished coordinator's lapsed owner
is released": `expected [] to include 'settled'`.
- GREEN on this branch: all pass. The restart test has two cases. When
the lease is released through `releaseLease`, the continuation starts
within 5 s. When the lease is settled by a write with no signal, the
backstop starts it within 75 s. A new report test puts three older held
runs ahead of a stranded run with a page size of 2, finds the stranded
run, and checks that the backstop signals once.
- `queued-run-dispatch-liveness.test.ts`: schedule bound, doubling, cap,
and going quiet.
- Focused suites pass (22 files, 495 tests):
recovery-deadline-acceptance, live-recovery-work,
heartbeat-run-dispatch-lifecycle, heartbeat-run-dispatch-work,
deferred-execution-receipt-work, heartbeat-native-cleanup-admission, the
ownership kinds suites, and others.
`heartbeat-native-restart-suspension` and
`ownership-reconciliation-report` pass in full. `tsc --noEmit` for the
server passes.

## Risks

- A queued run that is held on purpose (for example a temporary budget
hold or a case handoff waiting for release) is now re-checked at most
every 15 minutes after the first minute. Each re-check is one indexed
query and the normal claim gates. Such a run is also reported once as an
I7 violation.
- An instance with an undispatched queued run keeps a dispatch deadline,
so it does not idle while that run waits. An instance with no such run
is unchanged.
- Schema: one additive partial index, built concurrently. No data
change. State saved by an older version is read as is. Nothing runs
before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:04:58 -06:00
DottaandClaude Opus 5.5 776e900b48 fix(native): retire a finished run's idle Runner that still holds the sandbox Runner port before launching (#20995)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute through a Runner (`paperclip-runnerd`) inside a
remote sandbox. Daytona reaches every Runner through one fixed preview
port (43127), so a sandbox serves one listening Runner at a time.
> - A warm sandbox is reused by the next run of the same task. That run
often opens a new session, so it neither adopts the previous run's warm
Runner nor retires it. Retirement of another session's warm owner works
only through the in-memory warm registry of the server that launched it.
> - After a server restart (or on another server), that registry is
empty. The previous run's idle warm Runner keeps the port, and every new
launch fails with `runner_ingress_bind_conflict` until the bounded retry
runs out.
> - This pull request finds such a holder on the sandbox before launch
and proves from durable records whether it may be stopped. A finished
run's idle Runner is stopped through its verified process identity. Any
other holder sends the run to the fresh-sandbox fallback from #20863 and
is never stopped.
> - The benefit is that a follow-up run after a restart starts on its
reused sandbox instead of colliding with its predecessor and needing
Retry.

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261111T230537-s6402` (Claude through ACPX,
Daytona warm sandboxes, `--ephemeral-server`, K1/K2/K15 with
`--k15-fresh-home`, seed 6402, master, no injected fault) needed Retry
on 4 of 6 tasks. Each failing run's first Runner failed with
`runner_ingress_bind_conflict: failed to bind listener: Address already
in use`. The harness's sandbox captures name the process holding port
43127 at the time. In every case it was the previous, already
`succeeded` run of the same task, started under the server generation
before the restart:

| Task | Failing run | Port 43127 held by | Holder run status |
| --- | --- | --- | --- |
| t01 | b4769dfd | 395b202d (Runner started 23:11:30) | succeeded
23:12:19 |
| t05 | b6d466fd | 0969ad46 (23:11:29) | succeeded 23:12:18 |
| t02 | 4097fc1f | 527a1659 (23:11:28) | succeeded 23:12:17 |
| t03 | 60d821af | 1c06bf75 (23:11:28) | succeeded 23:12:25 |

**Root cause**
Warm owners are tracked only in the process-local `warmNativeSessions`
map in `server/src/services/native-runtime/native-session-executor.ts`.
`claimWarmNativeInstructionCopy`,
`reserveWarmNativeInstructionDirectory` and the same-session admission
path retire a predecessor only when it is in that map. A restarted
server has an empty map. The follow-up run uses a new session, so
`inspectRemoteWarmSessionRecovery` (same session only) does not adopt it
either. The launch then meets the old Runner on the fixed port.
`stopStaleRemoteRunnerBeforeBootstrap` stops only the run's own Runner,
so the same-run retries collide again.

**Expected behavior**
A finished predecessor's idle warm Runner is retired before the next
Runner launches on the same sandbox, wherever it was launched. A holder
that cannot be proven safe to stop is never stopped, and the run moves
to a fresh sandbox.

## What Changed

- New
`server/src/services/native-runtime/sandbox-runner-port-holders.ts`:
`retireSandboxRunnerPortHolders` lists every other session's Runner on
the sandbox (session roots with a `runner-process.identity`; this run's
own root and moved-aside roots are skipped). It checks each one with the
existing verified process check (`REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT`:
pid, runner id in the command line and, when recorded, the boot
generation). Then, for a live holder:
- Registered in this server's warm registry: it is retired through
`retireWarmPredecessor`.
- Its company-scoped run row is `succeeded` or `cancelled`, for the same
agent, and its Runner state is idle (empty outbox, no pending terminal
delivery, and both saved fences `pendingProviderCleanup` and
`warmTransition` clear): it is stopped through the verified identity
(`TERM`, then `KILL`, each with a bounded wait). An activity log entry
`native.runner_port_holder_retired` is written.
- When the inventory command itself cannot run, the holder state is
unproven, so the run takes the fresh-sandbox fallback.
- Anything else (a live run, a failed or interrupted run that may still
be resumed, an unknown owner, work in the outbox, an unverifiable or
changed identity marker, or a stop that is not confirmed):
`NativePredecessorRetirementUnprovenError`. Before any process start,
the existing #20863 path marks the lease replace-before-reuse and
re-queues the run in the uncharged lane on a fresh sandbox.
- `executePaperclipNativeSessionAttempt` runs this check before
`native.process_start_requested`, and only for a resumed (reused)
sandbox launched through a shell runner without a warm session to reuse
and without a reattach recovery. A created or replacement sandbox cannot
hold another session's Runner, so it adds no command there.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/sandbox-runner-port-holders.integration.test.ts`
(embedded Postgres, real processes): two Runners contend for one real
port. The second bind first fails with `EADDRINUSE`, the same failure as
staging. After `retireSandboxRunnerPortHolders` the finished run's
holder has exited and the bind succeeds, with the activity entry
recorded. The commands run through a confined runner that refuses any
path outside the sandbox workspace base. A holder owned by a live run,
by a finished run with work in its outbox, by a cancelled run with
pending provider cleanup, or by no recorded run is never stopped: the
call raises `NativePredecessorRetirementUnprovenError`
(`runner_port_holder_owner_not_settled` /
`runner_port_holder_unknown_owner`) and the port stays held. This run's
own session, moved-aside roots and exited holders are ignored.
- `native-warm-checkpoint-ownership.integration.test.ts`: its fixture
sandbox runner now answers the inventory command with an empty list
(that sandbox has no other Runner); all 9 cells pass.
- `native-session-executor.test.ts`: 1406 of 1406 pass. Server `tsc
--noEmit` passes.
- Chaos results are in a comment below.

## Risks

- A finished run's idle warm Runner is stopped earlier than its idle
timeout. It held no unsaved work (its run is `succeeded` or `cancelled`
and its outbox is empty), and SIGTERM is the same stop as its idle
close. Only that warm reuse is lost.
- Failed and interrupted holders are not stopped, because their run may
still reattach. Such a run goes to a fresh sandbox instead.
- One extra sandbox command before launch on a reused sandbox for a cold
launch, no wait and no file changes. A warm reuse and a reattach skip
it.
- Not changed: a bind conflict from a holder that is not a Runner (no
identity file) still uses the existing same-run bootstrap budget.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, test runs). Default context window,
extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20863, #20953, #20972, #20992.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 21:06:12 -06:00
DottaandClaude Opus 5.5 89ea418e76 fix(native): start the configured review stage when a worker reports finished work as needs_review (#21005)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks can have an execution policy with a review stage that names an
independent reviewer
> - #20882 made a native worker's done report start that configured
stage
> - A worker can also report finished work as needs_review and list the
review itself as the only blocking remaining work
> - That report skipped the configured-stage check. The generic
"needs_review requires a named reviewer" error made the worker file a
separate review card, and the configured stage never started
> - This pull request treats that report as finished work and gives the
configured-stage guidance first
> - The benefit is that the configured reviewer gates Done whichever way
the worker phrases its finished report

## Linked Issues or Issue Description

Refs #20882

**What happened**

A task had a review stage for a human reviewer. The native worker built
and verified the deliverable. It reported `needs_review` with
`objectiveSatisfied: true`, all criteria satisfied, and one blocking
remaining item: "Independent human review of the prepared website.html
deliverable." The completion feedback skipped the configured-stage check
because that item blocked completion. The generic needs_review error
then told the worker to name a reviewer. The worker added a review
attention request, and Paperclip made a separate "Review requested"
card. `executionState` stayed null, and the configured reviewer was not
assigned.

**Expected behavior**

On a task with a configured review stage, a worker that reports
finished, verified work is told to report done. Paperclip then starts
the configured stage with its reviewer.

**Steps to reproduce**

1. Create a task with `executionPolicy.stages = [{ type: "review",
participants: [{ type: "user", userId }] }]` for a native agent.
2. Have the agent report `needs_review` with a satisfied objective and
the review as its only blocking remaining work.
3. Before this fix, the feedback asks for a named reviewer. The task
gets a separate review card and no stage.

## What Changed

- `native-completion-feedback.ts`: on a task whose configured review
stage would start, a `needs_review` report with a satisfied objective
and satisfied criteria counts as finished work. It gets the
configured-stage guidance before the generic named-reviewer error. A
review needed before the work can continue (objective not satisfied)
keeps its own request path. Approval requests keep their own path.
- `native-completion-feedback.ts`: a done report on such a task that
still lists the review as outstanding gets the same guidance, not the
generic "done report includes unfinished work" error. So the worker is
not sent back to needs_review.
- The guidance now says to report done without listing the review as
remaining work or as an attention request.
- Tests: two new cases in `native-execution-policy-review-stage.test.ts`
use the recorded report shape.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-execution-policy-review-stage.test.ts`:
15 passed. The new recorded-shape test fails on master (red) and passes
with this change (green).
- `npx vitest run` for `native-completion-feedback`,
`native-current-wake-completion-feedback` and
`native-execution-policy-review-stage-race`: 42 passed.
- Paid chat eval run on a combined throwaway branch: results are in a PR
comment.

## Risks

- On a configured-review task, a worker that reports a satisfied
objective but has real non-review blocking work now gets the
configured-stage guidance. That guidance says to report done only when
the work is complete and verified, so the task does not close early.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:55:24 -06:00
DottaandClaude Opus 5.5 601babe322 fix(chat): let agents read saved trace export status and give chat real company page paths (#20998)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and agents answer from the
company's saved settings
> - A board member can save a company trace export (an OTLP collector or
Braintrust) in Company settings
> - The settings route is for board connection managers only, so an
agent cannot read it
> - In a chat eval, the board saved and tested a collector. The agent
checked only its own config and secrets, and then told the person that
nothing was configured
> - Chat guidance also names company pages as `/<prefix>/...`. In
another eval, the agent copied that placeholder into a link, and the
link opened nothing
> - This pull request gives agents a read-only status of the trace
export, and gives chat the real company prefix in page paths
> - The benefit is that agents answer from saved settings, and the links
they write open real pages

## Linked Issues or Issue Description

**What happened?**
1. A board member saved and tested a trace export destination. The chat
agent was asked to acknowledge the saved configuration. The agent said
that no collector endpoint or credentials existed. Agents get 403 from
`GET /api/companies/{companyId}/observability`, and no other route
reports the saved state.
2. An agent reply linked `/<prefix>/company/settings/instance`. The chat
rendered it as `/%3Cprefix%3E/...`, which is not a real page. The
placeholder came from the chat directive, which names pages such as
`/<prefix>/company/settings/instance/profile`.

**Expected behavior**
1. An agent can read whether the company's trace export is configured,
enabled and credentialed, and the result of its latest test. The agent
cannot read the destination or any credential.
2. The chat directive and API search results that an agent reads give
real company paths, such as `/COM/company/settings/instance/profile`.

**Steps to reproduce**
1. As a board member, save and test an agent observability destination
in Company settings.
2. In chat, ask the CEO agent to acknowledge that the test collector
configuration is saved.
3. See that the agent cannot read the saved settings and denies them.

**Paperclip version or commit**
master at 9dfdeaad86.

## What Changed

- Added `GET /api/companies/{companyId}/observability/status`. Agents of
the company and all members can read it. It returns `configured`,
`enabled`, `provider`, `credentialsConfigured`, `updatedAt` and the
latest test status. It never returns the destination or a credential.
- Added the shared `AgentObservabilityStatus` type and
`agentObservabilityStatus()` mapper, and registered the route in the
OpenAPI catalog so that `search_api` finds it.
- The chat task markdown now fills the conversation's own company prefix
(from its identifier, for example `COM-1` gives `COM`) into the chat
directives in place of `/<prefix>/`.
- `search_api` results now get the same substitution for the caller's
company. The shared catalog is not changed.
- Pinned the changed shared source in
`tests/runner-e2e/source-sdk-public-inputs.json`.

## Verification

- `cd server && npx vitest run src/__tests__/agent-observability.test.ts
src/__tests__/agent-observability-status-guidance.test.ts
src/services/heartbeat/task-markdown.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts`
- Red/green: with the product change removed, 10 of the new or updated
tests fail (the agent status read gets 403, search does not find the
route, and the chat markdown still contains `/<prefix>/`). With the
change, all pass.
- `npx tsc -p server/tsconfig.json --noEmit` passes.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
reports no change.

## Risks

- Low risk. The new route is read-only and returns no destination and no
credential. The existing settings route keeps its board-only rule.
- The chat prompt text changes for chats with an identifier. The text is
constant for each company, so prompt caching is not affected inside a
conversation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:55:21 -06:00
DottaandClaude Opus 5.5 50b79d711a fix(runner): keep call_api writes carrying their body (#21002)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agents use the `call_api` tool when no dedicated tool covers
an API operation
> - #20606 added an optional `saveResponse` flag to the shared
`call_api` schema for GET reads
> - After that change, OpenCode agents with DeepSeek v4.1 flash often
sent write calls without `body`
> - Body-less POST calls reached the routes and failed with 400; agents
then repeated the same call many times
> - This pull request moves the save option to its own read-only tool
and refuses body-less writes before they are sent
> - The benefit is that the write schema returns to its earlier shape,
and a missed `body` gets one clear, actionable error

## Linked Issues or Issue Description

Refs #20606

**What happened?**

In a full chat-eval cohort on master (OpenCode, DeepSeek v4.1 flash),
write calls through `call_api` arrived without `body`. The model's own
reasoning planned a body, but the emitted calls had only `operationId`,
`pathParams`, `query` and `saveResponse`. About 100 refused PATCH calls
listed exactly those arguments (the cohort before #20606 had none with
`saveResponse`). 72 POST calls reached the server with no body and got
400 (0 in the earlier cohort). One case sent the same body-less resolve
call 21 times.

**Expected behavior**

Write calls carry their request fields in `body`. When a body is
missing, the agent gets one clear error before anything is sent, and it
can correct the call.

**Steps to reproduce**

Run the OpenCode chat cases that resolve approvals, edit queued
messages, delegate with context, edit task properties or start preview
services on master after #20606. Count `call_api` writes without `body`
and POST 400 responses.

**Additional context**

The tool schema that OpenCode sends to the model is intact: `body` is
present. A short-context replay with the same schema did not reproduce
the omission (direct provider calls and local OpenCode runs: every write
carried `body`, with or without `saveResponse`). The omission appears in
long sessions, and it correlates with the added read-only flag. This
change removes that flag from the write schema and adds a safety net for
the remaining cases.

## What Changed

- Remove `saveResponse` from the advertised `call_api` schema and
description. The description now says that request fields go only in
`body`, and the `body` description says when it is required.
- Add a separate read-only `save_api_response` tool (`operationId`,
`pathParams`, `query`). It accepts only GET operation IDs, has no body
field, and uses the existing `saveResponse` export path with the same
access, sensitivity, budget and integrity rules.
- Refuse a write before dispatch when its body is missing: PATCH with a
required body (as before), and POST or PUT whose JSON request schema
names required fields. The error names the operation, the allowed
fields, the received arguments, the shape to resend, and that `query`,
`responseText` and `saveResponse` never carry request fields.
- Keep server-side acceptance of `saveResponse` on `call_api`, so saved
calls and receipts from earlier runs stay valid for recovery.
- Regenerate the runner contract, catalog, seeded fixture and manifest.
Update `doc/runner-api-tools.md`.

## Verification

- Red at master sources: the 3 replayed recorded writes (POST
resolve-from-comment, PATCH issue, POST comment, each with `query: {}`
and `saveResponse: false` and no body) were sent to the route instead of
being refused, and the schema test found no `save_api_response` tool. 4
tests failed.
- Green with this change:
`server/src/services/native-runtime/runner-api.test.ts` 2105/2105
passed; `runner-api.integration.test.ts` 61/61 passed;
`runner-bridge-tools.test.ts`, `task-document-key-contract.test.ts` and
`runner-api-artifacts.test.ts` passed; runner
`protocol-action-contracts.test.ts` 70/70 passed; server `tsc --noEmit`
passed.
- `runner-api-artifacts.integration.test.ts` now saves through
`save_api_response` and checks that a write operation ID or a body is
refused. Its download step needs Linux; it fails on macOS at master too,
so CI covers it.
- Paid E2E, OpenCode + DeepSeek v4.1 flash, [run
43138806990](https://github.com/paperclipai/paperclip/actions/runs/43138806990)
on this fix: 5/5 affected cases passed
(questions-and-approvals.reject-and-revise,
steering.edit-queued-message, task-coordination.delegate-with-context,
tasks.edit-properties, workspaces.preview-service). Body-less write
refusals went from 30 to 0, and body-less POST 400s went from 48 to 0,
compared with the earlier cohort on the same cases. The approval resolve
call was sent once (it was sent 24 times before). Every logged write
carried a body; the remaining non-2xx writes are the cases' intended
stale-edit and failing-fixture responses.
- Run-history redaction also covers `save_api_response` arguments; a
regression test fails on the earlier source and passes now.

## Risks

- Agents that learned `call_api` + `saveResponse: true` in a running
session will see a schema refusal and must use `save_api_response`.
Saved calls keep working for recovery.
- The new POST/PUT refusal applies only when the JSON request schema
names required fields, so writes whose body is optional are still sent.
- One more small read tool is advertised when API tools are enabled.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool
use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:46:14 -06:00
DottaandClaude Opus 5.5 85bb942aad fix(native): prove an unconnected bootstrap from the sandbox's own Runner state on a volume-free server (#20992)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute through a Runner (`paperclip-runnerd`) in a
remote sandbox. When a Runner exits before `session.open`, the run
restarts its bootstrap on the same run.
> - #20972 lets host preparation restart that bootstrap when the Runner
wrote `runner-state.json` but no provider state. It needs a proof that
this run's own interrupted Runner is stopped or gone.
> - That proof is recorded only when the server still has the run's
local Runner controller root. A volume-free server, or a root that held
no identity, skips it. The live chaos wave after #20972 still failed
with `runner_harness_state_mismatch: resumed_unavailable`.
> - This pull request lets host preparation establish the same proof
from durable sandbox evidence when no local proof exists.
> - The benefit is that the restart also works on volume-free servers,
which is how Cloud runs.

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261112T002811-s6402` ran #20972 (Claude through
ACPX, Daytona warm, `--ephemeral-server`, K1/K2/K15 with
`--k15-fresh-home`, seed 6402, `--runner-bind-fault`). 5 of 6 tasks
needed Retry, with 6 `resumed_unavailable` failures. Run `fbb1666d` in
t01: the sandbox held `runner-state.json` for this run with `lifecycle:
connecting`, the same Runner instance and session,
`lastConnectionProtocolVersion: null`, and no provider state. That is
exactly the state #20972 restarts, but its proof was absent.

**Root cause**
`recordRemoteBootstrapRestartProof` is called only inside the
`existsSync(scoped)` block of the restart path in
`server/src/services/native-runtime/native-session-executor.ts`, after
the local controller identity is read. With no local root (volume-free,
a fresh home, or an emptied root), `provenOwnBootstrapRestart` stays
false, so `inspectOwnUnpreparedBootstrap` from #20972 returns nothing
and the resumed branch throws `resumed_unavailable`.

**Expected behavior**
A run with a `bootstrap_incomplete` recovery claim whose sandbox Runner
state proves it never connected restarts its bootstrap without Retry.

## What Changed

- `inspectOwnUnpreparedBootstrap` now runs for any
`bootstrap_incomplete` claim of this run. When no proof was recorded, it
records one only if the sandbox Runner state proves it never connected
(`remoteRunnerStateNeverConnected`: `lastConnectionProtocolVersion:
null`, `lastControllerCommandSeq: 0`, `reconnectCount: 0`, empty outbox
and processed commands, no pending terminal delivery). The exact Runner
process must also be stopped or proven absent by the existing
`stopStaleRemoteRunnerBeforeBootstrap`, which uses the Runner state's
own identity. All earlier conditions from #20972 still apply: same run,
Runner instance and session, a lifecycle other than `suspended`, no
failover backup and no archived rotation snapshot.

## Verification

- Red on master (#20972 merged): `pnpm --filter @paperclipai/server exec
vitest run src/services/native-runtime/native-session-executor.test.ts
-t "bootstraps only an untouched provider session"`. The new
`interrupted_bootstrap_unprepared_unrecorded` cells failed: resumed
lease `Error: runner_harness_state_mismatch: resumed_unavailable`,
created lease no restart.
- Green: those cells pass. `_connected` (a Runner state that connected,
no local proof) and `_unproven` (the stop is not proven) keep the
refusal. The full file passes (1409 of 1409). Server `tsc --noEmit`
passes.
- Chaos, same cell with `--runner-bind-fault`. Wave
`w-20261112T002811-s6402` (#20972 only): 1 of 6 passed, 6
`resumed_unavailable`. Wave `w-20261112T010800-s6402` (this change): 3
of 6 passed, 2 `resumed_unavailable`. In this wave, run `0a080e83`
restarted its bootstrap three times on the same sandbox: three
`.bootstrap-restart-*` roots were moved aside. The baseline wave on
master before #20972 (`w-20261111T230537-s6402`, no injected fault)
passed 2 of 6.

## Risks

- The new path runs only for a `bootstrap_incomplete` claim of this run
when the full harness check fails, so the normal path is unchanged and
adds no wait before `turn.start`.
- A Runner state that ever connected, received a command or holds outbox
data keeps today's refusal. The moved-aside root is kept for inspection.
- Not fixed here, and seen in both waves: every Runner on a sandbox
listens on one fixed port (43127 in all captures). A live Runner of
another run or session on the same sandbox therefore makes each relaunch
fail with `runner_ingress_bind_conflict`, until the bounded retry is
exhausted (`native_session_retry_exhausted`). A later attempt can still
end with `resumed_unavailable` or `resumed_not_suspended`. That needs
its own fix in predecessor retirement or port selection.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, test runs). Default context window,
extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20972, #20928, #20953, #20863.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:07:02 -06:00
Nicky Leach 45170b0cac Reuse Runner plugin capability checks (#20987) 2026-11-11 17:21:07 -08:00
DottaandClaude Opus 5.5 5cd5adfccc fix(native): link a person-owned agent blocker to the agent's open question whatever owner kind the model reports (#20969)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #20931 lets a person's answer continue a task that its agent blocked
while it waited for that answer
> - That link is recorded only when the blocker's unblock descriptor is
bound to the board
> - Codex reports such a wait as `paperclip_block` with `owner: {"kind":
"user", "name": "Board"}` and scope `task_wide`. The evidence classifier
treats only `kind: "board"` (not a PRP owner kind) as board-owned, so
the arbiter binds this person-owned blocker to the reporting agent, and
no link is recorded
> - So every Codex W2Q task still loses the answer
(`issue_not_in_progress`) and needs Retry
> - This pull request classifies a `user` owner as a person
(board-bound) and records the link from the reported owner kind, not
from the spelling of the owner
> - The benefit is that the person's answer reaches the agent whatever
owner shape the model uses

## Linked Issues or Issue Description

Refs #20931

**What happened?**
Restart-chaos wave w-20261111T230717-s5201 on master 9c56e546 (Daytona
warm, Codex, no fault): all 3 W2Q tasks (t01, t05, t09) failed. Each
continuation after the answer (runs 8864fa72, 814b2cd0, 757c1c8f) was
cancelled with `issue_not_in_progress (current status: blocked)`. Each
agent blocked with `owner: {"kind": "user", "name": "Board"}`, scope
`task_wide` (reason codes `missing_required_answer`,
`awaiting_codename_answer`). The decision was `task_wide_blocker_bound`
with the unblock owner `{agentId}`, so `questionsAwaitedByAgentBlock`
recorded no `awaitingInteractionIds`.

**What did you expect to happen?**
The answer continues the task, as it does for a board-bound blocker.

**Steps to reproduce**
1. A Codex agent asks a question; the person comments before answering.
2. The agent calls `paperclip_block` with a `user` owner and scope
`task_wide`.
3. The person answers. The continuation is cancelled with
`issue_not_in_progress`.

## What Changed

- `evidence-classifier.ts`: new `nativeBlockerOwnerKind` and an optional
`ownerKind` on the blocker assessment (`agent`, `user`, `system`,
`external`, legacy `board`, or `unspecified`). A `user` owner is now
board-owned, like `board`: a person resolves it, so it is not bound to
the reporting agent.
- `status-arbiter.ts`: both agent-reported blocker decisions
(`task_wide_blocker_bound`, `current_track_blocker_waiting`) carry
`blockerOwnerKind`.
- `blocked-question-answer.ts` and the status committer:
`questionsAwaitedByAgentBlock` records the agent's own pending,
undismissed questions when the reported owner is one a person resolves:
`user`, `external`, legacy `board`, or no recognized kind. `agent` and
`system` owners and dependency blocks
(`durable_dependency_blocker_bound`) stay excluded. A decision without
`blockerOwnerKind` keeps the previous board-bound rule. All other guards
are unchanged: asked by this agent, answered by a person, decision still
current, `statusVersion` match, guarded checkout.
- `doc/execution-semantics.md` describes the rule.

## Verification

- `server/src/__tests__/heartbeat-blocked-question-answer.test.ts` now
builds the block decision with the real `classifyNativeEvidence` and
`arbitrateNativeStatus` from the exact Codex report shape, commits it
with `commitNativeStatusDecision`, and delivers the answer through
`heartbeat.wakeup`.
- Red on master (product hunks removed, test kept): `user` task-wide
fails because the unblock owner is `{ agentId }` instead of `board`;
`external` task-wide fails with `status: cancelled, errorCode:
issue_not_in_progress`.
- Green: 10/10. `user` task-wide, `external` task-wide and `user`
current-track are delivered. `agent` and `system` owners still cancel
with `issue_not_in_progress`, as do the earlier negative cases.
- `cd server && pnpm exec vitest run
src/services/native-runtime/status-arbiter.test.ts
src/services/native-runtime/evidence-classifier.test.ts
src/services/native-runtime/historical-task-questions.test.ts
src/modules/run-dispatch`: 257 passed. `recovery-observability`,
`heartbeat-issue-liveness-escalation`, `native-completion-feedback`,
`paperclip-control-plane-port`: all passed.
- `cd server && pnpm exec tsc --noEmit -p tsconfig.json`: passes.
- Chaos: see the comment below.

## Risks

- A task-wide blocker that names a person (`user`) no longer binds its
unblock descriptor to the reporting agent and no longer wakes that agent
about its own blocker. The board owns it, as for a current-track
blocker.
- An `external` owner can now link to the agent's open question. If that
blocker was about something else, the answer resumes the agent and it
reports the blocker again.
- Upgrade: `ownerKind` and `blockerOwnerKind` are optional. A decision
built by an older release keeps the previous rule. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:50:00 -06:00
DottaandClaude Opus 5.5 dfd2a6efca fix(native): restart a follow-up the closing server cut before it attached, instead of reattaching to its predecessor's Runner (#20979)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs on a remote sandbox keep a warm Runner between turns. A
follow-up run takes over its settled predecessor's provider session and
Runner.
> - When a deploy stops the server, a follow-up that is still preparing
fails with `local_runtime_api_listener_unavailable`. The executor saves
this failure as a same-run bootstrap retry (#20786). The saved failure
proves that the run did no provider work of its own.
> - After the restart, restart recovery saw a checkpoint and a provider
session, and chose `reattach_remote_runner`. But the checkpoint and the
Runner identity belong to the predecessor. The Runner identity check
then failed with `native_remote_recovery_lease_mismatch:
stage=runner_verification field=identity_run_id`. The run ended terminal
and the task needed Retry.
> - This pull request recognizes that durable shape in restart recovery
and restarts the run as a fresh bootstrap on its warm session. It does
not try to reattach to a Runner that belongs to a different run.
> - The pull request also stops the queue from claiming a queued run
after shutdown closed run admission during the queue scan.
> - The benefit is that a deploy during a follow-up's preparation no
longer ends the run terminal with a Retry.

## Linked Issues or Issue Description

Refs #20786, #20928, #20852, #20904

**What happened?**
In three Codex runs across two Daytona restart-chaos waves (warm
sandboxes, CR1 and K15 with overlap), a warm follow-up run
(`automation`, `retryOfRunId` = its predecessor) started 0–2 s before
its server got SIGTERM. Its bootstrap failed with
`local_runtime_api_listener_unavailable` and saved a `bootstrap_retry`
with `providerSessionEstablished: false`, `providerEventsExist: false`.
After the restart, `native.recovery.transition` chose
`reattach_remote_runner`. `verifyRemoteRunnerReattachment` then threw
`identity_run_id`, because the warm Runner's identity still named the
predecessor. The run ended with
`native_continuation_requires_reconciliation`. In every sandbox capture
the Runner was cleanly suspended, with no stderr.

**Expected behavior**
The run did no provider work and never wrote its own Runner identity.
The next server must restart it without a Retry.

**Steps to reproduce**
1. Finish a remote warm native run, so that its provider session and
Runner identity are saved.
2. Start a follow-up run. Admission carries the predecessor's provider
session and Runner process identity into the follow-up.
3. Send SIGTERM while the follow-up prepares, so that it fails with
`local_runtime_api_listener_unavailable`.
4. Restart the server. Restart recovery chooses
`reattach_remote_runner`, and the run fails with `identity_run_id`.

## What Changed

- `native-restart-recovery.ts` (`claimNativeRestartRecoveries`): add the
`remoteFollowUpNeverAttached` proof. It is true only when all of these
are true:
- the run is a failed, settled `bootstrap_retry` whose original failure
is `local_runtime_api_listener_unavailable`, with
`providerSessionEstablished: false`, `providerEventsExist: false`, no
successor and no lease owner;
- the run is on a prepared remote sandbox workspace with the current
preparation fencing version, it is not a plugin task, and it has no
Runner controller checkpoint and no staged attachment of its own;
- its session checkpoint is only the carried admission checkpoint (no
cursor, turn, result, terminal or pending request), and a settled
earlier run of the same session saved that exact provider session;
- its Runner process identity is absent or is exactly the identity of a
settled predecessor;
- it has no provider event, no Runner-originated event, and no
`native.process_identity_recorded` event of its own.
With that proof, the run gets a `bootstrap_incomplete` claim with the
reason `remote_follow_up_never_attached`, and it passes the same
current-admission check as the other settled bootstrap retries.
- `verifyRemoteRunnerReattachment` is unchanged. It still refuses a
Runner that names a different run.
- `heartbeat/queue.ts` (`startNextQueuedRunForAgent`): check run
admission again before each claim. A run that a scan reaches after
shutdown closed admission stays queued for the next server.
- Tests:
- `native-runner-restart-recovery.integration.test.ts`: eight cases for
the staging shape. Three restart (inherited Runner identity; inherited
identity plus a Runner request; a Runner request with no identity). Five
stay held (own recorded Runner identity; a Runner-originated event; a
carried active turn; a provider session that no settled run saved; a
saved failure that established a provider session).
- `queue.test.ts`: a queued run stays queued and is not executed when
admission closes during the scan.

### How this relates to #20852 and #20904

- #20852 hands a preparing run to the next controller and persists
launch intents, so that preparation itself survives a deploy. It does
not change restart recovery classification for a settled
`bootstrap_retry`, and it does not touch the run-admission gate. This PR
covers runs that already failed and saved a `bootstrap_retry` (including
state that older releases saved), so the two are complementary.
- #20904 changes startup readiness for native-controller preservation
obligations. It does not touch either code path in this PR.

### Finding about the "draining server" admission

In both waves the follow-up was claimed before the process observed
shutdown. In CR1 (seed 9511) the run was created at 21:54:04.695Z and
started at 21:54:04.854Z; the server logged "shutdown signal received"
at 21:54:05 (the harness sent SIGTERM at 21:54:03.256Z, under load
average 10). In the K15 overlap (seed 9513) the runs started at
23:31:10.9Z and 23:31:11.5Z, and the old server got SIGTERM at
23:31:13Z. During the overlap the old server is fully live and has no
signal that a newer deploy serves traffic. So the existing admission
gate (`closeProcessRunAdmission`) was not bypassed. The only gap in the
gate was the check-then-claim window inside
`startNextQueuedRunForAgent`, which this PR closes. Holding work on a
superseded but not yet signaled server needs a new "superseded deploy"
signal; this PR does not add one.

## Verification

- Red on master (`23909077ad`): `node cli/node_modules/tsx/dist/cli.mjs
packages/db/src/test-postgres-command.ts -- node
node_modules/vitest/vitest.mjs run --project @paperclipai/server
server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "closed the runtime listener before it attached"`: the three restart
cases failed with `"kind": "reattach_remote_runner"` (the staging
disposition); the five held cases passed.
- Red on master: `queue.test.ts -t "closes admission during the queue
scan"` failed (`expected [ { …(81) } ] to deeply equal []`).
- Green with the fix: both filters pass.
- `native-runner-restart-recovery`, `native-runner-restart-preparation`,
`native-startup-retry` and `native-controller-recovery` integration
files: 267 passed, 14 skipped.
- `queue.test.ts`: 89 passed.
- `cd server && pnpm typecheck`: passed.
- Chaos waves: see the PR comments for the reruns of seeds 9511 (CR1)
and 9513 (K15 overlap).

## Risks

- The new proof applies only to the closed-listener `bootstrap_retry`
that #20786 writes, on a remote sandbox, with a carried session that a
settled predecessor saved. Every other shape keeps its earlier
classification.
- A `bootstrap_incomplete` claim sends the run through the existing
bootstrap path, which verifies the prior owner of the session root
before it starts. The s9205 follow-ups already use this path with an
inherited Runner identity.
- Upgrade safety: no persisted shape changes. The proof reads only
fields that the current and recent releases write. A row saved without
these facts keeps the earlier verdict.
- No wait or file work is added before `turn.start`. The new queries run
only in restart recovery.
- The queue change only stops claiming after admission closed. Runs stay
queued for the next server, the same as the existing entry check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:36:45 -06:00
DottaandClaude Opus 5.5 66344b76ae fix(native): restart a bootstrap whose Runner exited before writing provider state on a resumed sandbox (#20972)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute through a Runner (`paperclip-runnerd`) inside a
remote sandbox. A warm Daytona sandbox is reused by later runs.
> - When a Runner exits before `session.open`, the run does a bounded
same-run bootstrap retry. Durable evidence proves that no provider
session and no provider event exist.
> - The retry fails when the Runner wrote `runner-state.json` but exited
before it wrote its provider state. Host preparation then calls the
sandbox state "unavailable" and refuses with
`runner_harness_state_mismatch: resumed_unavailable`. The task needs
Retry.
> - This pull request reads the Runner state on its own when the full
check fails, but only for a proven bootstrap restart of the same run. A
non-suspended Runner state of this run is then the run's own interrupted
bootstrap, and the existing restart path moves it aside and claims a new
session root.
> - The benefit is that a Runner that exits before `session.open` on a
reused sandbox recovers without a person. Saved state, a verified backup
and every other mismatch keep their refusal.

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261111T230537-s6402` (Claude through ACPX,
Daytona warm sandboxes, `--ephemeral-server`, K1/K2/K15 with
`--k15-fresh-home`, seed 6402) needed Retry on 4 of 6 tasks, with the
same signature after each restart kind. The first attempt of each run
reused the warm sandbox. Its Runner failed with
`runner_ingress_bind_conflict: failed to bind listener: Address already
in use` and exited with `lifecycle=connecting`. Recovery logged
"Restarting an incomplete native runner bootstrap on the same heartbeat
run". The second attempt reused the environment and failed with
`runner_harness_state_mismatch: resumed_unavailable`. The earlier wave
`w-20261111T212429-s6402` had one run with the same error code.

**Root cause**
`inspectRemoteHarnessState` in
`server/src/services/native-runtime/native-session-executor.ts` checks
the persistence directories, `runner-state.json` and the provider state
file in one shell test. A Runner that exits at bind time wrote
`runner-state.json` but no provider state, so the one test fails and the
result is `incompleteReason: "unavailable"` before the Runner state is
read. The restart path (`restartableBootstrapState`) accepts only
`not_suspended` and `provider_unopened`, so it is skipped.
`claimUntouchedRemoteSession` then cannot create the session root that
the first attempt created, and the resumed branch throws
`resumed_unavailable`. On a created lease the same state fell through
without a restart.

**Expected behavior**
A proven, interrupted bootstrap of the same run restarts on the same
sandbox. The run finishes without Retry.

**Steps to reproduce**
Run the s6402 cell above, or add `--runner-bind-fault` (this pull
request) to force the Runner exit on every run's first launch.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: when
the full harness check fails during host preparation,
`inspectOwnUnpreparedBootstrap` reads `runner-state.json` alone. It
returns `not_suspended` for this run only when all of these are true:
this run has a proven bootstrap restart (`bootstrap_incomplete` recovery
plus the stop proof), no failover backup exists, no archived rotation
snapshot is in use, and the Runner state names this run, this Runner
instance and this session with a lifecycle other than `suspended`.
Otherwise the result stays `unavailable`, as before. The existing
restart path then moves the session root aside (it is kept, not deleted)
and claims a new one.
- Chaos coverage: `--runner-bind-fault` in `tests/restart-chaos` gives
every server of the wave
`PAPERCLIP_TEST_RUNNER_BIND_FAULT=first_launch`. With `NODE_ENV=test`
only, the remote Runner launcher starts a short-lived listener on the
Runner's listen port before a Runner's first launch on that server, so
the Runner exits with a real `runner_ingress_bind_conflict`. Documented
in `tests/restart-chaos/README.md`.

## Verification

- Red on master: `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session"` failed 2 of the new
cells. Resumed lease: `Error: runner_harness_state_mismatch:
resumed_unavailable` (the staging error). Created lease: no restart
happened (`expected [] to have a length of 1 but got +0`).
- Green: the same cells pass. The negative cells keep
`resumed_unavailable`: no restart proof, a Runner state of another run,
a `suspended` Runner state with missing provider state, and a failover
backup present.
- `native-session-executor.test.ts`: 1406 of 1406 pass. A new test
proves the bind fault holds the listen port (a second bind gets
`EADDRINUSE`) and fires once per Runner instance.
- `tests/restart-chaos`: `runner-bind-fault.test.ts`,
`question-handoff.test.ts`, `shape-cells.test.ts` pass. Server and
harness `tsc --noEmit` pass.
- Chaos wave: see the comment below for the before/after counts.

## Risks

- The new read runs only for a proven bootstrap restart of the same run
with a failed full check, so the normal path has no extra command and no
wait before `turn.start`.
- A partly saved session (`suspended` Runner state without provider
state), a Runner state of another run and any failover backup keep
today's refusal. The moved-aside root is kept for inspection.
- A Runner that started under an older server and is restarted by this
version takes the same path, because it reads only files that older
Runners already write.
- The bind fault is test instrumentation. It is inert unless
`NODE_ENV=test` and the variable are both set.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, test runs). Default context window,
extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #20928 (restart a Runner that exited before session.open),
#20874, #20863, #20953.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:36:33 -06:00
Nicky LeachandPaperclip 38501b1170 Add turn-scoped attachment downloads to native Runner (#20956)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native agents open input images with a file reader.
> - Plugin Runner tasks use an authenticated Runner connection.
> - Runner needs a way to fetch an authorized file from the tenant app.
> - The tenant grants access to the exact attachment for the active run.
> - Runner downloads and verifies it before it returns a local path.

## Linked Issues or Issue Description

Refs #19740. Builds on the attachment authorization and command
transport added in #20939.

## What Changed

- Add the PRP v3 `runner.attachment.download` command for native
workspace providers.
- Issue short-lived grants for exact authorized wake and historical
attachments. Recheck the active run, source, size, and SHA-256 at the
tenant endpoint.
- Download current-wake inputs before provider dispatch. Download
historical files when the authorized chat reader asks for them. Preserve
SSH and sandbox command staging as a fallback.
- Run HTTPS transfers off the Runner control loop. Report completion
with one PRP event per transfer. Drop pending results on cancellation
and clear files at turn settlement, authority rotation, and restart.
- Grant restricted Codex sessions read-only access to the attachment
directory.

## Verification

- 69 targeted server attachment tests passed. They cover wake and
historical downloads, authorization failures, and command fallback.
- 22 focused historical attachment tests passed, including grant
redemption under a held database lock.
- Six focused Rust attachment tests passed, including a source-pinned
Codex sandbox check. The sandbox could read the staged file but could
not modify it or read adjacent Runner state.
- The real Runner transport test confirmed one durable command and one
completion event for a transfer. Restart-continuation tests confirmed
fresh attachment paths appear before dispatch.
- Server and Runner TypeScript checks, Rust formatting, and PRP
generated schema checks passed.
- Required GitHub CI passed on `7a33752bd7`. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/43122967837)
- Greptile scored the same revision 5/5 with no unresolved review
threads.
[Review](https://github.com/paperclipai/paperclip/pull/20956#issuecomment-6808582134)

## Risks

- Saved provider and Runner identities keep their existing formats. A
previous Runner falls back to command staging where the target supports
it. A restart continuation refreshes temporary attachment paths before
dispatch.
- Plugin Runner tasks require a compatible Runner ROM and outbound HTTPS
access to the configured tenant origin.
- Live agent-computer qualification remains a release check.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, shell tools, and
local test execution. Exact variant and context window are unavailable
in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #123` /
`Closes #123` / `Refs #123` or (b) described the issue in-PR following
the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 16:28:02 -08:00
DottaandClaude Opus 5.5 87fe4d9793 fix(workspaces): retry a workspace_finalize record write that loses a deadlock (#20971)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a run ends, Paperclip records a `workspace_finalize` workspace
operation. Dependent work reads that record to know if the workspace is
safe to use.
> - The record insert has foreign keys to `execution_workspaces`,
`companies`, `heartbeat_runs` and `issues`. Postgres takes a key-share
lock on each parent row, in trigger order: workspace, company, run,
issue.
> - The native finalizer and other writers lock the issue row and then
the run row (the order that #20819 documents). When such a transaction
holds the issue and then asks for the run, while the finalize insert
holds the run and waits for the issue, Postgres detects a deadlock and
aborts one side.
> - In restart-chaos wave `w-20261111T205259-s6301` (task t01, run
`e7129ec7`) the insert lost. The log says "failed to record
workspace_finalize=failed operation; dependents may remain gated". The
failed-finalize record was lost.
> - The insert's lock order comes from the foreign keys, so the code
cannot reorder it without a new explicit lock order that other writers
(for example the remote-workspace release path, workspace then run)
could invert again. A deadlock victim is safe to run again, because its
transaction rolled back completely.
> - This pull request retries the finalize record write on lock timeout,
deadlock and serialization failure (55P03, 40P01, 40001), with the same
bounded policy that #20850 added for the provider-pack admission write.
> - The benefit is that a short lock conflict can no longer delete the
record that releases dependent work.

## Linked Issues or Issue Description

**What happened?**
Run `e7129ec7` failed its native session recovery
(`runner_harness_state_mismatch: backup_without_reusable_lease`). The
failure path called `recordWorkspaceFinalize("failed")`. The insert into
`workspace_operations` failed with `PostgresError: deadlock detected`
(40P01). The server logged that dependents may remain gated. The run
then settled as failed with `native_session_interrupted`; the next
`finalizeNativeRun` call logged `native_finalization_missing` because
that run had no proposed result.

**Expected behavior**
A transient deadlock does not lose the finalize record. The write runs
again and the record is saved.

**Steps to reproduce**
1. Transaction A: lock the issue row `FOR UPDATE`.
2. Start a `workspace_finalize` record for a run on that issue. Its
insert takes key-share on the workspace, company and run rows, then
waits for the issue.
3. Transaction A: lock the run row `FOR UPDATE`.
4. Postgres reports a deadlock. Before this change the insert is the
victim and the record is lost.

**Additional context**
The new test reproduces this with two real transactions on embedded
Postgres. It gives transaction A a longer `deadlock_timeout`, so the
insert is always the victim, as on the chaos run.

Refs #20850
Refs #20819

## What Changed

- New `server/src/services/transient-lock-contention.ts`:
`postgresErrorCode`, `isTransientLockContention` and
`retryTransientLockContention` (5 attempts, 100 ms × attempt backoff,
only 55P03, 40P01 and 40001).
- `workspaceOperationService.createRecorder().recordOperation` wraps its
`workspace_finalize` state writes (the running-row insert and the
terminal update) in that retry. Each attempt is a new transaction, or a
new savepoint when the service runs inside a caller's transaction.
- `native-provider-pack-admission.ts` uses the shared helper. Its
behavior and its exported `nativeProviderPackAdmissionFailureCode` name
do not change.
- New test in
`server/src/__tests__/workspace-operations-reconciliation.test.ts`: a
real two-transaction deadlock between an issue-then-run locker and the
finalize insert.

## Verification

- Red on master: `cd server && npx vitest run
src/__tests__/workspace-operations-reconciliation.test.ts -t "loses a
deadlock"` failed with `deadlock detected`, code `40P01`, on the same
`insert into "workspace_operations"` query as the chaos log.
- Green: the same test passes; I ran it 3 times. The full file and the
provider-pack admission tests pass (49 tests).
`native-result-commit-work.test.ts` passes.
- `npx tsc --noEmit -p server` passes.
- I stopped `pnpm verify` after it selected 2533 test files; required CI
supplies broad coverage.
- Chaos: no new cell. The existing W2 × K1 cell (Codex, Daytona warm)
reached this path when the restart fault broke native session recovery.
I did not rerun a wave; both wave slots were busy.

## Risks

- Low risk. Only the `workspace_finalize` state writes retry, and only
on the three contention codes. Other errors throw as before.
- A deadlock victim's transaction is fully rolled back, so the retried
insert (same operation id) and the retried update are idempotent.
- The worst added delay is about 1 s of backoff across 5 attempts, on
the failure path only.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:09:57 -06:00
DottaandClaude Opus 5.5 2539004201 fix(ephemeral): start a restarted volume-free server over the Runner controller files it wrote itself (#20952)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Cloud servers run volume-free (`PAPERCLIP_EPHEMERAL_SERVER=true`).
Postgres and object storage hold the durable state. A startup gate
refuses legacy volume data that was not migrated.
> - The gate reads one family of data from disk: Runner controller files
under `runtime/paperclip-runner/durable-sessions/`. It treats a
controller file that the database does not prove as legacy data.
> - A volume-free server also writes these files during normal work. A
warm follow-up restores the previous run's checkpoint as a full
`durable.control-plane-state.v2` file. The file stays full until the new
controller registers its database checkpoint writer.
> - When the server crashes in that window and restarts on the same
filesystem, the gate finds its own file and refuses to start. Every
later start refuses the same way. This is a full outage that no Retry
can fix.
> - This pull request gives the gate a durable marker. A volume-free
server claims an empty state directory before any controller can write.
Files in a claimed directory belong to a volume-free server and do not
block startup.
> - The benefit is that a crash restart of a volume-free server starts
again. Genuine legacy data keeps its protection, and the refusal now
names the run, task and file.

## Linked Issues or Issue Description

**What happened?**
The first restart-chaos waves in volume-free server mode (harness flag
`--ephemeral-server`, #20930) found this outage. Wave
`w-20261111T212429-s6402` (Claude through ACPX, Daytona warm sandboxes)
did a fresh-home deploy and then a `kill -9` of that server. The
replacement server exited with `Volume-free server data is not eligible:
native_controllers_require_migration`. A second start at +608 s exited
the same way. The server stayed down.

**Root cause**
`hasLocalNativeControllerObligations` in
`server/src/services/native-runtime/native-controller-migration.ts`
scans every Runner session in the state directory. The kept instance
shows the cause. The session's newest run `4a4d7ca1` was `running`
(admitted 3 s before the kill). Its controller root held the full
`paperclip.runner.durable.control-plane-state.v2` file of its warm
predecessor `9c13b7f8`. A controller writes that full file when it opens
and replaces it with the `external-control-plane-state.v1` marker only
in `setCheckpointPersistence`. `preservedActiveCheckpoint` accepts only
the marker, so the gate returned `true`. A Codex wave with the same
fault sequence passed because no warm follow-up was in this window at
the kill. Claude W2 cells start a new warm follow-up every 15 to 35 s,
so the window is open much more often.

**Can staging hit it?**
Yes, with a restart that keeps the container. Railway documents that a
manual Restart "restarts the process within the deployment's container",
so the filesystem stays. Railway does not document whether an automatic
crash restart reuses the container. A redeploy makes a new container, so
it does not hit this.

**Expected behavior**
A volume-free server starts again after any restart. State that it wrote
itself is disposable, because the database checkpoint is the durable
record. Data that a volume-backed server wrote keeps its current
protection.

## What Changed

- Add
`server/src/services/native-runtime/volume-free-controller-state.ts`.
`claimVolumeFreeControllerState(true)` writes `.volume-free-owner.json`
atomically (temporary file, fsync, rename, directory fsync) when the
state directory is absent or empty. It returns `owned` for a directory
that has the marker and `unclaimed` for a directory with other files.
`claimVolumeFreeControllerState(false)` removes the marker.
- `server/src/index.ts` calls the claim right after the volume-free
configuration check. This is before any controller or recovery can
write. A server started without volume-free mode releases the claim,
because it may write files that exist only on its disk. An `unclaimed`
result logs a warning.
- `hasLocalNativeControllerObligations` ignores the marker entries. It
returns no obligation for a claimed directory. It now delegates to
`findLocalNativeControllerObligation`, which returns the run, task and
file (or `database_checkpoint`).
- `assertEphemeralServerData` adds that detail to the
`native_controllers_require_migration` refusal. The error says which run
and file the database does not prove and why startup refuses.
- Document the marker in `doc/DEVELOPING.md` ("Runner controller state
on a restarted volume-free server").

## Verification

- Red, before the gate reads the marker: `pnpm --filter
@paperclipai/server exec vitest run
src/services/native-runtime/native-controller-migration.integration.test.ts
-t volume_free_owner` failed with `AssertionError: expected true to be
false` at the first gate check. The fixture is a running warm run whose
root holds a full `durable.control-plane-state.v2` file, the same shape
as the incident.
- Green: the same test passes. It checks: claimed directory gives no
obligation; release gives the obligation back; a non-empty unclaimed
directory stays blocked; the refusal names the run; the controller file
is never changed or removed.
- `ephemeral-server-data.test.ts`, `ephemeral-server-contract.test.ts`
and all 57 cells of `native-controller-migration.integration.test.ts`
pass. Server `tsc --noEmit` passes.
- Probe on a copy of the kept instance database and the killed server's
state directory: the unmodified gate returned an obligation for run
`4a4d7ca1` (source file
`.../7dbf2d5c.../control-plane/control-plane-state.json`).
- Chaos: the s6402 shape (`--ephemeral-server --adapters claude
--workloads W2 --faults K1,K2,K15 --k15-fresh-home --seed 6402`) runs on
this branch merged with #20930. The result is added in a comment.

## Risks

- A filesystem that a volume-free server claimed and that a pre-marker,
volume-backed server then wrote is not detected. This needs a downgrade
and a mode change on one disk. A current server in volume mode removes
the marker, which closes the normal path.
- A volume-free server on a directory written before this change (no
marker) keeps today's refusal. A new deploy makes a new container, so
staging claims the empty directory on first start after this deploy.
- Other scanners of the state directory filter on the 64-hex session
pattern, so the marker file does not change them.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, test runs). Default context window,
extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:59:57 -06:00
DottaandClaude Opus 5.5 aaf76a3ca6 fix(native): start the configured review stage when a native worker reports done (#20882)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks can have an execution policy with a review stage that names an
independent human reviewer
> - The issue route starts that stage when the assignee moves the task
to done or in_review
> - Native runs do not use that route. The native finalizer writes the
task status through the issue service, which does not run the stage
transition
> - So a native worker that reports verified work as done closes the
task without the required review. A worker that reports needs_review
moves the task to in_review without the configured stage or its reviewer
> - This pull request makes the native completion path start the
configured review stage, as the route does
> - The benefit is that a configured independent review gates Done for
native workers too

## Linked Issues or Issue Description

**What happened**

A task had an execution policy with one review stage for a human
reviewer. A native worker (OpenCode on the Paperclip runner) finished
the work and reported it. When the worker reported done with verified
evidence, the task became done and the review stage never started. When
the worker reported needs_review, the task became in_review with a
generic review card, `executionState` stayed null, and the configured
reviewer was not assigned. The chat eval `tasks.completion-with-gate`
timed out in setup because the task never reached the pending review
stage.

**Expected behavior**

Verified work on a task with a configured human review stage goes to
in_review with that stage pending, the stage reviewer assigned and the
worker kept as the return assignee. The task becomes done only after the
reviewer approves.

**Steps to reproduce**

1. Create a task assigned to a native agent with `executionPolicy.stages
= [{ type: "review", participants: [{ type: "user", userId: <reviewer>
}] }]`.
2. Let the agent finish the work and report done with verified evidence.
3. Read the task. Before this fix it is `done` and `executionState` is
null.

## What Changed

- `server/src/services/issue-execution-policy.ts`: add
`completionReviewStageEntry`. It applies the full policy transition that
the route applies for `status: "done"` and returns `starts` (with the
stage patch), `none` or `rejected`.
- `native-run-finalizer.ts`: a done decision on a task whose policy
starts a stage becomes in_review with reason
`execution_stage_review_required`. When the policy refuses the
transition, the decision is in_progress with reason
`execution_stage_transition_rejected` and a recovery note. It is never
done. Deliverable checks still run first.
- `status-decision-committer.ts`: for a stage-entry decision, apply the
stale-owner check, start the stage from the locked task in the same
status update (this also clears a monitor that the new assignee cannot
use), and wake an agent participant with `execution_review_requested` or
`execution_approval_requested`, as the route does. If the policy
changed, the saved decision is stale and finalization retries.
- `native-completion-feedback.ts`: on such a task, tell the worker to
report done so the configured stage requests the review. Reject
needs_review with no request or with any review request. Approval
requests keep their own path, because they can be governed action cards.
- New regression test `native-execution-policy-review-stage.test.ts`
(embedded Postgres, 6 tests).

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-execution-policy-review-stage.test.ts`
(6 tests): no stage (done), human stage (in_review, reviewer assigned,
monitor cleared), agent stage (in_review, reviewer woken), refused live
review cycle (in_progress, not done), and feedback including mixed and
approval requests. The stage tests fail without the finalizer change
(the task becomes done).
- Related suites pass: `historical-task-questions`,
`native-completion-feedback`, `native-run-finalizer-telemetry`,
`native-terminal-status-projection`, `status-arbiter`,
`task-monitor.integration`, `__tests__/issue-execution-policy` (247
tests).
- Server `tsc --noEmit` shows no errors in changed files. The local
worktree has unbuilt packages, so CI supplies the full typecheck.

## Risks

- The agent-reviewer wake is a system workflow wake, like the route's.
Unit tests prove the wake row. An end-to-end native agent review after
this wake is not yet exercised in a paid eval.
- A decision saved with a new reason and resumed by an older release
projects in_review or in_progress without the stage. The task does not
close, so the older release fails safe.
- Agents on tasks with a configured review stage now get a rejection
when they report needs_review with no request or with a review request.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:43:58 -06:00
DottaandClaude Opus 5.5 c45f54ee74 fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown

## Linked Issues or Issue Description

**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.

**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:

1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.

The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.

**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.

**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.

## What Changed

- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.

## Verification

- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches 7e8901aa60, not current master, so each wave
ran this change cherry-picked onto 7e8901aa60:
- `w-20261111T210049-s6111` (first commit only): the close reason is now
logged: `close failed: Daytona sandbox lease 192606aa… is no longer
active.` The delete stayed 409 with that reason. The new check failed
the wave correctly.
- `w-20261111T222826-s6112` (all commits): two W2Q owners were released
with `sandbox was already destroyed`. The environment was deleted. 0
sandboxes left running. The warm-shutdown check passed. No 409 from a
warm session.
- Not verified: a wave where the graceful close fails and the forced
Launcher stop succeeds on a live sandbox (that path is covered by the
unit test). The wave's task failures (`T:first_run_succeeded`,
`R3:retry_needed`) are other open defects. They are also on master.

## Risks

- Environment delete now stops an idle Runner whose close did not
settle. The delete already has the operator's consent to destroy those
sandboxes. The workspace-export and restore-source guards on sandbox
destruction are unchanged. Instruction files are collected after the
confirmed stop, as in #20844.
- An owner is released without contact only when a lease for the same
company, environment and sandbox has a valid `destroyed` termination
receipt. A `stopped` receipt is not enough, because a stopped sandbox
can be resumed.
- `closeIdleWarmNativeSessions` results have a new `failures` field.
Callers that read only the counts are unchanged.
- No persisted state changes. Warm owners live only in controller
memory, so older saved state and mid-run deploys are not affected.
- The new harness check can fail a warm restart-fault wave that leaves a
warm session unclosed at controller shutdown. That is a real leaked
Runner.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs, chaos waves) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:12:21 -06:00