Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-13 07:12:00 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,464 Commits 2,004 Branches 2,451 Tags
1fa7e92fa83087fbf427ff18c2d11fa0a4a21d85
Commit Graph
3852 Commits
Author SHA1 Message Date
DottaandClaude 08ea18085e fix(workspaces): restore a saved workspace into one copy of sandbox disk (#20695) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task on a remote (Daytona) sandbox continues from its saved
workspace. Before the agent starts, the server uploads the saved archive
into the sandbox and a sandbox script extracts it.
> - The extraction wrote a full second copy of the workspace beside the
uploaded archive. A 7.5 GB workspace on a fresh sandbox then needed
about 15 GB, and the restore failed with `stage=extract code=ENOSPC
disk=full`.
> - The failure was classified as retryable, so every automatic retry
and every Retry repeated it on a same-size sandbox. The task could not
continue.
> - On a retained sandbox it is worse: the old root, the archive, and
the new tree all share the disk.
> - This pull request makes the restore need about one copy of the
workspace. The sandbox removes only data that control-plane storage
already holds durably, and it never removes a root whose save the server
did not prove.
> - The benefit is that a workspace that fits once on the sandbox disk
restores. A disk that is too small for one copy now gets a named failure
that does not loop.

## Linked Issues or Issue Description

**What happened?**
A saved remote workspace of about 7.5 GB could not be restored on a new
Daytona sandbox. The run failed with `remote_workspace_restore_failed:
step=sandbox_restore; remote_workspace_operation_failed:
operation=restore exit=1; remote_workspace_restore_failed: stage=extract
code=ENOSPC disk=full exit=1`. The run marked the failure as retryable,
and each retry failed the same way.

**Steps to reproduce**
1. Save a remote workspace whose archive is larger than half of the free
sandbox disk.
2. Start the next run on another sandbox, so that the server restores
the saved archive.
3. The upload succeeds. The extraction writes the second copy and fails
with ENOSPC.

**Expected behavior**
A workspace that fits once on the sandbox disk restores. A disk that
cannot hold one copy gives a specific failure that is not retried
automatically, and Retry works after the sandbox has more space.

**What the disk holds during a restore (from the code)**
- The uploaded archive
(`<parent>/.paperclip-runtime/workspace-snapshots/<operation>/workspace.pcws`,
about the workspace size, because the archive is not compressed).
- The extracted tree (`<root>.restoring-*`, the workspace size).
- On a retained sandbox only: the old root, which stayed until after the
new root was published.
- After a restore that died: its partial tree and its archive, which no
later attempt removed.

Refs #20678 (the save side of the same full-disk class).

## What Changed

- `workspaceArchive` restore has a `consumeArchive` mode. It creates the
directories and symlinks first, then writes the file bodies from the
last to the first. After it verifies each file against the manifest
hash, it truncates the archive behind that file. The archive and the
tree together need the workspace size plus the largest file. The default
mode, which the server uses for local restores, does not change.
- The sandbox restore script uses that mode on a private rename of the
uploaded archive. A later upload to the same path cannot change the file
that is being consumed. The script removes the consumed file when it
stops.
- The helper's own staging is now inside the root-specific
`<root>.restoring-*` folder, not in a shared `.workspace-restore-*`
sibling.
- While it holds the root's restore lock, and after the epoch fence, the
script removes leftovers of an earlier attempt on the same root that
died: `<root>.restoring-*` folders, `.consuming` archives in its own
staging, and archive staging folders that a `restore-target.json` binds
to the same root. Each restore writes that binding after the reclaim
step.
- When the server proved that a retained root is its own earlier, fully
saved allocation (`replaceRetiredRoot` and a Paperclip receipt, as
before), the script removes the root's files before the extraction, not
after the publish. The receipt and `.paperclip-runtime` stay, so an
interrupted restore proves the same root again and continues.
- A full disk at any restore stage reports `reason=sandbox_disk_full`.
The existing named-refusal rule makes it non-retryable, so it goes to
the board recovery action and does not loop.
- `doc/execution-semantics.md` describes the disk need, the reclaim, the
early removal, and the new reason.

## Verification

- `cd server && npx vitest run
src/services/remote-workspace-transport.test.ts
src/services/remote-workspace-snapshot-archive.test.ts` (287 passed).
- `cd server && npx vitest run
src/services/remote-workspaces.integration.test.ts
src/services/remote-workspace-lifecycle.test.ts
src/services/remote-workspace-archive-worker.test.ts
src/services/remote-workspace-snapshot-archive-environments.test.ts`
(122 passed, 16 skipped).
- `cd server && npx tsc --noEmit -p tsconfig.json` (pass).
- The new tests run the real sandbox scripts against a folder with a
byte capacity. A `--require` preload makes each file write fail with
ENOSPC when the folder would exceed the capacity.
- Red on `origin/master`, green with this change:
- Fresh sandbox, room for the archive plus one file: master fails with
the exact production error `stage=extract code=ENOSPC disk=full exit=1`.
The fix restores every byte and leaves only the restored files.
- Retained root that the server proved, room for the old root, the
archive, and one file: master fails with ENOSPC. The fix restores and
keeps the runner state.
- Too small for one copy: the failure is `stage=extract
reason=sandbox_disk_full`, with no partial tree or archive left. A retry
with more space then restores.
- Both sandbox processes are killed mid-extraction (a restart): a retry
with the same identity and a retry with a new identity and epoch both
reclaim the leftovers and restore into the same capacity.
- The command dies right after it removed a proven root's files: the
next restore proves the root again from its kept receipt, restores, and
keeps the runner state.
- Integration test through `prepareRemoteWorkspace`: a full disk gives
`retryable: false` with `reason=sandbox_disk_full`. On master the same
case is `retryable: true`.
- Guards (pass on master and with the fix): a retained root without the
server proof, or without a Paperclip receipt, is refused with
`reason=occupied_root`, and none of its files are removed.
- Previous-release story: the archive format and the saved state do not
change. A partial `<root>.restoring-*` tree from the previous release is
reclaimed (test). Archive staging from the previous release has no
binding, so it is kept (test). The previous release's shared
`.workspace-restore-*` siblings cannot be bound to a root, so they are
kept.

## Risks

- The consumed archive cannot be read again. Each restore uploads a new
copy from control-plane storage, and the early-return path for an
already restored root does not read the archive. Every file is still
checked against the manifest, and the manifest is checked against the
saved digest.
- The reclaim step removes data only under the root's restore lock and
after the epoch fence. It removes only names that a restore of the same
root creates. Staging that names another root, or no root, is kept.
- The early removal of a retained root depends on the existing server
proof (`isRetiredOwnAllocation`) and the receipt check. If a restore
fails after the removal, the saved version in storage is the source of
truth, and the next restore proves the same root again.
- Files in a lower overlay layer of a resumed sandbox image do not free
space when they are removed. Such a sandbox can still be full.
- A workspace that is larger than the free disk after the upload, plus
its largest file, still fails. It now fails with the named,
non-retryable `reason=sandbox_disk_full`.
- No change to `packages/paperclip-runner`, the database schema, or the
API.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 22:19:43 -06:00
DottaandClaude 855e241887 fix(server): record a long event-loop block with the deltas that name its cause (#20709) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process owns the controller leases of its runs, renews
them on timers, and serves every request
> - On a hosted instance, the process stopped for about 103 seconds
without a restart. Every lease lapsed, and two runs needed a manual
Retry (the symptom is fixed in #20694)
> - Nobody can say what blocked the process. The server records no
event-loop delay, and it has no CPU, GC or memory record for that minute
> - This pull request logs one local line for each event-loop block of 2
s or more. The line holds the deltas that tell the possible causes
apart, and a likely-cause class
> - The benefit is that the next stall names its class (main-thread CPU,
GC, paging, CPU throttling, host suspend, or a blocking sync call) and
its exact time, so we can match it to the run timeline and fix the cause

## Linked Issues or Issue Description

Refs #20694

**What happened?**

On a hosted instance, the merged run-event timeline of all active runs
had no events from 08:19:52.04Z to 08:21:35.37Z (103.3 s). Events from
streaming runs arrived about every 0.5 s before and after the gap. The
process did not restart (the run diagnostic shows an observer uptime of
about 24 minutes when the reaper ran). The first event after the gap was
a span with a 60 s timeout that ended after 103.5 s. Thus the timer
callback ran about 43 s late. This agrees with a blocked event loop, or
with a frozen process. Nothing in the process recorded which one.

**Expected behavior**

A long event-loop block is a product failure. It lapses every lease and
freezes every user. The server must record each long block with enough
data to identify its cause.

**Steps to reproduce**

1. Start the server.
2. Block the main thread for more than 2 s, for example with a
synchronous read on a slow disk or a busy loop.
3. Before this change: the logs show nothing. After this change: the
logs show `event loop blocked for <n> ms (likely <cause>)` with the
measured deltas.

## What Changed

- `server/src/event-loop-stall-monitor.ts` (new): one unref'd 1.5 s
timeout, re-armed after each check, measures how late it runs. (Server
startup creates no interval timers, so it is not a `setInterval`.) When
a check is 2 s or more late, it logs once with these fields:
  - stall duration, start and end time (wall clock and monotonic clock)
- process CPU (user and system), GC pause time and count (`perf_hooks`
GC entries)
- major and minor page faults, file-system reads and writes, voluntary
and involuntary context switches (`process.resourceUsage()`)
- cgroup v2 `throttled_usec` delta (Linux; read asynchronously, best
effort)
  - RSS, heap and external memory
- `likelyCause`: `host_suspended`, `gc`, `memory_paging`,
`cpu_throttled`, `main_thread_cpu`, or `blocked_off_cpu` (sync file I/O,
a sync child process, or another off-CPU wait)
- `server/src/index.ts`: starts the monitor when the listener binds. It
stops the monitor at the start of shutdown, and when a failed startup
closes the listener. It writes the line with the existing `logger.warn`.
- This is a local log line only. It is not the Telemetry path and not
the Observability (OpenTelemetry) path. No data leaves the process.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/event-loop-stall-monitor.test.ts
src/__tests__/server-startup-feedback-export.test.ts` (131 passed; the
monitor file passed 5 runs in a row):
- an 800 ms busy loop is recorded as `main_thread_cpu`, with the stall
and CPU time
- an 800 ms `Atomics.wait` (blocks without CPU, like a sync read on a
stalled disk) is recorded as `blocked_off_cpu`
- nothing is logged while the loop is responsive, or after `stop()`
(injected clock)
- the file-read delta of a stall is kept while the cgroup read is
pending
  - server startup still creates no interval timer
- a 100 s stall with 90 s of cgroup throttling is recorded as
`cpu_throttled`, with the stall start and end times
  - each cause class is selected from its deltas
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes.
- CI supplies the broad server suites.

## Risks

- Low risk. The timeout reads only process counters. Every 10 s it reads
one small cgroup file asynchronously. It does not query the database or
the network, so it does not keep an idle instance or its database awake.
It is unref'd and stopped on shutdown.
- `process.cpuUsage()` counts all threads of the process, not only the
main thread. A busy libuv thread pool during a stall can make an off-CPU
block look like `main_thread_cpu`. The raw deltas are in the line, so a
reader can check this.
- If a platform suspends and resumes the process, each resume can log
one `host_suspended` line. This is expected and useful.
- The monitor names the class of a block, not the call stack. If the
class is not enough, the next step is a bounded CPU profile around the
stall time.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use, shell, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (the
module comment carries the contract)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 22:19:13 -06:00
DottaandClaude 200c6e40e2 fix(native): restart a bootstrap whose Runner suspended before the provider opened (#20691) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a native run's first provider start fails before any session
opens, Paperclip retries the bootstrap on the same run. Durable
controller evidence proves that no provider session or provider event
exists.
> - A remote sandbox keeps the Runner state inside the workspace. A
replacement sandbox gets that state back when the saved workspace is
restored.
> - When the provider refused `session.open` (for example an Anthropic
429), the Runner still suspended cleanly. Its state says `suspended`,
but the provider state has no session identity.
> - The same-run restart handled a Runner that never suspended. It did
not handle a Runner that suspended before the provider opened a session.
It failed with `runner_harness_state_mismatch:
provider_identity_incomplete (provider_not_suspended)`, and the task
showed "Recovery needed" without Retry.
> - This pull request lets that proven restart set the unopened session
aside and start the provider again.
> - The benefit is that a rate-limited provider start continues by
itself after the limit resets.

## Linked Issues or Issue Description

Refs #20627 and #20683 (other restore paths). This PR changes only the
same-run bootstrap restart.

**What happened?**

Two ACPX Claude tasks on remote Daytona sandboxes, on staging:

1. The first attempt sent `session.open`. ACPX rejected it with
`classification=provider_rate_limit` (Anthropic 429). The Runner exited
with `lifecycle=suspended`.
2. Paperclip recorded `recoveryMode: bootstrap_retry`,
`providerSessionEstablished: false`, `providerEventsExist: false`, and a
retry after the limit reset.
3. One hour later the retry ran as `bootstrap_incomplete` on the same
run, in a new sandbox with the saved workspace restored.
4. Host preparation inspected the restored Runner state. The Runner was
`suspended`, but the provider had no session identity. The inspection
threw `provider_identity_incomplete (provider_not_suspended)` before the
bootstrap restart branch ran.
5. The run failed. The task stopped at "Recovery needed" with only
inspect actions.

**Expected behavior**

The restart sets the unopened session aside and opens a new provider
session, as it already does for a Runner that never suspended.

**Steps to reproduce**

1. Restart a remote run as `bootstrap_incomplete` with a proven
bootstrap restart.
2. Give the sandbox a Runner state for this run with `lifecycle:
suspended` and a provider state with no session identity.
3. Prepare the host. On `master` it throws
`runner_harness_state_mismatch: provider_identity_incomplete`.

The new `interrupted_bootstrap_unopened` cases of `bootstraps only an
untouched provider session in a $outcome workspace lease` run these
steps for a created and a resumed lease.

**Paperclip version or commit**

`master` at `7435abcc63`.

## What Changed

- During host preparation, a Runner state that is suspended with no
provider session identity returns the new incomplete reason
`provider_unopened`, but only when all of these hold:
  - the run is restarting as `bootstrap_incomplete`;
  - the state belongs to this same run;
- the controller already recorded the bootstrap restart proof (its
durable journal shows no provider turn, and the old Runner is stopped or
absent).
- the provider state is the one saved before any session opened
(`remoteProviderStateNeverOpened`): `prepared`, no session or identity,
no turn in progress, no unconfirmed exit. Only ACPX and Codex-family
states qualify.
- The created-lease and resumed-lease restart branches share one check
(`restartableBootstrapState`) and accept `provider_unopened` exactly
like `not_suspended`. They move the old session aside and claim a new,
untouched session root.
- Every other case still throws `provider_identity_incomplete`: another
run's state, a restart without the proof, a normal resume, and close
checkpointing.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session"`: 47 passed.
- Red on `master`: the `created` and `resumed`
`interrupted_bootstrap_unopened` cases fail with
`runner_harness_state_mismatch: provider_identity_incomplete
(identity_binding)`. Green with this change.
- Fail-closed cases stay red-proof:
`interrupted_bootstrap_unopened_unproven` (no restart proof) and
`interrupted_bootstrap_unopened_other_run` (another run's state) still
throw.
- Full `native-session-executor.test.ts` and `pnpm --filter
@paperclipai/server typecheck` pass.

## Risks

- Low risk. The new branch needs the same proof that already authorizes
a fresh bootstrap for a Runner that never suspended. The proof comes
from the controller's durable journal, not from the sandbox.
- The normal path is unchanged and has no added wait.
- Not proven live: I did not confirm on staging that the controller
recorded the restart proof for these runs. If it did not, the run still
fails closed as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 21:35:38 -06:00
DottaandClaude 4baff83225 fix(chat): never batch messages of two chat conversations into one wake (#20697) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat channels let a linked user talk to an agent from GitHub, Slack
and other platforms. Each external thread is a chat conversation. Many
conversations can share one task.
> - A reviewed chat run attests exactly one conversation. #17054 keeps
each conversation in its own wake batch, but it compares only the
conversation that a wake records.
> - The work-intent outbox saves unanswered user messages as a plain
comment wake. That wake records no conversation, even when its messages
are chat messages. A wake with no recorded conversation matched every
conversation, so a message from another conversation merged into it.
> - The promoted run held messages of two conversations. The binding
check correctly refused it
(`reviewed_chat_execution_binding_not_authorized`, `binding_denied`),
and the task needed Retry.
> - This pull request compares the conversations of the queued messages
themselves. Wakes of different conversations never merge. Each
conversation gets its own run, in order.
> - The benefit is that replies in several threads of one task each get
an attested run, without a failed run or a Retry.

## Linked Issues or Issue Description

Refs #17054 (the earlier per-conversation batching fix). Refs #20689 (a
different shape of the same error).

**What happened?**
On staging, a GitHub pull-request review task had two runs fail with
`setup_failed` / `reviewed_chat_execution_binding_not_authorized`
(`reviewedChatBinding.denialReason: binding_denied`). Each failed run
held the same two older saved messages plus one new message. The two
runs recorded two different `chatConversationId` values. The work-intent
outbox had saved the older messages as plain deferred wakes with no
conversation. A new message from each conversation then merged into one
of these saved wakes.

**Steps to reproduce**
1. Bind two chat conversations to one `in_review` task.
2. Let the outbox save an unanswered message of conversation B as a
deferred wake while a run is active.
3. As the linked user, send a message in conversation A while the run is
still active.
4. See that the message of A merges into the saved wake of B. The
promoted run fails the binding check.

**Expected behavior**
The saved message of B runs as its own wake. The message of A runs after
it, in its own attested run.

## What Changed

- New
`server/src/modules/wake-queue/adapters/chat-conversation-batch.ts`:
reads the inbound chat links (`chat_message_links`, direction `inbound`)
of queued messages and finds messages of a conversation other than the
expected one.
- Deferred-wake lookup (`findExistingDeferredWake`): when the incoming
wake has a conversation, a saved wake qualifies only if none of its
queued messages is an inbound message of another conversation. This is
one more predicate in the same query (no new round trip). It uses the
existing `chat_message_links_company_comment_idx` index.
- Coalescing into the active run (`coalesceIntoActiveExecutionRun`):
returns no merge when the merged batch would hold messages of a
conversation other than the one it records. The wake is then saved
behind the run, which is the existing fallback.
- Release drain (`findNextDeferredWake`), upgrade repair: a saved chat
wake that older code already merged across two conversations is split.
The other conversation's messages move to a plain saved wake (the outbox
shape), ordered just before it. The original wake keeps its conversation
and its own messages. The drain then reads the queue again.
- `doc/execution-semantics.md`: describe the comparison and the upgrade
repair.
- New test `chat-wake-conversation-batch.test.ts` (embedded Postgres,
real admission and release code, real
`authorizeChatConversationForBoundRun`).

## Verification

- Red on `origin/master` (test file only): `pnpm --dir server exec
vitest run
src/modules/wake-queue/adapters/chat-wake-conversation-batch.test.ts`
gives 4 failed, 6 passed. The 4 failures are the bug: the new message
merges into the other conversation's saved wake (`[[earlier, reply]]`),
two conversations cross-merge, a message coalesces into a queued run of
another conversation, and the old merged shape stays merged. The 6
passing cases pin unchanged batching.
- Green with the fix: 10 of 10 pass. Each separated wake passes
`authorizeChatConversationForBoundRun` for its own conversation. The old
merged shape fails it with
`paperclip_runner_chat_attachment_binding_denied` before the split.
- Batching that stays the same (pinned, pass before and after): comment
wakes with no conversation merge; messages of one conversation merge; an
outbox-saved wake of the same conversation still absorbs the reply; a
chat message and plain comments merge in both orders; same-conversation
messages still coalesce into a queued run; two recorded conversations
stay apart.
- Related suites pass locally: `src/modules/wake-queue`,
`src/modules/run-dispatch`, `chat-run-milestone-work`,
`durable-chat-wakeup`, `heartbeat-comment-wake-batching`,
`heartbeat-reviewed-chat-binding.integration`,
`slack-conversation-lifecycle`, `queued-comment-consumption`,
`issue-queued-comments-routes` (23 files, 1037 tests).
`heartbeat-process-recovery`: see the comment below.
- `tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. No schema change and no migration.
- Behavior change only for chat messages: a message no longer joins a
saved wake or a run that holds another conversation's inbound messages.
It gets its own wake behind them, and it runs after them. Before this
change, that batch always failed the binding check.
- Board comments have no inbound chat link, so they belong to no
conversation. Wakes without a conversation, wakes of one conversation,
and mixed board-comment and chat wakes batch exactly as before.
- Ordering: the deferred queue still promotes one wake per release,
oldest first. The second conversation waits for the issue execution
lock. Single-assignee and atomic checkout are unchanged.
- Upgrade: a saved wake that older code merged across conversations is
split at promotion. No message is dropped or delivered twice: each
message id moves to exactly one wake. A run that older code already
promoted in the merged shape is not changed; it fails the binding check
as before and keeps its existing Retry path.
- No wait is added before a turn starts. Admission adds one predicate to
an existing query. Coalescing adds one indexed read only when the merged
batch has a chat conversation.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with
tool use (shell, file edits, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 21:15:40 -06:00
DottaandClaude Opus 5.5 d79b7f29f2 fix(heartbeat): bind queued card-answer interrupts to the delivered answer (#20698) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can queue direction while a task's run is active, then use
Interrupt to stop the run and start a successor that receives the
direction
> - The queued direction can be an answer to a card, for example an
accepted confirmation of a saved plan
> - `explicitOperatorRunIdentity` verifies the click from the coalesced
queue receipt before the successor run can start
> - That check required comment ids in the receipt, but a card answer
carries its interaction and no comment ids
> - So every claim of the successor failed with 403, and the direction
never reached a run
> - This pull request binds a card-answer receipt to the answer that the
successor run delivers, and to the user who gave it
> - The benefit is that Interrupt works for queued card answers, and the
operator identity stays verified

## Linked Issues or Issue Description

Refs #14043, #14510

**What happened**

A person accepted a `request_confirmation` card on a task while its run
was active, and then clicked Interrupt. The active run stopped
("Interrupted to send queued messages"). The successor run was then
cancelled with "Cancelled because the queued run cannot be claimed:
Queued-message interrupt authority is unavailable". The coalesced
receipt had the interaction id and no `queuedCommentIds`. The successor
run's context had the same `interactionId` and the receipt id in
`queuedCommentSourceWakeIds`. The chat eval
`steering.interrupt-with-direction` recorded this.

**Expected behavior**

The successor run starts, receives the accepted direction once, and runs
with the clicking user as its responsible user. A receipt that the run
does not deliver, or an answer that another user gave, still fails
closed.

**Steps to reproduce**

1. Start a run on a task.
2. While the run is active, accept a confirmation card on that task. The
answer goes into the deferred queue.
3. Click Interrupt for the queued messages.
4. Before this change, the successor run is cancelled with
`queued_run_claim_rejected`.

## What Changed

- `server/src/services/run-identity.ts`: the shared receipt binding
(consumed `coalesced` receipt for the same company, agent and run; same
task; `user` request from the marker's actor) runs first, as before.
Then:
- A receipt with comment ids keeps the existing rule: all ids must be
delivered to the run.
- A receipt without comment ids must be a queued card answer
(`queuedInteractionId`). The run context must deliver the same
`interactionId`. The interaction must be on the same company and task,
with the receipt's status, and `resolvedByUserId` must be the clicking
user. Then the run gets the operator identity
(`queued_comment_interrupt`). Otherwise the claim still fails with 403.
- `server/src/__tests__/heartbeat-queued-run-claim-isolation.test.ts`: a
successor of a card-answer interrupt is claimed, runs, and has the
clicking user as responsible user. Two negative cases: the run delivers
a different answer, and another user resolved the answer. Both stay
rejected.

Relation to #14510: that open PR fixes the same claim failure (#14043)
by returning `null` for card-answer receipts, so the run starts without
the operator identity. Its Risks section notes that keeping the operator
identity needs a check that the actor resolved the interaction. This PR
adds that check instead, so the successor keeps the identity that
enqueue time records. If maintainers prefer #14510's approach, this PR's
regression test still applies to it.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-queued-run-claim-isolation.test.ts`: the new
positive test fails before the change (successor cancelled) and passes
after it. The negative tests and the existing tests pass.
- Suites that cover interrupt identity, each run separately:
`run-identity` (37), `heartbeat-responsible-user-invariant` (18),
`issue-queued-comments-routes` (129), `deferred-execution-receipt-work`
(50), `task-interruption-query` (31), `explicit-native-continuation`
(436), `run-cancellation` (8), `workspace-recovery-admission` (147),
`wake-queue/adapters/postgres` (113): all pass. In
`heartbeat-process-recovery`, the interrupt-named tests pass except
`dispatches interrupted CLI input and cleans up early fixture exits`,
which fails before any interrupt (the CLI child does not start:
`controller_lease_lost`). It fails the same way on unchanged master on
this loaded host (3 of 3 attempts failed at least one case), so it is
not caused by this change. CI covers it.
- `server` `tsc --noEmit`: no errors in the changed files.
- Live chat eval
`chat-features.runner-opencode.local.steering.interrupt-with-direction`
at this branch (run 42938970466): passed. The old run stopped with
"Interrupted to send queued messages". Exactly one successor run
started, delivered the accepted card answer, ran with the clicking user
as responsible user, and saved the approved single page and its delivery
document. Before this change, the same case failed with the successor
cancelled as `queued_run_claim_rejected`.

## Risks

- Low risk. The change adds a verified path only for receipts that
previously always failed. Comment receipts keep their checks.
- If a different user answered the card than the user who clicked
Interrupt, the claim still fails. The person who answered can click
Interrupt.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 21:10:28 -06:00
DottaandClaude ce3881f1b3 fix(skills): release a skill cache lock whose holder's deployment was replaced (#20688) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Run preparation publishes skill revisions into a shared runtime
skill cache. A file lock serializes each publication. A lock whose
holder provably exited is released at once.
> - That check used `controllerProcessExited`, a PID proof. It fails
closed whenever the recorded process space (host, kernel boot, PID
namespace) differs from this server's.
> - Every Railway deploy starts the server in a new process space. A
lock left by a server that a deploy stopped mid-publish can never be
released. Each later preparation of that skill waits 60 s and fails with
"Runtime skill cache publisher is busy … remove it", until an operator
deletes the file.
> - The rest of recovery already uses one shared prover
(`proveOwnerGone`). Across process spaces it accepts a replaced platform
deployment on the same exclusive volume, or a stopped or fenced server
boot.
> - This pull request makes the skill cache lock use that prover. It was
the last caller of the separate PID-only proof.
> - The benefit is one rule for "the controller from a previous
deployment is provably gone", and no manual file removal after a deploy.

## Linked Issues or Issue Description

**What happened?**
A census of staging failures after deploys showed that
`controllerProcessExited` always fails closed after a deploy, because
the process-space hash changes. On `master`, the only remaining caller
is the runtime skill cache lock check. The restart chaos harness
restarts in the same process space, so the PID proof succeeded there and
hid the gap. I did not find this failure in the staging run census. It
is a latent dead end with the same cause as the census class.

**Expected behavior**
A lock whose holder ran in a deployment that the platform replaced on
the same exclusive volume, or whose server boot stopped or was fenced,
is released at once. Any other holder is still waited for.

**Steps to reproduce**
1. Run the server on Railway with a volume.
2. Redeploy while a skill revision is being published (the lock file
exists).
3. Prepare a run that needs the same skill revision on the new
deployment.
4. Observe a 60 s wait and `Runtime skill cache publisher is busy; retry
preparation.` on every attempt.

**Paperclip version or commit**
`master` at `c444da7d05` (the new test fails there).

## What Changed

- `server/src/services/runtime-skill-cache.ts`:
`runtimeSkillCacheLockHolderExited` builds a local owner stamp from the
recorded controller (boot ID, PID, process space, start time, and the
Railway deployment and volume when recorded). It returns true only for
the `gone` verdict of `proveOwnerGone`.
- Same-space PID rules do not change. A dead PID is gone, a live PID
with no start time is unprovable, and this server's own boot is alive.
- `server/src/__tests__/runtime-skill-cache.test.ts`: a new test for a
holder in another process space whose deployment the platform replaced.
- The proof is bounded by the lock wait (5 s, never past the deadline).
A timed-out or failed proof keeps the lock held. A proof still running
is joined by later waits on the same holder, so a stalled read adds at
most one database query per holder.
- `server/src/services/controller-process-identity.ts`: removes
`controllerProcessExited`, which has no caller left; its test assertions
now check `sameProcessSpace`.

## Verification

- `npx vitest run src/__tests__/runtime-skill-cache.test.ts
src/services/controller-process-space-identity.test.ts` (from
`server/`): 31 passed.
- New: "releases a lock whose holder ran in a deployment the platform
replaced". It fails on `master` (`expected false to be true`) and passes
with this change.
- The same test keeps the lock held for: the same deployment ID, another
volume, a non-exclusive volume, another service, and no recorded
deployment.
- The existing test "never treats age, another host, or a legacy holder
as proof that a lock holder exited" still passes. Without a deployment
or controller lease proof, another host stays held.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` selected 2504 of 3294 test files (the change reaches
`owner-proof.ts` importers) and hit my 30 min local limit with no
failure seen. Required CI supplies the broad coverage.
- Live proof: not done (tests only).

## Risks

- Across process spaces, a held lock can now trigger a read of
`controller_instances` for the holder's boot. This happens at most once
per second, and only while a lock is held. Uncontended preparation does
no extra work.
- The prover may fence a boot row whose lease lapsed past its grace, as
it already does for every other recorded owner.
- No persisted shape changes. Older lock files without a `controller`
field stay held, as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 20:59:49 -06:00
DottaandClaude a46b9ac724 fix(native): wait out an ACP usage limit's reported reset, then Retry the task (#20692) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs (Paperclip Runner with an ACP harness, such as Claude)
can stop because the provider account runs out of its usage allowance
> - The ACP harness reports this as a "terminal limit failure".
Paperclip blocked the task behind a board hold that called the session
state ambiguous, and the only way forward was a manual Retry
> - The provider also says when the limit resets, for example "resets
Nov 12, 6pm (UTC)" for a weekly limit. Paperclip read only clock-only
resets ("resets 2am"), so the agent-level wait fell back to one hour.
Each later wake for the agent then hit the limit again and blocked
another task
> - This pull request reads calendar resets, says on the hold when the
limit resets, and requests the task's Retry itself at that time when the
reset is reported and bounded
> - The benefit is that a usage limit no longer needs a person to click
Retry, and the agent's other tasks wait instead of failing one by one

## Linked Issues or Issue Description

**What happened?**
A native ACP run failed with `native_provider_usage_limit` ("ACP agent
reported a terminal limit failure. You've hit your weekly limit · resets
Nov 12, 6pm (UTC)"). The task went to `blocked` with a board-owned hold.
The hold said "Inspect the original provider failure and explicitly
resolve the ambiguous session state". Nothing resumed the task after the
reset. The agent-level `provider_limit_wait` recorded "no reset time was
reported, so Paperclip waits one hour", so every hour another wake
dispatched, hit the limit, and blocked another task. In one 48-hour
window on a staging instance, this blocked 7 tasks for one agent.

**What did you expect?**
When the provider reports a bounded reset, the task waits and continues
after the reset, like the existing `provider_limit_wait` path. When the
reset is unknown, the task fails with a clear message and a working
Retry.

**Steps to reproduce**
1. Run a native ACP agent until its provider reports a weekly or session
limit with a reset time.
2. Look at the task: it is blocked with an "ambiguous session state"
hold.
3. Wait past the reported reset: the task stays blocked until a person
clicks Retry.

Refs #20666 (other recovery work in the same staging census; not changed
here)

## What Changed

- `provider-quota-reset.ts`: parse calendar resets such as "resets Nov
12, 6pm (UTC)", "resets Jul 17th at 4:30 PM (America/Chicago)" and
"Resets on August 2, 9am (UTC)". A past date rolls into next year. An
impossible date (Nov 31) is refused. Clock-only parsing is unchanged.
This also fixes the agent-level `provider_limit_wait` reset and the
legacy `provider_quota` reset for these messages.
- New `native-runtime/provider-limit-resume.ts`:
- `readNativeUsageLimitReset` reads the reset only from the run's own
durable events. The last turn event must be a provider-observed
`turn.failed` with a `rate_limit` or `quota_exhausted` provider failure,
and the reset must be at most eight days out.
- `requestProviderLimitResumeRetry` asks for the task's Retry
(`requestTaskRetry`, `initiatedBy: "provider_limit_reset"`) on behalf of
the user who requested the run, or the run's accountable user for
automatic wakes. It is idempotent per run and records its outcome on the
hold. At most three automatic Retries follow one another without a
successful run.
- `registerProviderLimitResumeWork` is a scoped deadline owner (shared
work scheduler, no polling). Each active usage-limit hold is due at its
reset. Startup rebuilds it from the holds, and the executor notifies it
in the hold's transaction.
- `native-session-executor.ts`: for a terminal usage limit, the hold and
run failure detail say when the limit resets and whether Paperclip
retries then, instead of "ambiguous session state". The evidence records
`providerLimitResetAt`.
- `task-retry.ts`: accept `initiatedBy: "provider_limit_reset"`. Its
activity is recorded as system actor `provider-limit-reset` on behalf of
the user, like `restart_recovery`.
- `doc/execution-semantics.md`: document the wait and its limits.

Previous-release story: holds saved by the previous release carry
forward. At startup the owner reads each active
`native_provider_usage_limit` hold. It rewrites the hold's message with
the reset when the reset is known and still ahead, and it requests the
Retry at the reset. Holds whose reset is unknown, more than a day in the
past, or that have no accountable user keep their Retry and are recorded
as not requested, so they are not read again.

## Verification

- `npx vitest run server/src/services/provider-quota-reset.test.ts` — 4
tests. Three fail on `origin/master` code (calendar resets return
`null`), all pass with this change.
- `npx vitest run
server/src/services/native-runtime/native-provider-capacity.integration.test.ts`
— 5 tests on embedded Postgres:
- the existing usage-terminal test (both replay modes) still blocks with
a Retry, and no automatic Retry is requested without a reported reset;
- "waits for a reported usage-limit reset and then requests the task's
Retry once": the hold says when the limit resets, an old-release hold is
rewritten, nothing starts before the reset, one Retry receipt is created
for the accountable user at the reset, and a second pass does nothing.
Fails on `origin/master` executor code;
- "starts the Retry from the hold's own deadline": a real delivery
coordinator rebuilds the deadline from the hold and calls Retry dispatch
once, after the reset;
- "leaves a visible Retry when no user is accountable for the run".
Fails on `origin/master` executor code.
- Focused regression suites pass (642 tests): `task-retry`,
`quarantined-task-retry`, `heartbeat-native-restart-suspension`,
`native-startup-retry.integration`, `provider-failure-classification`,
`heartbeat-provider-limit-wait`, `deferred-execution-receipt-work`,
`chat-run-publications`, `native-runner-restart-recovery.integration`,
`native-remote-retry-admission`.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes.
- `pnpm verify` selected a broad run (exit 2, because the docs file is a
global CI input). CI supplies the broad coverage.

## Risks

- The automatic Retry uses the accountable user's authority for
automatic (system) wakes. It is the same Retry the hold already offered
that user, and dispatch still checks Retry authority, stop proof, saved
work and action reconciliation. It is never broader than a click.
- A provider that keeps reporting resets that do not restore capacity
gets at most three automatic Retries in a row; then the Retry stays
visible.
- The deadline owner reads all active usage-limit holds at startup
(limit 500 rows). The number of such holds is small.
- Not changed: the agent-level wait still holds new wakes for at most
one day, as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only staging API reads). Extended
reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 20:57:03 -06:00
DottaandClaude 8c0dbffa70 fix(heartbeat): retry a legacy run whose lease this process lost before dispatch (#20694) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat orphan reaper settles legacy (non-native) runs that
lost their controller lease
> - #16355 taught the reaper to retry a legacy run that a restart
stopped while setup was still `preparing`, because revocation fences the
old controller and `preparing` proves no adapter dispatch
> - That proof was applied only when the lapsed lease belonged to a
different boot
> - A stall can make the current process lose its own lease while setup
is still `preparing`, and its executor can end without settling the row
> - The reaper then wrote "retrying once" but recorded no bootstrap
evidence, so the retry was refused and the task was blocked for
reconciliation
> - This pull request applies the same revocation proof to a lease from
the current boot
> - The benefit is that these tasks recover by themselves instead of
waiting for a person to press Retry

## Linked Issues or Issue Description

Refs #16355

**What happened?**

On a hosted instance, one server process stalled for about 100 seconds
(lease renewals were 42–46 s late). Several legacy `paperclip_runner`
runs on a remote sandbox were still in setup (`executionStage:
"preparing"`, no process id). Their controller leases expired. Some
executors settled their own runs as `controller_lease_lost`. Two runs
had no executor left, so the orphan reaper of the same process settled
them 4 minutes later. Each run got `error: "Process lost -- server may
have restarted; retrying once"` and
`processLossDiagnostic.retryEligible: true`, but no `executionRecovery`
evidence. `enqueueProcessLossRetry` then saw a run that needs
reconciliation and scheduled nothing. The next sweep recorded
`legacy_execution_requires_reconciliation` with
`preserve_without_replay_v1`, and both tasks stayed `blocked` until a
person pressed Retry.

**Expected behavior**

Revocation commits the stage. Renewal requires the old boot id, so no
controller can move a revoked run to `dispatching`. A revoked run that
is still `preparing` with no process identity did no provider work,
whichever boot owned the lease. The reaper should record the bootstrap
evidence and use the bounded retry.

**Steps to reproduce**

1. Claim a legacy run in this process: `controllerBootId` is the current
boot, `executionStage: "preparing"`, no process id.
2. Let its controller lease expire with no local executor (for example
after an event-loop or database stall).
3. Run the orphan reaper.
4. Before this change: `process_lost; retrying once`, no
`executionRecovery`, no retry, and a reconciliation hold. After this
change: bootstrap evidence and a `scheduled_retry` successor.

## What Changed

- `server/src/services/heartbeat/recovery.ts`:
`restartInterruptedBeforeDispatch` no longer requires the lease to come
from another boot. The other clauses stay: legacy runtime, a stamped
controller boot id, and `preparing` with no process id or group in the
state committed by the revocation. Locally tracked runs and runs with a
live lease are still skipped before this point.
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: two cases
in the hosted shape (assignment wake and due-monitor wake,
`paperclip_runner`, prepared remote workspace, same-boot expired lease,
`preparing`). They expect bootstrap evidence, a scheduled retry, no
recovery hold after a second sweep, and no adapter call. A control case
keeps a same-boot `dispatching` run held for reconciliation.

## Verification

- Red on `origin/master` (source change reverted): both new cases fail.
The monitor case shows the exact hosted shape: `retryEligible: true`
with no `executionRecovery`.
- Green with the change: `pnpm --filter @paperclipai/server exec vitest
run src/__tests__/heartbeat-process-recovery.test.ts -t "lease this
process lost|this process dispatched held|restart stopped before adapter
dispatch|reached adapter dispatch held|lost monitor|older build without
a controller stamp|failure budget"` (10 passed).
- Related files, run together: `heartbeat-process-recovery`,
`live-recovery-work`, `interrupted-preparation-hold-settle`,
`resource-wait-stop`, `legacy-controller-lease`,
`execution-recovery-dead-controller`, `legacy-process-settlement`,
`run-failure-report` (876 passed; 2 timing failures under load in
unrelated setup-failure tests, both pass when run alone).
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes.

## Risks

- Low risk. The proof is the one #16355 already relies on across boots:
the revocation compare-and-set replaces the boot id, so the in-memory
controller of this process also fails its next renewal and cannot
dispatch.
- A same-boot run reaches this code only when it is not in
`activeRunExecutions` or `runningProcesses` and its lease expired. If an
executor is stalled before it registers, it aborts at its first renewal.
Setup work can run again in the retry; provider work cannot.
- Upgrade safety: no persisted shape changes. The reaper reads the same
fields (`executionStage`, `controllerBootId`, process ids) that older
releases write. Runs that older code already parked keep their hold and
their working Retry.
- Runs that reached `dispatching`, recorded a process, or have no
controller stamp keep the current behavior.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use, shell, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (the
code comment carries the contract)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 20:54:53 -06:00
DottaandClaude a1ec05a1a4 fix(native): read settled harness state again after a dropped sandbox connection (#20690) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner keeps one provider conversation across the per-turn
runs of a task. After each turn, the controller reads the settled Runner
and provider state out of the sandbox and saves it as the harness
backup.
> - A per-turn sandbox is released after the turn. The harness backup is
then the only copy of the provider session.
> - On staging, one dropped sandbox connection (`write ECONNRESET`)
during that read lost the backup. The run itself succeeded, because its
result was already accepted.
> - The next run of the task could not resume the conversation: `PRP
provider resume state is unavailable`. When the task was waiting for a
reply, the run stopped at "Recovery needed" with no Retry. A person had
to send a message to start a new conversation.
> - This pull request reads the settled state again when the sandbox
connection drops during that read.
> - The benefit is that one network reset no longer loses a task's
conversation.

## Linked Issues or Issue Description

Refs #20627 (warm backup restore when the source sandbox is idle) and
#20683 (relaunch of a settled runner). This PR changes a different step:
the backup capture at the end of a turn.

**What happened?**

A per-turn OpenCode task ran on a remote Daytona sandbox with reuse
disabled.

1. Run A accepted its result. The Runner suspended.
2. The controller read the settled state out of the sandbox. The read
failed with `remote checkpoint failed: write ECONNRESET`.
`harness_state.backup.persist` and `session.checkpoint.persist` failed.
3. Run A still finished as `succeeded`, because the result was durable.
The sandbox was released. No harness backup of turn A existed.
4. Run B started in a new sandbox. It failed in 0.2 s with
`native_session_recovery_failed: Error: PRP provider resume state is
unavailable`. When the prior turn yielded for a reply, the checkpoint
policy does not allow a replacement session, so the task showed
"Recovery needed" with only inspect actions.

On staging this happened on 4 tasks in 48 hours. In all 3 predecessor
runs that I could inspect, the backup capture failed. 2 of them show the
`ECONNRESET` in the run log. No deploy happened between run A and run B
in one case, so a deploy is not the cause.

**Expected behavior**

A transient connection drop while reading settled state does not lose
the provider conversation. The read is repeated, and the backup is
saved.

**Steps to reproduce**

1. Run the first durable turn of a remote OpenCode session.
2. Make the sandbox `syncOut` of the backup directories fail once with
`ECONNRESET`.
3. Call the settled checkpoint. On `master` it throws `write
ECONNRESET`, and no backup is saved.

The new test `uses the bundled Linux image authority for remote
opencode-new-durable-reset without controller overrides` runs these
steps.

**Paperclip version or commit**

`master` at `7435abcc63`.

## What Changed

- The settled-state capture (inspect plus `syncOut` of each persistence
directory into a pending directory) is now one step that can repeat. The
step writes nothing durable, so it is safe to repeat.
- A dropped, reset or refused sandbox connection repeats the step, at
most 3 attempts, with 1 s and 2 s delays. Each repeat clears the pending
directory and inspects the live state again, so the copy always matches
the inspected identity.
- Any other failure fails at once, as before: a state mismatch, a
missing directory, an unsafe archive, or a TLS failure.
- The durable backup upload and the local publish run only after a
complete capture. They did not change.
- `isTransientRemoteCheckpointReadFailure` classifies the error with the
existing `classifyRemoteConnectionError`, plus a message match for SDK
errors that lose their `code`.
- The first completeness inspection and its visibility-window re-reads
use the same bounded retry (`retryTransientRemoteCheckpointRead`). A
dropped provider-state read keeps its `provider_state_unreadable`
refusal for every other caller, but the checkpoint retry looks through
it to the dropped read. Undici `fetch failed` and `no response returned`
also count as drops.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-session-executor.test.ts`: 1343
passed.
- Red on `master`: the new `opencode-new-durable-reset` case fails with
`Error: write ECONNRESET`. Green with this change.
- The `isTransientRemoteCheckpointReadFailure` test checks that state
mismatches, unsafe archives and TLS failures do not repeat.
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Low risk. The normal path has no added wait. A repeat adds at most 3
s, and only after a connection failure.
- A repeated read inspects the live state again. If the state changed
between attempts, the existing identity checks still refuse it.
- A connection that stays down for all 3 attempts fails the same way as
before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 20:46:29 -06:00
DottaandClaude Opus 5.5 69c69e4c20 fix(recovery): finish the passive recovery pass before restarting it (#20680) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The workspace recovery worker saves the files of failed runs. One
passive pass of the worker visits holds with obligations: saved input
that waits, cleanup, and due re-arms and fallbacks of exhausted holds
> - #17741 restarts that pass after each committed workspace-recovery
change, so new work is found quickly
> - On staging, changes commit continuously (stop receipts, new holds,
recovery progress). Every pass restarted from the first candidates,
which are settled holds whose saved input waits behind another hold, and
it never reached later candidates
> - After #20295, #20569 and #20661 were deployed, no due re-arm ran:
five stop-proof holds and about twenty capture-policy holds kept their
exhausted state, with no marker and no activity
> - This pull request restarts the pass only after it reaches its tail,
and it continues an unfinished pass on the next run
> - The benefit is that every candidate is visited on a busy instance,
so the exhausted holds re-arm, save their files and free their tasks
without an administrator

## Linked Issues or Issue Description

Refs #17741
Refs #20295
Refs #20569
Refs #20661

**What happened?**
Exhausted workspace holds were not re-armed on staging, even after the
fixes that add their re-arm were deployed. Their recovery actions stayed
unchanged for days (`updatedAt` on Nov 5 or Nov 10). No
`stopProofRecovery` or `capturePolicyRecovery` marker and no
`issue.workspace_recovery_rearmed` activity was recorded. The worker did
process new restore holds, which are selected by its unfinished pass.

**What did you expect to happen?**
The worker's passive pass reaches every hold with a due re-arm or
fallback, re-arms it, and saves the files.

**Steps to reproduce**
1. Create several settled holds with saved user input that waits, with
action ids that sort before an exhausted stop-proof hold.
2. Start the real workspace recovery worker.
3. Commit a workspace-recovery notification every 50 ms, as stop
receipts and new holds do on a busy instance.
4. The worker never re-arms the exhausted hold. Without the
notifications, it re-arms the hold.

## What Changed

- `workspace-recovery-work.ts`: a committed change marks the passive
pass for restart (`restart`). It no longer resets the cursor. While a
restart is pending, an unfinished pass that made progress continues on
the next run.
- `workspace-recovery.ts`: when the passive cursor passes the last
candidate and a restart is pending, the same run starts a fresh pass
from the first candidate, so a new change is not delayed by another run
interval. Without a pending restart the pass records `complete` and does
not wrap, so paused input does not keep the worker active.
- `doc/execution-semantics.md`: document that a pass is never restarted
midway.
- `workspace-recovery-stop-proof.test.ts`: add a real-worker regression
in the staging shape (six earlier settled holds with waiting user input,
one exhausted stop-proof hold, continuous notifications).

## Verification

- Red: on master, the new test "re-arms the hold on a busy instance
whose passive sweep keeps selecting earlier settled holds" fails
(`stopProofRecovery` stays undefined). Control: the same test without
the notifications passes on master. This shows that the restarts cause
the starvation.
- Green: with the fix, the test passes. These suites pass:
`workspace-recovery-stop-proof` (12), `execution-control-work` (27),
which includes "visits a newly committed earlier Retry during a busy
recovered-source pass, then becomes quiet", plus
`workspace-recovery-capture-policy`,
`workspace-recovery-preparation-dispatch`,
`workspace-recovery-warm-handoff`, `workspace-recovery-preparation` and
`server-startup-feedback-export` (279). The timing-sensitive
`heartbeat-observed-workspace-preservation` suite passed 6 of 6 runs (an
earlier revision that restarted only on the next run made it flaky).
- Staging evidence (read-only GET requests): the 28 exhausted holds of
the company have no re-arm marker and no fallback check. Many blocked
tasks with settled holds have deferred wakes.

## Risks

- A change committed during a pass is now picked up at the next pass,
not at once. Unfinished physical work still uses its own pass first, so
new recoveries are not delayed. Saved input in a passive obligation can
wait up to one full pass.
- Previous-release story: no persisted state changes. Holds saved by
earlier releases are selected on the first full pass after deploy, with
no migration.
- Quiet behavior stays: a completed pass with no new change keeps the
worker idle.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
code editing, browser GET reads) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 20:44:09 -06:00
DottaandClaude Opus 5.5 a8f21a171c fix(cases): hold direct case status writes to terminal handoff authority (#20686) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The experimental Cases feature closes a case through a terminal
handoff to `done`
> - The terminal handoff lets an agent close a case only when the agent
owns the case's primary task
> - The direct case write routes (`PATCH /api/cases/:id` and the keyed
upsert on `POST /api/companies/:companyId/cases`) also write `status`
and `completedAt`
> - These routes did not apply the primary-task ownership rule to agent
status writes
> - This pull request applies the same ownership rule to those routes,
and sends `done` through the terminal handoff
> - The benefit is that a case closes only through the authority and
gates that the product describes

## Linked Issues or Issue Description

No public issue exists. This is the bug description.

**What happened**

An agent that did not own a case's primary task tried `handoff_case` to
`done` and got "Agents may complete only their own case work". The same
agent then sent `PATCH /api/cases/:id` with `status: "done"`. The route
returned 200 and closed the case. A chat eval (`cases.close-case`)
recorded this sequence.

**Expected behavior**

An agent changes a case with a primary task to or from
`done`/`cancelled` only with the authority that the terminal handoff
requires. `done` goes through the terminal handoff, so primary-task
completion keeps its normal gates (`doc/PRODUCT.md`, Cases section). The
dedicated `update_case` tool already excludes `status`.

**Steps to reproduce**

1. Create a case with a primary task that is assigned to agent A.
2. As agent B in the same company, send `PATCH /api/cases/:id` with
`{"status":"done"}`.
3. Before this change, the response is 200 and the case status is
`done`.

## What Changed

- `server/src/services/case-handoffs.ts`: add
`assertCaseStatusWriteAuthority`. For an agent actor, on a case that has
a primary task, a status change into or out of `done`/`cancelled` must
pass the same check as the terminal handoff (`authorizeTerminalTask`:
the agent is the primary task's assignee). A change to `done` then
returns 422 and names the terminal handoff. Board actors, non-terminal
status changes and fact edits do not change.
- `server/src/routes/cases.ts`: call the check inside the case write
transaction in `PATCH /api/cases/:id` and in the keyed upsert branch of
`POST /api/companies/:companyId/cases`.
- The terminal ownership denial now names the owning agent and the
primary task, and asks the caller to contact that agent on the task. The
message keeps its old first sentence.
- The keyed upsert falls back to the locked case row (not the pre-lock
read), so an upsert without `status` cannot undo a concurrent close.
- `server/src/routes/experimental-api-paths.ts`: add a catalog summary
and description for `PATCH /api/cases/{id}` that states the closure
rule.
- `server/src/__tests__/cases-routes.test.ts`: regression test for
non-owner `done`/`cancelled` through PATCH and upsert, the owner's
`done` through PATCH (422), owner cancel, non-owner reopen, and board
close.

## Verification

- `cd server && npx vitest run src/__tests__/cases-routes.test.ts -t
"terminal handoff's primary-task"`: fails before the route change
(`expected 403, got 200`) and passes after it.
- `cd server && npx vitest run src/__tests__/cases-routes.test.ts
src/__tests__/case-mutation-fence.integration.test.ts
src/__tests__/case-handoff-native.integration.test.ts
src/__tests__/case-handoff-legacy.integration.test.ts
src/services/case-handoffs.test.ts
src/services/native-runtime/native-case-tools.integration.test.ts
src/__tests__/heartbeat-case-terminal-handoff.test.ts
src/__tests__/openapi-routes.test.ts
src/__tests__/case-history-api-discovery.test.ts
src/services/native-runtime/runner-api.test.ts`: all pass. (One local
run of the legacy integration suite failed in fixture scratch
preparation under load; two reruns with the change pass.)
- `server` `tsc --noEmit`: no errors in the changed files. The fresh
worktree had unrelated errors from unbuilt workspace packages; CI covers
the full typecheck.
- Live chat eval
(`chat-features.runner-opencode.local.cases.close-case`, run
42936492869, at the first commit): the non-owner agent's terminal
handoff was denied with the owner named, and the agent did not change
case status through another route. The eval did not pass: the first chat
turn did not finish inside the scenario's 180-second observation window.
That is a separate latency issue in the eval.

## Risks

- Behavior change for agents: an agent can no longer set `done` through
PATCH on a case with a primary task, and a non-owner agent can no longer
cancel or reopen it. The agent skill reference already says "Do not
change case status separately".
- Cases without a primary task keep the old behavior, because they have
no owner to check and the terminal handoff does not apply to them.
- The check reads the primary task without a row lock, to keep the task
-> run -> case lock order. A concurrent reassignment can race one
request. The terminal handoff itself still locks and rechecks.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 20:41:27 -06:00
DottaandClaude b93df4ca38 fix(chat): consume a saved chat wake whose message a completed run answered (#20689) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - External chat (for example GitHub review requests) wakes the
assigned agent with a saved chat message
> - When the agent is already running, the chat wake is saved behind
that run and released after it finishes
> - The running agent often reads the new message as part of its current
turn, so the message is already answered when the saved wake is released
> - The release kept that chat wake anyway, removed its now-delivered
message ids, and started a reviewed chat run with no message
> - A reviewed chat run with no message cannot be attested, so it failed
with `reviewed_chat_execution_binding_not_authorized` and the task
became blocked
> - This pull request treats a chat message wake like a comment wake:
when a completed run already received all its messages, the saved wake
is consumed
> - The benefit is that a message that arrives during an active run no
longer blocks the task after it was answered

## Linked Issues or Issue Description

**What happened?**
A GitHub chat message arrived for a task while the assigned agent's run
was active. The wake was saved behind that run. The run read the message
and answered it. When the run finished, the saved chat wake was still
promoted. Its message ids were removed because a completed run already
received them. The new run had source `chat:<provider>` and no message,
so reviewed chat attestation denied it with `context_mismatch`. The run
failed with `reviewed_chat_execution_binding_not_authorized`.
Review-path recovery inherited the same context and failed too, and the
task was blocked.

**Steps to reproduce**
1. Assign a task in review to an agent that is running.
2. Deliver an external chat message for the task. The wake is saved
behind the active run.
3. Let the active run receive the message as wake input and succeed.
4. Release the task's execution lock.
5. See that the saved chat wake is promoted with no message, instead of
being consumed.

**Expected behavior**
The saved chat wake is consumed with "Queued messages were already
delivered to a completed run". No new run starts. The chat does not get
a "not started" notice for a message that was answered.

**Paperclip version or commit**
`origin/master` at 6e524d8a69.

**Additional context**
This was seen on a staging deployment on five runs across three tasks in
one day. It is a message timing race, not a deploy or authorization
problem.

## What Changed

- `server/src/modules/wake-queue/adapters/postgres.ts`: a saved wake
whose reason is `External chat message received` and whose payload
mutation is `chat_message_received` is treated as message-only. If a
completed run already received all of its messages, the release consumes
it, the same as a comment wake. If any message is still unread, it is
promoted as before.
- `server/src/services/chat-inbound-wakeup-publications.ts`: a chat
receipt consumed for that reason does not publish "This follow-up was
not started". Other declines keep that notice.
- `server/src/services/queued-comment-delivery.ts`: constants for the
chat message wake reason and the delivered-message reason.
- New test
`server/src/modules/wake-queue/adapters/chat-wake-delivered.test.ts`.

## Verification

- `cd server && npx vitest run
src/modules/wake-queue/adapters/chat-wake-delivered.test.ts`
- RED on `origin/master`: the release promotes the consumed chat wake
("Unexpected promotion of a consumed chat wake"), and the receipt
resolves to a `not_started` notice.
- GREEN with this change: the wake is cancelled with the delivered
reason, no run starts, no "not started" notice, and a chat wake with an
unread message is still promoted.
- Related suites: `src/modules/wake-queue`,
`heartbeat-reviewed-chat-binding.integration.test.ts`,
`durable-chat-wakeup.test.ts`,
`heartbeat-native-status-context.test.ts`,
`external-chat-wait.integration.test.ts`,
`heartbeat-held-comment-wake.test.ts`.
- `npx tsc --noEmit -p server` passes.

## Risks

- A chat wake that merged with another kind of wake takes the merged
wake's reason and mutation. Only a wake whose current reason and
mutation are both the chat message values is treated as message-only, so
other merged intents keep promotion.
- A visible "Your follow-up is queued." notice is not followed by a "not
started" notice when the message was already answered. That matches the
existing behavior when the wake is promoted.
- Not changed: a separate case where messages from two different chat
conversations are merged into one saved wake. That case fails with
`binding_denied` and needs its own change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, read-only API access). Extended
reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 20:15:13 -06:00
DottaandClaude 06af074452 fix(native): deliver the retry after a failed native turn, once per chain (#20681) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime decides a task's next status after each run,
through the status arbiter and the status-decision committer
> - When a native turn fails without a classified provider error, the
arbiter keeps the task's status and asks for one retry
(`schedule_retry`)
> - The native-status wake dispatcher refused that retry wake, because
its provenance check accepted only `enqueue_continuation` effects
> - So the task stayed `in_progress` with no live run, no queued wake
and no visible cause, until a person noticed it
> - This pull request lets the dispatcher accept the committed retry
effect, keeps the retry on its chat route, and limits the automatic
continuation to one per chain
> - The benefit is that a failed turn always ends with either queued
work or a blocked task with a working Retry

## Linked Issues or Issue Description

**What happened?**
A native run that ends with a failed turn and no classified provider
error leaves its task stranded. Examples: an OpenCode server that the
sandbox killed with SIGKILL during a turn (`opencode_server_exited`), or
an OpenCode turn that ended without an answer
(`opencode_empty_response`). The run's accepted result is finalized as
`run_failed_partial_evidence_preserved`. The committer queues a
`schedule_retry` wake. The dispatcher then skips the wake with
`chat_control_recovery_proof_unresolved`. The task stays `in_progress`
with no run, no wake and no blocker.

**Steps to reproduce**
1. Run a native task. Make the provider turn fail with
`opencode_server_exited` or `opencode_empty_response` and an accepted
`needs_review` result.
2. Finalize the run.
3. Run the native-status wake dispatcher.
4. See that the retry wake and its dispatch receipt are `skipped`, and
that no run starts.

**Expected behavior**
The task gets its automatic continuation. If that continuation fails the
same way, the task is blocked with a clear board blocker and a Retry
that starts a new run.

**Paperclip version or commit**
`origin/master` at c444da7d05.

**Additional context**
The task stays `in_progress` for hours with no owner. On a staging
deployment, two tasks stayed in this state for more than 7 and 11 hours.

## What Changed

- `server/src/services/chat-control-recovery-stop.ts`: the native-parent
proof accepts the `schedule_retry` effect, as well as
`enqueue_continuation`, when the effect targets the wake intent. Other
effect kinds (for example `notify_owner`) are not changed. The other
checks (applied decision, delivered effect, same company, task and
agent) do not change.
- `server/src/services/native-runtime/status-arbiter.ts`: new input
`failedTurnContinuationUsed`. When it is true, a failed run is decided
as `native_run_failed_continuation_exhausted`: the task is blocked with
a board blocker and no wake. New constants
`NATIVE_RUN_FAILED_RETRY_CAUSE` and
`NATIVE_RUN_FAILED_CONTINUATION_EXHAUSTED_ACTION`.
-
`server/src/services/native-runtime/native-failed-turn-continuation.ts`
(new): `isFailedTurnContinuationRun` reads the run's dispatcher-set
`nativeStatusWakeIntentId` and proves from the intent row (company,
agent, task, committer actor, `retryCause`) that the run is the
automatic continuation of a failed turn. A copied or unknown id does not
count.
- `server/src/services/native-runtime/native-run-finalizer.ts`: passes
`failedTurnContinuationUsed` to the arbiter for failed runs that the run
still owns; the committer rechecks that ownership under the issue lock
(`requireProviderFailureOwner`).
- `server/src/services/native-runtime/native-continuation-delivery.ts`:
the failed-turn retry keeps the failed run as its chat conversation
source, so its answer reaches the original chat.
- `doc/execution-semantics.md`: documents the retry wake, its provenance
and the one-continuation bound.
- New test `native-failed-turn-continuation.integration.test.ts`.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-failed-turn-continuation.integration.test.ts`
- RED on `origin/master` (c444da7d05): the dispatcher reports
`dispatched: 0` and the wake is skipped with
`chat_control_recovery_proof_unresolved`.
- GREEN with this change: for both `opencode_server_exited` and
`opencode_empty_response`, the dispatcher starts one continuation run.
When that continuation fails the same way, the task is blocked with the
board blocker, no new wake is queued, and the task Retry is accepted and
starts a new run. A copied wake id does not exhaust the continuation.
- Related suites pass: `status-arbiter.test.ts`,
`native-provider-failure.integration.test.ts`,
`heartbeat-dependency-scheduling.test.ts`,
`chat-control-admission-retry.test.ts`.
- `npx tsc --noEmit -p server` passes.
- `pnpm verify` selects broad verification (exit 2) because
`doc/execution-semantics.md` changed. Required CI supplies that
coverage.

## Risks

- Behavior change: a failed native turn now gets the automatic
continuation that the arbiter already requested. That continuation is a
new run of the same task, with the normal admission gates. It does not
replay the failed turn.
- The continuation is bounded to one per chain, so a deterministic
failure cannot make a wake loop.
- Tasks that are already stranded by the old behavior keep their skipped
wake. Their task Retry still works.
- Upgrade safety: an old pending `schedule_retry` intent is accepted by
the new dispatcher. A run that started before this change has no proven
intent, so it gets one continuation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, read-only API access). Extended
reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 20:10:48 -06:00
DottaandClaude 7435abcc63 fix(native): wait out a run-event append before saving provider-pack admission (#20682) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native OpenCode run in a remote sandbox records a provider-pack
admission receipt on its run row after the provider session opens.
Artifact retention then requires that receipt.
> - `recordNativeProviderPackAdmission` locked the run row with `FOR
UPDATE NOWAIT`.
> - A run-event append locks the same run row for a few milliseconds
(`FOR NO KEY UPDATE` since #20670, `FOR UPDATE` before). Both conflict
with `FOR UPDATE`. Startup appends many events at the same moment.
> - When the two met, the admission write failed with `55P03`. The
admission state stayed `pending`. Artifact retention then failed the run
with `native_provider_artifacts_admission_unverified`. The task stopped
as "Recovery needed" with no Retry.
> - This pull request makes the admission write wait for the run-row
lock within the existing 1000 ms `lock_timeout`. The coordinator claim
keeps its `NOWAIT` fence.
> - The benefit is that an ordinary concurrent event append no longer
fails a healthy run before its first turn.

## Linked Issues or Issue Description

Refs #20670 (changed the append lock mode; the conflict with `FOR UPDATE
NOWAIT` remains).

**What happened?**
On a hosted staging instance, three OpenCode runs on remote sandboxes
failed in the first minute with
`native_provider_artifacts_admission_unverified`. Each run log shows
`native provider-pack admission receipt could not be saved` about
0.2–0.7 s after the runner authenticated. The run profile kept
`nativeProviderPackAdmissionState.state = "pending"`. The tasks stayed
blocked.

**Steps to reproduce**
1. Start a native run and hold its `heartbeat_runs` row lock in another
transaction, as a run-event append does.
2. Call `recordNativeProviderPackAdmission` for that run.
3. Observe `could not obtain lock on row in relation "heartbeat_runs"`
(`55P03`).

**Expected behavior**
The admission waits a short, bounded time for the append to commit, then
saves the receipt.

## What Changed

- `recordNativeProviderPackAdmission` sets `lock_timeout = 1000ms` for
every call, not only `waitForClaim` calls.
- The run-row lock is now `FOR UPDATE` without `NOWAIT`, so it waits for
a concurrent append within that bound.
- The coordinator (`native_run_finalizations`) lock keeps `NOWAIT` when
the caller does not wait for the claim. The existing fence tests still
pass.
- New PostgreSQL test: `waits for a concurrent run-event append's
run-row lock instead of losing the admission` (both `waitForClaim`
modes).

## Verification

- RED on `origin/master` (`c444da7d05`): the new test fails with `55P03`
in both modes.
- GREEN with this change: `pnpm --filter @paperclipai/server exec vitest
run src/services/native-runtime/native-provider-pack-admission.test.ts`
(37 passed).
- `pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify`: see the result below.

## Risks

- Low risk. The normal path has no contention and does not wait. Under
contention the admission waits at most 1000 ms, before any provider
turn.
- A lock wait between the coordinator row and the run row can now meet a
deadlock. PostgreSQL detects it and raises an error. That error takes
the same caught path as the earlier `55P03`, so the result is no worse
than before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 19:39:42 -06:00
DottaandClaude 6e524d8a69 fix(workspaces): stop retrying a copy-back that hit a full sandbox disk (#20678) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a native run, Paperclip copies the agent's sandbox workspace
back to durable storage. This is finalization.
> - The copy-back builds an archive inside the sandbox. When the sandbox
disk is full, the copy-back fails with
`remote_workspace_sandbox_disk_full`.
> - Finalization treated that failure as transient. It retried three
times. Each attempt took about 15 minutes for a large workspace, and
nothing freed space between attempts.
> - Saved-work recovery already reclaims the retained sandbox's inactive
provider cache when a save hits a full disk. It runs only after
finalization ends.
> - This pull request ends finalization on the first full-disk copy-back
failure, with the accepted result kept, so saved-work recovery starts at
once.
> - The benefit is that the task gets its files saved sooner, and the
user sees what happened instead of "failed after 3 attempts".

## Linked Issues or Issue Description

Refs #19619 (related: records why the read-only capture fallback failed;
this PR does not change that path)

**What happened?**
On staging, a native run finished its turn and its result was accepted.
The workspace copy-back then failed three times with
`remote_workspace_sandbox_disk_full` (about 7.5 GB of workspace, about 6
GB free). The three attempts spanned about one hour across deploys.
Finalization then ended as `native_workspace_sync_out_retry_exhausted`.
Saved-work recovery reclaimed the inactive provider cache (about 3 GB
freed) and saved the files.

**Expected behavior**
A full sandbox disk is recognised on the first failure. Paperclip does
not spend the finalization budget on attempts that cannot succeed. The
agent's result is kept. The user sees that the disk is full and that
Paperclip saves the files again without another provider turn.

**Steps to reproduce**
1. Run a native task on a remote sandbox whose workspace is larger than
the free sandbox space.
2. Let the provider turn finish.
3. Observe three copy-back attempts that each fail with
`remote_workspace_sandbox_disk_full`.

## What Changed

- `recordRetryableFailure` treats a workspace failure whose cause starts
with `remote_workspace_sandbox_disk_full:` as an exhausted budget on the
first attempt. The code stays
`native_workspace_sync_out_retry_exhausted`, so the existing export
hold, saved-work recovery, and export Retry apply unchanged.
- The run error for that case says that the sandbox disk is full and
that Paperclip saves the files again without repeating the agent's work.
It no longer says "after 3 attempts".
- Other copy-back failures keep their bounded retry.
- `captureRemoteWorkspaceArchive` rethrows the read-only fallback's own
failure when that fallback only lost its connection (`transport_closed`,
`provider_unavailable`, `provider_rate_limited`,
`sandbox_state_transition`). That failure is not disk-full, so
finalization keeps its bounded retry for it. Other fallback failures
keep the disk-full code.
- `remoteWorkspaceFailureCause` records the server-built read-only
capture failure, so a failed save names where the fallback stopped.
- The transport exports `REMOTE_WORKSPACE_SANDBOX_DISK_FULL_CODE`. The
finalizer and tests use it.
- `doc/execution-semantics.md` describes the new rule.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-run-finalizer-telemetry.test.ts
src/services/remote-workspace-transport.test.ts` (195 pass).
- New test "hands a full sandbox disk to saved-work recovery on the
first copy-back failure" fails on `origin/master` (the failure stays
`retryable_failure`) and passes with this change. It completes an
accepted result first and checks that the result stays linked and that
no agent wakeup is created.
- New transport test "keeps a full disk only when the read-only fallback
did not just lose its connection": the two dropped-connection cases fail
on `origin/master` and pass with this change.
- New test "keeps other copy-back failures on their bounded retry"
passes before and after.
- `cd server && npx tsc --noEmit` passes. `pnpm verify` selected 2504
test files because the finalizer is widely imported; its first selected
batch passed, and required CI supplies the full run.

## Risks

- Low risk. The terminal state is the same state that three failed
attempts produce today. Only the timing changes.
- A sandbox whose disk frees space by itself between attempts now waits
for saved-work recovery instead of the next finalizer attempt.
Saved-work recovery repeats the save with its own bounded attempts, so
this does not lose work.
- Upgrade safety: a run saved by an older release in `retryable_failure`
with this cause ends on its next full-disk failure. No migration is
needed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits) in Claude Code. Extended reasoning was not configured.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 19:31:33 -06:00
DottaandClaude Opus 5.5 4a61a2501e fix(storage): never promise a storage retry no storage wait owns (#20676) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A run that hits a file-storage outage gets a bounded storage wait: a
retry that starts again after storage is back
> - The run error says "Paperclip will retry automatically when storage
is back"
> - On staging, a server restart reattached a remote native runner while
its turn was still running. A storage outage then failed the
reattachment
> - Restart recovery kept the workspace under a no-replay hold, so the
run took the code `native_continuation_requires_reconciliation`. Only a
`storage_unavailable` run gets a storage wait, so no retry was
scheduled. The run still showed the storage retry promise, and the task
stayed blocked for about 15 hours
> - This pull request makes the storage text follow the run's final code
and the real storage wait outcome, and reads old rows truthfully
> - The benefit is that every storage message is true: it promises a
retry only when one is scheduled, and it offers a working Retry when
nothing retries the run

## Linked Issues or Issue Description

**What happened?**
A native run on a remote sandbox was reattached after a server restart.
Its provider turn was still in flight. File storage failed during the
reattachment. Restart recovery settled the run as
`native_continuation_requires_reconciliation`, kept its files, and did
not replay the unfinished turn (this is the existing policy for a turn
with an unknown outcome). The run error said "Paperclip will retry
automatically when storage is back". No storage wait was scheduled, so
nothing retried. The task stayed blocked until an operator clicked
Retry.

**Root cause**
`server/src/services/heartbeat/run-completion.ts` chose the message and
the error code separately. The message used
`isStorageUnavailableError(err)`. The code used a chain where
`nativeTerminalFailureCode` wins over `storage_unavailable`. The storage
wait is scheduled only when the code is `storage_unavailable` (and no
native terminal failure is recorded). So the message promised a wait
that the code made impossible. The same false promise stayed when the
storage wait was refused (`not_scheduled`) or its write failed.

**Expected behavior**
A storage failure promises an automatic retry only when the storage wait
is scheduled. Otherwise it offers Retry. Rows saved by earlier releases
read the same way after deploy, with no migration.

**Steps to reproduce**
Fail a native run with a storage-unavailable error after its native
coordinator is settled as `terminal_failure` /
`native_continuation_requires_reconciliation`. See the new regression in
`run-completion.test.ts`.

## What Changed

- `storage-unavailable.ts`: add the held messages (outage and rejected
credentials). They say Paperclip is not retrying automatically and offer
Retry or a message. Add `storageUnavailableHeldMessage()` and
`presentStorageFailureMessage()`.
- `run-completion.ts` (`failRun`): compute the error code first. Use the
storage retry promise only when the code is `storage_unavailable`. Under
any other code (for example a native recovery hold), use the held
message. No storage wait is created for such a run, because it could
only be refused.
- `run-completion.ts` (`failRun` and `failRunSetup`): replace
`markStorageWaitExhausted` with `settleStorageWaitMessage`. It keeps the
promise only when a retry is scheduled. It writes the exhausted text
when the budget is spent, and the held text when the wait was refused,
its write failed, or a case handoff took over.
- `run-completion.ts`: when the promise is withdrawn, the failure's
`error` event (written before the retry decision) gets the same final
text, so the transcript does not keep the promise after reload.
- `run-completion.ts` (`failRunSetup`): a lost controller lease keeps
its own code and text, not the storage promise.
- `run-lifecycle.ts`: `decorateHeartbeatRunRuntimeStatus` presents a
saved storage retry promise under a code other than
`storage_unavailable` with the held text. This covers run detail and
issue run presence. `listRuns` in `run-state.ts` applies the same
projection, so the company run list and the Inbox failure read agree.
Rows from earlier releases read truthfully with no migration.
- `doc/run-log-events.md`: document which runs get a storage wait and
what each message means.

Previous-release story (upgrade-safe run state): this change adds no
persisted shape. Rows saved by the previous release carry forward. A row
with the storage promise under another code reads with the held text. A
`storage_unavailable` row keeps its text. The native hold policy is
unchanged: an unfinished turn with an unknown outcome is not replayed
automatically, and the existing Retry continues it.

## Verification

- Red before the fix (assertion failures, not setup errors): the new
`run-completion.test.ts` cases received "Paperclip will retry
automatically when storage is back" under
`native_continuation_requires_reconciliation`, and the
`run-lifecycle.test.ts` cases read the saved promise unchanged.
- Green after the fix:
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts
src/services/heartbeat/run-lifecycle.test.ts -t storage` (13 passed)
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts
src/services/heartbeat/run-lifecycle.test.ts
src/storage/storage-unavailable.test.ts` (213 passed)
  - `cd server && npx tsc --noEmit -p tsconfig.json` (clean)
- Existing storage cases still pass: a scheduled storage wait keeps its
promise and suppresses rapid recovery; an exhausted wait writes the
exhausted text.
- `pnpm verify` exited 2 (a doc is a global CI input), so required CI
supplies the broad coverage.
- The full `run-completion.test.ts` file passed every test locally. Its
`afterAll` hook timed out under machine load (load average about 17),
also with the new tests skipped.

## Risks

- Low risk. The change affects only the text of failed runs and one read
projection. It does not change retry scheduling, recovery ownership, or
replay policy.
- The read projection matches this repository's own two promise
constants under a code other than `storage_unavailable`. A future change
to those constants must keep the projection in sync.
- Error events already written by earlier releases keep their original
text in the run log. Live events already published before a withdrawn
promise are not resent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, browser reads for GET-only diagnostics).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 19:22:26 -06:00
DottaandClaude 2eede8b318 fix(native): give a relaunched settled runner its admitted provider pack (#20683) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - After a server restart or a same-run retry, a remote native run
reattaches to its runner in the sandbox (`reattach_remote_runner`).
> - When the controller saved the runner artifact, the executor prepares
that run as an adoption. It skips provider-pack selection, because a
live adopted runner keeps its launched provider.
> - The reattach probe can instead find the runner settled and exited
(`verifyRemoteRunnerOrSettledExit`). Then nothing is adopted, and the
transport relaunches the runner from its durable state. That relaunch
must start the provider again, so it needs the provider-pack launch
inputs.
> - The executor did not give them. OpenCode failed with "OpenCode
executable is missing from the provider pack". ACPX failed with "ACPX
launch profile omitted its provider-pack authority or verified artifact
digests". Both ended the run.
> - This pull request selects the saved (admitted) provider-pack
authority on that settled-exit relaunch and verifies it as an exact
installed identity, as an ordinary reattach does.
> - The benefit is that a restart or a same-run retry can resume an
exited runner with the same provider pack, instead of failing.

## Linked Issues or Issue Description

**What happened?**
On a hosted staging instance, two OpenCode runs and one ACPX (Claude)
run were interrupted by a deploy (SIGTERM) or ended a turn without a
result. The server scheduled a same-run retry that reattached to the
existing remote runner. The runner had exited cleanly
(`runner_remote_process_exited lifecycle=suspended`). The retry then
failed within three seconds with
`runner_remote_provider_artifact_incompatible`.

**Steps to reproduce**
1. Prepare a remote OpenCode or ACPX run with a persisted runner
artifact (`controllerPersistence.runnerArtifact`).
2. Reattach with `restartRecovery.kind = "reattach_remote_runner"`,
where the runner process is not running and its state is `suspended` and
settled.
3. Observe that the transport options have no `adoptExistingRunner`, no
`providerPackAuthorityDigest` and no `opencodeCommand` /
`acpxSidecarSha256`.

**Expected behavior**
The relaunch gets the admitted provider pack's launch inputs, or it
fails with a clear pack-identity error.

## What Changed

- `createRunnerdBackend` (native session executor): when a
persisted-runner reattachment finds no live runner to adopt, select the
saved provider-pack authority (`reattach: true`), so
`expectedProviderPackManifest` is the admitted pack.
- `prepareRemoteRunnerAssets` no longer returns early for that relaunch.
It verifies the installed pack as an exact identity and sets the active
pack root, the same path as a reattach without a persisted artifact.
- A live adopted runner is unchanged: it still skips pack selection.
- The test `uses the bundled Linux image authority for remote
opencode|claude-x64` now also covers the settled-exit relaunch for
OpenCode and ACPX.

## Verification

- RED on `origin/master` (`c444da7d05`): the extended test fails for
`opencode` and `claude-x64` (the relaunch options have no provider-pack
inputs).
- GREEN with this change: `pnpm --filter @paperclipai/server exec vitest
run src/services/native-runtime/native-session-executor.test.ts -t "uses
the bundled Linux image authority"` (14 passed).
- `pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify`: see the result below.

## Risks

- Upgrade safety: the relaunch reads the saved authority, so a newer
controller does not select a newer pack for an older session. It
requires an exact installed identity. If the sandbox no longer has that
pack, the relaunch fails with a pack-identity error, and the existing
Retry path applies, as before.
- No change to a live adoption, to fresh runs or to the normal path
before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 19:14:43 -06:00
DottaandClaude e33ede9a8a fix(native): settle a restart-orphaned write_document from its transactional receipt (#20674) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run calls Paperclip tools such as `write_document`. The
runner saves each tool input durably before the server runs the tool,
and the server saves a result receipt
> - A control-plane restart can cut a `write_document` call after the
runner saved its input and before any result was saved
> - Recovery then has no proof of the outcome. A call from an earlier
server process settles as "unknown outcome" (#20628). A call the current
server process admitted stays pending, close cannot prove that the
provider drained, and the session quarantines
(`native_session_cleanup_quarantined`). A person must click Retry
> - `write_document` saves its revision and its idempotency receipt in
one database transaction, under the run and issue row locks. The
revision is also compare-and-swap on `baseRevisionId`. So the receipt is
a durable outcome record: present means applied, absent means not
applied
> - This pull request makes recovery use that proof. Recovery runs the
same receipt-guarded write. It replays the saved result, or it applies
the write exactly once
> - The benefit is that an orphaned `write_document` settles with its
real result. The document is never written twice, and the run does not
need Retry for this call

## Linked Issues or Issue Description

**What happened?**
Daytona restart-chaos wave `w-20261110T174930-s9205`, task t01 (codex,
workload C1, fault RR2). A restart cut a `write_document` call. The
close settlement showed `semanticTools.pending: [{ operationId:
"write_document", callId: "exec-8b79ab78-…" }]` and `providerDrained:
false`. The session quarantined with
`native_session_cleanup_quarantined`, and the task needed Retry. Since
#20628, a call that an earlier server process admitted settles as
`tool_call_interrupted_by_restart` with an unknown outcome. The agent
then has to guess whether the write applied.

**Expected behavior**
When a durable record proves the outcome, recovery uses it. A write that
applied returns its saved result. A write that did not apply runs once.
When the record cannot prove it, the call stays as it is today.

**Steps to reproduce**
1. On master, run `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/paperclip-runner-tool-authority.test.ts -t
"orphaned"`.
2. `authority.recover()` returns `null` for a `write_document` that has
no receipt, so the call stays unsettled.

Refs #20628, #20639, #20507

## What Changed

- `PaperclipRunnerToolAuthority.recover()` now settles `write_document`
through its transactional receipt (`#recoverTransactionalWrite`). It
runs the same receipt-guarded mutation as an ordinary call. If the
receipt exists, it returns the saved result and writes nothing. If no
receipt exists, it applies the write once.
- A definite refusal that rolls the transaction back (a 4xx such as a
stale `baseRevisionId`, a reused idempotency key, invalid input) settles
as a tool error. Any other failure leaves the call unsettled, as before.
- The set of operations with this proof is explicit:
`TRANSACTIONAL_RECEIPT_OPERATIONS` contains only `write_document`. Other
tools keep their current recovery rules.
- New test: an orphaned `write_document` with no receipt applies once; a
second recovery and a late original call replay the receipt; an applied
write recovers its saved result; a stale base settles as a tool error
with no change.
- Updated test: a reused idempotency key with other input now settles as
an `paperclip_runner_tool_idempotency_conflict` tool error instead of
staying pending. The receipt proves that this call can never apply under
that key.
- Docs: `packages/paperclip-runner/docs/durable-recovery.md` (semantic
operations that outlive a turn) and a new design note,
`doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md`. The
note also covers the second change (an ACPX turn start cut off before
the prompt reached the provider), which comes in a separate pull
request.

## Verification

- Red on master, green here: `pnpm --filter @paperclipai/server exec
vitest run
src/services/native-runtime/paperclip-runner-tool-authority.test.ts` (43
passed).
- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
178 passed. One unrelated ACPX timing test failed once under load and
passed when run alone.
- Server typecheck: `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` exits 2 (a runner doc changed), so required CI supplies
the broad coverage.
- Live restart-chaos wave: see the comment on this pull request.

## Risks

- Recovery can now apply a write that never applied before the restart.
This is the call the agent made, applied once. The write is
compare-and-swap on `baseRevisionId`, so it cannot overwrite a revision
that someone wrote after the agent read the document. That case settles
as a conflict error.
- Upgrade safety: no saved shape changes. Receipts saved by the previous
release have the same shape. A server on the previous release keeps its
current recovery for these calls.
- Time to first token: no change. This code runs only during recovery.
- No runner source changes, so no new runner image is needed.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file editing) in Claude Code. Extended reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 19:05:15 -06:00
DottaandClaude cec903c6f0 fix(native): retire a finished remote owner whose gone Runner left a delivery unacknowledged (#20677) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native follow-up run on a remote sandbox first proves that the
previous per-turn owner finished. Then it retires that owner's
controller journal and starts a new Runner.
> - The proof requires a settled journal. A command that the Runner did
not acknowledge holds the owner.
> - On staging, a per-turn owner committed its result and its Daytona
sandbox was destroyed after the turn
(`paperclip_runner_destroy_after_turn`, cleanup `success`). Its journal
still held a `run.attach` or a `semantic_tool.result` that the Runner
never acknowledged.
> - The next run failed with `prior_owner_terminal_state_indeterminate;
retirement_refused=journal_commands_unsettled(run.attach.pending)` (or
`semantic_tool.result.pending`). A Retry or a new message gets the same
journal, so it fails the same way. The task stays blocked.
> - Retirement already excuses the Runner's own lifecycle commands when
the prior sandbox lease ended on another sandbox. This pull request
applies the same rule to deliveries that only that gone Runner could
receive.
> - The benefit is that these follow-ups start by themselves. Commands
that carry user intent still hold the owner.

## Linked Issues or Issue Description

**What happened?**
A census of staging failures (48 h, Nov 9–10) found 6 failed runs with
`prior_owner_terminal_state_indeterminate` and
`retirement_refused=journal_commands_unsettled(run.attach.pending |
semantic_tool.result.pending)`. The last one was at 18:56 UTC on Nov 10.
In two examples that I checked:
- The previous run succeeded. Its result was committed and its
coordinator lease was released.
- Its lease policy was `ephemeral`. The lease is `expired`, with
`failureReason=paperclip_runner_destroy_after_turn` and
`cleanupStatus=success`.
- The follow-up got a new sandbox (a different provider lease) and
refused in about 11 s. It started no Runner. The task became `blocked`,
and the next action was "Send a message to continue".

**Expected behavior**
When the record proves that the previous Runner is gone, an undelivered
attachment, tool answer or read cannot run. The follow-up retires the
finished owner and starts a new Runner. Retirement replays no command.

**Steps to reproduce**
1. Finish a per-turn remote native run whose controller journal keeps a
`run.attach` or `semantic_tool.result` command in `pending`.
2. Let the sandbox lease of that run end (`expired` or `released`) on
its own sandbox.
3. Start a follow-up run that gets a new sandbox.
4. Observe
`retirement_refused=journal_commands_unsettled(run.attach.pending)`.

**Paperclip version or commit**
Staging builds of Nov 10. The refusal still reproduces on `master` at
`48c6e996da` (the new tests fail there).

## What Changed

- `native-session-executor.ts` (`readSettledRunnerdProjection`): when
the prior run's sandbox leases ended on another sandbox
(`remoteRunnerGone`, the existing proof), unfinished `run.attach`,
`semantic_tool.result`, `session.snapshot` and `session.goal.get`
commands no longer hold the retirement. These join the lifecycle
commands that the rule already excused.
- In the same case, a journaled `semantic_tool.result` answers its
`semantic_tool.input`. The controller journals the answer after the tool
ran. Only its delivery was outstanding.
- No other proof changes. The prior run must still be terminal, with a
committed result, a released coordinator lease, no recovery state, a
stopped or fenced controller, a delivered `run.terminal`, and no live
work after it. Quarantine copies must pass the same checks.
- `doc/execution-semantics.md`: describe the rule.

## Verification

-
`server/src/services/native-runtime/native-recovered-owner-followup.test.ts`
(real Postgres):
- New: "retires a finished remote owner with an unacknowledged
pending-attach / pending-tool-result when its sandbox lease ended on
another sandbox". It fails on `master` with the exact staging errors
`journal_commands_unsettled(run.attach.pending)` and
`journal_commands_unsettled(semantic_tool.result.pending)`. It passes
with this change, and the log shows the retirement.
- Held cases stay held: a lease that is still retained, a lease on this
run's sandbox, no lease record, and a tool input with no journaled
answer (`journal_tool_input_unanswered`).
- New: "keeps a stopped local owner held by an unacknowledged
run.attach". A local Runner is not gone by record.
  - Whole file: 95 passed.
- Neighbouring suites pass: `explicit-native-continuation.test.ts` and
`native-controller-recovery.integration.test.ts` (460 tests).
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` exits 2: a global CI input changed
(`doc/execution-semantics.md`). Required CI supplies the broad coverage.
- Live proof: not done. The redeploy chaos scenario (K15) is not on
`master` yet. Tests only.

## Risks

- The rule applies only when every sandbox lease of the prior run is
`released` or `expired`, has a release time, and names a different
provider lease than this run's. A reusable sandbox that this run resumed
keeps the existing hold.
- A command that carries user intent (`turn.start`, `turn.steer`,
`request.resolve`, `interaction.receipt`, `external_provider.operation`,
`session.goal.set`/`clear`, `session.close`, `run.cancel` and others)
still holds the owner.
- Retirement archives the old journal under `continuity-breaks` and
replays nothing, as before.
- No persisted shape changes. Old saved journals are read by the same
rule.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 18:49:26 -06:00
DottaandClaude Opus 5.5 ad0543ae90 test(recovery): re-arm a capture-policy save whose finished coordinator names a lapsed owner (#20672) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Workspace recovery re-arms a save that an older capture policy
exhausted (#20569), so the files are saved without a person
> - On staging, one save of this kind was not re-armed after the deploy.
Its export hold also carries a workspace finalization block, and it has
a resolved no-replay sibling for the same run
> - #20661 found that discovery skipped a hold when the run's finished
coordinator still named a lease owner whose lease had lapsed. The same
discovery clause gates the capture-policy re-arm
> - This pull request adds a regression test for that exact
capture-policy shape. It shows that #20661 alone makes the worker re-arm
and save it
> - The benefit is that the capture-policy path keeps this guarantee in
later changes

## Linked Issues or Issue Description

Refs #20661
Refs #20569

**What happened?**
An exhausted capture-policy save (`export_interrupted`, cause
`provider_error`, 6 of 6 attempts, phase `preserve_files`) was not
re-armed on the worker's first pass after #20569 was deployed.

**What did you expect to happen?**
The worker re-arms the save once under the current capture policy, saves
the files, and returns the hold to the system.

**Steps to reproduce**
1. Exhaust a save under capture policy 1 with `export_interrupted`. Add
a workspace finalization block and a resolved no-replay sibling for the
same run.
2. Leave a lapsed `native-finalizer:` lease owner on the run's
`terminal_failure` coordinator.
3. Run the workspace recovery worker. Before #20661,
`nextWorkspaceRecoveryAt` returns null and the hold is never selected.

## What Changed

- `workspace-recovery-capture-policy.test.ts`: add "re-arms a save whose
finished coordinator still names a lapsed finalizer owner". The test
uses the staging record: export hold with `historicalWorkspaceSource`
and `workspaceFinalizationBlock`, a resolved no-replay sibling, waiting
user input, and a lapsed finalizer owner. It drives the worker's own
discovery loop until the worker is idle.
- No production code changes.

## Verification

- Red: with the source files from before #20661 (`git checkout
686893636d^ -- <the four source files>`), the new test fails.
`nextWorkspaceRecoveryAt` returns null, so the worker is never woken.
- Green: on current master, `pnpm --filter @paperclipai/server exec
vitest run src/services/workspace-recovery-capture-policy.test.ts`
passes (5 tests). The file passed 4 runs in a row.
- The test asserts that the save completes under policy 2, the hold
returns to the system, `releasedLapsedOwner` is recorded, and the lapsed
owner is released.

## Risks

- Low risk. This change adds a test only.
- Previous-release story: the test uses rows in the shape an older
release saved, with no capture-policy marker. No migration is needed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
code editing, browser GET reads) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 18:47:29 -06:00
DottaandClaude Opus 5.5 e9db5e0d03 perf(github): read the inherited review waits of an issue list in one pass (#20673) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can review a GitHub pull request. After the agent publishes
the review, the task can wait in review for the next GitHub reply. A
later run of the same task can inherit this wait.
> - Issue lists and inbox attention show these waits. For each waiting
task, the server reads the inherited GitHub review wait: the source
comment, the latest review, and the review publication.
> - The server did these reads one time for each waiting task. A list
request with N waiting tasks sent N copies of the prepared review
statement and N copies of the publication read.
> - On a hosted database, the prepared review statement ran about
270,000 times in about 2.5 hours (up to 87 calls per second). The run
identity read before it ran the same number of times. The list read that
calls it ran only about 1,400 times, so each list read started about 190
review reads on average. No other caller reads the waits of many runs in
a loop.
> - This pull request reads the reviews and the publications of all
waits of one list request with one statement each. Each wait gets the
same result as before.
> - The benefit is about 200 times fewer review statements and fewer
pool connections for each issue list.

## Linked Issues or Issue Description

Refs #20400 (shares review-wait reads between the callers of one
request).
Refs #20551 (authorizes the waits of one issue page with one read; this
change does the same for the inherited GitHub review reads of that
page).

**What happened**

`listExternalReviewWaits` called `readInheritedGitHubReviewWait` one
time for each candidate wait, inside `Promise.all`. Each call sent a run
identity read, a source-message read, the prepared
`inherited_github_review_wait` statement, and a publication read. A list
with N waits sent each statement N times.

**Expected behavior**

One list request sends one review statement and one publication read for
each company. The result of each wait does not change.

**Steps to reproduce**

1. Put many tasks of one company in review with an inherited GitHub
review wait.
2. Open the issue list or the inbox.
3. Count the statements of the database. The review statement runs one
time for each waiting task.

## What Changed

- `server/src/services/chat-channels/github/review-wait.ts`:
- Split `readInheritedGitHubReviewWait` into shared steps: the source
comment read, the review checks, the publication key, and the published
result. The single-run read uses the same prepared statement and the
same checks as before.
- Add `readInheritedGitHubReviewWaits(db, runs)`. It reads the source
comment of each run with the same lineage check as before. Then, for
each company, one `DISTINCT ON` statement reads the latest review for
each (source comment, task, agent) key, with the same joins, filters and
order as the prepared statement. One read finds the processed
publications. Each run gets the same result as a separate call: `null`,
`revoked`, or `published`.
- `server/src/services/native-runtime/external-review-wait.ts`:
`listExternalReviewWaits` uses the batch read. The rest of the list and
its authorization do not change.
- `server/src/__tests__/github-review-wait.test.ts`: a new test reads 18
runs in one batch and compares each result with a separate single-run
read. The runs include each revocation case (connection disabled or
inactive, review state, review agent, repository, repository id, thread,
publication, identity, company), a newer unreviewed head, no identity,
no task, an endpoint of another agent, a run in another company and a
duplicate run. The test also makes sure that the batch sends one review
statement and one publication read for each company. The existing
revocation tests also check the batch read.
- `server/src/__tests__/helpers/github-review-wait.ts`: the seed helper
can add a wait to an existing company and agent.

## Verification

- `pnpm vitest run server/src/__tests__/github-review-wait.test.ts
server/src/__tests__/github-review-check-state.test.ts
server/src/__tests__/issue-list-review-wait-authorization.integration.test.ts
server/src/__tests__/blocked-inbox-review-wait-scope.test.ts`
- Focused local run of every test that imports the changed modules or
the seed helper (7 files, 335 tests passed), including
`external-chat-wait.integration.test.ts`, `external-review-wait.test.ts`
and `legacy-execution-recovery.test.ts`. The change to the shared seed
helper makes `pnpm verify` select the full server suite, so CI supplies
that coverage.
- Local measurement on real PostgreSQL (embedded PostgreSQL, `ANALYZE`
after seeding). One company and agent, each wait has 5 reviews and 10
outbound messages. "Before" is the previous list code: one
`readInheritedGitHubReviewWait` call for each run in `Promise.all`.
"After" is `readInheritedGitHubReviewWaits`. The runs alternate order in
each pair. Both return equal results in every sample.

| Waits | Version | Statements | Review statements | Median ms | Min–max
ms | Samples |
| --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 800 | 200 | 106.8 | 98.0–117.5 | 10 |
| 200 | after | 402 | 1 | 44.3 | 41.6–50.9 | 10 |
| 20 | before | 80 | 20 | 9.8 | 8.6–14.7 | 12 |
| 20 | after | 42 | 1 | 6.3 | 5.3–7.0 | 12 |

- `EXPLAIN (ANALYZE, BUFFERS)` of the batch review statement for 200
waits: index scans on each join, 6.7 ms execution. The single statement:
0.05 ms execution and about 2.7 ms planning for each call.
- The run identity read and the source-message read still run one time
for each wait. They are cheap on the hosted database (about 0 ms each),
and they keep the lineage check for each run.

## Risks

- Low risk. The batch statement uses the same joins, filters and order
as the prepared statement, and `DISTINCT ON` keeps the first row for
each key. The array filters can also match a mix of two keys. The code
uses only rows of exact requested keys.
- The single-run read and its prepared statement do not change. The
finalizer, the status committer, and the attachment reuse paths still
use it.
- Errors do not change: a 403 from the lineage check gives `revoked`,
and other errors are thrown.
- No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code reading, editing, shell, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 18:35:39 -06:00
DottaandClaude Opus 5.5 c444da7d05 perf(run-events): stop blocking checkpoint FK checks during event appends (#20670) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run sends a stream of runner events. The server appends
each event to the run log. The runner controller then saves a checkpoint
for recovery.
> - Each checkpoint save inserts the new `event:N` row and updates the
`cursor` row in `native_runner_checkpoints`. The new row has a foreign
key to the run row in `heartbeat_runs`.
> - The event append locks the same run row `FOR UPDATE`. That lock
blocks the `FOR KEY SHARE` lock that the foreign key check takes. So
each checkpoint save waits for the event append that is in progress.
> - On a hosted database, the two-row checkpoint upsert had the largest
total SQL time. The one-row (cursor-only) upsert, which has no foreign
key check, was much cheaper.
> - This pull request changes the append lock to `FOR NO KEY UPDATE`.
Appenders still serialize. Child foreign key checks no longer wait.
> - The benefit is shorter checkpoint saves, shorter waits before each
runner ACK, and less lock wait on the database.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The controller checkpoint save for native runs
(`nativeRunnerCheckpointWriter`), and the native run-event append
(`appendHeartbeatRunEvent`).

**Subsystem affected**
Server run-event log and native runtime checkpoint persistence.

**Current behavior**
On one hosted database, Neon query performance showed this statement as
the largest total: `insert into native_runner_checkpoints ... values
(...), (...) on conflict (run_id, part) do update`. It ran 6,582 times
at 62.6 ms (412 s) in the first 38 minutes after a compute restart, and
9,435 times at 31.8 ms (300 s) about two hours later. An earlier day
showed 100,650 calls at 102 ms. The one-row form of the same upsert ran
965 times at 13.4 ms.
The runner already writes only changed records (`captureCheckpoint`
compares plaintext), so unchanged material is not rewritten. The two-row
form is a new event row plus the cursor. Only the new row fires the
foreign key check. `appendHeartbeatRunEvent` locks the run row `FOR
UPDATE` for each native event, in a transaction with several statements.
`FOR UPDATE` conflicts with `FOR KEY SHARE`, so the check waits for that
transaction.

**Proposed behavior**
`appendHeartbeatRunEvent` locks the run row `FOR NO KEY UPDATE`. This
mode still conflicts with other appenders, `FOR SHARE`, `FOR UPDATE`,
deletes and updates. It does not conflict with `FOR KEY SHARE`. The
append never changes key columns: `runUpdate` is limited to process
fields, and the native and retry-exhaustion paths reject it. The fast
path already uses a plain `UPDATE`, which takes the same lock mode. The
execution-admission adapter already locks this row `FOR NO KEY UPDATE`.

**Reason and benefit**
The wait goes in both directions. A checkpoint save waits for an append.
An append also waits for an open checkpoint transaction that holds `FOR
KEY SHARE`. Both waits stop. The controller sends each ACK only after
its checkpoint commits, so shorter saves also shorten the event loop.

## What Changed

- `server/src/services/heartbeat-run-events.ts`: lock the run row `FOR
NO KEY UPDATE` instead of `FOR UPDATE` in the transactional append path.
Add a comment that names the contract.
- `heartbeat-run-event-atomic-append.test.ts`: add a regression test. An
append transaction stays open. A checkpoint row insert for the same run
must finish under a 2 s `lock_timeout`. A second appender must still
wait. After release, the sequence is 1, 2 with no gap and no duplicate.
The existing test with 40 concurrent appenders on two connections still
proves contiguous sequence numbers. Update the existing lock-statement
assertion to the new clause.
- Add `scripts/benchmark-run-event-checkpoint-lock.ts`. It replays a
native event stream with the real append and the real checkpoint writer
on separate connections. It times each checkpoint save for a baseline
append module and the current one.

## Verification

- Red: with the old `FOR UPDATE` clause, the new test fails with
`canceling statement due to lock timeout`. With the change it passes.
- `npx vitest run
src/__tests__/heartbeat-run-event-atomic-append.test.ts
src/__tests__/heartbeat-run-event-sequencing.test.ts
src/services/native-runtime/native-runner-checkpoint-authority.integration.test.ts
src/__tests__/native-recovery-notice-work.test.ts` (server): 4 files, 42
tests passed.
- `pnpm --filter @paperclipai/server typecheck`: passed.
- `pnpm verify` exits 2, because a new script under `scripts/` is a
global CI input. Required CI supplies the broad coverage.

Local benchmark. Embedded PostgreSQL 18.1 on a laptop. 400 events per
pass, 2 KiB event payload, 6 KiB header, real encryption. Order:
warm-up, baseline, current, baseline, current. `HOLD_MS` keeps each
append transaction open longer. It stands in for hosted round trips.

| Hold | Pass | Median save | p90 | Max | Saves > 2 ms | Total save time
|
| --- | --- | --- | --- | --- | --- | --- |
| 5 ms | Baseline (1) | 8.16 ms | 9.50 ms | 129.48 ms | 396 / 400 |
3,551 ms |
| 5 ms | Current (2) | 1.44 ms | 2.08 ms | 12.66 ms | 44 / 400 | 677 ms
|
| 5 ms | Baseline (3) | 8.12 ms | 9.64 ms | 22.08 ms | 398 / 400 | 3,453
ms |
| 5 ms | Current (4) | 1.50 ms | 2.37 ms | 10.54 ms | 80 / 400 | 700 ms
|
| 0 ms | Baseline (1) | 1.71 ms | 2.50 ms | 11.05 ms | 105 / 400 | 780
ms |
| 0 ms | Current (2) | 0.97 ms | 1.42 ms | 10.28 ms | 22 / 400 | 458 ms
|
| 0 ms | Baseline (3) | 1.49 ms | 2.02 ms | 6.45 ms | 41 / 400 | 667 ms
|
| 0 ms | Current (4) | 0.91 ms | 1.24 ms | 9.80 ms | 19 / 400 | 420 ms |

- Event order: the controller frame chain awaits each event's append and
its local commit, but not its checkpoint save. The ACK goes through
`#sendJson`, which queues the save and does not block the next frame.
Only semantic inputs and command results await the save. So the save for
event N overlaps the append of event N+1. Streaming events
(`item.delta`) also go through the same locked append, with no
checkpoint save, so a save can also overlap them. The benchmark uses
this order.
- Control without overlap (`SERIAL=1`, each save finishes before the
next append, 5 ms hold): baseline median 1.15 / 1.07 ms, current 1.03 /
1.00 ms. There is no material difference. The gain comes only from the
removed lock wait.
- Results are the same in every pass. The script asserts that all event
rows, the header and the cursor are stored, and that the run-log
sequence is contiguous.
- Command: see the script header. It needs a baseline copy of
`heartbeat-run-events.ts` from the base revision.

## Risks

- Low risk. The SQL change is one lock clause. Appenders still serialize
with each other, with `FOR UPDATE` and `FOR SHARE` lockers, and with
updates and deletes of the run row. Run deletion and key changes take
`FOR UPDATE`, so they still wait for an append, and an append still
waits for them.
- The only new concurrency is with `FOR KEY SHARE` lockers. In this code
base those are foreign key checks of child inserts. The one explicit
`FOR KEY SHARE` read is on `issues`, not on `heartbeat_runs`. A foreign
key check only needs the run row and its key to exist. The append does
not delete the row or change its key.
- The hosted cause is inferred. Neon shows calls and mean time only. We
did not read lock waits on the hosted database. The local proof shows
the mechanism and the size of the effect. A new query-performance window
after deployment must confirm the result.
- Persisted run state: no saved run, session, event or checkpoint shape
changes. State saved by the previous release carries forward unchanged.
The lock mode applies only inside one transaction, so a deploy during a
run needs no special case.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, local PostgreSQL).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 18:27:58 -06:00
DottaandClaude Opus 5.5 81c52e3666 perf(chat): find terminal GitHub responses by issue run indexes (#20671) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel posts a terminal response when an agent run
for a GitHub request stops. A background sweep finds acknowledged
response comments (`processed`, not final) whose issue is blocked or
whose latest run is terminal
> - For each candidate, the sweep reads the status of the latest run of
the assigned agent for the issue. It matches runs with
`coalesce(native_issue_id::text, context_snapshot->>'issueId')`
> - No index has that expression. So PostgreSQL reads every run of the
agent for each candidate, and it decompresses the large legacy
`context_snapshot` of each run
> - On a hosted test database, this statement ran 13 times at a mean of
11 s in a 38-minute window (149 s in total), with single runs of 16 s
and 41 s
> - This pull request splits the run match into its two disjoint
branches. Each branch uses an existing issue index
> - The benefit is that each candidate reads a few index entries instead
of the agent's full run history. The returned rows do not change, and no
migration is necessary

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The discovery page query in `registerGitHubTerminalResponseWork` in
`server/src/services/chat-channels/github/terminal-response-work.ts`.
The query runs when the sweep reconstructs its candidates, and once or
twice for each candidate check.

**Subsystem affected**
GitHub chat channel terminal-response recovery.

**Current behavior**
The `WHERE` clause has `issues.status = 'blocked' or (select r.status
from heartbeat_runs r where r.company_id = ... and r.agent_id = ... and
coalesce(r.native_issue_id::text, r.context_snapshot->>'issueId') =
<issue id> order by r.created_at desc, r.id desc limit 1) in (<terminal
statuses>)`. The planner can only use
`heartbeat_runs_company_agent_started_idx`. It reads all runs of the
agent, then evaluates the `coalesce` on each row. That decompresses the
context of each legacy run. The cost is the number of candidates times
the number of agent runs.

**Proposed behavior**
The subquery takes the latest run from two branches and then the latest
of the two:

- runs with `native_issue_id = <issue id>`, through
`heartbeat_runs_company_native_issue_contract_id_uq`
- runs with `native_issue_id is null and context_snapshot->>'issueId' =
<issue id>`, through `heartbeat_runs_company_ctx_issue_created_idx`

The two branches are disjoint, and together they are exactly the rows
where the `coalesce` is equal to the issue ID. Both branches and the
outer query use the same order (`created_at desc, id desc`). So the
selected run is the same, including ties.

**Reason and benefit**
This statement was the largest single-statement cost in the hosted
window after the earlier database fixes. A local reproduction is about
490x faster on the default fixture, and the gain grows with the agent's
run history.

**Breaking changes**
None. The API, the candidate rules and the results do not change.

## What Changed

- Rewrite the latest-run status subquery in the GitHub terminal-response
discovery page as two index-backed branches (native binding, and legacy
context without a native binding).
- Add two integration tests for the sweep. In the first, newer runs of
the same agent that do not bind the issue (one with a native binding to
another issue, but a legacy context that names this issue) must not hide
the stopped run. In the second, a newer active run with a native binding
to the issue keeps the request waiting.
- Add `scripts/benchmark-github-terminal-response-sweep.ts`. It captures
the SQL that the real worker generates at a baseline commit and at
`HEAD`, compares all rows, and measures both.

## Verification

Cause and thesis. The cost comes from the `coalesce` run match, which
has no index and makes each candidate scan the agent's run history.
Thesis: if each precedence branch uses its own issue index, the cost of
the page no longer depends on the agent's run count, and the rows stay
identical. The benchmark confirms the thesis.

Local SQL benchmark:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-github-terminal-response-sweep.ts --baseline-ref=7b33384446
```

Fixture (synthetic): two companies, each with one GitHub endpoint and
one assigned agent. Each issue has one GitHub response comment. The
first N comments are acknowledged and not final; the others are final.
Every tenth issue is blocked. Runs alternate between native binding and
legacy context, and each run has a 4 KiB incompressible context. The
fixture also has these edge cases: newer active runs with a native
binding to another issue but a legacy context that names this issue,
active runs that tie on `created_at` with the latest run (between native
and legacy runs), and newer active runs of another agent. The benchmark
compares every column of every candidate row (the page limit is lifted
for this check). Embedded PostgreSQL 18.1 on an Apple laptop. The hosted
database is Neon with different compute, so absolute times are local SQL
measurements only. Each size has 2 warmups and then alternates the
baseline and the treatment.

| Size | Runs (both companies) | Candidate rows equal | Baseline median
(range) | Treatment median (range) | Speedup |
| --- | --- | --- | --- | --- | --- |
| Small: 200 issues, 3 runs each, 20 candidates, 7 samples | 1,372 | 26
of 26 | 18.7 ms (17.7–20.8) | 3.8 ms (3.3–4.3) | 4.9x |
| Default: 1,500 issues, 8 runs each, 150 candidates, 7 samples | 25,300
(156 MB) | 200 of 200 | 3,697 ms (2,875–4,032) | 7.6 ms (6.6–9.6) | 490x
|
| Large: 3,000 issues, 10 runs each, 400 candidates, 5 samples | 62,600
(382 MB) | 538 of 538 | 57,055 ms (48,343–66,079) | 31 ms (16–79) |
1,800x |

Sizes are set with `BENCH_ISSUES`, `BENCH_RUNS_PER_ISSUE`,
`BENCH_CANDIDATES` and `BENCH_ITERATIONS`. The timed query is the real
first page (limit 100).

Plan (default size): the baseline runs `SubPlan` 270 times. Each run is
a `Bitmap Heap Scan` on `heartbeat_runs_company_agent_started_idx` that
removes about 12,000 rows by filter, for about 8.2 million buffer reads
in total. The treatment runs an `Append` of two `Limit` index scans
(`heartbeat_runs_company_native_issue_contract_id_uq` and
`heartbeat_runs_company_ctx_issue_created_idx`), for about 3,800 buffer
reads in total.

Tests:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "terminal GitHub"`:
13 passed, including the 2 new tests.
- Mutation check: when the legacy branch does not exclude runs with a
native binding, the first new test fails. It passes with the fix.
- `cd server && npx tsc --noEmit`: passed.

## Risks

- Low risk. The change is one SQL subquery. The match is the same set of
runs: `native_issue_id::text = <uuid text>` is equal to `native_issue_id
= <uuid>`, and a null `native_issue_id` makes the `coalesce` use the
legacy context.
- The candidate set itself is not bounded by this change. Acknowledged
responses that never become final stay in the candidate set. Each
candidate is now cheap, but the count still grows with that history.
- `readGitHubResponseRun` in `response-run-state.ts` uses the same
`coalesce` match. The sweep calls it once for each candidate row, and
other GitHub paths also call it. Each call is a separate, smaller
statement. This pull request does not change it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 18:25:59 -06:00
DottaandClaude c14f2fe45d fix(native): read the scheduled-resume evidence the resume reconciler claimed (#20656) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #20645 lets a local same-run scheduled resume send the one
same-conversation continuation for a turn lost on restore. It applies
only when durable records prove three facts: the previous runner
stopped, the attempt was admitted within the budget, and it is the same
checkpointed conversation.
> - It read the third proof from the coordinator's saved
`failure_detail` while the phase was `retryable_failure`.
> - In production, the resume reconciler
(`claimNativeSessionResumptions`) claims the run first. It moves the
coordinator to `observed` and clears `failure_detail`. So the executor
never saw that evidence, and #20645 refused every real scheduled resume.
> - The restart-chaos L7 wave showed this live. Codex B5 TM2 hit the
exact path (`runner.drain` timeout, scheduled resume,
`provider_turn_lost_on_restore`), sent no continuation, and needed
Retry.
> - This pull request makes the reconciler record that evidence durably
when it claims the run, and makes the executor read it from there.
> - The benefit is that #20645 works on the real path. The proof rule is
unchanged.

## Linked Issues or Issue Description

Refs #20645

**What happened?**
Wave `w-20261110T212307-s9107` (L7, seed 9107, local, master plus
#20645), task codex B5 TM2:
- `native_session_recovery_failed: runner_local_connect_failed: PRP
command runner.drain timed out`;
- `retryable_failure` with `recoveryMode: "exact_checkpoint_resume"`,
`recoverable`, `providerSessionEstablished` and `checkpointExists` all
set;
- `native.local_process_stopped`;
- attempt 2 started;
- `turn.failed` with `provider_turn_lost_on_restore` for the same turn
ID that the checkpoint was running.

No continuation was sent. The run failed with
`native_provider_terminal_failed` and was preserved with
`uncertain_provider_action`.

**Expected behavior**
The proven scheduled resume sends the single continuation.

**Steps to reproduce**
The new integration test below. The live path is timing-dependent (the
L7 command in #20645).

## What Changed

- `native-finalization-reconciler.ts`: when it claims a same-run resume,
the reconciler appends one `same_run_resume` entry to
`recovery_history`. The append is bounded like other history writes, and
it runs in the same update that clears the failure. The entry copies
`recoveryMode`, `recoverable`, `providerSessionEstablished`,
`checkpointExists`, `successorRunId` and `replacementDenied` from the
saved failure, with the controller boot and provider attempt.
- `scheduled-resume-continuation.ts`: the evidence for the third proof
now comes from either of two durable records:
- the saved failure, when the executor claims a `retryable_failure`
coordinator directly (unchanged); or
- a reconciler claim: an `observed` coordinator whose last history entry
is a `same_run_resume` entry naming the executor's own controller boot.
The two other proofs are unchanged: the stop receipt, and an admitted
attempt from 1 to 3.
- `native-session-executor.ts`: passes the coordinator's last history
entry and the controller boot to the decision.
- `doc/execution-semantics.md`: describes the recorded evidence.

Readers of `recovery_history` that look at the last entry do so only for
restart claims, Stop recovery and shutdown suspension. Restart
recovery's previous-claim lookup skips non-claim dispositions, so a
`same_run_resume` entry does not change those paths.

**Still refused** (tested):
- every #20645 case;
- the last history entry is not the resume claim, or there is no
history;
- the resume claim was recorded by another boot;
- the claimed evidence is a bootstrap retry, has no checkpoint, or
records a replacement;
- a reconciler-claimed resume without a stop receipt.

## Verification

- `native-scheduled-resume-continuation.integration.test.ts` now also
runs through the real reconciler claim (`claimNativeSessionResumptions`)
before `executePaperclipNativeSession`. On master the reconciler-claimed
allowed case fails (`resumeInterruptedTurn` is false). With this change
it is true, and the claim admits attempt 2. A reconciler-claimed resume
without a stop receipt still passes false.
- `scheduled-resume-continuation.test.ts`: decision cases for the
reconciler-claim entry.
- `native-session-resumption.test.ts` passes. 54 tests pass across the
three files.
- `cd server && npx tsc --noEmit` passes.
- Live: not yet. The failing wave above is the reproduction. A rerun is
timing-dependent.

## Risks

- One more `recovery_history` entry per scheduled resume, inside the
existing 20-entry bound.
- The proof rule itself is unchanged. This only reads the same evidence
from the record where it now survives.
- No schema change. A run claimed by an earlier build has no
`same_run_resume` entry, so it keeps the existing failure: refused, with
Retry offered as today.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 18:21:25 -06:00
DottaandClaude Opus 5.5 48c6e996da fix(native): continue a task whose quarantined completed turn's holds settle (#20662) 
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a remote run starts, workspace preparation attaches the
verified saved work of earlier runs of the same task.
> - Preparation failed with `remote_workspace_recovery_receipt_invalid`
when any earlier run had a recovery receipt that it could not verify.
> - A refusal cannot change the saved state, so every Retry failed in
the same way, and the task stayed blocked. Recoveries must fall back
automatically.
> - This pull request skips an unverifiable source. It attaches nothing
unverified and starts from the workspace's newest verified save.
> - The benefit is that Retry and continuations start again, while
strict receipt checks still decide what is attached, and no saved work
is changed.

## Linked Issues or Issue Description

Refs #20631

**What happened?**
On staging, after the deploy that included #20631, a user pressed Retry
on a task whose previous run failed with
`native_controller_recovery_missing_evidence:runner_not_settled`. The
new run failed in setup in under a minute with `setup_failed` /
`remote_workspace_recovery_receipt_invalid` (`executionRecovery:
bootstrap`, `providerWorkStarted: false`). The task stayed blocked.
Another Retry would fail the same way. #20631 accepted one shape of this
refusal (a source that saved only the snapshot its own restore loaded).
This task had a different shape.

**Expected behavior**
A Retry starts. A source whose receipt cannot be verified is not
attached, and its saved state stays intact for workspace recovery.

**Steps to reproduce**
1. Let a remote native run fail before provider work and leave a
recovery receipt or restore hold that does not verify (for example a
receipt on an unsaved reference, a receipt from another epoch, or a
saved reference without a receipt).
2. Press Retry on the task.
3. See the new run fail in setup with
`remote_workspace_recovery_receipt_invalid`.

**Paperclip version or commit**
Staging deploy that includes `e3bc4ca798`. The gap remains on `master`
at `a3b34a9d88`.

## What Changed

- `attachPreservedRemoteWorkspaceSnapshots` no longer throws for an
unverifiable source. Each of the five refusal shapes now skips the
source and logs `remote_workspace_recovery_receipt_invalid` with a
reason: `receipt_without_reference`, `reference_unreadable`,
`receipt_on_unsaved_reference`, `receipt_unverified`, and
`receipt_snapshot_missing` (was
`remote_workspace_recovery_snapshot_missing`).
- `retainRemoteRecoveryAttachments` no longer throws for a retained
recovered copy that it cannot verify. It does not publish that copy
again. The files stay in the workspace.
- The writer check (`remote_workspace_recovery_writer_changed`), the
workspace access check, and snapshot read and digest failures still
refuse.
- `doc/execution-semantics.md` describes the fallback rule.

## Verification

- Red on unfixed code:
`remote-workspace-recovery-receipt-fallback.test.ts` (moved out of
`workspace-recovery-preparation.test.ts` to keep that file under 1,000
lines) "starts a Retry from the newest verified save when a predecessor
has %s" (6 shapes: `receipt_without_reference`,
`unreadable_reference_with_hold`, `receipt_on_prepared_reference`,
`superseded_snapshot_with_hold`, `receipt_from_other_epoch`,
`receipt_for_other_run`). All 6 failed with
`remote_workspace_recovery_receipt_invalid`. The source is a native run
that failed with `runner_not_settled`, and the writer is a Retry
(`retryOfRunId`, `taskRetryRequestId`) that follows the real ancestry
query.
- Green: the same 6 tests pass. They also assert that nothing is
attached, and that the source run row, its workspace operations, and its
snapshot row are unchanged.
- Updated safety tests: the canonical-attachment fault matrix now
expects no attachment for receipt faults, and still expects a refusal
for a missing or corrupt archive and for a wrong company or task. The
retained-history test expects an unverifiable retained copy to be left
out of the published list, while verified copies stay published.
- `npx vitest run
src/services/remote-workspace-recovery-receipt-fallback.test.ts
src/services/workspace-recovery-preparation.test.ts`: 91 passed.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` returns exit 2, because a global doc input changed
(`doc/execution-semantics.md`). Required CI supplies the broad coverage.
- Not verified live on Daytona. I could not read the staging rows for
this task: the browser had no free tab. The tests cover every refusal
shape of this function, so the exact staging shape is one of them.

## Risks

- Previous-release state: the rule reads the same rows as before (run
reference, result hold, workspace operations, recovery actions,
snapshots). Rows that earlier releases saved carry forward. No
migration. A blocked task recovers on its next Retry or wake.
- A Retry can now start without files from an unverifiable source. Those
files are not lost: the source's snapshot, restore hold, receipts and
recovery action stay unchanged, and workspace recovery still owns them.
The skip is logged with a named reason.
- A retained copy that fails verification is dropped from the published
list. Its directory stays in the workspace and is no longer retired
automatically.
- A missing or corrupt snapshot archive still refuses. A permanently
lost archive therefore still blocks attach. This PR does not change
that.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits). Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 18:06:41 -06:00
DottaandClaude Opus 5.5 8f6ca25976 fix(workspaces): skip unverifiable recovery sources instead of failing Retry (#20666) 
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a remote run starts, workspace preparation attaches the
verified saved work of earlier runs of the same task.
> - Preparation failed with `remote_workspace_recovery_receipt_invalid`
when any earlier run had a recovery receipt that it could not verify.
> - A refusal cannot change the saved state, so every Retry failed in
the same way, and the task stayed blocked. Recoveries must fall back
automatically.
> - This pull request skips an unverifiable source. It attaches nothing
unverified and starts from the workspace's newest verified save.
> - The benefit is that Retry and continuations start again, while
strict receipt checks still decide what is attached, and no saved work
is changed.

## Linked Issues or Issue Description

Refs #20631

**What happened?**
On staging, after the deploy that included #20631, a user pressed Retry
on a task whose previous run failed with
`native_controller_recovery_missing_evidence:runner_not_settled`. The
new run failed in setup in under a minute with `setup_failed` /
`remote_workspace_recovery_receipt_invalid` (`executionRecovery:
bootstrap`, `providerWorkStarted: false`). The task stayed blocked.
Another Retry would fail the same way. #20631 accepted one shape of this
refusal (a source that saved only the snapshot its own restore loaded).
This task had a different shape.

**Expected behavior**
A Retry starts. A source whose receipt cannot be verified is not
attached, and its saved state stays intact for workspace recovery.

**Steps to reproduce**
1. Let a remote native run fail before provider work and leave a
recovery receipt or restore hold that does not verify (for example a
receipt on an unsaved reference, a receipt from another epoch, or a
saved reference without a receipt).
2. Press Retry on the task.
3. See the new run fail in setup with
`remote_workspace_recovery_receipt_invalid`.

**Paperclip version or commit**
Staging deploy that includes `e3bc4ca798`. The gap remains on `master`
at `a3b34a9d88`.

## What Changed

- `attachPreservedRemoteWorkspaceSnapshots` no longer throws for an
unverifiable source. Each of the five refusal shapes now skips the
source and logs `remote_workspace_recovery_receipt_invalid` with a
reason: `receipt_without_reference`, `reference_unreadable`,
`receipt_on_unsaved_reference`, `receipt_unverified`, and
`receipt_snapshot_missing` (was
`remote_workspace_recovery_snapshot_missing`).
- `retainRemoteRecoveryAttachments` no longer throws for a retained
recovered copy that it cannot verify. It does not publish that copy
again. The files stay in the workspace.
- The writer check (`remote_workspace_recovery_writer_changed`), the
workspace access check, and snapshot read and digest failures still
refuse.
- `doc/execution-semantics.md` describes the fallback rule.

## Verification

- Red on unfixed code:
`remote-workspace-recovery-receipt-fallback.test.ts` (moved out of
`workspace-recovery-preparation.test.ts` to keep that file under 1,000
lines) "starts a Retry from the newest verified save when a predecessor
has %s" (6 shapes: `receipt_without_reference`,
`unreadable_reference_with_hold`, `receipt_on_prepared_reference`,
`superseded_snapshot_with_hold`, `receipt_from_other_epoch`,
`receipt_for_other_run`). All 6 failed with
`remote_workspace_recovery_receipt_invalid`. The source is a native run
that failed with `runner_not_settled`, and the writer is a Retry
(`retryOfRunId`, `taskRetryRequestId`) that follows the real ancestry
query.
- Green: the same 6 tests pass. They also assert that nothing is
attached, and that the source run row, its workspace operations, and its
snapshot row are unchanged.
- Updated safety tests: the canonical-attachment fault matrix now
expects no attachment for receipt faults, and still expects a refusal
for a missing or corrupt archive and for a wrong company or task. The
retained-history test expects an unverifiable retained copy to be left
out of the published list, while verified copies stay published.
- `npx vitest run
src/services/remote-workspace-recovery-receipt-fallback.test.ts
src/services/workspace-recovery-preparation.test.ts`: 91 passed.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` returns exit 2, because a global doc input changed
(`doc/execution-semantics.md`). Required CI supplies the broad coverage.
- Not verified live on Daytona. I could not read the staging rows for
this task: the browser had no free tab. The tests cover every refusal
shape of this function, so the exact staging shape is one of them.

## Risks

- Previous-release state: the rule reads the same rows as before (run
reference, result hold, workspace operations, recovery actions,
snapshots). Rows that earlier releases saved carry forward. No
migration. A blocked task recovers on its next Retry or wake.
- A Retry can now start without files from an unverifiable source. Those
files are not lost: the source's snapshot, restore hold, receipts and
recovery action stay unchanged, and workspace recovery still owns them.
The skip is logged with a named reason.
- A retained copy that fails verification is dropped from the published
list. Its directory stays in the workspace and is no longer retired
automatically.
- A missing or corrupt snapshot archive still refuses. A permanently
lost archive therefore still blocks attach. This PR does not change
that.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits). Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 17:47:54 -06:00
DottaandClaude 9296c74e8d fix(native): let a restart claim take back its lapsed lease at budget admission (#20650) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a control-plane restart, restart recovery claims an
interrupted native run (`resume_dead_runner`) and grants a 20-minute
lease. Budget admission then re-checks that the claim still owns the run
before the resumed run may start provider work.
> - A host sleep, or the restart-chaos fault TM2, can make that stored
lease look expired while its owner is alive. #20625 lets the owner take
back such a lease when the row still shows exactly its own claim. That
rule did not cover budget admission.
> - So a claim whose lease lapsed before dispatch was refused with
`Native recovery no longer owns provider dispatch`. The run failed with
`adapter_failed`, recovery held it with
`native_continuation_requires_reconciliation`, and the user had to click
Retry.
> - This pull request applies the #20625 rule in budget admission: the
claim takes its lease back through the existing re-acquire condition.
> - The benefit is that a run resumed after a restart survives a time
jump before dispatch with no Retry. Every fence and every budget check
stays the same.

## Linked Issues or Issue Description

Refs #20625, #20474

**What happened?**
Restart-chaos wave L7 (seed 9107, local) on a branch with #20625, task
codex B5 TM2. A TM3 restart claimed the run (`resume_dead_runner`,
generation 2). TM2 then aged every database time by one hour before the
resume reached dispatch. Budget admission
(`lockRunBudgetRecoveryOwnerInTransaction`) required a live lease and
threw `Native recovery no longer owns provider dispatch`. The run failed
with `adapter_failed`, recovery stopped with
`native_continuation_requires_reconciliation` /
`preserve_without_replay_v1`, and the task needed Retry.

**Expected behavior**
The claim's own lapsed lease is taken back, and the resume is admitted
when the budget allows it.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--env local --adapters codex,claude --workloads A1,B5,C1 --faults
TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107`
(timing-dependent).

## What Changed

- `native-execution-lease.ts`: two new helpers. Both use the #20625
own-claim rule. Neither takes a lock out of order.
- `readOwnNativeRecoveryClaimLease` (read-only) returns `live`,
`lapsed_own_claim` or `lost`. It compares the lease against the database
clock. A lapsed lease counts as `lapsed_own_claim` only when all of
these hold:
    - the same lease owner;
    - this process's controller boot, pid and process start;
    - a non-null `recovery_state`;
- `nativeExecutionLeaseHeldCondition({ remote: false,
controllerGeneration })`, which applies the #20625 own-claim rule and
every #20474 untouched check;
    - no non-local environment lease on the run.
- `extendOwnNativeRecoveryClaimInTransaction` extends such a lease by
the local execution TTL, with one conditional UPDATE on the same
predicate.
- `budget-reservations.ts`:
- `lockRunBudgetRecoveryOwnerInTransaction` still refuses another owner,
and refuses a lapsed lease unless it is the run's own untouched claim.
It no longer locks the run row, so the documented coordinator → issue →
run order holds.
- The lease is extended only at the end of an admitted reservation
(`reserveRunBudget`) or an admitted dispatch recheck
(`recheckRunBudgetReservationInTransaction`). At those points the caller
already holds the run row lock. A refused admission, for example one
over budget, leaves the lease lapsed.
- `heartbeat-native-restart-suspension.test.ts`: the test "still settles
a recovery whose claim merely expired" encoded "expired means not the
owner". #20474 and #20625 changed that rule for an untouched own claim.
The test now has two cases:
- an expired claim that another actor recorded on (`awaiting_evidence`)
still settles with no provider session;
  - an untouched expired claim is dispatched.
- `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1
lists the new user of the rule.

## Verification

- New
`server/src/__tests__/native-recovery-dispatch-lease-reacquire.test.ts`:
- `reserveRunBudget` takes back the claim's own lapsed lease and admits
the resume. This fails before the change.
  - An over-budget resume is still refused, and its lease stays lapsed.
- Fenced variants each throw `Native recovery no longer owns provider
dispatch` and leave the row unchanged: a graceful-shutdown suspension,
an appended failure settlement, `awaiting_evidence`, `blocked`, a claim
from an earlier generation, another boot, a remote sandbox lease, and
another lease owner.
- At the dispatch gate (`recheckRunBudgetReservationInTransaction`), an
admitted own claim gets its lease extended, and a fenced claim is
refused with the row unchanged.
- Related suites pass (308 tests):
`native-execution-lease-reacquire.test.ts`,
`closed-native-receipt-accounting.test.ts`,
`heartbeat-native-predispatch-retry.test.ts`,
`run-dispatch/adapters/postgres.test.ts`,
`heartbeat-native-restart-suspension.test.ts` and
`heartbeat-restart-claim-lapse.test.ts`.
- `cd server && npx tsc --noEmit` passes.
- Live: one L7 wave (seed 9107) ran on the first commit of this branch
(`w-20261110T212307-s9107`). It logged 0 `Native recovery no longer owns
provider dispatch` refusals, against 1 in the previous wave. Codex B5
TM2 got past dispatch, but it still needed Retry for another reason:
after a `runner.drain` timeout, the scheduled same-run resume did not
send the continuation of #20645. That case is tracked separately. The
other five tasks passed.

## Risks

- This widens when a restart claim can take back its own lapsed lease,
and only at budget admission. It is local only. It needs the same owner,
boot and generation, and the row must show exactly that claim. A claim
that any other actor touched still fails as before.
- Budget hard-stop behaviour is unchanged. The lease is extended only
after budget admission passes.
- No schema change and no migration. Rows saved by earlier releases
carry the same history shape.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 17:43:46 -06:00
DottaandPaperclip de7ceac0de fix: preserve compatible historical Codex WEB evidence (#20665) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native session migration preserves local evidence in encrypted
archives.
> - Each WEB receipt binds the exact provider directory and exclusion
policy.
> - The Codex credential refresh change added two excluded files and
changed that policy hash.
> - A new reader then rejected old receipts even when their facts,
files, skills, and authority still matched.
> - This pull request accepts the exact prior Codex policy while
checking archive paths against the current credential policy.
> - Operators can keep valid saved evidence across the upgrade without
copying it again.

## Linked Issues or Issue Description

Refs: #19814, #20646.

**What happened?**

A reader upgrade rejected saved Codex WEB evidence because the profile
hash changed when `auth-refresh.json` and `auth-refresh.json.tmp` became
excluded credentials. The other receipt bindings still matched. This
blocked the native harness migration gate.

**Expected behavior**

The exact known prior profile remains readable when the current archive
path checks accept every saved entry. An archive that includes either
newly excluded credential file must still fail. Unknown profiles and
changed authority must fail.

**Steps to reproduce**

1. Preserve WEB evidence with the frozen prior Codex profile and no
credential refresh files.
2. Read its unchanged receipt with the current profile.
3. Observe that the old reader returns false solely because the profile
hash changed.

**Paperclip version or commit**

Reproduced against `a3b34a9d88`; final branch is based on `e9343d8501`.

## What Changed

- Accept the exact prior Codex profile hash only when the complete
current profile has the reviewed successor hash and the provider is
Codex.
- Keep the current entry and excluded-path checks. Keep facts,
controller, archive, skill, and ownership checks.
- Add a frozen prior policy fixture with its literal hash. Test old
receipt reads, both credential files, unknown policies, changed
authority, and current writes.

## Verification

- Real PostgreSQL baseline: the frozen old-policy receipt test failed
because the reader returned false. The unchanged fixture passes with
this fix.
- All 18 selected PostgreSQL cases pass on final head after the master
rebase: six new cases and twelve existing WEB archive and authority
guards (14.93 seconds).
- Full layered server typecheck passes on final head after the rebase.
The module boundary check passes.
- A single read-only native-only candidate check on the deployed
dependency graph validated all 54 retained scopes. The 29 prior failures
passed, and the other 25 stayed valid. The check made no storage calls
or data writes. It did not replace final deployed enrollment acceptance.
- Final head `14112d0e3abb6e318a86fe9799d95bddd2dbf76b` passes all
[required CI
checks](https://github.com/paperclipai/paperclip/actions/runs/42919867108):
55 successful checks, four expected skips, and one neutral review. The
legacy security status also passes. Exact-head Greptile review is 5/5
with no open review threads.

## Risks

The compatibility branch accepts only the two reviewed policy hashes. A
future policy change can require another explicit compatibility
decision. The reader still rejects saved credential refresh files. It
does not rewrite receipts or change execution authority. No stored
schema or operator command changes.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window capacity are not exposed in this session. Used reasoning,
source editing, shell execution, and tool calls.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 17:22:08 -06:00
DottaandPaperclip aa2a7be8b4 fix: recognize provider-owned Runner workspace storage (#20663) 
## Thinking Path

> - Paperclip manages agent work and preserves its execution state.
> - A server replacement must retain every workspace that depends on the
old server disk.
> - Runner task environments can own their workspace outside the Core
server.
> - These environments save an adapter-managed workspace with a
runnerTaskWorkspaceOwner binding.
> - Readiness and maintenance treated that provider path as a Core
filesystem dependency.
> - This change validates the saved owner and shares that classification
between both paths.
> - Local, malformed, and foreign workspace bindings still block
enrollment.

## Linked Issues or Issue Description

Related enrollment safeguards: #18835. A GitHub search found no
duplicate fix for this provider-owned class.

**What happened?**

Volume-free readiness returned workspaces_require_migration for an
adapter-managed Runner workspace owned by a plugin task environment.
Maintenance also selected it and failed with
workspace_provider_unsupported. The provider owned the path, which did
not exist on the Core server.

**Expected behavior**

A valid provider-owned workspace does not require migration from the
Core server disk. Ordinary local workspaces and invalid ownership
bindings remain blockers.

**Steps to reproduce**

1. Run a paperclip_runner agent in a plugin task environment.
2. Retain its adapter-managed workspace and complete
runnerTaskWorkspaceOwner binding.
3. Read startup eligibility for volume-free enrollment.
4. Observe the workspace blocker before this fix.
5. Run workspace maintenance and observe workspace_provider_unsupported
for the same row.

**Paperclip version or commit**

Reproduced on 353945c86d. The regression
also fails on the branch base a3b34a9d88.

**Deployment mode**

Docker with external PostgreSQL. Local regression coverage uses
disposable PostgreSQL.

## What Changed

- Share one metadata predicate between startup eligibility and offline
workspace migration selection.
- Validate a complete saved owner against a same-company
paperclip_runner agent and a plugin environment with matching nonempty
keys. Per-run environment overrides remain supported.
- Keep ordinary local workspaces and invalid ownership bindings in the
migration set.
- Add real PostgreSQL coverage for provider-owned readiness, zero-import
maintenance selection, unchanged retained rows, and
local/foreign/malformed ownership controls.

## Verification

- Readiness RED: the provider-owned fixture failed with
workspaces_require_migration on unchanged source.
- Maintenance RED: the provider-owned fixture failed with
workspace_provider_unsupported before the shared selector was applied.
- GREEN: 29 selected tests passed across three files. This covers 8
eligibility/configuration tests and 21 migration/offline-admission
cases, including eleven invalid/local owner controls inside the
eligibility test. The affected migration file has 207 cases; broad
coverage remains with required CI.
- Commands: `pnpm exec vitest run
server/src/__tests__/ephemeral-server-data.test.ts
server/src/__tests__/ephemeral-server-contract.test.ts` plus focused
selections in `server/src/services/remote-workspace-migration.test.ts`
for provider ownership, offline admission, local imports,
interrupted-batch resume, and unresolved workspace semantics.
- Server layered typecheck passed after the final source change.
- Final head `2fb98dbc664da14e6f60d812e1e10877b2b7f00a`: [required
CI](https://github.com/paperclipai/paperclip/actions/runs/42920262773)
completed with 55 successful checks, 1 neutral review, and 4 intended
skips. Greptile and Paperclip Review both rated this head 5/5. No
unresolved review threads remain.
- No hosted data migration or metadata rewrite was performed.

## Risks

- This changes a readiness classification. Incomplete, foreign,
non-Runner, or mismatched environment bindings remain conservative
blockers.
- The check validates saved ownership through current database bindings.
It does not inspect provider files or grant execution authority.
- Existing saved rows retain their shape. No migration or archive format
changes are required.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
editing, and real PostgreSQL tests. The host does not expose the exact
model ID or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 17:22:01 -06:00
Nicky Leach 0d4b1d8c89 Gate agent plugin tabs by execution environment (#20657) 2026-11-10 15:04:44 -08:00
Devin FoleyandPaperclip e9343d8501 fix(heartbeat): release task Retries held by a drain as soon as it ends (#20640) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person clicks Retry on a stopped task, Paperclip saves the
request as a held wake and dispatches it right away, unless admission is
held. Admission is held during a task drain, for example while a
deployment restarts the server
> - A Retry held by a drain is released later by the deferred-receipt
worker. That worker shares one queue with every other held wake: comment
waits, execution waits, and interrupted queues
> - On a busy instance that queue holds hundreds of passive waits that
come due every 30 seconds. A Retry is checked in due-time order behind
all of them. Ending a drain does not wake the worker either
> - In a real case, a Retry clicked during a deployment drain was still
held 40 minutes later, after the drain expired and the server restarted.
It ran only when the person clicked Retry again
> - This pull request checks operator Retries ahead of passive waits,
and wakes the worker as soon as a drain stops or expires
> - The benefit is that a Retry held by a drain starts within one worker
pass after the drain ends

## Linked Issues or Issue Description

**What happened?**

A person clicked Retry on a blocked task 14 seconds after a deployment's
task drain started. The Retry was saved as `deferred_issue_execution`,
and the immediate dispatch returned early because of the drain. The
drain had a 20-minute TTL, and the deployment restarted the server about
6 minutes in. The held Retry still did not start, and its `updated_at`
never changed. Forty minutes later the person clicked Retry again, and
the new request started a run and merged with the held one.

That instance had 390 held wakes: 271 execution waits and 116 others,
against 6 task Retries. The execution waits come due every 30 seconds.
Measured on the instance, the deferred-receipt worker took about 7
minutes to get through that backlog once.

**Expected behavior**

A Retry that a drain held starts soon after the drain stops or expires,
or after the server restarts, whatever the size of the passive backlog.

**Steps to reproduce**

1. Start a non-idle task drain (`POST /instance/task-drain`).
2. Click Retry on a stopped task. The receipt is
`deferred_issue_execution`.
3. Stop the drain. On `master`, the deferred-receipt worker only notices
on its next 30-second re-check. It then checks the Retry in due-time
order behind every passive wait that came due before it.

**Paperclip version or commit**

`master` at 8148fc0b1e.

## What Changed

- `server/src/services/scoped-deadline-work.ts`:
- New optional `priority(entry)`. Due entries are sorted by priority
first, then by due time and id as before.
- New `refresh(hint)`. It clears a scope's reconstruction delay and
invalidates it. Failure backoff is kept.
- New optional `onStop` hook, called when the coordinator stops the
worker.
- `server/src/services/heartbeat/deferred-receipt-work.ts`:
- A `task_retry` receipt that a person requested is an operator Retry,
with priority 1. Passive receipts keep priority 0, so their order does
not change.
- Each deferred scan loads operator Retries first, with their own query,
then pages by id as before. A Retry is therefore in memory in the first
pass even beyond the first 100-row page. A receipt loaded twice merges
by id.
- New optional `owner.subscribeRefresh`. The work calls `refresh()` on
all three of its queues when it fires, and releases the subscription
from `onStop`, so the coordinator's own stop removes it.
- `server/src/services/heartbeat.ts`: passes `subscribeRefresh` that
listens to task-drain changes and expiry. Before this change, a deploy
drain left the worker waiting for its 30-second re-check, behind the
scan's reconstruction delay.
- `server/src/__tests__/deferred-execution-receipt-work.test.ts`, four
new tests:
- A real Retry, requested during a deploy drain, starts a run (to
`succeeded`) within 5 seconds after the drain stops.
- An operator Retry whose id sorts after 120 passive execution waits
(more than one load page) is resumed first, in the first worker pass.
- A held Retry inserted with no notification right after a full scan is
resumed as soon as the drain refresh fires.
  - Stopping only the coordinator removes the drain listener.

## Verification

- `npx vitest run src/__tests__/deferred-execution-receipt-work.test.ts
-t "held by a deploy drain|ahead of a backlog|ordinary committed task
Retry"`: 3 pass. The third is the existing ordinary Retry dispatch test.
- Red checks, removing one piece at a time:
- Without the drain subscription, the drain test fails: the Retry is
still `deferred_issue_execution` with no run after 5 seconds.
- Without the priority, or without the prioritized load, the backlog
test fails: the first resumed request is the passive `00000000-…` wake.
- With `invalidate()` instead of `refresh()`, the post-scan test fails
after 5 seconds.
  - Without the `onStop` release, the listener test fails.
- Full `deferred-execution-receipt-work.test.ts`: 50 pass.
- `chat-channels.integration.test.ts -t "still batches several replies
in one finding thread"`: it timed out once on CI and passes 2 of 2 times
locally with this change.
- `scoped-deadline-work`, `heartbeat-task-drain`,
`heartbeat-task-drain-admission-release`, `cloud-control-task-drain`,
`task-admission-deadline`, `onboarding-first-task-admission` and
`delivery-work-coordinator` suites: 164 pass.
- `pnpm run typecheck` in `server` (layered): clean.

## Risks

- Operator Retries are a small set; there were 6 on the instance above.
They now take the first slots of a worker pass, so passive waits can
move by at most that many places. Passive waits keep their own order.
- A Retry that keeps failing its check still gets the existing failure
backoff, so it cannot take every pass.
- A drain start also wakes the worker. That pass ends at once, because
the worker's run gate still reads the drain.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use
(shell, file editing, vitest runs against embedded PostgreSQL, read-only
production log and database analysis).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 15:00:50 -08:00
DottaandClaude Opus 5.5 686893636d fix(recovery): release a lapsed owner of a finished coordinator so stop-proof holds re-arm (#20661) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Workspace recovery saves the files of a run whose copy-back failed,
and it holds the shared workspace until the save is done
> - #20295 re-arms a save that was exhausted with
`agent_stop_unconfirmed`, so the run's stop is proven again and the
files are saved without a person
> - On staging, four holds of this shape were never re-armed. Their runs
ended six days ago, their sandbox leases are released with a provider
stop receipt, and the servers that ran them were replaced many times.
Their tasks stay blocked, and so do the parent tasks that wait for them
> - The finished coordinator of each run still names a lease owner whose
lease lapsed. Discovery treats that owner as "waiting for the
abandoned-owner settlement", but that settlement handles only observed
(unfinished) coordinators. Nothing ever releases the owner, so the
worker never selects the hold, the re-arm refuses it, and the attempt's
stop check refuses it too
> - This pull request makes a lapsed owner of a finished coordinator
count as stopped. The re-arm releases it in the same transaction, and
then the new attempt proves the sandbox stop and saves the files as
#20295 designed
> - The benefit is that these holds settle on the worker's next pass
after deploy, without a migration and without an administrator

## Linked Issues or Issue Description

Refs #20295

**What happened?**
An exhausted workspace save
(`native_workspace_sync_out_retry_exhausted`, last failure
`agent_stop_unconfirmed`) stayed blocked for six days. The task showed
"Paperclip tries again automatically", but no re-arm was ever recorded
(no `stopProofRecovery` marker, no `issue.workspace_recovery_rearmed`
activity), and the shared workspace stayed held by the failed run.

**What did you expect to happen?**
The recovery worker re-arms the save once the run is provably over,
saves the dirty files, and frees the workspace so the task and its
parent can continue.

**Steps to reproduce**
1. Exhaust a workspace save with `agent_stop_unconfirmed` (one attempt,
window spent).
2. Leave a lease owner on the run's `terminal_failure` coordinator with
an expired lease (or no expiry), as a finalizer pass of a replaced
server does.
3. Run the workspace recovery worker. `nextWorkspaceRecoveryAt` returns
null, discovery does not select the hold, and `rearmExhaustedStopProof`
refuses it.

## What Changed

- `workspace-recovery-stop-proof.ts`: add
`lapsedFinishedCoordinatorOwner` and `releaseLapsedCoordinatorOwner`. A
lease owner on a finished coordinator (committed, terminal failure,
applied) whose lease lapsed, or that has no expiry, does not block
`sourceRunProvablyOver`. The stop-proof re-arm clears that owner with a
compare-and-swap in its locked transaction, and records
`releasedLapsedOwner` in the marker and the activity.
- `workspace-recovery-capture-policy.ts`: the capture-policy re-arm uses
the same rule, because it shares the same check and discovery clause.
- `workspace-recovery-fallback-discovery.ts`: the discovery clause still
waits for the abandoned-owner settlement for a stale owner of an
unfinished coordinator, but it no longer excludes a hold whose finished
coordinator has a lapsed owner. A live lease still defers the re-arm to
its expiry.
- `system-recovery-state.ts`: add `FINISHED_COORDINATOR_PHASES`.
- `doc/execution-semantics.md`: document the rule.
- Tests: two red-then-green regressions in the staging shape (an expired
finalizer lease, and an owner without an expiry saved by an older
release), one strictness case (a live lease on a finished coordinator
keeps the hold and is not released), and the existing stale-owner case
now uses an observed coordinator, which still waits for the
abandoned-owner settlement.

## Verification

- Red before the fix: `pnpm --filter @paperclipai/server exec vitest run
src/services/workspace-recovery-stop-proof.test.ts` failed the two new
staging-shape cases (`nextWorkspaceRecoveryAt` returned null: the worker
was never woken for the hold).
- Green after the fix: `workspace-recovery-stop-proof.test.ts` and
`workspace-recovery-capture-policy.test.ts` pass (15 tests). The
staging-shape cases assert that the hold is re-armed, the save
completes, the dirty files are saved, the workspace hold is free, and
the lapsed owner is released.
- `pnpm verify`: exit 2 (broader verification required, because
`doc/execution-semantics.md` is a global CI input). Focused suites pass:
`workspace-recovery-stop-proof`, `workspace-recovery-capture-policy`,
`workspace-recovery-fallback`, `workspace-recovery-retry-copyback` (113
tests). Required CI supplies the broad coverage.
- Staging evidence (read-only GET requests): the run is `failed`,
finished six days ago, `controllerLeaseExpiresAt` is null, its only
lease is `released` with `cleanupStatus: success`, a `stopped`
termination receipt and a stop-and-retain receipt. The action's evidence
has no re-arm marker, and four holds of the same shape were not touched
after #20295 was deployed.

## Risks

- Previous-release story: rows saved by earlier releases carry the
lapsed owner and no marker. They are selected and re-armed on the
worker's first pass after deploy. No migration.
- Strictness stays where an owner could be live: a live lease on any
coordinator defers the re-arm to its expiry, and a stale owner of an
unfinished (observed) coordinator still waits for the abandoned-owner
settlement, which proves its controller stopped. The new attempt still
requires a provider stop receipt for the allocation before it reads
files.
- Clearing a lapsed owner matches the existing token-lease contract: an
expired finalizer or cleanup lease is free for the next claim, and a
holder's later write fails its token compare-and-swap.
- Low risk otherwise: the change is limited to the two exhausted-hold
re-arms and their discovery clause.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
code editing, browser GET reads) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 17:00:46 -06:00
DottaandPaperclip a3b34a9d88 fix(ui): allow queued message deletion before queue acknowledgement (#20553) 
## Thinking Path

Paperclip lets a user queue a message while an agent works. The user
must be able to remove that message before delivery starts. A new local
message can clear the displayed queue ID until the server accepts it.
This also disables trash for older saved messages. The page rejects the
action before it sends a request.

This change keeps trash available when the server says the user can
discard the message. When the queue ID is absent, the page uses the
existing comment cancellation route. The server finds the queue and
checks the author and dispatch state in the same transaction.

## Linked Issues or Issue Description

**What happened?**

Trash stays disabled for a saved queued message while another message
waits for queue acknowledgement. The page handler also throws before it
sends a cancellation request.

**Expected behavior**

Trash removes the selected queued message before dispatch. A message
that has started dispatch stays visible with an error. Other queued
messages stay queued.

**Steps to reproduce**

1. Send a follow-up while an agent works.
2. Send another follow-up before the queue read includes it.
3. Try to discard the first saved message.

**Paperclip version or commit**

Reproduced from source at `0e3fdebea825`.

**Deployment mode**

Source checkout on Linux. UI tests use jsdom. Route tests use isolated
embedded PostgreSQL.

Related work: #15865 changes mixed-queue interrupt handling. #15939
changes recovery after a queued message is discarded. #15465 changes
discard permissions for agent-authored entries. This change fixes trash
for a user's own saved messages while queue acknowledgement is pending.

## What Changed

- Keep trash enabled for a saved message with discard permission when
the displayed queue ID is absent.
- Use the existing author-only cancellation route when queue identity is
pending. Keep the revision-checked route when queue identity is present.
- Keep failed cancellations visible. Refresh the queue after
cancellation and remove successful deletions from comment caches.
- Cover deletion, dispatch conflicts, permission failures, network
failures, history reloads, final-message cleanup, and concurrent
dispatch.

## Verification

- Red: three regression cases failed on the original production code.
The trash button was disabled. Both page cases made no cancellation
request.
- Green: five complete UI suites pass, with 767 tests and one existing
skipped test. These cover the page, queue component, thread, queue
merge, and optimistic comments.
- Green: 11 focused PostgreSQL route cases pass. They cover both routes,
queue promotion, final-message cleanup, authorship, dispatch conflicts,
and a concurrent dispatch claim.
- `pnpm check:token-gates` and `git diff --check` pass.
- `pnpm --filter @paperclipai/ui typecheck` passes.
- The GitHub tree is `d4f0d61cbef2620c1fd62e6988ad80ccb1ba6173`. It is
identical to the tested local tree after merging master `902f3767960d`.
- `pnpm verify` selected more than 1,500 server test files. That broad
local run was stopped without a pass claim. CI will supply broad build
and test coverage. The preflight helper could not build Rust because
Cargo is absent.
- [CI
Build](https://github.com/paperclipai/paperclip/actions/runs/42835584447/job/129897150252)
and [browser
E2E](https://github.com/paperclipai/paperclip/actions/runs/42835584447/job/129901676175)
pass for head `44dea22faea51a6664a6f45852a42f62f5c72bbd`.
- [Greptile
review](https://github.com/paperclipai/paperclip/pull/20553#issuecomment-6770202777)
gives the current head 5/5 with zero comments and no unresolved review
threads.
- [Final CI
run](https://github.com/paperclipai/paperclip/actions/runs/42835584447)
passes for head `44dea22faea51a6664a6f45852a42f62f5c72bbd`: 55 check
runs passed and 4 were skipped. No failures or pending checks remain.
Typecheck, Build, all general test shards, serialized suites, Runner
checks, and browser E2E passed. Snyk also passed.

## Risks

The fallback has no client queue revision. The existing server
transaction finds the queue for the selected comment and checks
ownership and dispatch state. Tests cover rejection after dispatch and
when another user owns the message. No API contract or schema changes
are required. No visual tokens change. Existing user documentation
remains accurate.

## Model Used

OpenAI GPT-6 in Codex. The runtime does not expose a more specific
provider revision or context window size. Used reasoning, repository
tools, code edits, and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 16:38:58 -06:00
Nicky LeachandPaperclip f30ac226d6 Keep project environment mappings independent of shared defaults (#20651) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A project supplies work context and repository storage to task runs.
> - Creation selected a storage owner but did not retain the selected
environment on the project.
> - Configuration stored an environment inside checkout policy, but
policy reads and run selection dropped it.
> - Projects therefore continued to follow shared agent and instance
defaults.
> - This pull request saves placement on each project and uses it for
new runs.
> - The benefit is an independent Project to Environment association
with the existing storage, trust, and recovery safeguards.

## Linked Issues or Issue Description

Refs #20459 and #11644. Follow-up to #20587.

The reported journey is Project > Configuration: changing one project's
environment must not change another project's mapping or shared
defaults.

Related open PR #18176 repairs placement through the legacy
workspace-policy field. This follow-up introduces a separate project
association, saves creation defaults, and migrates saved legacy
references. It overlaps that PR's placement, permission, and recovery
safeguards. The changes should be reviewed together. #9588 concerns
environment-variable response privacy and does not address placement.
ROADMAP.md has no separate project-association work planned.

## What Changed

- Add nullable `projects.environment_id` and expose
`Project.environmentId`. Save the effective creation default once on the
project. Keep environment placement separate from checkout/isolation
policy.
- Read and write Environment directly in Project Configuration. Keep the
control available with isolated workspaces disabled. Each form reads its
project's value, even when environment choices share a query cache.
- Offer environments compatible with the current storage owner. Change
the target and storage together when the existing storage selector
attaches a core project to a plugin. Preserve retained-data restrictions
and validate the observed storage owner again on save.
- Resolve new task runs from their organizational project's target
before agent and instance defaults. File sources from another project do
not change placement. Explicit low-trust sandbox choices and enforced
sandbox policies retain precedence. External provider placement remains
separate.
- Restore admitted native runs from their owned environment lease. Later
project edits apply to new admissions. Require workspace-write
permission for environment changes and clears. Check company bindings
before runtime admission and during lease creation.
- Migrate valid saved policy references into the project column.
Normalize legacy API writes. Count and clear matching project references
when an environment is deleted. Update API and behavior documentation.
- Refresh the maintained SDK source registry for the two changed shared
files. Runtime pins are unchanged.

## Verification

- [Hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/42908045146)
is green on `8b3b7d5b5aed51fc4c8e9c2e3506508bbaf9e00c`: **60 successful
checks**, three skipped gates, no failures, and `ci / verify` passed.
GitHub reports no merge conflicts. Standard Greptile scored the same
commit **5/5** with both review threads resolved.
- Source checkout on macOS arm64, Node 26.12.0, with fresh isolated
PostgreSQL. No production data or live provider calls.
- The two-project component test changes the first form with isolation
disabled and a shared query cache. The second form keeps its target.
Storage changes update only the selected project. A saved target remains
selected while the list loads, and an explicit clear remains possible.
- API integration creates two projects from the same default, patches
one, reads both, and checks that agent and instance defaults remain
unchanged. A later instance-default change does not rewrite either saved
project target. Tests cover unavailable/incompatible choices, anonymous
and cross-company denials, and storage preconditions.
- The migration test runs the actual SQL against an existing project row
and verifies its environment and disabled checkout policy survive.
Deletion accounting and clearing affect only matching project
references.
- The final guard set passes **53 tests across four files**:
`project-environments.test.ts`,
`environment-selection-route-guards.test.ts`,
`heartbeat/environment-selection.test.ts`, and
`ProjectProperties.environment.test.tsx`. All **64
environment-run-orchestrator tests** pass, including direct lease
acquisition without an extra caller argument. The broader affected set
passed **109 tests across eight files** before the additional admission
guards. These sets overlap.
- UI typecheck, migration numbering/safety, token gates, module
boundaries, query-error rendering, and diff checks pass. `pnpm verify
--explain` returns exit 2 and requires broad CI because shared
schema/runtime paths and behavior documentation changed. Broad coverage
is delegated to current-head CI.
- Early iterations failed due to a legacy parser that drops the
environment field, a fixture without real company membership, and
incorrect expectations for privacy-preserving 404 denials. The service
now reads legacy references before normalization, the API fixture has
real membership, and denial assertions match the existing privacy
contract. The orchestrator mock also needed the reused company-binding
guard in its service contract. The focused suites above pass after those
corrections.
- Local server typecheck remains blocked by installed ACPX, Slack, AI
SDK, and Runner type mismatches in unchanged dependency-facing code. It
reports no errors in the changed production paths. This is not a full
local typecheck or suite pass. Hosted CI supplies full current-head
coverage.
- SDK source-proof and privacy qualification passes **56 tests across
three files**, including the real built SDK. The maintained source
registry matches the committed revision; runtime pins are unchanged.
- Review reproduced two failures: an incoming policy without a storage
owner could be validated before the existing owner was restored, and
uppercase legacy UUIDs missed migration matching. Preserve the owner
before validation and compare normalized UUID text. Both regressions now
pass. The post-fix project/storage/orchestrator batch passes **107
tests**: `node node_modules/vitest/vitest.mjs run
server/src/__tests__/project-environments.test.ts
server/src/__tests__/project-workspace-provider.integration.test.ts
server/src/__tests__/environment-run-orchestrator.test.ts --silent`.
- The first hosted run failed build, typecheck, and canary at the same
`run-completion.test.ts` call because lease acquisition required a new
argument. The lease method now enforces binding internally. Its existing
caller compiles without a new field, and the orchestrator tests prove
the binding flag reaches the runtime. The full server check no longer
reports that error; the local dependency errors above remain.

## Risks

- Apply the normal database migration before serving the new revision.
The migration adds a nullable foreign key and copies valid legacy
project selections. It does not move repository data or resize provider
storage. The five newest schema snapshots are retained.
- New projects retain their creation target when shared defaults change.
Existing projects without a saved selection continue to inherit
defaults. A null target explicitly opts into inheritance.
- Existing saved project selections now affect placement. Missing,
archived, incompatible, and foreign-company targets fail validation;
they do not silently choose another storage owner. A running admitted
process keeps its original owned lease.
- Local dependency limitations are listed above. No live-provider,
deployed-browser, or production rollout qualification is claimed.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, and code
execution. The exact serving snapshot and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 14:31:48 -08:00
DottaandClaude Opus 5.5 358977454b test(native): cover unknown-outcome recovery and hold settlement without a recorded PID space (#20649) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #20628 settles a host-tool callback that a server restart orphaned,
and settles the no-replay holds that earlier releases left for a
completed turn
> - Review of #20628 asked for two more cases. That PR merged before the
commit with those cases reached it
> - This pull request adds only those tests
> - The benefit is that the two edge rules of #20628 stay covered

## Linked Issues or Issue Description

Refs #20628

**What happened?**
The review of #20628 named two untested rules. First, a recovery
authority that throws `semantic_tool_outcome_unknown` must settle only
for a callback from an earlier controller, and must rethrow for this
controller's own callback. Second, a coordinator row saved before
`controller_host_hash` existed must still settle its holds through the
stopped controller boot.

**Expected behavior**
Both rules have regression tests.

**Steps to reproduce**
1. Read `recoverToolCallback` and `native_completed_turn_cleanup_hold`
on master.
2. Search the tests for a thrown `semantic_tool_outcome_unknown`, or for
a row with a null `controllerHostHash`. Neither exists.

**Paperclip version or commit**
master at 2150820bd8.

## What Changed

- `durable-prp-control-plane.test.ts`: an authority that throws
`semantic_tool_outcome_unknown` settles as
`tool_call_interrupted_by_restart` for `earlier_controller`, and
rethrows for `this_controller`. Any other rejection rethrows.
- `completed-turn-cleanup-hold-settle.test.ts`: a coordinator row
without `controllerHostHash`, whose server boot is recorded stopped in
`controller_instances`, settles the run's holds through the
controller-instance proof. No attestation is needed.

## Verification

- `npx vitest run src/control-plane/durable-prp-control-plane.test.ts -t
"orphaned by a server restart"` in `packages/paperclip-runner`: 7
passed.
- `npx vitest run
src/__tests__/completed-turn-cleanup-hold-settle.test.ts` in `server`: 5
passed.
- `tsc --noEmit -p packages/paperclip-runner/tsconfig.test.json` and
`tsc --noEmit -p server/tsconfig.json` pass.

## Risks

- Low risk. Test-only change. No production code changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 15:42:00 -06:00
DottaandClaude 30403781b9 fix(native): keep crediting proven unstarted restarts past the generation limit (#20647) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run survives a control-plane restart through restart
recovery, which resumes the interrupted provider turn
> - Restart recovery charges a provider attempt only when the provider
did work that could be lost, and it stops after three charged attempts
> - From controller generation 12 on, every restart charged an attempt
before that work rule was checked
> - In a busy restart wave, one long task sees 12 to 15 restarts, most
caused by other tasks. A restart that hit before the provider accepted a
turn was charged, the budget ran out, and a person had to click Retry
> - This pull request keeps the work rule past generation 12 for
restarts that a durable record proves did not start the provider, and
bounds a crash loop with a count of consecutive restarts without
provider progress
> - The benefit is that a long task survives many unrelated restarts
without a manual Retry, while a run that never makes progress still
stops

## Linked Issues or Issue Description

Refs #20625 (same restart recovery path). No public issue exists for
this case.

**What happened?**

`restartResumeKeepsAttempt` returned "charge" for every restart once the
run reached `NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT` (12) controller
generations. It did this before it read the work-based outcome. In a
local restart chaos wave (L6, seed 9106, task t03, Claude B5 with RR1)
the run reached generation 13. The last restart interrupted a resume
whose runner started but whose provider had not accepted a turn. That
restart was charged at the last attempt, so the claim failed with
`execution_recovery_budget_exhausted` and
`native_restart_recovery_blocked`. The task needed a manual Retry.

**Expected behavior**

A restart that a durable record proves did not start provider work does
not spend an attempt, also past generation 12. A crash loop that never
makes provider progress still stops automatically. When it stops, the
task shows a Retry that works.

**Steps to reproduce**

Run a native task past controller generation 12. Make the last restart
hit after `native.process_start_requested` but before `turn.accepted`,
with the provider attempt at 3. The new integration test `does not
charge a restart past the generation limit whose resume never started
the provider` reproduces the failure before the fix.

**Paperclip version or commit**

`master` at ed5e7493bb.

## What Changed

- `restartResumeKeepsAttempt` now returns `{ keepsAttempt,
noProgressRestarts }`. A new helper judges the interrupted attempt as
`provider_not_started`, `provider_progressed` or `charged`, with the
same proofs as before.
- `noProgressRestarts` is 0 when the interrupted attempt is proven to
have made provider progress. Otherwise it is the previous claim's count
plus one. A clean-drain claim carries the previous count.
- Before generation 12 the decision does not change.
- From generation 12 on, a restart keeps the attempt only when the
interrupted attempt is proven `provider_not_started` and the count is
below the new `NATIVE_NO_PROGRESS_RESTART_LIMIT` (12). Provider work
(progress, or an accepted turn without progress) and unprovable cases
still charge.
- Each restart claim writes `noProgressRestarts` into its
`recovery_history` entry. No schema change: `recovery_history` is JSONB.
- Claims saved by earlier releases have no count. They are read as zero.
- Integration tests: the three generation-cap remote tests now check the
new bound. New tests cover the unstarted restart past the limit, the
consecutive-limit stop with a working Retry, the progress reset, an
accepted turn without progress, an unrecorded previous claim, and an
old-format claim.
- The durable-ownership plan (section F) records the rule.

**Why 12.** It equals the existing generation limit
(`PROVIDER_ATTEMPT_CREDIT_GENERATION_LIMIT`). The count includes
restarts before generation 12. Thus a run that never makes provider
progress is bounded exactly as before: the existing test `bounds a crash
loop that never starts the provider` passes unchanged. Only proven
progress resets the count. In the failing wave, the task had made
progress shortly before its last restart, so its count was 1.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/__tests__/heartbeat-native-restart-suspension.test.ts`
- Red before the fix: with only the new constant added to the old code,
the 9 new and changed tests fail. With the fix, they pass. The existing
crash-loop, progress and unstarted-resume tests pass without change.
- The consecutive-limit test checks that the stopped run is
`terminal_failure` / `native_restart_recovery_blocked` with no lease
owner and nothing scheduled, that `taskInterruptionPresentation` shows
`failed` with `canRetry: true`, and that `requestTaskRetry` accepts the
Retry.
- `cd server && npx tsc --noEmit -p .`
- Full local run of the three suites above: 322 passed, 14 skipped.
- Live: local restart chaos wave L6 (seed 9106, concurrency 3, `--env
local --adapters codex,claude --workloads A1,B5,C1 --faults RR1,RR2,RR3
--tasks 6`) at 7f7803ab0c: 6 of 6 tasks passed, and no task needed
Retry. Task t03 (Claude B5 RR1) reached controller generation 13 again.
The claim at generation 13 resumed the run (`resume_dead_runner`), and
the server log has no `execution_recovery_budget_exhausted`. The same
task on master was blocked at that claim and needed Retry.

## Risks

- A run that makes provider progress between restarts past generation 12
still charges each of those restarts. This is the same as before, by
design.
- An upgrade or a mid-run deploy reads old claims as count zero. A run
that was already in a no-progress loop can get up to 12 more
proven-unstarted restarts before the bound applies. It cannot be charged
where the old rule did not charge.
- A rollback ignores the new field and restores the old rule.
- Past generation 12 the claim now reads the event log, as it already
does before generation 12. This is not on the turn-start path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 15:27:45 -06:00
DottaandClaude Opus 5.5 4f33bb3578 fix(workspaces): release the Blocked status of a sibling-settled no-replay hold (#20648) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a remote workspace save fails, recovery actions hold the task
until the run's files are saved
> - #20641 lets a resolved no-replay hold settle when a sibling action
of the same run saved the files
> - Review found that the settlement did not release the task's Blocked
status, and that a settled hold with a historical source stayed
selectable
> - A new test with the waiting user comment also showed that the worker
skipped the second hold of the same run in every pass, so it never
settled
> - #20641 was merged before these fixes were pushed
> - This pull request completes #20641 with those fixes
> - The benefit is that the task leaves Blocked and continues without a
person

## Linked Issues or Issue Description

Refs #20641, Refs #20569

**What happened?**
#20641 merged at its first head. The review follow-ups for it were
pushed later and are not on master:
- A settled no-replay hold that owned the task's Blocked status
(`nativeFailureBlock`) left the task `blocked`.
- A hold settled by a sibling stayed selectable through
`workspaceRecoverySourceEligible` when it named a
`historicalWorkspaceSource`.
- With saved user input waiting, the export hold stayed selectable and
came first in each worker pass. `reconcileRetainedWorkspaces` skipped
the other hold of the same run, so the no-replay hold never settled.

**Expected behavior**
After the save, the no-replay hold settles in the same or next pass. The
task returns to `todo` when that hold owns its Blocked status. The
settled hold is never selected again.

**Steps to reproduce**
1. Use the staging-shape test in
`workspace-recovery-capture-policy.test.ts`, with a waiting "try again"
comment.
2. Run the worker until nothing is due.
3. On master, the worker never goes idle and the no-replay hold stays
`resolved`/`blocked`.

## What Changed

- `workspace-recovery-fallback.ts`: new shared `holdOwnsBlockedStatus`,
used by `continueFromLastSavedWorkspace` and
`settleSiblingRecoveredHold`. A settled no-replay hold that owns the
Blocked status returns the task to `todo`. A newer decision's Blocked
status is kept. An export hold's status stays with its own result
finalization.
- `workspace-recovery-source.ts`: `workspaceRecoverySourceEligible`
excludes a resolved hold that carries `siblingWorkspaceRecovery`.
- `workspace-recovery.ts`: `reconcileRetainedWorkspaces` runs several
holds of one run one after another in that run's job, instead of
skipping the later ones.
- `workspace-recovery-capture-policy.test.ts`: the staging test now has
the waiting comment, and a variant with `nativeFailureBlock` and
`historicalWorkspaceSource`. It checks `todo`, no re-selection, and no
repeat handoff in a later pass.
- `doc/execution-semantics.md`: documents these rules.

## Verification

- Red, with each rule disabled: the status stayed `blocked`; the settled
hold was selected again; the worker never went idle (starvation).
- Green: `npx vitest run --root server` on 14 workspace recovery suites
(including `workspace-recovery-capture-policy`, `-fallback`,
`-admission`, `-source`, `execution-blocker-read`,
`preserved-native-response-wait.integration`): all passed.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.

## Risks

- Several holds of one run now run in sequence in one job. Concurrency
per run is unchanged (one job per run).
- A no-replay hold releases the Blocked status only when it still owns
it. Otherwise the status is kept.
- Previous-release story: no new persisted state. Holds saved by earlier
releases are handled on the first worker pass after deploy.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 15:24:04 -06:00
DottaandClaude 805531cc71 fix(native): never leave a cleanup-gated task without Retry or an owner (#20639) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs close their provider session at the end of each attempt.
When close cannot prove that the runner stopped, the runner keeps a
cleanup gate (`native_session_cleanup_quarantined`) in server memory
> - #20572 hid Retry for every gated run and said "Paperclip finishes
the cleanup automatically". Only a remote environment release settles
the gate. A local gate that needs an operator never settles, and a
server restart loses it
> - A restart-chaos wave on master found two tasks that rested blocked
with no Retry, no scheduled retry and no recovery owner. A person could
not continue them
> - This pull request makes the run's own recovery retire the gate when
the host proves the old processes stopped. It also offers Retry whenever
nothing settles the gate, and it makes that Retry work
> - The benefit is that a cleanup-gated task either continues by itself
or shows a Retry that starts. It never rests without both

## Linked Issues or Issue Description

Refs #20572, #20578

**What happened?**
Restart-chaos wave `w-20261110T185144-s9107` (local, claude, fault TM2:
the server is paused with SIGSTOP for about 33 s). Tasks t04 and t05
completed their provider turn. The runner checkpoint owner lapsed while
the server was paused (`native_runner_checkpoint_owner_lost`). The
session close then failed with `native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`. The provider had
drained and no tool call was pending. The runner kept an
operator-required cleanup gate. Thirty seconds later the run's own
same-run recovery verified that the runner process and group had stopped
(`native.local_process_stopped`). The gate still refused the recovery,
and the run ended `failed` with `native_session_cleanup_quarantined`.
The task showed `canRetry: false` and a cleanup wait until the deadline.
Nothing owned that wait.

**Expected behavior**
When the old runner and provider processes are proven stopped, the run's
own recovery continues. When nothing settles the gate, the task offers a
Retry that starts.

**Steps to reproduce**
1. Run restart-chaos L7: `--env local --adapters codex,claude
--workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1
--concurrency 3 --tasks 6 --seed 9107`.
2. Observe a claude TM2 task rest `blocked` with `canRetry=false`,
`retryWait=true` and no scheduled retry.

**Paperclip version or commit**
master at ed3ffc3bc7.

## What Changed

- `native-session-executor.ts`: add `provenStoppedOwnNativeSession`. It
returns proof only for a local run with a pid-bound
`native.local_process_stopped` record (no later launch) and with every
recorded provider process gone. An ACPX session open names no provider
PID; its sidecar counts only when it shares the runner's process group,
which the stop record proves empty.
- `native-session-executor.ts`: add
`retireStoppedOwnSessionCleanupGate`. With that proof, it retires the
run's own operator-required gate through the existing
`completeTerminatedLocalNativeSessionCleanup` boundary and records
`native.session_cleanup_gate_retired`.
- `heartbeat.ts`: the native recovery preamble calls it right after it
records the local process stop, before admission. The recovery decision
itself does not change.
- `explicit-native-continuation.ts`: Retry or a new message accepts the
run's own stop proof when the retained-state inventory proof does not
apply. It still starts a fresh session, and it never resumes the old
turn.
- `task-interruption-presentation.ts`: hide Retry for a gated run only
while a live owner settles the gate: an unreleased remote environment,
or a scheduled close in this process. Otherwise offer Retry. The same
rule applies to a Retry held on `cleanup_quarantined`.
- `doc/execution-semantics.md`: describe the new rule.

## Verification

- Red before the fix, green after (5 tests failed on master sources):
- `server/src/services/task-interruption-presentation.test.ts`: "offers
Retry when nothing settles the gate by itself", "holds a Retry on a
cleanup gate only while an owner settles it", and an invariant test: a
failed presentation without Retry must name a wait, and a cleanup wait
needs a settling owner.
- `server/src/__tests__/task-interruption-query.test.ts` (real
Postgres): an unreleased remote lease hides Retry; the released lease
offers it.
- `server/src/services/quarantined-task-retry.test.ts`: "starts a task
Retry for a run's own quarantine proven stopped by its own process
record" (stopped, and provider alive stays held).
- New: "retires a run's own failed-close gate once its runner and
providers are proven stopped" for codex PIDs, an ACPX sidecar in the
runner's group, a sidecar with its own group, a live provider, and no
stop record.
- `npx vitest run` of `quarantined-task-retry`,
`task-interruption-presentation`, `task-interruption-query` and
`explicit-native-continuation`: 575 passed.
- `npx tsc --noEmit -p server` passes.
- Live: LIVE_PROOF_PLACEHOLDER

## Risks

- The gate retirement removes process ownership only. It runs only for a
local run whose exact runner and group and every recorded provider
process are proven gone. A provider without a recorded process identity
keeps the gate.
- Retry on a gated run still runs every existing stop check. If the stop
cannot be proven, the Retry waits and the task keeps a Retry, instead of
a wait that nobody settles.
- No persisted shape changes. Runs saved by earlier releases use the
same fields.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 15:20:42 -06:00
DottaandClaude 37c50f70c2 fix(native): continue a turn lost on restore in a proven same-run resume (#20645) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run keeps its provider conversation when its local runner
dies. When the restored runner finds that the active turn was lost, it
reports `provider_turn_lost_on_restore` and does not invent a result.
> - After a control-plane restart, the admitted restart attempt may send
one continuation in that same conversation (#15297). The agent then
reconciles unfinished work. The task is not resent.
> - A runner can also be lost without a server restart, for example when
`runner.drain` times out. The run then resumes itself after the retry
delay from its exact checkpoint. That path never allowed the
continuation, because the executor passed `resumeInterruptedTurn` only
for a `resume_dead_runner` restart claim.
> - So the lost turn failed with `native_provider_terminal_failed`,
automatic replacement was refused (`uncertain_provider_action`), and the
user had to click Retry.
> - This pull request allows the same single continuation for that
resume, but only when durable records prove the same three facts that a
restart claim proves.
> - The benefit is that a runner lost mid-turn continues in the same
conversation with no Retry. No case that is refused today becomes
allowed without that proof.

## Linked Issues or Issue Description

Refs #15297, #20625

**What happened?**
Restart-chaos wave L7 (seed 9107, local), task codex B5 TM2, run on
master after #20625. A TM3 restart resumed the run (attempt 1). During
the TM2 freeze the TS runtime lost its runner:
`native_session_recovery_failed: runner_local_connect_failed: PRP
command runner.drain timed out`. The executor saved `retryable_failure`
with `recoveryMode: "exact_checkpoint_resume"`. After the delay,
recovery verified the runner process stopped
(`native.local_process_stopped`). The same run then resumed as attempt 2
(`native.process_start_requested`, `providerAttempt: 2`). The restored
runner reported `turn.failed` with `provider_turn_lost_on_restore`
(`recoverable: true`). No continuation was sent. The run failed with
`native_provider_terminal_failed`, recovery stopped with
`uncertain_provider_action` / `preserve_without_replay_v1`, and the task
needed Retry.

**Expected behavior**
The run sends the one same-conversation continuation, exactly as it does
after a restart claim, and finishes without Retry.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--env local --adapters codex,claude --workloads A1,B5,C1 --faults
TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107`
(timing-dependent: the drain timeout must land during an active turn).

## What Changed

- New
`server/src/services/native-runtime/scheduled-resume-continuation.ts`.
It decides whether a same-run scheduled resume may continue a lost turn.
Each of the three proofs comes from a durable record:
1. **The previous runner is stopped.** The latest process record of the
run in `heartbeat_run_events` (`native.process_start_requested`,
`native.process_identity_recorded`, `native.local_process_stopped`) must
be `native.local_process_stopped`. Its `processPid` must equal its
`processGroupId`, and neither the pid nor the group may be alive now.
This is the existing `readNativeLocalProcessStop` check, read in the
claim transaction before this attempt records its own
`native.process_start_requested`.
2. **The attempt was admitted under the bounded budget.** The executor's
claim moves `native_run_finalizations` from phase `retryable_failure` to
`observed`. It charges a provider attempt, and the claim refuses any
attempt above 3 (`native_session_retry_exhausted`). The decision also
requires `1 <= attempt <= NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT`.
3. **It is the same provider conversation.** The saved `failure_detail`
must show `recoveryMode: "exact_checkpoint_resume"`, `recoverable:
true`, `providerSessionEstablished: true` and `checkpointExists: true`,
with no `successorRunId` or `replacementDenied`. At run time the
existing runtime guards still apply: the backend must recover the
checkpointed session (`recovered`, not a continuity-break replacement),
and the lost turn must equal the checkpoint's `activeTurnId` and the
snapshot's interrupted turn. The provider kinds stay codex, acpx and
opencode, with no compaction, goal control, held result or pending
runtime request.
- `native-session-executor.ts`: computes that decision in the
coordinator claim. It passes `resumeInterruptedTurn` when either the
restart claim is `resume_dead_runner` (unchanged) or the decision allows
it.
- `doc/execution-semantics.md`: describes the rule and the refused
cases.

**Cases that stay refused** (each has a test in
`scheduled-resume-continuation.test.ts`):
- a restart-recovery claim (it keeps its own existing rule);
- a remote execution target;
- a coordinator that is not `retryable_failure`, including a fresh run;
- `recoveryMode` `bootstrap_retry` or `ambiguous_state`, or no saved
failure detail;
- a failure that is not recoverable;
- no established provider session;
- no checkpoint;
- a recorded replacement (`successorRunId` or `replacementDenied`);
- an attempt outside 1..3;
- no stop receipt, a stop receipt superseded by a newer launch, a
stopped process that is still alive, or a run that never had a process.

The runtime-side refusals are unchanged and already covered by
`native-session-runtime.test.ts`: real provider errors, historical
error-null failures, and no restart permission.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/scheduled-resume-continuation.test.ts`: 22
tests, covering the decision table and the stop receipt in embedded
Postgres.
- `cd server && npx tsc --noEmit` passes.
- Live: not proven. An L7 wave on this branch (seed 9107) did not
reproduce the drain-timeout path, so this pull request has unit and
database tests only. In that wave the same cell (codex B5 TM2) needed
Retry for a different reason: TM2 aged its restart-recovery claim lease
before dispatch, and budget admission refused with `Native recovery no
longer owns provider dispatch`. This pull request does not change that
path. Four other tasks passed. Claude C1 TM2 rested blocked without a
Retry button, which is a separate cleanup-quarantine issue.

## Risks

- This allows one more path to send the same-conversation continuation.
It is limited to local runs. It needs the same proofs as a restart
claim, and it uses the attempt the resume already charged. It does not
change the generation cap or `restartResumeKeepsAttempt`.
- A rolling deploy reads `failure_detail` and stop events written by
earlier releases. Those fields and events already have this shape, so an
older saved failure is judged by the same rule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 15:16:51 -06:00
DottaandClaude Opus 5.5 353945c86d fix(tool-gateway): let agents create gateway tokens without receiving the value (#20613) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool gateways let external MCP clients use a company's tools with a
gateway client token
> - Since delegated connection management (#17737), an agent can create
and revoke gateway client tokens for its responsible person
> - #20581 stopped agents from getting the one-time token value, but it
did this by making token creation person-only
> - The product decision is that agents keep their delegated authority
to create and revoke tokens
> - This pull request restores agent token creation, but the agent never
receives the token value: the person reveals the value once on the
gateway's Tokens page
> - The benefit is that an agent can do the rotation work for the
person, and the secret still never enters chat, run events, logs or tool
output

## Linked Issues or Issue Description

Refs #17737
Refs #20581

**What happened**
After #20581, `POST /api/tool-gateway/gateways/{gatewayId}/tokens`
returns 403 `gateway_token_requires_person` to every agent caller.
Agents lost the delegated authority to create gateway client tokens that
#17737 gave them. Before #20581, an agent that created a token received
the plaintext value in its tool result and could repeat it in chat.

**Expected behavior**
An agent with delegated connection management can create and revoke
gateway client tokens. The agent never receives a token value. The
responsible person gets the value once, on the gateway's Tokens page.
Person callers keep today's behavior.

**Steps to reproduce**
1. Give an agent delegated connection management for a responsible
person with `tools:admin`.
2. As the agent, call `POST
/api/tool-gateway/gateways/{gatewayId}/tokens` (for example through
`call_api`).
3. See 403 `gateway_token_requires_person`.

## What Changed

- Agent create: `POST /api/tool-gateway/gateways/:gatewayId/tokens` from
an agent with delegated authority returns 201 with token metadata only
(`id`, `tokenPrefix`, `expiresAt`, `pendingReveal: true`), plus
`tokensPath`, a `reveal_credential` `requiredUserAction` for the
responsible person, and a `nextStep` instruction. The saved row has no
usable value: its stored hash is of a value that is generated and then
discarded. Gateway authentication also rejects `pendingReveal` rows.
- Person reveal: new person-only `POST
/api/tool-gateway/gateway-tokens/:tokenId/reveal`. It generates the
value, stores its hash, clears `pendingReveal` and returns the value
once. A conditional update makes a second or concurrent reveal return
409. Only the token's responsible person can reveal it (403
`gateway_token_reveal_requires_responsible_person`). Agent callers get
403 `gateway_token_requires_person`. A token revoked before reveal
cannot be revealed.
- Why a reveal that generates the value: no stored plaintext existed to
reveal, and storing one (even encrypted) for later display adds a secret
at rest. Generating the value at reveal time keeps the existing one-time
display and stores only a hash.
- Schema: `tool_mcp_gateway_tokens.pending_reveal boolean not null
default false` (migration `0460_gateway_token_pending_reveal`). Shared
`ToolMcpGatewayToken.pendingReveal`, and `ConnectionRequiredUserAction`
gets `reveal_credential` / `gateway_token`.
- UI: the Tokens page shows a "Waiting for you to reveal" section for
pending tokens with a "Reveal token" button. The value appears in the
existing "New token — copy now" one-time block. Pending tokens show the
status "Not revealed".
- Agent gateway create: the 201 for agent callers also includes
`gatewayPath`, `tokensPath` and a `nextStep` that tells the agent to
link the saved gateway.
- The agent create-token `nextStep` and the catalog say not to revoke
the old token in the same turn: its client keeps working on it until the
person installs the new value.
- The reveal update and its `tool_gateway.token_revealed` activity entry
commit in one transaction. The Tokens page offers Reveal only for tokens
that act for the signed-in person.
- API reference: new `docs/api/tool-gateway-tokens.md` (create, reveal,
revoke) in the docs nav.
- Revoke stays allowed for agents with delegated authority (unchanged
route).
- API catalog: create-gateway, create-token, reveal and revoke
descriptions now say who acts, that agents never receive values, the
page paths, and that the old token is revoked only after the client
works with the new one.
- Agent access map: create-token policy note and a new reveal entry.
- Evals `gateways-profiles.create-gateway` and
`gateways-profiles.rotate-token`:
- Remove the keyword grader (`human|operator|admin|manage`) and the
handoff text checks. Grade saved state, rendered links and secret
absence. A plain reply or a question card is accepted.
- Remove the setup probes that called the create routes as the agent.
Under delegated authority, those probes created a real gateway
("Forbidden agent gateway"), which failed create-gateway before the
agent ran.
- create-gateway: exactly one gateway for the intended profile and
company, created by the agent (or by the person after a handoff); the
person opens it from a rendered link; attribution matches the creator;
any agent-created token is still pending reveal; no token value in chat,
cards or run events.
- rotate-token: after the agent's turn, both setup tokens still work and
any replacement the agent created is pending the person's reveal; the
person follows the rendered Tokens page link and reveals (or issues) the
value in a private browser; the new value works before the old token is
revoked; the unrelated client keeps access; no token value appears
anywhere, including a regex scan for any full `pcgw_<id>.<secret>`
value.
- The changed scenario context and outcomes are recorded in
`chat-full-catalog-testops-manifest.json` `changedFields` and pinned in
the manifest test. The inventory digest, descriptors (`oracleVersion`
v2, `sourceFiles`) and SDK public input pins are updated. The unused
`external-gateway-handoff.ts` module is removed.
- Fixture realism: the create-gateway profile now allows a real read
tool (an empty profile made the agent ask which apps to add instead of
finishing), and the rotation tokens get a neutral owner note instead of
"Disposable rotation fixture" (the agent read that note as an eval
hint).
- Shared private-browser helpers (`external-private-browser.ts`,
`external-private-browser-worker.ts`) get one optional `pendingTokenId`
input and one `revealedPending` receipt field.

## Verification

- Red: on `master` route code, the new integration tests `lets agents
create and revoke gateway client tokens without ever receiving a token
value` and `reveals an agent-created gateway token value only to its
responsible person` fail (403 `gateway_token_requires_person`).
- Green: `cd server && npx vitest run
src/__tests__/connection-management.integration.test.ts -t gateway` (25
passed). The test checks that the agent `call_api` result has no `token`
field and no token-shaped value, that the agent cannot reveal, that the
person reveals once (second reveal 409) and the value authenticates,
that agent revoke works and stops the value, that a token revoked before
reveal never gets a value, and that person create still returns the
value.
- `npx vitest run src/__tests__/tool-gateway.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts` (2247 passed).
- `cd ui && npx vitest run
src/pages/apps/gateways/panels/TokensPanel.test.tsx
src/pages/apps/gateways/gateway-helpers.test.ts` (16 passed); `pnpm
check:token-gates` clean.
- Eval graders and catalog: `pnpm exec vitest run --config
tests/runner-e2e/vitest.config.ts` on the gateway oracle tests,
`external.test.ts`, catalog consistency/contract, TestOps
export/retained, phase/clock callers and SDK input/type tests (all
pass); `node --test` on the TestOps manifest, adapter, contract,
streaming and reconcile tests (all pass); `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
(unchanged).
- `node scripts/check-maps.mjs` passes. `tsc --noEmit` for `server`,
`ui` and `tests/runner-e2e`: no errors in changed files.
- Paid end-to-end run of both eval cases, run
[42872921890](https://github.com/paperclipai/paperclip/actions/runs/42872921890)
on 39f38a7f6c (before the last three commits): both FAILED, for
agent-behavior reasons the later commits address. Not rerun yet.
- rotate-token: the agent created the replacement (pending reveal, no
value), then revoked the old token in the same turn, despite the create
response saying to wait. Its reasoning cited the eval-looking fixture
note. The grader rejected the premature revocation. The reply linked the
Tokens page correctly.
- create-gateway: the agent created the gateway and one pending token,
then asked a question card about which apps to add to the empty profile,
with no link to the gateway. The grader rejected the missing link.
- Token values: a scan of all evidence files of both cells found no full
`pcgw_<id>.<secret>` value and no `pcgw_<uuid>` form. Agent `call_api`
payloads in run events are redacted; the integration test proves the
agent tool result has no value.

## Risks

- Behavior change for agents: token create returns metadata and a Tokens
page path, not a value. Agents that expect `token` in the response must
send the person to the Tokens page.
- Migration adds one boolean column with a default. Existing tokens get
`false` and keep working.
- A pending token that is never revealed stays as an unusable row until
someone revokes it. It cannot authenticate.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 15:13:34 -06:00
DottaandPaperclip 2150820bd8 perf(native): parallelize durability eligibility checks (#20646) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A web server must prove that retained run state is durable before
volume-free enrollment.
> - The proof reads selected native sessions and validates their
existing recovery receipts.
> - Serial database checks made a valid inventory take 59.665 seconds.
> - The signed preparation request has a 55-second limit.
> - This pull request overlaps independent reads within fixed limits.
> - The benefit is a faster eligibility check with the same preservation
rules.

## Linked Issues or Issue Description

Refs #20599 and #20601. A search found no duplicate of this remaining
serial-read bottleneck.

**What happened?**

An actual full startup eligibility check returned eligible with no
blockers in 59.665 seconds. Controller checks took 44.134 seconds and
harness checks took 8.592 seconds. Signed preparation timed out at
55.002 seconds.

**Expected behavior**

A durable instance can complete signed preparation within its existing
deadline.

**Steps to reproduce**

1. Retain many native task sessions with database-backed controller
checkpoints and valid filesystem-independence receipts.
2. Run the full startup eligibility check.
3. Observe serial checkpoint and receipt reads consume the signed
request budget.

## What Changed

- Check the existing terminal-controller metadata and
checkpoint-existence predicate within the existing eight-reader source
batch.
- Validate harness independence receipts with at most four concurrent
readers per existing page.
- Stop admitting more harness reads after an invalid result or error.
Join all started reads before returning or throwing.
- Keep large controller, full-checkpoint and legacy archive reads
serial.
- Add real PostgreSQL regressions for overlap, concurrency limits and
failure settlement.

## Verification

- Genuine controller regression RED: all 16 checkpoint reads ran, but
peak overlap was one. GREEN: 10 focused PostgreSQL controls passed,
including bounded overlap, joined failures, active and warm handoffs,
and failed-source preservation. The other 45 cases were filtered, not
claimed as passes.
- Genuine harness regression RED: overlap was one instead of four.
GREEN: all three new PostgreSQL controls passed. Six existing authority,
replacement and refusal controls also passed.
- Final head `c059e031`: the same 10 controller and 9 harness PostgreSQL
controls passed after rebasing onto current master and refreshing Runner
artifacts. `node node_modules/typescript/bin/tsc --noEmit -p
server/tsconfig.json` also passed. The first `pnpm exec` attempt could
not start the compiler because package-manager bootstrap fetches failed;
it is not counted as a typecheck pass.
- `git diff --check` passed.
- Actual private-process acceptance with deployed `8e4c5e81`
dependencies and only the two changed owners: the complete check passed
in 37.596 seconds with no blockers. Nine runs were running before and
after; 1,750 sessions were selected. This is 17.4 seconds inside the
unchanged request limit. The earlier complete baseline was 59.665
seconds on 1,745 selected sessions, so these are observed timings on a
changing dataset, not a controlled exact speedup. No serving process was
patched.
- The first private candidate check stopped on a native-controller
blocker in 18.216 seconds. A targeted follow-up found no blocker without
any preservation or business write. Its cause was not identified. It is
retained as a nonpass and is not a speedup result.
- Current-head Greptile review is 5/5 with no actionable findings.
Required CI is running:
https://github.com/paperclipai/paperclip/actions/runs/42898013184.

## Risks

- The scans now use more simultaneous database reads. Limits remain
eight for controller source readers and four for harness proofs.
- Invalid or failed proofs still refuse eligibility. All started readers
settle before failure leaves the scan.
- Previous-release state carries forward unchanged. This changes no
stored shape, checkpoint identity, lease rule, schema, deadline or
provider protocol.
- Large legacy payload reads remain serial, so their existing memory
bound is unchanged.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. This is a measured
performance defect in the existing enrollment path.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, code execution and
parallel agents. The runtime does not expose the exact model variant or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes:`
/ `Refs:` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 14:57:11 -06:00
DottaandClaude Opus 5.5 4398a723c0 fix(native): settle tool callbacks and holds that a server restart orphaned (#20628) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs send provider tool calls to the server as durable
semantic inputs. The server runs the tool and saves the result in the
controller journal
> - Close must prove that every admitted tool call has a saved result.
Otherwise it reports `native_session_settlement_incomplete` and the
runner holds a cleanup gate (`native_session_cleanup_quarantined`)
> - A server restart in the middle of a tool call leaves its input
without a result. The new controller asks the tool authority for a
receipt. Tools without a receipt (assigned app tools,
`connections_search`) return nothing, so the input stays pending forever
> - Runs that already failed this way kept no-replay holds for a turn
that had completed. The task stayed blocked with no Retry and no
continuation
> - Restarts are normal. Work that a restart interrupts must settle by
itself, not wait for a person
> - This pull request settles such a callback as interrupted, with an
unknown outcome, when its owner is provably gone, and settles the holds
that earlier releases left once their controller is gone. It never runs
the tool or the turn again
> - The benefit is that close drains after a restart, and tasks that
earlier releases blocked this way continue on their own

## Linked Issues or Issue Description

Refs #20572 (that PR fixed the presentation of this quarantine. Its
Risks section names this runner drain rule as a separate follow-up.)
Refs #20641 (it settles a resolved no-replay hold when a sibling action
saved the run's files. This PR covers a different case: the hold's own
recovery saved the files, and its run's turn completed. Both use the
same recovery receipt, `completedWorkspaceRecoveryCondition`.)

**What happened?**
A remote native run started an assigned app tool call. The server
restarted 3 seconds later. The new server adopted the runner, and the
provider turn completed about 9 minutes later. The tool call never got a
saved result. Close reported `native_session_settlement_incomplete` with
`runnerSuspended: true`, `providerDrained: false` and one pending
callback. The session quarantined, and the same-run recovery was refused
at admission. A second run showed the same shape with a
`connections_search` callback after a later restart.

The first run then had two resolved-but-blocked holds with
`preserve_without_replay_v1` and an unknown action outcome: the
cleanup-quarantine hold, and a continuation hold whose workspace
recovery saved the files. The task view said "Saved work recovered. Send
a message to continue." with no Retry and no continuation. The task
waits on a child task and stays blocked.

**Expected behavior**
After a restart, a tool call without a saved result settles
automatically. A durable receipt is replayed. Without a receipt, the
agent is told that the outcome is unknown. Close proves the drain, and
no cleanup gate is left. Holds that earlier releases left for a
completed turn settle without a person, and the task continues when its
own wait ends.

**Steps to reproduce**
1. Start a native run and call an assigned app tool.
2. Restart the server while the tool call runs.
3. Let the adopted turn complete, then close the session. Close reports
`native_session_settlement_incomplete` and quarantines. The task stays
blocked by the no-replay holds.

**Paperclip version or commit**
master at fd6e71a7ff.

## What Changed

- Root cause: `DurablePrpControlPlane.#recoverPendingSemanticInputsOnce`
(`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`)
keeps an input pending when `recoverSemanticToolInput` returns null.
`PaperclipRunnerToolAuthority.recover` returns null for every tool
without a durable receipt. Nothing else could settle the input, so
`semanticToolResultsSettled()` stayed false and `#closeOnce` in
`runnerd-codex-transport.ts` threw
`NativeSessionCloseUnrecoverableError`.
- The control plane now passes `admission` to
`recoverSemanticToolInput`: `this_controller` when this instance
dispatched the callback live, `earlier_controller` when it restored the
input from an earlier controller's journal. The field is not written to
the result command or the journal. A live dispatch leaves this record
when its result is journaled.
- New
`packages/paperclip-runner/src/control-plane/orphaned-tool-callback.ts`:
- `dispatchToolCallback` records the callbacks that this process runs,
and the ones it saw fail.
- `recoverToolCallback` replays a durable receipt first. Without one, it
settles an `earlier_controller` callback that this process does not run
as `tool_call_interrupted_by_restart` (error result, `outcome:
"unknown"`, `retryable: false`). A `semantic_tool_outcome_unknown`
rejection from the authority settles the same way, only under the same
owner-gone proof.
- `native-session-executor.ts`: `SessionAuthorityEpoch.execute` runs
through `dispatchToolCallback`. `SessionAuthorityEpoch.recover` runs
through `recoverToolCallback`. The three recovery call sites pass
`admission`.
- New ownership kind `native_completed_turn_cleanup_hold`
(`server/src/services/native-runtime/completed-turn-cleanup-hold-settlement.ts`,
registered in `recovery/ownership-registry.ts`). It lists a native run
that failed with `native_session_cleanup_quarantined` after its saved
checkpoint recorded a completed turn, and that has a system-recorded
no-replay hold. After `proveOwnerGone` proves the closing controller
gone (its in-process cleanup gate went with it), it settles each such
hold as `restored`, removes its replay block, and logs
`issue.recovery_action_resolved`. A hold that still protects files not
verified as saved stays for workspace recovery. Operator decisions are
never reopened. It starts no turn. `heartbeat.ts` registers the ordinary
resume path (`resumeRemoteStopComments`) for saved user input.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md`.

## Verification

- Red before the fix, green after:
-
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts`
"host-tool callback orphaned by a server restart". It uses the real
control plane and an authenticated runner. A callback starts 3 seconds
before the restart, the old controller retires with no result, the new
controller adopts the runner, the turn completes, and then recovery
runs. `paperclip_call_assigned_tool` and `connections_search` settle as
`tool_call_interrupted_by_restart`, the runner accepts the late result,
`semanticToolResultsSettled()` becomes true (the condition close
requires), and the tool is not dispatched again. Before the fix, 4 of 5
cases failed: recovery received no admission and the callback stayed
pending.
- `server/src/__tests__/completed-turn-cleanup-hold-settle.test.ts` uses
the staging shape of both holds (real Postgres, rows as earlier releases
saved them). Before the fix, the ownership reconciler did not list them
(4 of 4 failed), and the execution blocker was the files hold with
`canRetry: false` and `canContinue: false`. After the fix, both holds
settle as `restored`, the execution blocker is null, no run is created,
the task keeps `blocked`, and the next pass lists nothing.
- Safety cases:
- A callback that this process still runs stays pending. After it
finishes, the next recovery pass settles it.
- A callback that this controller dispatched and saw rejected stays
pending, as before.
  - A durable receipt is replayed and not replaced.
- Holds stay while the closing controller is alive (its gate may still
exist).
- A hold whose files are not saved stays; the run's other hold settles.
  - Holds of a run whose turn did not complete are never listed.
- Focused suites pass: `durable-prp-control-plane.test.ts` and
`runnerd-codex-transport.test.ts` (506 tests); runner adoption,
release-resume, semantic readiness, ACPX reconciliation and external
driver tests; `native-session-executor.test.ts` and
`native-runner-restart-recovery.integration.test.ts` (1496 tests);
`ownership-reconciliation-report.test.ts` and
`interrupted-preparation-hold-settle.test.ts`.
- `tsc --noEmit -p packages/paperclip-runner/tsconfig.test.json` passes.
`pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify` returns exit 2 (a global doc input changed), so required
CI supplies the broad coverage.

## Risks

- Previous-release story: no persisted shape changes. `admission` is
computed at recovery and never saved. A journal that an earlier release
saved with a pending callback restores as `earlier_controller`, so that
session settles at its next adoption. Holds that earlier releases saved
settle on the reconciler's first pass after deploy. No migration.
- No double execution: recovery never calls the tool again. It replays a
durable receipt or reports an unknown outcome. The agent is told to
check the current state before it repeats the call. Hold settlement
starts no turn and replays nothing.
- The owner-gone proof for a callback: the new controller holds the
run's checkpoint fence, and the earlier controller retired its journal
writer. If the earlier process is still alive during an overlapping
deploy, it can still finish the tool, but it can never record a result
in this session. The unknown outcome is accurate in that case.
- Hold settlement requires `proveOwnerGone` on the closing controller's
stamp, because the cleanup gate lives in that process. On a server that
has not restarted since the quarantine, the holds stay.
- The in-process failure memory is bounded to 4,096 entries. Eviction
only lets a much older failed callback settle as interrupted later. It
never replays it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 14:39:45 -06:00
DottaandClaude Opus 5.5 d06f43b14f fix(workspaces): settle a resolved no-replay hold once a sibling saved its files (#20641) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a remote workspace save fails, recovery actions hold the task
until the files are saved
> - One failed run can have two holds: its export hold, and a no-replay
hold that a later "try again" opened for the same run
> - The no-replay hold is recorded resolved-but-blocked, and it is the
task's execution blocker
> - #20569 lets the export hold retry under the new symlink capture
policy and save the files
> - The sibling settlement that should then release the no-replay hold
only looked at active holds, so the task stayed blocked with no working
Retry
> - This pull request lets the sibling settlement and its discovery
include resolved no-replay holds
> - The benefit is that the task continues without a person once the
files are saved

## Linked Issues or Issue Description

Refs #20569

**What happened?**
On staging, a native run on a Daytona sandbox failed to save its
workspace (`remote_workspace_archive_unsafe_symlink`). Its export hold
(`native_workspace_sync_out_retry_exhausted`) was exhausted after 6
attempts and handed to the board. A user comment then opened a second
hold for the same run (`native_continuation_requires_reconciliation`,
`preserve_without_replay_v1`). That hold also failed 6 times and was
recorded `resolved` with outcome `blocked`. The task's execution blocker
is this second hold.

**Expected behavior**
After #20569 is deployed, the worker re-arms the export hold and saves
the files. The no-replay hold of the same run then settles, and the task
is no longer blocked.

**Steps to reproduce**
1. Create both holds with the exact staging evidence (see the new test
`stagingIncident`).
2. Run the recovery worker until nothing is due.
3. Before this change, the export hold completes, but the no-replay hold
stays `resolved`/`blocked` and `getExecutionBlocker` still returns it
with `canRetry: false`.

## What Changed

- `workspace-recovery-fallback.ts` (`settleSiblingRecoveredHold`): a
resolved no-replay hold (`isRetainedFileHold`) can now settle when a
verified receipt of a sibling action saved its run's files. Settling
removes `automaticRecovery`, so the hold stops blocking the task. It
starts no turn.
- `workspace-recovery-fallback-discovery.ts`: the worker discovers such
a resolved hold when a sibling of the same run is recovered.
- New test in `workspace-recovery-capture-policy.test.ts` with the exact
staging shape of both holds: board owner, `active_run_watchdog`, system
recovery policy version 1 without a capture policy version, stage
`restoring_files`, `export_interrupted` with cause `provider_error`, 6/6
attempts, `historicalWorkspaceSource` and `workspaceRestoreRecovery`
lease ids.
- `doc/execution-semantics.md`: states that a resolved no-replay hold
also settles from a sibling save.

## Verification

- Red before the fix: the new test failed. The no-replay hold stayed
`resolved`/`blocked`, and it stayed the execution blocker.
- Green after the fix: `npx vitest run --root server
src/services/workspace-recovery-capture-policy.test.ts` (3 passed). The
test drives the worker's own discovery (`nextWorkspaceRecoveryAt` +
`reconcileRetainedWorkspaces`) until nothing is due. It checks: the
export hold is re-armed once and saves the files; the no-replay hold is
`resolved`/`restored` by the sibling; `getExecutionBlocker` returns
null; the continuation hook ran when the task was free; no new run was
created; the worker goes idle.
- Related suites: `workspace-recovery-fallback`, `-admission`,
`-stop-proof`, `-source`, `-preparation`, `-preparation-gates`,
`execution-blocker-read`, `preserved-native-response-wait.integration`:
409 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify` exits 2 (global doc input), so CI supplies the broad
coverage.

## Risks

- A resolved no-replay hold is now released when another action's
verified receipt saved the same run's files. The receipt check is
unchanged, so an unverified sibling still does not release it.
- Settling starts no turn. Waiting input and the run's own result
continue the task through normal admission, which still checks every
other hold.
- Previous-release story: holds saved by earlier releases need no
migration. The worker discovers them on its first pass after deploy.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 14:39:24 -06:00
DottaandClaude Opus 5.5 e3bc4ca798 fix(native): settle warm follow-ups that a restart left behind (#20631) 
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native warm runs on a remote sandbox continue the saved session of
the previous run. Admission first proves what happened to the previous
Runner.
> - A deploy restarted the staging server while warm Runners were idle.
Three follow-up runs then failed before provider work, each with a
different refusal, and their tasks stayed blocked.
> - The refusals asked for proof that a restart had made impossible. A
restart is a product failure, so recovery must settle it automatically
and keep the strict checks for real uncertainty.
> - This pull request accepts the proof that the restart leaves behind,
in three places: the warm owner on a replaced sandbox, the suspended
Runner with an undelivered suspend receipt, and the restore hold of a
run that saved nothing new.
> - The benefit is that these follow-up runs start, and state saved by
older releases carries forward without a migration.

## Linked Issues or Issue Description

Refs #20567, #20346

**What happened?**
On staging, a deploy replaced the webserver process at about 18:04 UTC
on 2026-11-10 (lease controller boot `348ec0bb` on one deployment, then
boot `3e84c195` on a new deployment that started at 18:04:16). Three
runs of one Codex agent on Daytona failed between 18:08 and 18:15:

1. A warm follow-up got a newly created sandbox. The previous warm
Runner stayed `ready` in its own retained sandbox. Live adoption denied
it with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate`
(`remote-warm-session-recovery.ts:127`), because only a resumed sandbox
may be adopted. The per-turn retirement path refused warm owners
(`lifecycle_not_per_turn`).
2. A warm follow-up resumed the same sandbox and directory. The previous
Runner had been suspended by the shutdown, but the controller stopped
before it acknowledged the suspend result. The Runner kept its terminal
fence (`pendingTerminalDelivery` for `runner.suspend`). The read refused
it with `native_controller_recovery_missing_evidence:runner_not_settled`
(`native-session-executor.ts:7032`).
3. A continuation into a fresh workspace failed with
`remote_workspace_recovery_receipt_invalid`
(`remote-workspace-recovery-attachments.ts:60`). Its predecessor had
restored the committed save, failed before provider work, and kept a
restore hold with no recovery receipt. Its saved reference named the
snapshot it restored.

**Expected behavior**
After a restart, the next run continues. It retires a finished owner
that nobody can adopt, reads a finished suspension, and skips a source
that left no unsaved files. Real uncertainty still refuses with a named
reason.

**Steps to reproduce**
1. Run a warm native Codex turn on a reusable Daytona sandbox and let it
commit.
2. Restart the server (deploy) while the Runner is idle.
3. Start the next turn. Get a new sandbox (case 1), or resume the same
sandbox after the shutdown suspended the Runner (case 2).
4. For case 3, continue a task whose previous run restored the last save
and failed before provider work, into a fresh workspace.

**Paperclip version or commit**
Staging before `2e3cbe57` (the runs predate that deploy). The gaps
remain on `master` at `9b21dcbd`.

## What Changed

- `inspectRemoteWarmSessionRecovery` returns `null` when this run's
sandbox is `created` or `replacement`. Such a sandbox holds no Runner to
adopt. The caller then uses ordinary admission.
- `inspectFinishedPerTurnRunnerdOwner` also retires a **warm** owner on
a remote sandbox runner when this run's sandbox is `created` or
`replacement`. It uses all the existing proofs: terminal run, committed
result, released coordinator lease, stopped or fenced controller,
settled journal, and quarantine copies. Local warm Runners and resumed
sandboxes keep the live path.
- `restoreRemoteSuspendedRunnerCheckpoint` accepts a suspended Runner
whose only open fence is its own undelivered `runner.suspend` receipt.
The fence must name `runner.suspend` with lifecycle `suspended`, it must
be the last controller command, and the restored journal must not name
that command differently. A provider cleanup, unsent events, a turn's
terminal fence or a warm transition still refuse with
`runner_not_settled`. A later authorized Runner process reconciles the
receipt, as the Runner already does.
- `attachPreservedRemoteWorkspaceSnapshots` skips a source with a
receipt-less restore hold when its saved reference names the exact
committed snapshot that its own succeeded `remote_workspace_restore`
loaded at the same epoch. Any other saved snapshot still needs the
canonical receipt.
- `doc/execution-semantics.md` describes the three rules.

## Verification

- Red (unfixed code), each in the exact staging shape:
- `native-recovered-owner-followup.test.ts` "retires a finished warm
owner left behind by a restart when this run's sandbox is
created/replacement" failed with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate`.
- `native-controller-recovery.integration.test.ts` "defers a remote warm
continuation…" failed: live adoption rejected a created/replacement
sandbox with the same error.
- `native-controller-recovery.integration.test.ts` "reads a resumed
sandbox's suspended Runner with an undelivered suspend receipt" failed
with `native_controller_recovery_missing_evidence:runner_not_settled`.
It reads through a sandbox runner that refuses paths outside the
workspace base.
- `workspace-recovery-preparation.test.ts` "decides a receipt-less
restore hold of a source that restored_unchanged" failed with
`remote_workspace_recovery_receipt_invalid`.
- Green: the same tests pass. Focused files pass:
`native-recovered-owner-followup.test.ts` and
`native-controller-recovery.integration.test.ts` (104 tests),
`workspace-recovery-preparation.test.ts` (85 tests).
- Safety cases (strict checks stay strict):
  - A warm owner on a resumed sandbox is not retired.
- A warm owner with an unsettled journal (pending command, unanswered
tool input) stays held and names `retirement_refused=journal_…`.
  - A warm owner whose finalization lease is still held stays held.
- A turn terminal fence, a provider cleanup, unsent events, a fence
behind a later command, or a journal that names the command differently
still fail with `runner_not_settled` and cache nothing.
- A restore-hold source that saved another snapshot, whose restore did
not succeed, or whose restore was at another epoch still fails with
`remote_workspace_recovery_receipt_invalid`.
- Neighbouring suites pass: `native-session-executor.test.ts`,
`plugin-runner-release-resume.integration.test.ts`,
`explicit-native-continuation.test.ts` and
`native-harness-filesystem-independence-continuation.integration.test.ts`
(1786 tests). The last one asserted the old warm refusal on a created
sandbox. It now shows that the warm follow-up resumes the same provider
session for the new run.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` returns exit 2, because a global doc input changed
(`doc/execution-semantics.md`). Required CI supplies the broad coverage.

## Risks

- Previous-release state: all three rules read state that earlier
releases already save (lease outcome, coordinator rows, Runner state in
the sandbox, journal commands, run results and workspace operations).
Rows saved by older releases carry forward. No migration.
- Case 1 starts a new Runner for a warm session whose old Runner may
still exist, idle, in its retained sandbox. Its controller is gone and
its result is committed, so it can accept no new commands. The provider
session follows the existing resume-or-replace policy.
- Case 2 trusts a suspend fence only in the exact shape that the Runner
writes before it sends the result. Other fences still refuse.
- After case 1, a warm follow-up resumes from its own warm checkpoint.
That checkpoint carries no replacement policy, so a failed provider
resume is still reported as `native_session_recovery_failed` and is not
replaced. This is the existing warm behavior.
- Case 3 skips a source only when it provably saved nothing beyond a
committed snapshot. That snapshot stays in its own workspace.
- The three tasks that already failed stay blocked until their next wake
or a user message. This PR does not add an automatic continuation for a
completed no-replay hold. A follow-up can add that.
- `native-runner-restart-recovery.integration.test.ts` (runs only with a
runner container) no longer probes a replacement sandbox, because that
probe now retires the owner by design.
- Not verified live on Daytona.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only browser access to staging).
Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 14:38:30 -06:00
DottaandClaude 24661683db fix(recovery): bootstrap a remote follow-up that a restart cut before its Runner launched (#20642) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a control-plane restart, restart recovery classifies each
claimed native run: reattach a live Runner, resume from a checkpoint, or
bootstrap again when durable records prove no provider work
> - A follow-up run on a continued session gets a rebound admission
checkpoint. It names the new run, but it keeps the previous turn's
provider session ID
> - Recovery read that copied provider session as this run's provider
work. A follow-up killed after `native.process_start_requested` but
before its remote Runner launched was classified
`reattach_remote_runner`
> - The reattach then had no Runner to find, failed with
`native_continuation_requires_reconciliation`, and the task needed Retry
> - This pull request stops the copied session from counting as this
run's work, and keeps the existing durable proofs as the authority
> - The benefit is that these restarts bootstrap again by themselves and
do not repeat provider work

## Linked Issues or Issue Description

Refs #20561, #20555

**What happened?**
Two Daytona restart-chaos tasks needed a Retry click:
- Wave `w-20261110T173243-s9202` t02 (claude C1, K2): the follow-up run
was killed at `stage.asset.home`, after
`native.process_start_requested`. Recovery chose
`reattach_remote_runner`. Verification failed with
`native.remote_runner_recovery_failed {stage: runner_state, reason:
state_read_failed}` (`runner_remote_state_unavailable: exit=1`):
`runner-state.json` did not exist because the Runner never launched. The
storage fault window (ST7) had ended 30 s before, so it was not the
cause.
- Wave `w-20261110T174930-s9205` t04 (codex C1, RR1): the scheduled
retry of a `process_lost` run was killed at
`session.checkpoint.restore`, after `native.process_start_requested`.
Recovery chose `reattach_remote_runner`. Verification failed with
`native_remote_recovery_lease_mismatch: stage=runner_verification
field=identity_run_id`: the local Runner identity belonged to the
previous run, because this run's Runner never started.

**Expected behavior**
A run whose durable records prove that no Runner received a provider
turn bootstraps again without a person.

**Steps to reproduce**
1. Start a remote follow-up run on a continued session (for example, a
comment after a completed turn).
2. Kill the server after `native.process_start_requested` and before the
remote Runner launches.
3. Observe `reattach_remote_runner`, a failed verification and a Retry
banner.

**Paperclip version or commit**
master close to `6d6b2d0e` (waves); still reproduced by the new tests on
master `90dd864bff`.

## What Changed

- `native-restart-recovery.ts`, no controller checkpoint for the run
(`remote_runner_never_received_commands`): a follow-up with
`sessionIdBefore` qualifies when it carries only its continued-session
checkpoint, and a predecessor's settled Runner identity
(`carriesInheritedRunnerIdentity`) no longer disqualifies it. A recorded
process identity, a Runner-sent event, a provider event or a pending
request still keeps the run held.
- `native-restart-recovery.ts`, controller checkpoint exists: the
existing Postgres proof (`nativeRunnerCheckpointRemoteBootstrapProof`)
now also runs when the run carries only a rebound checkpoint whose
provider session a settled earlier run of the same session saved
(`carriesSettledPredecessorSession`). The copied provider session then
does not count as this run's provider evidence. The controller
checkpoint stays the authority: any turn command or turn event keeps the
run held.
- `doc/execution-semantics.md`: describe the rule.

## Verification

-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
(real Postgres):
- "restarts a follow-up whose controller never saved a checkpoint (s9205
t04)" and the variant "with its predecessor's remote runner identity":
red on master (`reattach_remote_runner`), green now.
- "restarts a follow-up whose saved controller checkpoint proves no
provider turn (s9202 t02)": red on master, green now.
- Held cases stay held: own recorded Runner identity, a carried active
turn, a provider session that no settled earlier run saved, a
`turn.start` command, a projected `turn.started` event.
- Whole file: 171 passed.
- `npx tsc --noEmit -p server` passes.
- The executor's existing `bootstrap_incomplete` path handles both
shapes: a scoped root that belongs to the previous run goes through the
existing prior-owner verification, and a root of this run restored from
the controller checkpoint goes through
`runnerdStateProvesIncompleteBootstrap` /
`runnerdStateProvesNoRemoteProviderTurn`.

## Risks

- A run with any recorded Runner process, Runner-sent event, provider
event, turn command or pending request keeps today's reattach or hold.
- A bootstrap claim charges one provider attempt, as today. The bounded
limit (3) still applies.
- No persisted shape changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 14:37:56 -06:00
DottaandClaude ed5e7493bb fix(native): take back a lapsed lease held under the run's own restart claim (#20625) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run holds an execution lease on its
`native_run_finalizations` row. The owner renews it every 5 minutes.
After a host sleep (or the restart-chaos time fault TM2), the stored
lease looks lapsed although the owner is alive.
> - #20474 lets the same owner take back a lapsed lease when no other
actor recorded anything on the row. One of the checks is
"`recovery_state` is null".
> - Restart recovery writes `recovery_state` (`resuming_session`,
`awaiting_runner_reattach` or `bootstrap_incomplete`) in the same update
that grants the lease. The executor keeps that value until a result is
accepted.
> - So a run that a restart resumed can never take back its own lapsed
lease. Renewal interrupts the turn, and failure settlement is refused
with `native_session_lease_lost`. The run fails with `adapter_failed`,
recovery holds it with `preserve_without_replay_v1`, and the user must
click Retry.
> - This pull request counts the claim's own record as untouched when
the row still shows exactly that claim and nothing recorded since.
> - The benefit is that a resumed run survives a later host sleep with
no Retry. Every real takeover still fences the late owner out.

## Linked Issues or Issue Description

Refs #20474

**What happened?**
The zero-retry restart-chaos matrix (local wave L7, seed 9107: TM1, TM2,
TM3 with 1 h aging) had 3 of 6 tasks that needed Retry. All three were
Codex runs. In each run, a TM3 kill made restart recovery resume the run
(`resume_dead_runner`, generation 2). The runner restored the
conversation, reported `provider_turn_lost_on_restore`, and the
same-conversation restart continuation started as designed. Then the TM2
fault aged every database time by one hour. The execution lease lapsed
while its owner was alive. The run then failed with
`native_session_lease_lost` (`adapter_failed`), and recovery stopped
with `native_continuation_requires_reconciliation` /
`preserve_without_replay_v1`. One run had already proposed its result
and completed its turn. The Retry click worked every time.

**Expected behavior**
A run whose lease lapsed only because time jumped, and that no other
actor claimed, continues with no Retry. This must also be true after a
restart resumed that run.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--env local --adapters codex,claude --workloads A1,B5,C1 --faults
TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107`

## What Changed

- `server/src/services/native-runtime/native-execution-lease.ts`: the
"untouched" part of `nativeExecutionLeaseHeldCondition` now also accepts
the run's own local restart-recovery claim. All of these must be true:
- the last `recovery_history` entry is that claim: same controller boot
and generation as the row;
- its disposition matches the current `recovery_state`
(`resume_dead_runner` → `resuming_session`, `reattach_existing_runner` →
`awaiting_runner_reattach`, `bootstrap_incomplete` →
`bootstrap_incomplete`);
  - `recovery_request_id` equals the request that claim recorded.
Any later actor appends a history entry (shutdown suspension, failure
settlement), moves the state (`awaiting_evidence`, `blocked`), or
replaces the owner, boot or generation. Each of these still fences the
late owner out. All other untouched checks are unchanged.
- `native-execution-lease-reacquire.test.ts`: new cases for each claim
kind, a requested claim, failure settlement under the claim, and eight
fenced variants.
- `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1
lists the new rule.

## Verification

- Red/green: `cd server && npx vitest run
src/services/native-runtime/native-execution-lease-reacquire.test.ts`.
Without the source change, 5 new cases fail (the three claim kinds, the
requested claim, and failure settlement). With it, all 37 pass.
- Related suites pass: `native-execution-lease.test.ts`,
`heartbeat-restart-claim-lapse.test.ts`,
`heartbeat-native-orphan-sweep-lease-race.test.ts`,
`native-runner-restart-lease-renewal.integration.test.ts`,
`native-runner-restart-preparation.integration.test.ts`.
- `cd server && npx tsc --noEmit` passes.
- Live: the same L7 wave (seed 9107, local) on this branch. The server
log has 0 `native_session_lease_lost` (the baseline had 3). The two
tasks whose cells failed this way in the baseline (codex C1 TM1, codex
C1 TM3) now pass with no Retry. Two other tasks needed Retry for
different causes, which this pull request does not change: codex B5 TM2
(a `runner.drain` timeout, then a same-run retry that may not continue a
turn lost on restore) and claude C1 TM2 (`ACPX session is not open` on
`turn.start` until the retry budget ran out). The L6 wave (seed 9106)
also ran on this branch: 5 of 6 passed. Its one failure is
`execution_recovery_budget_exhausted` and is not related to leases.

## Risks

- The rule widens when an owner may take back its own lapsed lease. It
applies only to local runs, before a result, with the same owner token,
attempt, boot, pid, process start and generation. A claim that any other
actor changed afterwards does not match.
- Rows saved by earlier releases carry the same history entry shape, so
a run claimed before a deploy is handled the same way. A deploy changes
the boot, so the new process never takes back the old process's lease.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 14:31:15 -06:00
DottaandClaude 6505538fc5 fix(realtime): limit a shared live-event read set to 100 events and 250 ms (#20632) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server sends the output of a running agent to the browser on the
company WebSocket.
> - #20624 lets the events that are already in the queue share one set
of permission and redaction reads.
> - That set could serve a full queue of 1,000 events for one second.
> - If access is revoked during that time, the principal still receives
the queued output. The security scan on #20624 reported this.
> - This pull request limits one set to 100 events and 250 ms.
> - The benefit is a small, stated limit on output that a revoked
principal can receive, with the same delivery speed.

## Linked Issues or Issue Description

Refs #20624

**What happened?**

After #20624, one set of permission reads can authorize each event that
was in the queue when the set opened. The queue holds at most 1,000
events or 1 MiB. The set stays in use for one second. If a membership,
an API key or a task grant is revoked in that time, the server still
sends the remaining queued events.

**Expected behavior**

A revocation stops delivery after a small, fixed quantity of queued
output. The documentation states that quantity.

**Steps to reproduce**

1. Open a company WebSocket as a user who can read a private task.
2. Publish 1,000 transcript events for a run of that task in one burst.
3. Revoke the task grant while the server delivers the burst.
4. The server delivers the remaining queued events.

**Paperclip version or commit**

`master` at 90dd864bff.

## What Changed

- `live-event-delivery.ts` replaces a shared read set after 100 events
or 250 ms. Before, the limit was one second with no event limit.
- `live-event-delivery.test.ts` pins the exact limits. One set serves
events 2 to 101 and event 102 opens a new set. Access is revoked during
that set, and event 102 is not delivered. A set is kept at 249 ms and
replaced at 250 ms.
- `doc/live-transcript-delivery.md` states the limit and that queued
output from before a revocation can still be sent up to that limit.

## Verification

- `npx vitest run src/realtime
src/__tests__/live-events-run-read.test.ts` in `server/`. 66 tests pass.
- The boundary tests fail when either limit changes by one in either
direction (99, 101 events; 249, 251 ms).
- The 200-event burst test against PostgreSQL issues 22 statements with
this limit. Separate reads issued 1,401.
- Not run locally: a clean server typecheck. The local worktree reports
type errors in `server/src/services/chat-channels/slack/adapter.ts`,
which this change does not touch. CI supplies the typecheck.
- Not verified: delivery delay on a hosted instance after deployment.

## Risks

- More permission reads for large bursts: one set for each 100 queued
events. A burst of 1,000 events uses 10 sets, not 1,000.
- Output produced before a revocation and already in the queue can still
be sent, up to 100 events or 250 ms. Before #20624 the limit was the one
event in flight.
- No schema, API or protocol change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude, model ID `claude-fable-5-2`, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 14:18:20 -06:00
DottaandClaude Opus 5.5 fd6e71a7ff fix(native): restore a warm backup when its source sandbox is idle (#20627) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes keep a warm sandbox and a verified
harness backup so the next run continues the same provider session
> - A run that starts while the previous run's workspace export is still
saving gets a new (created) sandbox lease, because reuse skips an
exporting sandbox
> - When that run prepares the harness, it finds a backup but no
recorded reusable lease, and it fails closed with
`runner_harness_state_mismatch: backup_without_reusable_lease`
> - #20278 accepted a destroyed source and a settled restore hold as
retired backup sources, but not an idle warm source that is still
`retained` for reuse
> - This pull request accepts an idle warm source under the same proofs,
claims it so it is never resumed again, and covers the resumed lease
that the failed run leaves behind
> - The benefit is that these recoveries continue the same session
automatically instead of blocking the task

## Linked Issues or Issue Description

Refs #20278

**What happened?**
A monitor wake started about 90 seconds after the previous run on the
same task finished. The previous run's warm sandbox lease was `retained`
with policy `reuse_by_environment`, but its workspace export was still
saving. Reuse skipped it and created a new sandbox. Workspace
realization waited for the export (about 136 s) and restored the saved
files. Then harness preparation found the previous run's backup (its
lease stamp names this exact manifest, and no provider work followed
it). The source lease was not destroyed and was not a restore hold, so
the run failed with `native_session_recovery_failed: Error:
runner_harness_state_mismatch: backup_without_reusable_lease` and the
task was blocked.

**What did you expect?**
The run continues the same provider session. The idle source sandbox
holds exactly the backed-up state, so restoring the backup into the new
sandbox is safe.

**Steps to reproduce**
1. Finish a warm native run on a remote sandbox. Its workspace export is
still saving.
2. Start the next run on the same task before the export settles.
3. Reuse skips the exporting sandbox and creates a new lease. Harness
preparation fails with `backup_without_reusable_lease`.
4. A retry resumes that new lease, which has no harness state, and fails
with `resumed_unavailable`.

## What Changed

- `native-retired-backup-source.ts`: an idle warm source is a third
retired backup source. It must be `released` or `retained` with policy
`reuse_by_environment`, its own run must be terminal and not the current
run, and it must have no cleanup in flight, no restore hold or pending
shutdown settlement, and no pending native workspace export. The backup
stamp, single-owner chain, and later-provider-work checks still apply.
- Before the restore, the idle source is claimed with a compare-and-swap
on the checked row. The claim moves its `reusableSandboxLease` scope
into a `reusableSandboxLeaseSuperseded` receipt. Reuse never resumes
that older state again, and the next acquisition treats it as obsolete
and destroys it. If a run resumed or changed the source first, the claim
matches no row and the backup is not restored.
- `environments.ts`: a cross-run reuse handoff now requires that the
source row has no superseded receipt. The supersede claim and the
handoff compete on the same row, so an acquisition that selected the
source before the claim cannot resume it afterward.
- The proof now rejects a backup whose source is the target allocation
itself (`targetProviderLeaseId`).
- `native-session-executor.ts`: a resumed lease whose sandbox has no
harness state at all (`unavailable`) restores the backup under the same
proof. This recovers the lease that the failed run left behind, so rows
saved by older releases recover after deploy without a migration.
- `doc/execution-semantics.md`: documents the idle-source rule, the
supersede claim, and the resumed stateless lease.

## Verification

- Red before the fix, green after:
- `native-session-executor.test.ts` "restores a warm backup into a
created lease only when an idle warm source finished its run and its
export" failed with `backup_without_reusable_lease`.
- "... a resumed lease never held the session and the warm source is
idle" failed with `resumed_unavailable`.
- `native-retired-backup-source.integration.test.ts` (real Postgres)
"authorizes an idle retained/released warm source and claims it as
superseded" failed (returned false).
- Integration test "refuses a reuse handoff of a source superseded while
its provider resume was pending" fails without the handoff guard and
passes with it.
- Safety rows stay refused: source run still running, workspace export
pending, cleanup in flight, restore into its own allocation, stamp names
another checkpoint, provider work after the backup, two host leases for
one allocation, claim lost to another run, and a resumed lease that lost
its own state. Existing destroyed-source and restore-hold rows are
unchanged.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-retired-backup-source.integration.test.ts
src/services/native-runtime/remote-warm-session-recovery.test.ts` passes
(1408 tests).
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` exits 2 because a global CI input
(`doc/execution-semantics.md`) changed. Required CI supplies the broad
coverage.

## Risks

- Previous-release story: no persisted shape changes. Lease rows saved
by earlier releases carry forward. The new
`reusableSandboxLeaseSuperseded` metadata key is additive.
- The idle source is destroyed at the next acquisition for its
workspace. Its state is the verified backup, which the restored lease
now holds, so no work is lost.
- Not covered: when the source cannot be proven retired or idle (for
example a live owner or later provider work), the run still fails closed
with the same named code. An automatic fresh-session fallback for that
case is not in this PR.
- The root trigger (reuse skips a sandbox while its export saves, then
waits for that export) is unchanged. This PR does not change the reuse
lookup.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, with tool use
and code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 14:11:10 -06:00
DottaandPaperclip 76c787fffc fix(runner): report monitor notes length limits (#20604) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use task monitors to resume a future check.
> - The monitor schema limits notes to 500 characters.
> - The field description did not state the limit. Rejection showed only
the schema path.
> - This pull request adds the limit to the description and to safe
length errors.
> - Agents can correct notes before they retry a rejected call.

## Linked Issues or Issue Description

Related prior fix: #16868.

**What happened?**

A native `set_task_monitor` call with 501 characters in `monitor.notes`
failed with a path that ended in `notes/maxLength`. The error did not
state 500. The schema already had `maxLength: 500`, but the field
description did not state it.

**Expected behavior**

State the limit in the field description and in the rejection. Accept
500 characters. Reject 501 characters before dispatch. Preserve a saved
monitor when a replacement is invalid.

**Steps to reproduce**

1. Supply a future `monitor.nextCheckAt` and 501 characters in
`monitor.notes`.
2. Read the native provider error.
3. Before this fix, it shows the constraint path without the maximum.

**Paperclip version or commit**

Reproduced on `7368a0b15c2f02107778a961144d86e2eb59a952` in a Linux
source checkout, with Node 25.9.0 and Rust 1.97.1.

## What Changed

- Use one local value for the published notes limit and its description.
The server test checks agreement with the persisted shared validator.
Keep the standalone Runner package boundary.
- Read the numeric maximum from the authorized JSON Schema validation
error. Keep submitted values private and preserve the 512-byte feedback
bound.
- Test the published schema and real bridge dispatch path at 500 and 501
characters. Test saved contents and rejection without monitor changes
with real PostgreSQL.
- Refresh generated contracts and document the limit.
- Sync two SDK proof lock identities after the reviewed master lock
refresh in #20615. This fixes the remaining CI proof mismatch. Keep
compiler, parser, Zod, and all rejection checks unchanged.

## Verification

- Final head: `bac60234ab3a8fc7043a90c7f9c81b02e250428b`.
- Red: the server regression failed because the field description
omitted `500`. The Rust regression failed because the error omitted
`maximum 500 characters`.
- Local passes after the final base update: 44 PostgreSQL
monitor/scheduler tests, 14 real-source Rust bridge tests, 11
cache/Docker guard tests, and 56 SDK proof tests. Runner TypeScript
build and generated contract checks pass.
- The focused Rust check compiles the actual source in a small Cargo
workspace. The full Rust CI suites also pass on Linux and Windows.
- Final-head [PR CI
run](https://github.com/paperclipai/paperclip/actions/runs/42881633978)
passes. This includes typecheck, build, Runner support, Runner Rust and
Vitest, server tests, serialized tests, and all eight e2e shards.
- Final-head [Windows Rust
run](https://github.com/paperclipai/paperclip/actions/runs/42881633419)
passes.
- Greptile: 5/5 on the final head, with no open findings. There are 61
successful checks and five conditional skips. The separate Paperclip
Review remains queued; its comment says work is paused. No CI or test
check is pending or failed.
- `pnpm verify` returned exit 2 and selected broader CI. The full local
suite was not run. Final-head CI supplies that coverage. Earlier failed
attempts were repaired and are not counted as passes.

## Risks

- Low risk. The existing 500-character bound, scheduling semantics, and
permissions stay in place.
- Length feedback uses only a schema path and numeric bound. It does not
print submitted text.
- The Runner binary and tool contracts must reach an instance for the
new guidance to apply. This PR does not deploy or merge.

## Model Used

OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The
runtime does not expose a more specific model ID or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 13:59:58 -06:00
DottaandClaude Opus 5.5 38944cff1a fix(environments): retry and safely replace a reusable sandbox that will not resume (#20626) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable (warm) cloud sandbox resumes the same
sandbox on the next wake
> - When a follow-up arrives while the provider is auto-stopping or
archiving the idle sandbox, the resume can fail
> - Each failed attempt waited the provider's whole five-minute start
deadline and then failed with `reusable_sandbox_resume_failed`, even
when the workspace was saved and nothing lived only on the sandbox
> - Two such runs in a row made a user wait for many minutes and
recorded two failed runs
> - This pull request bounds the resume, retries a state change inside
the run, and replaces the sandbox from the saved snapshot when that is
safe
> - The benefit is that recovery works automatically and the first turn
is not delayed by minutes

## Linked Issues or Issue Description

**What happened?**
Two consecutive runs failed before provider dispatch with: `Reusable
sandbox lease "<id>" could not be resumed (cause=provider_error); the
lease was preserved and no replacement was created.` Each run took about
5.5 minutes (320s and 330s) in preparation. The third run resumed the
same sandbox from `stopped` in 4.7 seconds after the provider settled.
The previous run had ended with a clean final save and a verified
harness backup.

**Steps to reproduce**
1. Let a task's reusable sandbox go idle so that the provider auto-stops
or archives it.
2. Send a follow-up while the provider still reports the sandbox as
changing state.
3. The Daytona worker waits in `ensureSandboxStarted` until its 300s
start deadline and throws "is still <state> at its start deadline". The
host does not classify that as transient and it has no safe fallback, so
the run fails.

**Expected behavior**
The run retries briefly. If the sandbox still cannot start and nothing
lives only on it, the run continues on a replacement sandbox that
restores the saved workspace.

**Root cause**
`server/src/services/environment-runtime.ts` (reusable resume block,
about line 1820): the resume deadline was the provider's full start
deadline (300s plus RPC margin), the transient classifier did not
recognize a state transition, and the catch always threw
`ReusableSandboxResumeError`. #20212 waits out a state change inside the
same 300s start deadline, so it covers the start but not the time budget
or the fallback.

## What Changed

- Add `server/src/services/reusable-sandbox-resume-replacement.ts`:
- `REUSABLE_SANDBOX_RESUME_BUDGET_MS` (90s): the total resume time for
one run, retries included.
- `sandboxStateTransitionFailure`: "state change in progress" and "is
still <state> at its start deadline" are transient.
- `retireUnresumableReusableSandboxLease`: replace only when the lease
is `released`/`retained` with no live owner, the task workspace is clean
with a committed saved snapshot and no other writer, and a
`paperclip_runner` session has a verified lease-bound harness backup.
Access refusals (401/403) and an unavailable worker stay strict. After
the replacement lease (which carries the harness backup) is recorded,
the old lease moves to `pending_cleanup` (reason `resume_failed`) in one
guarded write, so the existing cleanup sweep destroys the old sandbox
and no provider delete runs before the first turn. A failed creation or
a restart before that write leaves the old lease `retained` for the next
attempt.
- `environment-runtime.ts`: cap the resume deadline at the budget, treat
a state transition as transient, and call the replacement policy before
failing. A replacement records `sandboxLeaseAcquisition: { outcome:
"replacement", reason: "resume_failed" }`; workspace preparation then
restores the saved snapshot.
- `remote-workspace-transport.ts`: add the closed failure label
`sandbox_state_transition`, so a strict failure names the cause instead
of `provider_error`.
- `doc/execution-semantics.md`: document the budget, the in-run retry,
the replacement conditions, and the strict cases.
- Update the two slow-start tests: the resume RPC timeout is now 90s
instead of 330s/150s. A 61s start still resumes the same sandbox.

## Verification

- Red before the fix (new file
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`,
staging shape: lease created by the real acquisition path, then retained
after a clean save with a backup stamp, 300s provider deadline, worker
refusal with no status): 3 tests failed with `could not be resumed
(cause=provider_error); the lease was preserved and no replacement was
created.`
- Green after the fix: 10/10 pass, including a case where the
replacement creation fails and the next run replaces the sandbox with
the backup carried forward. The safe case replaces the sandbox, keeps
every resume RPC at or below 90s, makes no synchronous destroy call, and
moves the old lease to `pending_cleanup`. A single state-change refusal
resumes the same sandbox inside the run. Six strict cases keep the lease
and fail with `cause=sandbox_state_transition`: unsaved work, no saved
version, an uncommitted saved version, a live owner, no verified backup,
and a 403.
- `pnpm exec vitest run src/__tests__/environment-runtime.test.ts` (197
pass), `heartbeat-retry-scheduling`, `legacy-execution-recovery`,
`remote-workspace-transport` (251 pass).
- `tsc -p server/tsconfig.json --noEmit` passes.
- `pnpm verify` exits 2 (a doc change is a global CI input). Required CI
supplies the broad coverage.
- Not run: a live Daytona run on staging. The provider message is kept
off the run record, so the exact transition state on staging (stopping
or archiving) is inferred from timing, not observed.

## Risks

- A strict resume now gives up after 90s instead of 300s. The provider
start continues in the background, and the bounded bootstrap retry
resumes the same sandbox. A stopped sandbox resumes in seconds and an
archived one in about 25s on Daytona, so 90s leaves headroom.
- A replacement leaves the old sandbox to the cleanup sweep. If the old
start finishes later, the sweep still destroys it. The guarded write
only claims a `released`/`retained` lease with no live owner.
- Upgrade safety: no new persisted fields are required. Leases saved by
older releases are read with the same rule; one without a verified
backup stays strict. The replacement reason `resume_failed` already
exists in the acquisition outcome.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 13:51:10 -06:00