Commit Graph
2317 Commits
Author SHA1 Message Date
Devin FoleyandPaperclip ad320aeb71 fix(native): wait for a dated provider limit reset and keep it out of Sentry (#19775)
Keep native quota cooldowns tied to valid dated reset messages. Parse explicit years and strict month names while preserving already-expired dates. Keep typed native quota and rate-limit failures on their tasks in unexpected Sentry reporting mode. Preserve setup, process, malformed and all-mode reporting.

Verified current-head CI, Greptile 5/5 at Medium impact, local parser/report red-green tests and PostgreSQL integration coverage.

Co-Authored-By: Devin Foley <devin@devinfoley.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-12 17:06:22 -08:00
DottaandClaude Opus 5.5 334b2ae066 fix(recovery): continue a task blocked behind its workspace's ended writer once the hold releases (#21221)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote task workspaces have one writer run at a time, and an ended
writer keeps the workspace until its files are saved or set aside
> - When a task's next runs fail setup on that hold ("The workspace
still has an unfinished writer or save operation."), stranded-work
recovery blocks the task with a board escalation
> - The retained-file recovery from #21136 later released such a hold on
staging (it continued from the last save), but nothing settled the board
escalation, so the task stayed Blocked until a person pressed Retry
> - Recovery must work without a person: the block belonged to the hold,
and the hold is gone
> - This pull request adds a pass to the active recovery-action sweep
that settles that escalation once the workspace's writer hold is free
> - The benefit is that such tasks continue by themselves with one plain
notice and one run, with no Retry

## Linked Issues or Issue Description

Refs #21136
Refs #21143

**What happened?**
On staging, an ended run held its task's own remote workspace after its
superseded workspace export failed. The task's queued runs failed setup
with the unfinished-writer error, and stranded-work recovery blocked the
task with a board-owned escalation. After #21136 was deployed, its
retained-file recovery recorded the hold 77 seconds after server start
and, 19 minutes later, continued the task from its last save and
released the workspace. The task still stayed Blocked behind the board
escalation, and no run was queued. A board user had to press Retry.

**What did you expect to happen?**
Once the hold that caused the setup failures is released, the task
returns to `todo` and its assignee continues it, with no Retry.

**Steps to reproduce**
1. Give a task a remote workspace whose writer is an ended run with
unsaved files.
2. Let the task's next run fail setup with the unfinished-writer error,
and let stranded-work recovery block the task.
3. Release the hold (save the files, or continue from the last save).
4. Run the active recovery-action sweep. Before this change the task
stays Blocked with an active board escalation.

**Why the merged fix did not unblock the task**
The retained-file fallback only releases a Blocked status that its own
hold set (`holdOwnsBlockedStatus`). Here stranded-work recovery set the
block with a different, board-owned escalation, and a late status
decision 1.4 seconds later kept that block. So the fallback correctly
did not take the status, and no code settled the board escalation after
the hold was gone.

## What Changed

- Add `continueTasksAfterWorkspaceWriterReleased`
(`server/src/services/recovery/released-writer-escalation.ts`). It
selects board-owned `stranded_assigned_issue` escalations whose latest
run failed setup with the unfinished-writer error before any provider
work, by the current assignee. In one transaction it checks that the
block belongs to the escalation, that no newer run exists for the task,
that dependencies allow work, and that the workspace's writer hold is
now free. Then it resolves the escalation with a named outcome, returns
the task to `todo`, records one plain activity notice, and records a
pending wake.
- A later status decision that only kept the Blocked status it found
(from `blocked` to `blocked`) does not own the block. A decision that
chose the block still keeps the task blocked.
- Run the pass in `reconcileActiveRecoveryActions`, next to the legacy
rehome pass, and include its next wake retry time in the pass deadline.
- Generalize the legacy rehome wake delivery to
`deliverPendingRecoveryWakes`, keyed by the evidence key, so both passes
share one crash-safe, backed-off wake delivery. The rehome wake keeps
its fresh session. The new wake resumes normally.
- Keep the company selected by the scheduled sweep while a
released-writer wake is owed (`recoveryActionSweepCondition`).
- Document the rule in `doc/execution-semantics.md`.

## Verification

- `cd server && npx vitest run
src/__tests__/released-writer-escalation.test.ts
src/__tests__/legacy-workspace-rehome.test.ts
src/__tests__/issue-blocker-stall-wakeups.test.ts
src/__tests__/issue-blocker-stall-waiter-statement.test.ts
--maxWorkers=2`: all pass.
- Red/green: with the sweep hook removed, "continues the task once the
hold is released" fails (the task stays `blocked`). With the hook, it
passes.
- The new test seeds the staging row shape: a native writer run ended
with `native_finalization_superseded` still writing the task's own dirty
workspace, three legacy setup failures with the unfinished-writer error
and `{ kind: "bootstrap", providerWorkStarted: false }`, a board
escalation with `latestRunErrorCode: "setup_failed"`, the block 4.4
seconds after the last failure, and a later `blocked` to `blocked`
status decision.
- The tests also cover: no change while the writer still holds the
workspace, one wake only, a pending wake after a crash, a refused wake
that backs off, a later decision that chose the block, a dependency that
forbids work, a newer run, and another setup error.
- `cd server && npx tsc --noEmit -p tsconfig.json`: passes.

## Risks

- Low risk. The pass only acts on board escalations whose evidence is
exactly the unfinished-writer setup failure, and only when workspace
reservation would now admit a new writer. It never changes the
workspace, never applies a result, and never releases a writer fence.
- The task gets one normal wake through the issue-state guard (`todo`,
same assignee), so it cannot start twice.
- The pass adds one bounded query per sweep of a company with active
recovery actions.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only browser API reads of staging) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:58:06 -06:00
DottaandClaude Fable 5.2 9a9cd20d91 feat(ci): send one in ten CODEOWNER PR runs and their outside checks to Honeycomb (#21192)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Contributors need fast, predictable CI feedback.
> - PR #16378 added a Honeycomb trace of the main PR workflow, but only
for PRs with the `profile-ci` label.
> - Ordinary CI time was therefore never recorded. All of the last 300
`CI profile` runs were skipped.
> - Reviews and scans outside the `PR` workflow, such as Greptile, were
not in the trace.
> - This pull request exports one CODEOWNER PR run in ten without a
label, and adds outside checks to every trace.
> - The benefit is a continuous record of total CI time and of which job
or check was slowest. The Honeycomb key stays out of reach of outside
contributors.

## Linked Issues or Issue Description

Refs #16378

**What existing behavior does this improve?**

The Honeycomb CI trace that the `CI profile` workflow exports.

**Current behavior**

- The exporter runs only for a PR with the `profile-ci` label. The label
also forces the full suite.
- The trace has the jobs of the `PR` workflow only. Greptile, security
scans, commit statuses and other workflows are absent.
- The only total is the `PR` workflow time.

**Proposed behavior**

- One `PR` run in ten is exported without a label. The run number must
end in 0. The trace has every job and step. Each span has a sample rate
of 10.
- Each trace has one span for each outside check and each other workflow
on the head commit.
- The root span has `ci.workflow_ms`, `ci.all_checks_ms`,
`ci.last_to_finish` and `ci.slowest_check`.

**Reason and benefit**

Maintainers can see how long CI takes and what is slow, for example a
long Greptile review, without a label on each PR. Sampling keeps the
volume near 1.7 million events each month. Export of every run is near
17 million.

## What Changed

- `scripts/ci-profile/authorize.mjs`: add `exportMode` (label or
one-in-ten sample) and `associatedOwnerPr` (the CODEOWNER checks without
the label).
- `scripts/ci-profile/trace.mjs`: add `normalizeChecks` and spans for
outside checks. Add the sample rate and the new root fields. Sampled
runs do not warn about absent test artifacts.
- `scripts/ci-profile/export.mjs`: select the mode. Read check runs,
commit statuses and workflow runs for the head commit. Wait a maximum of
15 minutes for outside apps that are not complete. Sampled runs read no
artifacts.
- `.github/workflows/ci-profile.yml`: no new job. The job condition now
requires a branch of this repository, an actor and rerun actor on the
`AWS_CI_ROUTING_USER_IDS` roster, and a labelled or sampled run. Add
`checks: read` and `statuses: read`. Increase the timeout to 30 minutes
for the bounded wait.
- `doc/ci-profiling.md` and `doc/DEVELOPING.md`: describe sampled runs,
outside checks, the totals and the job condition.

## Verification

- `node --test .github/scripts/tests/ci-profile.test.mjs
.github/scripts/tests/ci-routing.test.mjs`: 163 pass, 0 fail.
- `actionlint .github/workflows/ci-profile.yml`: no findings.
- Dry runs against real runs with the real CODEOWNERS check (`node
scripts/ci-profile/export.mjs --run-id=ID --attempt=1 --dry-run`):
- A sampled CODEOWNER run (run number 64780): 1,144 spans, sample rate
10, 5 outside checks and 1 other workflow.
- A sampled run from an outside contributor (run number 64800): "No
CODEOWNERS-authorized PR at this revision; no data exported."
- A CODEOWNER run that is not in the sample: "This attempt was not
selected for profiling; no data exported."
- New tests cover: the sample rule, fork and outsider PRs, a bot rerun
actor, a stale head, outside-check spans and totals, the bounded wait,
and the job condition.
- Not run locally: `scripts/ci-profile/reporters.test.mjs`. It needs
installed dependencies, and this change does not touch the reporters. CI
runs it.
- No data was sent to Honeycomb before merge. The workflow runs from
`master`, so the first real export occurs after merge.

## Risks

- The label path now also needs the `AWS_CI_ROUTING_USER_IDS` roster. If
that variable is empty, nothing is exported. All four CI CODEOWNERS are
on the roster today.
- The roster is only a first filter. The exporter still checks the PR
author and both actors against CODEOWNERS on `master` before it reads or
sends data. The key stays in the `ci-profiling` environment, which
permits only `master`.
- The export job can now run for a maximum of 30 minutes on a
GitHub-hosted runner while it waits for an outside check.
- GitHub gives no completion time for a workflow run. The span for
another workflow ends at the last update of that run.
- Honeycomb volume increases by about 1.7 million events each month.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with
tool use, code execution and repository inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 16:57:23 -06:00
DottaandClaude Opus 5.5 0904a65d82 fix(server): local board boundary identifies agent processes on Linux it cannot inspect (#21209)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In `local_trusted` mode, the operator's own local processes get
implicit board access without credentials
> - #21181 added a local board boundary. It refuses implicit board
access to requests that come from an agent run's process tree
> - On Linux CI hosts, the boundary sometimes did not decide a loopback
request. It logged a warning and kept the earlier permissive behavior
for that request
> - The cause was not IPv6. IPv4 and IPv6 loopback lookups worked. The
gap was a connection that belonged to a same-user process that the
server cannot inspect
> - This pull request makes the Linux boundary decide those connections
from the process ancestry that the server can read, and refuse them when
that ancestry is not clearly the operator's
> - The benefit is that the local board boundary now holds on Linux for
IPv4 and IPv6 loopback, and the operator's browser and CLI keep working

## Linked Issues or Issue Description

Refs #21181

**What happened?**
On Linux CI hosts, the local board boundary sometimes could not
attribute a loopback request to a process. It then logged that it could
not identify the local process and kept the earlier permissive behavior
for that request. In one chat eval cohort, 37 of 191 server logs
contained this warning. The warning is logged once per server process.

**Expected behavior**
The boundary decides every uncredentialed loopback request on Linux. A
request from an agent run gets no implicit board access. Requests from
the operator's browser and CLI keep implicit board access.

**Actual behavior**
Some requests on Linux were not decided by the boundary, and they got
implicit board access.

## What Changed

- The Linux inspector now records which same-user processes it cannot
inspect. Before, it only recorded that one existed.
- The Linux TCP table lookup now also returns the socket's owner UID. It
covers `/proc/net/tcp` and `/proc/net/tcp6`, including IPv4-mapped
addresses.
- If no inspectable process owns the connection, the server reads the
TCP tables again. If the connection is gone, has changed, or belongs to
a different user, the request is refused.
- If the connection is still open and same-user, the server checks every
process it cannot inspect. Implicit board access stays only if each one
has a readable parent chain that does not belong to an agent run.
Otherwise the request is refused.
- The "cannot identify" fallback now applies only when the platform has
no inspector, or when Linux cannot read the TCP tables. The server keeps
logging a warning in that case.
- `doc/DEPLOYMENT-MODES.md` describes the new Linux behavior.

**How the operator is recognized**
- Browser and CLI processes are normal same-user processes. The server
reads their open sockets and environment, so their requests are
classified as before.
- A browser helper process that the server cannot inspect is still the
operator when its readable parent chain leads to the browser and has no
agent-run markers.
- The server's own requests and requests from the server's ancestors
stay the operator, as before.
- Connections owned by other users were refused before, and they are
still refused.

## Verification

- `pnpm --dir server exec vitest run
src/__tests__/local-agent-process-boundary.test.ts` passes on macOS (15
passed, 10 Linux-only skipped).
- The same file ran on Linux (Docker, `node:24-bookworm`, non-root user,
IPv6 loopback on). Result: 25 passed.
- Red check on Linux with the previous boundary and the new tests: the
agent-run cases for processes the server cannot inspect failed for both
`127.0.0.1` and `::1`. The operator cases passed.
- New Linux-only tests run real processes over `127.0.0.1` and `::1`:
  - an operator process keeps board access
  - an agent-run process is refused
  - an operator process the server cannot inspect keeps board access
  - an agent-run process the server cannot inspect is refused
- New platform-independent tests cover the classification rules and the
`/proc/net/tcp6` fixture parsing.
- A paid chat-feature run on Linux CI, with temporary diagnostic logging
on a separate branch, checks the server logs for the warning and for
refused operator requests. The results will be in a PR comment.

## Risks

- A rare operator tool that the server cannot inspect, and whose parent
chain the server cannot read, now loses implicit board access. Such a
tool can use a board API key.
- The extra TCP table read happens only when a connection has no
inspectable owner. Normal requests do no extra work.
- Platforms without process inspection keep the previous behavior.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, Docker on Linux) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:52:06 -06:00
DottaandClaude Opus 5.5 6ac7211cb9 fix(heartbeat): stop re-delivering a remote-stop follow-up that a succeeded run already read (#21203)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person's follow-up comment must reach the agent once
> - #21175 made a succeeded run's mid-turn read of a comment (through
`get_task_history`) count as delivery for the queued-comment paths
> - The remote-Stop resume path decides delivery with
`undeliveredLegacyUserCommentIds`, which ignores those reads
> - So a follow-up that a succeeded run already read was sent again, and
the agent posted a second DONE on a finished task
> - This pull request applies the same rule on that path and adds a read
helper for a later change
> - The benefit is one answer per follow-up on every resume path, with
no dropped messages

## Linked Issues or Issue Description

Refs #21175

**What happened?**
A saved follow-up comment was resumed after a remote Stop (wake reason
`issue_commented`, idempotency key `remote-stop-comment:…`). A succeeded
run had already read this comment mid-turn. The resumed wake started a
new turn, and the agent posted a second DONE on a finished task. The
cause: `undeliveredLegacyUserCommentIds`
(explicit-native-continuation.ts) checks only wake input and accepted
first turns. It does not check the `task_history.read` records that
#21175 added.

**Expected behavior**
A follow-up that a succeeded run read at its current revision is
delivered. The remote-Stop resume path skips it. A read by a run that
did not succeed does not count, so that follow-up is still delivered
once.

**Steps to reproduce**
1. A native run reads the task history after a person posts a follow-up.
2. The run succeeds. The follow-up stays saved as a deferred wake.
3. The remote-Stop resume path runs for that wake.
4. Before this change, the follow-up is delivered again.

**Paperclip version or commit**
`master` at 549816e6a0.

## What Changed

- `server/src/services/queued-comment-delivery.ts`:
`historyReadDeliveredCommentIds` exposes the #21175 rule for a list of
comment IDs. `taskHistoryReadsForRun({ companyId, runId })` returns each
comment one run read, with the read time and the comment revision that
the read returned. It returns these whatever the run's outcome. It reads
only that run's events through `heartbeat_run_events_company_run_idx`.
No delivery decision uses it.
- `server/src/services/explicit-native-continuation.ts`:
`undeliveredLegacyUserCommentIds` also removes comments that a succeeded
run read at their current revision. Its two callers are the remote-Stop
resume path and the queued explicit-continuation check, and both now use
the #21175 rule.
- Deliberate asymmetry: a read by a run that did not succeed never
counts. A dropped person's message is worse than a duplicate answer.
- `doc/run-log-events.md`: documents both points.
- New test file
`server/src/services/queued-comment-history-read.test.ts`.

## Verification

- RED on `549816e6a0` (`npx vitest run
src/services/queued-comment-history-read.test.ts`): 2 failed, 1 passed.
  - The follow-up that a succeeded run read was still returned as owed.
  - `taskHistoryReadsForRun` did not exist.
- The case where a failed run read the follow-up (still owed) passed, as
intended.
- GREEN: 3/3 pass.
- Neighbouring suites (`first-turn-queued-input`,
`explicit-native-continuation`, `heartbeat-agent-directory-busy`,
`heartbeat-held-comment-wake`, `execution-control-work`): 517/517 pass.
Server `tsc` is clean.

## Risks

- Low risk. The function and its callers keep their contract. It returns
fewer comment IDs only when a succeeded run of the assignee read that
exact revision.
- Comments read through the REST API by non-native adapters are not
recorded, so their behavior does not change.
- No migration. The added lookup uses the time-bounded, index-backed
query from #21175.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:23:34 -06:00
DottaandClaude Opus 5.5 de675d924a fix(heartbeat): stop a comment the closing run already read from reopening its finished task (#21175)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can add a comment to a task while the agent's run on that
task is still working
> - The comment must reach the agent once: in the running turn, or in
one later turn
> - In a restart-chaos wave, a comment that the running turn already
read reopened the finished task, and the agent did the completion turn a
second time 15 minutes later
> - This pull request makes a read by the running turn count as
delivery, and holds the comment behind the running run so the run's own
release settles it
> - The benefit is no reopened done task, no repeated completion turn,
and an unread comment that is delivered when its run finishes, not after
a slot wait and a stale cancellation

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261112T181948-s7101`, task t005 (OpenCode,
native Runner), during a two-server overlap deploy (K15).
- 18:26:56 run A started on the old server. 18:26:55 traffic switched to
the new server.
- 18:27:03.953 a board user posted a follow-up comment. The new server
handled it. Run A was not in that process's `activeRunExecutions`, so
`filterZombieCoalesceTarget` treated it as a zombie. Admission returned
`proceed`, and the comment got its own queued run B (18:27:04.252). The
queued run is not part of the release drain.
- 18:33:32 run A called `get_task_history`, received the comment,
applied it, and finished the task (`done`, 18:34:20).
- Run B waited for an agent slot until 18:41:49 and was cancelled as
`continuation_task_ownership_changed`.
- `deferUndeliveredCommentsOfStaleRun` then saved the comment again. Its
"unread" test is only "a run started before the comment and finished
after it". `deliveredQueuedCommentIds` had no record of run A's read.
The release drain reopened the task (done to todo, 18:41:51), and run C
(18:49:22) posted a second "Task complete" summary. The harness grade
`done_comment_once` failed.

**Expected behavior**
- A comment that the running turn received is consumed. No queued work
outlives that run, and the task stays done.
- A comment that the running turn did not receive is delivered once,
through the normal comment-on-done-task path, when that run finishes.
- A stale continuation is never re-issued for a comment that a completed
run already read.

**Steps to reproduce**
1. Start a native run on a task.
2. Post a person's comment while another server instance executes the
run (or seed a queued comment run behind it).
3. Let the run read `get_task_history`, then finish the task.
4. Before this change, the queued comment run is cancelled as stale, the
task reopens, and the agent runs a second completion turn.

**Paperclip version or commit**
`master` at ef6d347f79.

## What Changed

- `server/src/services/queued-comment-delivery.ts`: new run-log event
`task_history.read` and `recordTaskHistoryRead`.
`deliveredQueuedCommentIds` also counts a comment as delivered when a
succeeded run of the assignee on that task read the comment's current
revision (`historyReadDeliveredSql`). All existing consumers use this
one check: the release drain liveness check, the stale-continuation save
(`saveUndeliveredUserComments`), the ownership reconciler, and
first-turn queued input.
- `paperclip-runner-tool-authority.ts` and
`runner-semantic-authority.ts`: `get_task_history` records the returned
comment IDs and `updatedAt` values. The tool output does not change. A
failed record is logged and leaves the comment owed, so it is delivered
again.
- `server/src/modules/wake-queue/application/use-cases.ts`: a comment
follow-up for the same agent's `running` run is held as a deferred wake
even when this process does not track that run. Before, a run executed
by another server instance was treated as a zombie, and the comment got
its own queued run that bypassed the release drain. Coalescing into an
untracked run is still refused.
- `doc/run-log-events.md`: documents `task_history.read`.
- Regression tests in
`server/src/__tests__/heartbeat-held-comment-wake.test.ts`.

Intended behaviour chosen: in t005 run A received the comment mid-turn
through its own `get_task_history` call (run events at 18:33:32 show the
comment ID in the tool result, and the result says it added the
follow-up line), so (a) applied. The fix makes that read the durable
proof of delivery. A comment that no run read still follows the existing
outbox: it is held behind the run and delivered once at the run's
release, reopening the done task as a person's comment does. Envelope
task history of another message's turn still does not count (unchanged
doctrine and test).

## Verification

- RED on master (`ef6d347f79`), `npx vitest run
src/__tests__/heartbeat-held-comment-wake.test.ts -t "mid-turn|another
server|held message"`: 3 failed, 1 passed.
- "consumes a message the closing run read mid-turn, so its stale queued
run reopens nothing": delivered 1, expected 0.
- "holds a message behind a run another server is executing instead of
queueing a run of its own": a queued run was created.
- "settles a held message as soon as its run closes the task (read
mid-turn: true)": delivered 1, expected 0.
- The unread variant (delivered once, no cancelled stale run) passes on
both, as intended.
- GREEN with this change: all 4 pass.
- Review follow-up (f43e79c3): a history read counts as delivery only
for a succeeded run on the read revision. Tests cover an edit after the
read (the stale queued run delivers the comment once) and reads by an
edited or failed run (the comment stays owed). Full file: 37/37 pass.
- Security-scan follow-up (239f0597): the delivery check probes each run
binding (`native_issue_id`, `context_snapshot->>'issueId'`) through its
own existing index with `offset 0`. It reads only the assignee's
succeeded runs of the task with `finished_at >= ` the comment's current
revision. It does not use `started_at`, because the t005 reader started
before the comment. It stops at the first matching `task_history.read`
event of each run through `heartbeat_run_events_company_run_idx`. No
migration is needed.
- Seeded plan (local PostgreSQL: 200 tasks, 4,000 runs, 1.2M run events,
one comment read by one run), `EXPLAIN (ANALYZE, BUFFERS)`:
- Before: 1.202 ms, 6,097 shared buffers. It scanned every event of all
21 runs of the task.
- After: 0.030 ms, 23 shared buffers. It reads one index-matched run and
one event. The full `deliveredQueuedCommentIds` call takes 1.55 ms.
- New case: a succeeded run's read is not delivery when the comment was
edited after that run finished. Full held-comment file: 38/38 pass.
- `muse-control.test.ts` compared the whole run row after a
`get_task_history` call. It now allows that read's own run-log append
(`nextEventSeq` +1, `updatedAt`), and every other column must be
unchanged. That CI failure (General tests server 2/12) came from this
PR. "e2e shard (5/8)" failed because the self-hosted runner lost
communication; that is not related to this PR.
- Runner and Muse suites (`dot-runner`, `paperclip-control-plane-port`,
`runner-prp-coordinator`, `historical-task-questions`,
`paperclip-runner-tool-authority`, `muse-native-follow-up`,
`muse-control`): 509/509 pass. Server `tsc` is clean.
- Neighbouring suites: `heartbeat-held-comment-wake`,
`src/modules/wake-queue/**`, `execution-continuation`,
`paperclip-runner-tool-authority`, `first-turn-queued-input`,
`heartbeat-comment-wake-batching`, `ownership-reconciliation-report`,
`heartbeat-zombie-guard`: 555/556 passed in one loaded run. The 2
timeouts (one batching test, one reconciler `afterAll` hook) pass when
run alone (1/1, 44/44).
- `server` `tsc --noEmit`: clean.
- The restart-chaos harness already grades this class
(`done_comment_once` in staging-mix). No new cell is necessary.

## Risks

- Product choice: a comment counts as delivered when a succeeded run
read it through `get_task_history`, even if the model ignored it. This
is the same trust that applies to a comment delivered as wake input. The
alternative fails worse: it reopens a done task and repeats a completed
turn. An edit after the read, or a reader run that fails, leaves it
owed.
- Legacy (non-native) adapters that read comments through the REST API
do not write the record. Their behavior does not change.
- A comment follow-up behind a truly dead running run now waits as a
deferred wake until recovery releases that run. Before, its queued run
also could not start until the lock was released.
- Each `get_task_history` call appends one small run-log row. This is
not in the path before `turn.start`.
- Upgrade-safe: continuations queued by older builds still go through
the dispatch stale check, and that path now reads the same delivery
record.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:31:11 -06:00
DottaandClaude Opus 5.5 f92f91f0fb fix(native): finish a parent claim when its child results arrived before the run started (#21189)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task gets a wake when one of its child tasks completes. The
wake carries the child results.
> - When the parent already has a run, the child result wake waits
(`deferred_issue_execution`) behind that run.
> - At finalization, a parent completion claim stays open
(`native_child_completion_pending`) while such a wake is still queued,
claimed or deferred. This guard protects child results that arrive while
the parent works.
> - The guard did not check when the child result arrived. A child
result saved before the parent run started was already visible to that
run, but it still blocked the claim.
> - So the task stayed `in_progress`, the deferred wake started a second
run, and the agent repeated its completion summary.
> - This pull request counts a child result as undelivered only when it
was saved after the finishing run started.
> - The benefit is that a parent that already read its child results
finishes in one turn, and child results that arrive during the run still
get a fresh turn.

## Linked Issues or Issue Description

Refs #15372 (added the pending child-completion guard). Refs #20935
(same family: a self-wake after a settled run).

**What happened?**

Restart-chaos wave `w-20261112T181948-s7101`, task t001 (OpenCode,
native runner, staging-mix parent with two subtasks). Both subtasks were
done before parent run `8006a205` started at 18:45:29. The run read
them, posted its final comment and claimed `done` at 18:46:15.860. A
child-completion wake for the parent had been deferred since about 18:42
behind that run. The finalizer found that wake and committed
`native_child_completion_pending`, so the task stayed `in_progress`. 63
ms later the release promoted the deferred wake into run `9282de34`
(`invocationSource: automation`, `wakeCommentId: null`). Its
`paperclip.native-continuation.v1` turn carried the subtask results that
the first run had already used, plus the agent's own two comments
(resume delta since the base run). That run posted a second "complete"
summary and only then moved the task to `done`.

Root cause: `hasPendingNativeChildCompletion()`
(`server/src/services/native-runtime/native-child-completion-delivery.ts`)
treated every queued, claimed or deferred child-completion wake for the
issue as unread, except a wake bound to the finishing run itself. It did
not compare the wake's save time with the run's start.

**Expected behavior**

A child result that was saved before the finishing run started is not
new input for that run. The completion claim completes the task. The
deferred wake is cancelled at release as a wake on a terminal task. A
child result saved during the run still keeps the task open and gets a
fresh parent turn.

**Steps to reproduce**

1. Create a parent task with a done child. Save a deferred
`issue_children_completed` wake for the parent.
2. Start a native parent run after that wake was saved. Let it claim
`done`.
3. Finalize the run. On `master` the decision is
`native_child_completion_pending` and the task stays `in_progress`; the
deferred wake then starts a second run.

## What Changed

- `native-child-completion-delivery.ts`:
`hasPendingNativeChildCompletion()` counts a child-completion wake only
when its `createdAt` is at or after the finishing run's `startedAt`. A
run without a recorded start keeps the old behavior: every such wake
stays pending. The finalizer, the status-commit recheck and status
reassessment all use this function.
- `heartbeat-process-recovery.test.ts`: a regression test finalizes a
real native done claim with a deferred child-completion wake saved
before (task becomes `done`) or after (task stays `in_progress` as
`native_child_completion_pending`) the run started. It then releases the
issue execution through the wake queue. In the `before` case the
deferred wake is cancelled and there is one run total. In the `after`
case the wake is promoted into a fresh run that carries the child
results.
- `doc/architecture/native-status-arbitration.md`: describes when a
child result counts as undelivered.

## Verification

- RED on `master` (`ef6d347f79`): `heartbeat-process-recovery.test.ts`
"treats a child result saved before the parent run started…" fails with
decision `reasonCode: "native_child_completion_pending"`, `toStatus:
"in_progress"`. This is the same decision as the wave. The "after" case
passes on `master`.
- GREEN: both cases pass with the fix.
- Neighbouring suites pass: `native-status-arbiter-corpus` (includes the
queued / claimed / deferred / consumed child-delivery cases from
#15372), `heartbeat-dependency-scheduling` (revised child completion
delivered after a running parent), `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, and `src/services/native-runtime`.
- `pnpm --filter @paperclipai/server typecheck` passes.
- The regression test runs finalization and release end to end. The
cancel-at-release step itself is existing behavior, also covered by
`modules/wake-queue/application/use-cases.test.ts` ("cancels stale
assignee continuations before claiming promotion on %s tasks", `done`
case).

## Risks

- A child result saved before the run started no longer blocks the
claim. The child status was already `done` when the run started, so the
run could read it. If the clocks of two server instances differ, a child
result saved during the run could look older than the run start; the
window is the clock skew.
- A re-dispatched native status wake gets a new row with a later
`createdAt`. Such a row stays pending, which is the old behavior.
- No migration. Saved wakes from older builds work unchanged, because
the check reads only existing columns.
- Scope: this change does not alter the comment-wake promotion path or
the resume-delta builder.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:19:32 -06:00
DottaandClaude Opus 5.5 f95bbe081e fix(auth): agent shells can only reach the local API with their own run credentials (#21181)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In `local_trusted` mode, the server gives board access to local
requests that carry no credentials, so the operator can use the app
without a login
> - Agent runs execute on the same machine, so their tools can also send
local requests
> - An agent run must act only with its own run credentials and
permissions
> - This pull request makes implicit board access belong only to the
operator's own local processes
> - The benefit is that agent shells and tools can only reach the API
with their own run credentials

## Linked Issues or Issue Description

**What happened?**
In `local_trusted` mode, a process inside an agent run could receive the
board identity for an API request that had no credentials. A permission
denial for the agent then did not stop the change.

**Expected behavior**
A request from inside an agent run uses the run's own credentials and
permissions. Implicit board access stays available to the operator's
browser and terminal tools.

**Deployment mode**
`local_trusted` (private, loopback). Authenticated deployments do not
use implicit board access and do not change.

**Additional context**
Found by the chat feature evals. The bundled board skill also described
implicit board access in words that an agent could follow.

## What Changed

- Add `server/src/middleware/local-agent-process-boundary.ts`. For a
request without credentials, it identifies the local process behind the
TCP connection (Linux: `/proc`; macOS: `netstat -anv` and `ps`). If that
process or one of its ancestors belongs to an agent run, the request
gets no implicit board access. An agent run is a descendant of
`paperclip-runnerd`, or a process that carries a run identity variable
(`PAPERCLIP_RUN_ID`, `PAPERCLIP_AGENT_ID`, `PAPERCLIP_AGENT_KEY_ID`,
`PAPERCLIP_RUNNER_INSTANCE_ID`) with a value that is different from the
server's own value.
- The walk stops at the server process and its ancestors. Requests from
the server process itself, and from an operator whose own development
server runs inside the same agent execution, keep board access.
- The result is cached for each connection. Only one process lookup runs
at a time. Connections that arrive during a lookup wait for the next
lookup, which handles all of them together (one `netstat` and two `ps`
calls on macOS, one `/proc` walk on Linux). All lookups are
asynchronous, so they do not block other work in the server.
- `actorMiddleware` and the live events WebSocket upgrade apply the
check in `local_trusted` mode. A refused request is unauthenticated
(`actor.type = "none"`), the same as an anonymous request in
authenticated mode.
- Platforms where the server cannot identify the process (for example
Windows) keep the previous behavior and log one warning.
- Rewrite the `paperclip-board` skill authentication section. It no
longer describes implicit board access to agents, and it says that a 401
or 403 inside an agent run is final.
- Document the behavior in `doc/DEPLOYMENT-MODES.md`.

## Verification

- `cd server && npx vitest run
src/__tests__/local-agent-process-boundary.test.ts`. The test starts a
real `local_trusted` server and real child processes:
- an operator process without credentials gets board access (green
before and after);
  - an in-process request gets board access;
- a process with a run identity gets no board access (red before this
change, green after);
- an agent process that starts a tool with an empty environment gets no
board access (red before, green after);
- an operator inside the same execution as its own server keeps board
access.
- A burst test with a slow inspector: 12 parallel connections and 2
connections that close early. It checks that only one lookup is active
at a time, that the burst needs at most two lookups, that the event loop
stays responsive, and that later requests still work.
- Parser tests for the Linux TCP table and the macOS `netstat` format,
and a test that the runner daemon itself is not treated as an agent
process.
- Ran `agent-auth-middleware`, `agent-identity`, `live-events-ws`,
`live-events-toast`, `live-events-run-read` and
`issue-create-deduplication-routes` tests locally. `npx tsc --noEmit -p
server/tsconfig.json` passes.
- Paid runner E2E evidence for the affected chat cases is linked in a
comment.

## Risks

- Medium. The check runs once for each new local connection without
credentials. On macOS it starts `netstat` and `ps` (about 30 to 80 ms).
On Linux it reads `/proc`. Bearer requests do not run it.
- If a process can not be identified on a supported platform, the
request gets no implicit board access. On Linux, when a same-user
process hides its sockets, the check keeps the previous behavior so that
the operator's browser continues to work.
- This is a boundary for the documented path, not a sandbox. Agent runs
use the operator's OS account. A process that intentionally leaves its
run's process tree, or reads the instance files directly, is outside
this check. The broader option is a product decision: bind implicit
board access to a capability that only the operator's browser and CLI
hold, or run agents under an OS sandbox that blocks the server port.
This pull request does not change `local_trusted` semantics for the
operator.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:05:15 -06:00
Nicky LeachandPaperclip 2168f50741 Forward provider credentials from verified image catalogs (#21186)
## Thinking Path

> - Paperclip manages AI agents for work.
> - Application images can ship plugins that provide task environments.
> - Paperclip starts plugin workers with a restricted process
environment.
> - An image author can know which provider credentials an image-bundled
worker needs.
> - The image catalog already identifies selected bundles and verifies
their content.
> - This change adds a way for the image catalog to declare the provider
keys that a verified worker receives.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Image-bundled environment drivers can use the worker credential delivery
already available to built-in providers.

**Current behavior**

The host provides fixed credential forwarding for certain built-in
providers.

**Proposed behavior**

A verified image bundle can receive provider API keys declared on its
own catalog entry.

**Reason and benefit**

Image authors can scope deployment credentials to the verified worker
that needs them.

## What Changed

- Add an optional provider credential declaration to the image-owned
distribution catalog.
- Scope forwarding to the verified worker for the selected bundle.
- Publish distribution catalog support for image builders.

## Verification

- The new loader regression failed before the change because the old
catalog rejected the credential declaration.
- Ran six focused Vitest suites for bundled activation, package
identity, distribution catalog validation, worker environment, and
managed configuration. 166 tests passed and one existing test was
skipped.
- Ran `pnpm --filter @paperclipai/server typecheck`. It passed.
- Reviewed the diff for secret values and personal information. Test
keys are synthetic.
- A downstream image build fixture exercised the catalog reader, bundle
verification, and worker environment together. The supported build
passed only the declared provider key to the verified worker. The build
for an older target omitted the declaration.
- Local verification used embedded PostgreSQL and prebuilt fixtures.

## Risks

- Deploy images with catalog credential declarations only to Core
versions that support the field. Older hosts reject unknown catalog
fields.
- The image author controls which verified bundle receives each provider
API key.

## Model Used

- Implementation: GPT-6.1 Sol (`gpt-6.1-sol`), delegated through Codex.
Independent Codex review and verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs
- [x] I have described the issue in the PR
- [x] I have used public references
- [x] My branch name describes the change
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address reviewer comments before requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 20:53:08 +00:00
DottaandClaude Opus 5.5 267983356c feat(mcp): speak stateless MCP 2026-07-28 to provider servers, keep older servers working (#21140)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through Paperclip's MCP client. The client calls
each provider's remote (Streamable HTTP) or local (stdio) MCP server.
> - The client spoke only protocol 2025-06-18. It always did the
`initialize` handshake and it sent client capabilities only in that
handshake.
> - MCP 2026-07-28 removes the handshake and sessions. Every request
carries its version, capabilities and client identity in `_meta`.
Servers that only speak 2026-07-28 reject the old requests, and they
never see that Paperclip renders MCP Apps views.
> - Public servers already speak it: DeepWiki and Context7 answer
`server/discover` with 2026-07-28 today.
> - This pull request makes the client negotiate the version per server.
It speaks stateless MCP to 2026-07-28 servers and keeps the handshake
for 2025-11-25 and older servers.
> - The benefit is that new and old providers both work, and a tool call
to a stateless server is one HTTP request with no handshake.

## Linked Issues or Issue Description

Refs #21119 (MCP Apps host). This PR sends its MCP Apps capability to
stateless servers too.

**Subsystem affected**
Server: the MCP client in `server/src/services/mcp-http.ts`, catalog
discovery, the tool gateway (`tools/call`, context reads, stdio), and
the remote MCP OAuth flow.

**Problem or motivation**
The client used protocol 2025-06-18 only. MCP 2026-07-28 makes MCP
stateless: no `initialize`, no `Mcp-Session-Id`, per-request `_meta`,
`server/discover`, `resultType` with multi round-trip requests (MRTR),
cacheable lists, required `Mcp-Method`/`Mcp-Name` headers, and RFC 9207
`iss` validation. Servers that speak only 2026-07-28 cannot be used.
Paperclip advertised MCP Apps support only in `initialize`, so stateless
servers never saw it.

**Proposed solution**
Probe each server once with `server/discover`, pick the newest mutually
supported version, and save it on the connection. Send stateless
requests to modern servers and keep the handshake for legacy servers.
Tool calls use the saved version and never probe. The saved version is
checked again only when a request built from it is refused.

**Alternatives considered**
Inline negotiation (send each request in modern shape first and fall
back on `400`) adds a failed request to every call to a legacy session
server. Probing on every tool call adds a round trip before each tool
runs. Saving the negotiated version keeps tool calls at one request.

**Roadmap alignment**
MCP protocol currency for the Apps and connections substrate. It follows
the MCP Apps host work in #21119.

**Additional context**
Spec:
https://modelcontextprotocol.io/specification/2026-07-28/changelog. Out
of scope (separate work): the Tasks extension, Skills over MCP,
`subscriptions/listen`, progress and cancellation, icons,
enterprise-managed authorization.

## What Changed

- `mcp-http.ts`: `negotiateMcpProtocolVersion` probes with
`server/discover`. A `DiscoverResult` or a modern error
(`-32020`/`-32021`/`-32022`) means a modern server. Anything else means
a legacy server, which gets `initialize`. A `401`/`403` keeps the
challenge for OAuth discovery. The era is remembered per connection and
endpoint.
- `mcpHttpRequestInit` builds every request. It always sends
`Mcp-Method` and `Mcp-Name` (Base64 sentinel when unsafe). For modern
versions it adds `_meta` (`protocolVersion`, `clientCapabilities` with
`extensions["io.modelcontextprotocol/ui"]` and URL elicitation,
`clientInfo`) and `Mcp-Param-*` headers from `x-mcp-header`.
- Legacy `initialize` now requests 2025-11-25 and advertises the MCP
Apps extension.
- Catalog refresh saves `config.mcpProtocolVersion`. It drops tools with
invalid `x-mcp-header` annotations. It reuses a fresh `tools/list`
result (`ttlMs`, `cacheScope`). It probes again when the saved version
is refused, and it negotiates again after a credential refresh.
- `tools/call`: a saved modern version is called directly (no probe, no
handshake). A modern rejection of a legacy-shaped call, or `-32022`,
gets one retry with a negotiated version. The server rejected the first
request before it ran, so the retry cannot repeat a side effect. A
remembered legacy server that refuses `initialize` is probed again.
- MRTR: `input_required` with URL requests becomes the existing provider
link flow (`provider_interaction_required`). The next identical call
sends `requestState` and `inputResponses`. Form, sampling and roots
requests stop with `provider_input_required` and a plain message. A
missing `resultType` means `complete`.
- Context reads (`resources/*`, `prompts/*`) use the saved version,
cache fresh results, and stop on `input_required`. The error code is
kept, so resource not found (`-32602` or `-32002`) is visible.
- stdio: probe with `server/discover` (2 s limit). Restart a server that
exits on the probe. Send `_meta` to modern servers.
- OAuth: refuse a callback without `iss` when the server advertises
`authorization_response_iss_parameter_supported`. Check `iss` before
acting on a provider error. Send `application_type: "native"` for
localhost callbacks and `web` otherwise. Issuer-bound clients and the
CIMD-first order already existed.
- Docs: `doc/connections/MCP-PROTOCOL-VERSIONS.md`, linked from
`doc/connections/README.md`.

## Verification

- New suite `server/src/__tests__/mcp-protocol-versions.test.ts` (19
tests) uses a fake 2026-07-28 server and a fake legacy session server.
It covers discovery, `_meta`, headers, version fallback, `-32022`,
`resultType`, MRTR link and retry, form stop, cache TTL, legacy flow,
and recovery when a saved legacy server moves to 2026-07-28. Embedded
Postgres runs the end-to-end cases.
- New OAuth tests in `generic-mcp-connection.test.ts`: `iss` required
when advertised, `iss` checked on provider errors, `native` for
localhost, and a stateless server listed again after a revoked token.
- Red/green: with the master service code, all 21 new protocol and OAuth
tests fail. With this change they pass. The token-refresh regression
test fails without its fix and passes with it.
- Existing suites: 12 touched files, 1088 tests pass. `tsc --noEmit`
passes for `server`. Updated expectations count the one-time
`server/discover` probe, and the Calendly DCR test now expects `native`
for its localhost callback.
- `pnpm verify` exits 2 (a doc is a global CI input), so CI supplies the
broad coverage.
- Live check with the changed client, without credentials:
- DeepWiki (`mcp.deepwiki.com/mcp`): discover 200 → 2026-07-28, no
session, `tools/list` 200 (3 tools).
- Context7 (`mcp.context7.com/mcp`): discover 200 → 2026-07-28, no
session, `tools/list` 200 (2 tools).
- GitMCP (`gitmcp.io/docs`): discover 400 → `initialize` (2025-03-26) →
session → `tools/list` 200 (5 tools).
- Circleback (`circleback.ai/api/mcp`): discover 401 with its
`WWW-Authenticate` challenge kept for sign-in.

## Risks

- A legacy server that is not saved as session-required gets one extra
request (`server/discover`) the first time it is refreshed. Then the
version is saved. Saved session-required connections are not probed.
- DCR from a localhost callback now registers a `native` client. This is
what the spec asks for. Existing registrations are reused.
- An authorization server that advertises `iss` support but does not
send `iss` now fails sign-in with a clear message. The spec requires
this.
- MRTR answers are kept in memory for 30 minutes. After a restart the
server asks again, and the user sees the link again.
- No schema change. `config.mcpProtocolVersion` is a new key in the
existing JSON config.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits, web fetch of the MCP spec), extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 14:34:26 -06:00
DottaandClaude Fable 5.2 31c3412bbb perf(db): build tx.query on first read instead of for every transaction (#21148)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Every server module reaches PostgreSQL through one client,
`createDb` in `packages/db`. The server has about 1,260 `transaction(`
call sites. Run events, runner checkpoints and the run queue use a
transaction for each write.
> - Drizzle builds one relational query builder for every schema table
when it creates a database object. It creates a database object for each
transaction and for each savepoint.
> - The Paperclip schema has 321 tables. One `BEGIN` therefore allocates
about 100 KB and uses about 80 microseconds of CPU before the first
statement. One `SAVEPOINT` costs the same again. Five places read the
relational API inside a transaction.
> - On a hosted instance the server main thread was near one full core,
and garbage collection used a quarter of it. A fixed 100 KB for each
transaction is garbage that no caller asked for, and it grows with every
new table.
> - This pull request makes `createDb` build `tx.query` the first time a
transaction reads it.
> - The benefit is 82% less allocation and 7 times fewer garbage
collections for an empty transaction, and 45% less allocation for one
native run event. Transaction behavior does not change.

## Linked Issues or Issue Description

Related: #21131 (one transaction of two statements for a native run
event). That pull request reported this cost as a follow-up. No issue
exists for it.

**What existing behavior does this improve?**
The cost of `db.transaction()` and of a nested `tx.transaction()`
(savepoint) on the client that `createDb` returns.

**Subsystem affected**
Database client (`packages/db`).

**Current behavior**
`createDb` calls `drizzle(sql, { schema })`. Drizzle's `PgDatabase`
constructor then runs this loop for the root client, for every
transaction and for every savepoint:

```js
for (const [tableName, columns] of Object.entries(this._.schema)) {
  this.query[tableName] = new RelationalQueryBuilder(...);
}
```

A transaction is a `PgTransaction`, which extends `PgDatabase`
(`drizzle-orm/pg-core/session.js`). `PostgresJsSession.transaction` and
`PostgresJsTransaction.transaction` construct one for each `BEGIN` and
each `SAVEPOINT` (`drizzle-orm/postgres-js/session.js`).

Measured on `master` at 4e932fd178, Node 26, local PostgreSQL, 3,000
iterations, median of 5 passes:

| Scenario | Allocated | CPU | Collections per 1,000 |
| --- | --- | --- | --- |
| Statement outside a transaction | 11.5 KB | 27 µs | 0.3 |
| Empty transaction | 125.8 KB | 131 µs | 5.0 |
| Transaction with one statement | 136.6 KB | 153 µs | 5.3 |
| Transaction with an empty savepoint | 231.6 KB | 235 µs | 9.7 |
| Empty transaction, Drizzle without a schema | 18.5 KB | 53 µs | 0.7 |

The allocation profile of an empty transaction attributes 96.8 KB of
115.6 KB sampled to the `PgDatabase` constructor (`pg-core/db.js:17`),
reached from `PgTransaction` and `postgres-js/session.js:109`. About 50
KB is the builders and the `query` object. About 46 KB is
`Object.entries` of the schema.

The relational API is in use, always as a single-table `findFirst({
where[, columns] })`. It has 22 call sites. Five of them run on a
transaction: `services/announcements.ts:19` and `:25`,
`services/decision-training.ts:331`, and
`services/resource-memberships.ts:361` and `:369` (the service is built
on a transaction in `updateAgent` and in
`services/primary-agent.ts:60`). The other 17 run on the root client.

**Proposed behavior**
A transaction or savepoint that does not read `tx.query` builds no
relational query builder. A transaction that reads `tx.query` gets the
same builders as before, bound to that transaction.

**Reason and benefit**
Less garbage and less CPU on the server main thread for every write
path. Hosted statement statistics showed 13.4 native run events per
second and 5.3 checkpoint inserts per second. Each is one transaction,
so those two paths alone discarded about 2 MB per second. The total
transaction rate of the hosted instance was not measured.

**Breaking changes**
None. No schema change, no API change, and the `Db` type is identical.

## What Changed

- `packages/db/src/lazy-transaction-queries.ts` (new):
`drizzleWithLazyTransactionQueries(client, schema)`.
- It creates the root client with the schema, as before. `db.query` and
`db._` do not change.
- It creates a second Drizzle client on the same postgres.js client,
without the schema, and `db.transaction` opens transactions through it.
`BEGIN`, `SAVEPOINT`, commit, rollback and connection reservation are
still Drizzle's and the driver's own code. No Drizzle internals are
patched.
- Each transaction and savepoint gets a `query` accessor. The first read
builds one `RelationalQueryBuilder` for each table from the root
client's schema description and the transaction's session, then stores
the result as a plain property. `tx._.schema`, `tx._.fullSchema` and
`tx._.tableNamesMap` keep their values.
- `packages/db/src/client.ts`: `createDbClient` calls the new function.
The work-signal and response-capture wrappers install on top of it in
the same order as before.
- `packages/db/scripts/benchmark-transaction-allocation.ts` (new): the
benchmark in this description.
- `doc/DATABASE.md`: new section "Relational queries inside
transactions".

## Verification

Benchmark, same script before and after (`createDb` client):

```sh
cd packages/db
TSX_DISABLE_CACHE=1 node --expose-gc --import tsx scripts/benchmark-transaction-allocation.ts 3000 5
```

| Scenario | Allocated before | Allocated after | CPU before | CPU after
| Collections per 1,000 before | after |
| --- | --- | --- | --- | --- | --- | --- |
| Statement outside a transaction | 11.5 KB | 11.3 KB | 27 µs | 31 µs |
0.3 | 0.3 |
| Empty transaction | 125.8 KB | 22.4 KB | 131 µs | 54 µs | 5.0 | 0.7 |
| Transaction with one statement | 136.6 KB | 35.6 KB | 153 µs | 82 µs |
5.3 | 1.0 |
| Transaction with an empty savepoint | 231.6 KB | 35.6 KB | 235 µs |
111 µs | 9.7 | 1.3 |
| Transaction with a savepoint of one statement | 242.7 KB | 44.8 KB |
331 µs | 217 µs | 10.7 | 2.0 |

After the change, no Drizzle frame is in the allocation profile of an
empty transaction (19.8 KB sampled). The script also runs Drizzle with
the schema and Drizzle without a schema in the same process, so the
comparison can be repeated on any later revision.

The machine was shared and under load (load average 6 to 35 on 18 CPUs).
Allocation and collection counts were stable across four runs. CPU
values moved by a factor of two between runs. The CPU columns come from
the two runs with the lowest load.

One native run event, through the real append path:

```sh
TSX_DISABLE_CACHE=1 EVENTS=1500 PASSES=3 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-run-event-append.ts
```

`native-stream`: 220.4 -> 120.9 KB allocated for each event, 12.0 -> 5.3
collection pauses per 1,000 events. `native-burst`: 221.1 -> 121.0 KB,
10.6 -> 5.3 pauses. Statements for each event do not change (4.6).

Tests, all on real PostgreSQL
(`packages/db/src/lazy-transaction-queries.test.ts`, 15 tests):

- Red and green for the mechanism. On `master` these fail, and with the
change they pass:
- `builds none for a transaction or savepoint that never reads
tx.query`: 321 builders for the transaction and 321 for the savepoint on
`master`, 0 with the change.
- `allocates the same for an empty transaction as a client without the
schema`: 101 KB more than a schema-less client on `master`, below the 16
KB limit with the change. It reads V8's allocation counter, not a timer,
and it skips on a Node.js without that counter.
- Semantics. These pass on `master` and with the change, which shows the
behavior is the same:
  - `commits and returns the callback's value`
- `rolls back and rejects with the thrown error` (thrown error,
`tx.rollback()`, SQL error)
- `keeps the outer transaction usable after a savepoint rolls back`
(three levels)
- `discards a released savepoint when the outer transaction rolls back`
  - `applies the isolation level and access mode`
- `runs concurrent transactions on separate reserved connections and
releases them` (12 transactions on a pool of 3)
- `returns the only connection to the pool after a commit, a rollback
and a failed savepoint` (pool of 1)
  - `behaves the same with prepared statements enabled`
- Instrumentation. These pass on `master` and with the change:
- `registers each transaction and savepoint with the client's work
signals`: the transaction and the savepoint have the client's
work-signal scope, the registration statement runs on the transaction's
own connection, listeners get `intent` at once and `settled` only after
the outer commit, and a rollback reconciles. When transactions bypass
the work-signal wrapper, this test and five tests in
`work-signals.test.ts` fail.
- `response capture counts every statement of a transaction, a savepoint
and tx.query`: six statements, counted for each operation and table,
including two relational reads.
- Every `tx.query` pattern in the codebase:
- `reads tx.query through the transaction, as the announcement and
decision-training services do`
  - `reads tx.query inside a savepoint and follows its rollback`
- `gives tx.query one builder per schema table, the same object on every
read`

Commands run locally on the final revision:

- `pnpm exec vitest run src/lazy-transaction-queries.test.ts
src/work-signals.test.ts src/response-capture-postgres.test.ts` in
`packages/db` on the final revision: 43 passed.
`prepared-queries-postgres.test.ts` passed on the first commit.
- Before the rebase onto current `master`, also
`response-capture.test.ts`, `response-capture-bundle.test.ts`,
`client-options.test.ts`, `client-teardown-registry.test.ts`,
`checkpoint-pool.test.ts`, `client-disconnect-replay.test.ts`: 111
passed in 10 files.
- On the first commit, server callers of `tx.query` and `db.query`, and
hot transaction paths: `announcements-routes`, `decision-training`,
`heartbeat-run-event-atomic-append`, `heartbeat-run-event-sequencing`,
`primary-agent`, `resource-memberships-routes`, `workspace-preferences`,
`sidebar-preferences-routes`, `paperclip-control-plane-port`,
`native-runner-checkpoint-authority.integration`, `dispatch-work`,
`decision-target-work`, `issue-thread-interactions-service`: all passed.
`workspace-preferences` timed out once in its database start hook while
eight suites ran in parallel, and passed alone.
- `pnpm exec tsc --noEmit` in `packages/db`: passed. A type probe showed
that `Db` is the same type as before.
- `pnpm check:module-boundaries`: passed.
- `pnpm verify` exits 2: `packages/db` is a broad input. CI supplies the
full suites. I did not run the full local suite, repo-wide typecheck or
build.

Not verified: the hosted transaction rate, and the effect on a hosted
instance. CI uses Node 24, which has no V8 allocation counter. On CI the
allocation test skips itself, and the builder-count test covers the
mechanism. The benchmark needs Node.js 25 or later and stops with that
message on Node 24. I did not run that error path, because this machine
has Node 26 only.

## Risks

- Every module uses this client. The mitigation is that the transaction
path is still Drizzle's own `transaction()` on the same postgres.js
client. Only the schema argument is absent. The semantics tests pass
unchanged on `master` and on this branch.
- The change relies on Drizzle surface that is not documented API. The
lazy `tx.query` calls the `RelationalQueryBuilder` constructor from
`drizzle-orm/pg-core/query-builders/query`. That subpath is in Drizzle's
export map and the constructor is typed, but `drizzle-orm/pg-core` does
not re-export it. The change also writes the three schema fields of
`tx._`, which the types mark `readonly`. In `drizzle-orm@1.0.0-beta.22`
the same path exports a different class. An upgrade that changes the
constructor fails the typecheck and the `tx.query` tests. An upgrade
that renames `db._.schema` makes the function return the plain eager
client: behavior stays correct, the saving is lost, and the
builder-count test fails. The repository owner decides whether this
dependency on Drizzle is acceptable. A version that uses only documented
API removes the 22 relational reads from the server and creates the
client without the schema.
- `tx.query` is now an accessor until its first read. Code that reads
the property descriptor of `query` sees the difference. No code in the
repository does that.
- The second client and the builders use Drizzle's default dialect
options, as the root client does. If `createDb` ever passes `casing`,
`logger` or `cache` to Drizzle, the same option must go to all three.
The source comment says so.
- A transaction that reads `tx.query` pays the old cost once, on first
read.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code, with tool use
and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 14:31:21 -06:00
Nicky LeachandPaperclip 7f69ef8b44 Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path

> - Paperclip lets plugins own managed Git workspaces.
> - A portable Runner task can move to a different physical allocation.
> - The checkout must be present on the allocation that launches the
Runner.
> - Workspace preparation and finalization need exact placement when
they retry.
> - This change checks the placement at Runner admission and records it
before finalization.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Workspace preparation and finalization for portable Runner tasks.

**Current behavior**
A task replacement can change the allocation after Git preparation. A
lost finalization reply can leave retry without the original placement.

**Proposed behavior**
Prepare the checkout on the actual allocation before admission. Record
finalization placement so an exact retry can use it.

Portable ownership and controller recovery are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066).

## What Changed

- Reserve the managed workspace before acquiring the portable task
lease.
- Pass the task placement to the workspace provider. Reuse the described
task while placement stays the same and prepare again before admitting a
replacement.
- Record the initial preparation's run, agent, generation, and optional
task placement before dispatch. Inspection and retries on the same
Launcher replay the same request; VM and sandbox preparation use the
same recovery path.
- Keep provider request identity separate from Git credential authority:
reservation reconciliation uses the existing project-consent path in a
host-only scope bound to the plugin and workspace. Foreground
preparation still requires an active run; consent, repository, and
revocation checks are unchanged.
- Acquire the current Launcher before retrying preparation. If placement
changes, use ordinary preparation on the replacement so the provider can
settle the prior writer and recover the retained checkout. No additional
provider discovery call is added.
- Select finalization placement before publishing the operation record,
then dispatch and recover using that saved choice. Retain the Launcher
during bounded finalization retries.
- Preserve interrupted pre-upgrade finalization requests without
placement, including their original operation IDs and provider routing.
No migration or new durable mode flag is required.

## Verification

- Final revision `74ea3e6712`: server typecheck and exact SDK source-pin
verification (538 inputs) passed.
- All 185 tests passed across the full plugin worker manager, project
Git authorization, and plugin workspace integration suites (including 56
workspace tests). Independent verification reran the three lost-reply
cases: inspection, same-Launcher retry, and replacement after deleting
the original Launcher. The tests assert the correct request identity,
durable intent, and clearing that intent after success.
- Before the final retry-accounting fix, all 323 tests passed across
workspace admission, runtime execution, native workspace finalization,
plugin workspace integration, and terminal cleanup retry. Regressions
exercise crashes after operation insertion but before dispatch in both
finalization paths, exact replay after lease release, and retry after
ambiguous placement clears without publishing an incomplete operation.
Pre-upgrade interrupted requests retain their operation IDs despite an
available portable lease.
- Final retry-accounting fix: all 84 native workspace finalizer,
failure, and supersession tests passed. A real missing-plugin regression
first failed on the prior commit, then verified healthy-run progress,
three-attempt exhaustion, terminal cleanup, and absence of a misleading
workspace operation. Independent verification also passed the legacy and
crash-before-dispatch cases.
- Worker-process tests cover both preparation RPC methods: the provider
still receives the original run and agent, maintenance receives
project-scoped credential authority, and foreground or mismatched-scope
requests remain denied. Database tests confirm a stopped run cannot
authorize foreground Git access while valid project consent can
authorize maintenance.
- Existing metadata protection coverage proves callers cannot overwrite
the private preparation intent or read it through workspace responses.
- Two existing Launcher Git tests passed: retry preserves dirty files
and the original base commit without fetching, and retry resumes the
saved checkout intent while rejecting a changed pending branch.
- Implementation was delegated to GPT-6.1 Sol and independently
reviewed. The GitHub stack's automatic rebase preserved the original
workspace-placement patch.

- [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706)
passed on `74ea3e6712`, including server and workspace tests, Runner,
end-to-end tests, typecheck, build, and canary dry run. Ordinary
Greptile and
[Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388)
both scored `74ea3e6712` 5/5, with no unresolved findings.

## Risks

- Live Daytona end-to-end qualification is pending. The checkout record
persists on the mounted volume; replacement preparation relies on the
provider confirming the previous writer is settled before using that
retained checkout. Local tests cover Core request routing and Launcher's
durable checkout recovery separately.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 12:12:41 -08:00
DottaandClaude Fable 5.2 eaef46b60b feat(ui): browser page-load tracing to Honeycomb for hosted instances (#21156)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI is the main surface. Its load speed is a product
requirement.
> - The team wants to track page-load times over time and give each
metric an owner and an SLO in Honeycomb.
> - Server traces show only the server part of a page load. They do not
show JavaScript start-up, lazy route code, rendering, or client-side
navigation.
> - Paperclip is also self-hosted by others. Browser tracing must stay
off there, and it must not slow any page.
> - This pull request adds real-user browser tracing for hosted
Paperclip Cloud instances only. It loads lazily and sends a closed set
of attributes.
> - The benefit is one `page.load` measurement per page (named by route
pattern), plus Web Vitals and API spans that join the server trace,
ready for Honeycomb queries and SLOs.

## Linked Issues or Issue Description

No public issue exists. Feature description:

**Subsystem affected**

Board UI (`ui/`), the authenticated session route
(`server/src/routes/auth.ts`), and the shared session schema.

**Problem or motivation**

We cannot measure how long the all-tasks page or a task page takes to
load for real users. Server traces stop at the API response. Most
navigation is client side, and the standard OpenTelemetry document-load
instrumentation does not cover it. Without a per-page browser
measurement, no agent can own a page-load SLO.

**Proposed solution**

The server sends a `browserTracing` config in the authenticated session
only when the instance is cloud managed and
`PAPERCLIP_BROWSER_TRACES_ENDPOINT` is set. The browser then measures
every hard load and client navigation in one shared recorder tied to the
router's own route tree. After the page settles, it lazily loads an
OpenTelemetry chunk that exports `page.load`, `web_vital.*`, and
same-origin `/api/` fetch spans to Honeycomb. An allowlist removes every
other attribute before export.

**Alternatives considered**

`@honeycombio/opentelemetry-web` was the first choice. Its defaults add
the full page URL, query string, hash, referrer, and user agent to every
span and to the resource, and it bundles log and metric pipelines. We
would have to replace its exporter and remove most of its defaults.
Plain OpenTelemetry web packages plus `web-vitals` are smaller and send
only what we choose. Honeycomb needs only OTLP/HTTP and its ingest-key
header.

**Roadmap alignment**

This is internal hosted-instance observability. It changes no product
behavior and no self-hosted behavior.

## What Changed

- `server/src/browser-tracing-config.ts`: resolves the browser tracing
config. It returns `null` unless `isCloudManagedInstance()` is true and
`PAPERCLIP_BROWSER_TRACES_ENDPOINT` is set. It accepts only a Honeycomb
ingest key (`hcaik_` prefix), requires `https` (except loopback), and
fails closed with one value-free warning on any invalid setting.
Optional: `PAPERCLIP_BROWSER_TRACES_INGEST_KEY`, `_SAMPLE_RATE` (default
1), `_SERVICE_NAME` (default `paperclip-web`), `_ENVIRONMENT` (default
`SENTRY_ENVIRONMENT`).
- `GET /api/auth/get-session` adds `browserTracing` (config or `null`),
the same delivery path as the front-end Sentry DSN. The shared
`authSessionSchema` has the new optional field
`browserTracingConfigSchema`.
- `ui/src/lib/browser-tracing/page-load-recorder.ts` and `controller.ts`
(main bundle, no OpenTelemetry imports): one `page.load` measurement per
hard load (from navigation start) and per client navigation (from the
history change). A measurement ends at the first painted frame where no
full-page loading state is mounted and either every page-declared
content signal is ready or no query without data is fetching. Redirects
continue the measurement. A new navigation interrupts it. 30 s times
out. The sample decision is made once per page session before any
download.
- `ui/src/components/PageLoadTracker.tsx`: `App` passes the same
`<Route>` tree it renders, so every page is reported by route pattern
(for example `/:companyPrefix/issues/:issueId`) with no per-page code.
- `usePageContentReady` hook and blocking gates: `PaperclipLoading`
(also the route `Suspense` fallback) blocks readiness. The task page
uses the existing `issue-detail:navigate→content-paint` point. The plain
all-tasks list signals when its first rows render.
- `ui/src/lib/browser-tracing/sdk.ts` (lazy chunk, loaded after the page
settles): OpenTelemetry web tracer, OTLP/HTTP exporter, fetch
instrumentation for this origin's `/api/` only with `traceparent` sent
only to this origin, `web-vitals` LCP/INP/CLS, and a context manager
that makes the open `page.load` span the parent of its API calls.
`span-sanitizer.ts` applies the closed attribute allowlist (no URLs,
ids, query strings, status messages, or event attributes). Uncaught
errors are not sent because Sentry already reports them.
- New `ui` dev dependencies, pinned exactly. Each is needed, and the
OpenTelemetry ones are the same versions the server already resolves:
- `@opentelemetry/api@1.9.1` (already in the workspace): tracer and
context API.
- `@opentelemetry/sdk-trace-web@2.10.0`: the browser tracer provider,
batch processor, and stack context manager.
- `@opentelemetry/instrumentation-fetch@0.221.0`: fetch spans and
`traceparent` injection.
- `@opentelemetry/resources@2.10.0` (already a server dependency): the
resource.
- `@opentelemetry/otlp-transformer@0.221.0` (already a server
dependency): OTLP JSON serialization for a 60-line `OtlpFetchExporter`.
This replaces `@opentelemetry/exporter-trace-otlp-http`, which is a
server optional peer: installing it in the workspace made the server's
exact-version gate test see it as present.
  - `web-vitals@6.2.3`: LCP, INP, and CLS.
All of them are in the lazy chunk only. The lockfile is not committed:
`pr-trusted` rejects lockfile edits in pull requests, CI installs with
`--no-frozen-lockfile` after the frozen attempt reports
`ERR_PNPM_OUTDATED_LOCKFILE`, and the lockfile bot refreshes it after
merge.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the three shared files (the session schema is part of the pinned public
SDK surface; the contract must live in `packages/shared`).
- Docs: new `doc/browser-performance-tracing.md` (enablement, config,
key handling, span and attribute list, page-load definitions, example
Honeycomb queries and an SLO SLI). Pointers in `doc/observability.md`
and the Honeycomb tracing skill.
- Two App routing tests mock `PageLoadTracker` because they render `App`
without a `QueryClientProvider`.

## Verification

- `pnpm verify` returns exit 2 (broader verification required) because
docs are global CI inputs. Required CI supplies the broad coverage.
- Focused local tests, all pass:
- `ui`: `vitest run src/lib/browser-tracing
src/components/PageLoadTracker.test.tsx
src/components/SentryGate.test.tsx src/api/auth.test.ts
src/lib/issue-detail-performance src/pages/IssueDetail src/pages/Issues
src/App src/components/CloudAccessGate.test.tsx
src/components/IssuesList`: 17 files, 484 tests.
- `server`: `browser-tracing-config.test.ts`, `auth-routes.test.ts`,
`sentry.test.ts`: 63 passed, 5 skipped.
  - `packages/shared`: `validators/access.test.ts`: 19 passed.
- New tests prove: the SDK module is never imported when the session
config is `null`, absent, or signed out, or when the session is not
sampled; route templates contain no ids, slugs, or tokens; a client
navigation produces a `client` measurement under
`/:companyPrefix/issues/:issueId`; `traceparent` goes only to this
origin's `/api/` and not to a third-party or non-API URL; exported spans
carry only allowlisted attributes. A mutation check (tracker forced on)
turned the "never imports" tests red.
- Typecheck: `ui` (`pnpm typecheck`), `server` (`pnpm --filter
@paperclipai/server typecheck`), and `packages/shared` pass.
- Real-browser run: a local dev server with a local OTLP collector and
headless Chromium. The self-hosted session returned `browserTracing:
null` with the endpoint set, and the page made 0 SDK and 0 collector
requests. With a cloud session config injected into the session
response, the collector received a hard `page.load` (route
`/:companyPrefix/issues`, `ready_signal=idle`), a client `page.load` to
`/:companyPrefix/issues/:issueId` (`ready_signal=content`), a back
navigation, LCP/INP/CLS spans, and `/api/` fetch spans parented to the
open `page.load` span. No task identifier, company id, task title, host,
or URL appeared in the exported payload. No third-party request carried
`traceparent`.
- Bundle size (`ui` production build, initial JavaScript closure of
`index.html`): before 3,813,986 B raw / 1,153,991 B gzip; after
3,820,743 B / 1,156,590 B (+6.8 KB raw, +2.6 KB gzip, +0.2%). The lazy
SDK chunk is 84 KB raw / 27 KB gzip and is not fetched unless tracing is
on and sampled.
- Review fixes (each red on the old code, green on the new): the SDK now
loads by a 35 s deadline even if the page never settles, so a timed-out
hard load is exported (`controller.test.ts`); an agent chat with no
saved task uses the idle rule instead of a content gate that never fires
(`usePageContentReady.test.tsx`); the sample rate is rounded to exactly
1/N so Honeycomb's `SampleRate` weight matches
(`browser-tracing-config.test.ts`). The real-browser run was repeated
after the exporter change with the same result.

## Risks

- Low risk for self-hosted installs: the server never sends a config, so
no tracing code beyond the small recorder runs, and the SDK chunk is
never fetched.
- On hosted instances the feature is off until an operator sets
`PAPERCLIP_BROWSER_TRACES_ENDPOINT` and an ingest key. When on, the
controller wraps `history.pushState`/`replaceState` to time navigation
starts and restores them on sign-out.
- Every signed-in browser can read the ingest key, like the Sentry DSN.
Only Honeycomb ingest keys (send-only) are accepted. A collector
endpoint can keep the key server-side.
- Hard-load API calls happen before the SDK loads, so they have no fetch
spans. The hard-load `page.load` span is still recorded.
- The browser-to-server trace join needs server tracing. The hosted
image does not install the OpenTelemetry server peers today
(`CLOUD_BUNDLED_SERVER_DEPS` is `@sentry/node` only). Browser spans work
without it.
- `ui/src/pages/IssueDetail.tsx` was already over 1,000 lines. This
change adds 5 lines there for the task page content signal, next to the
existing content-paint measure.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use
(shell, file edits, local browser automation). Reasoning effort: low.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 13:48:35 -06:00
DottaandClaude Fable 5.2 8990e94005 fix(ci): keep community PR runs from delaying CODEOWNERS jobs in the hosted queue (#21166)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Its pull request CI runs on two pools: an AWS fleet for CODEOWNERS,
and GitHub-hosted runners for every other contributor
> - Some short jobs for CODEOWNERS pull requests also run on
GitHub-hosted runners: the commitperclip review and the merge review
notifier
> - GitHub starts about 550 hosted jobs each hour for this repository,
and all hosted jobs wait in one first-come queue
> - A community pull request run puts 52 jobs in that queue at one time,
so the short CODEOWNERS jobs wait behind them
> - This pull request limits how many jobs one hosted run can queue, and
lets the short CODEOWNERS jobs use the reserved AWS metadata fleet
> - The benefit is that community pull requests do not delay CODEOWNERS
work

## Linked Issues or Issue Description

**What happened?**

On 2026-11-12 the repository had about 348 queued workflow runs. Jobs on
the AWS fleet started in a median of 29 seconds. Jobs on `ubuntu-latest`
did not:

| Job | Median wait | Job length |
| --- | --- | --- |
| PR merge review notification | 42 min | 9 s |
| commitperclip PR Review | 36 min | 15 s |
| Community PR test jobs | 57 min (p90 76 min) | 7 min |

Between 12:00Z and 18:00Z, hosted job starts stayed between 492 and 567
each hour while the number of running hosted jobs changed from 63 to
134. The start rate is the limit. Community PR runs supplied 3,462 of
the 3,744 hosted jobs in that window (92%).

**Expected behavior**

A short job for a CODEOWNERS pull request starts in about one minute,
whatever the community pull request load is.

**Steps to reproduce**

1. Open several pull requests from forks in a short time.
2. Open a pull request as a CODEOWNER.
3. Look at the start time of `commitperclip PR Review` and `PR merge
review notification` for the CODEOWNER pull request.

## What Changed

- `pr-trusted.yml`: each of the four test matrices has `max-parallel: 4`
when the run uses `ubuntu-latest`. The run has the same jobs. The other
jobs wait in the run, not in the shared queue. AWS runs have no limit
(256).
- `commitperclip-review.yml` and `pr-merge-review.yml`: the job uses the
reserved `paperclip-ci-routing-x64` fleet when
`AWS_CI_SHORT_JOBS_ENABLED` is `true`, the pull request branch is in
this repository, and the author and the sender (or the run actor and the
triggering actor) are in the AWS admission list. The admission variables
are the same as in `ci-routing.yml`. All other events use
`ubuntu-latest`.
- Check-run and commit-status notifications stay on `ubuntu-latest`.
Those events do not identify the author.
- `ci-routing.test.mjs`: 24 placement cases for the two jobs, and a
contract test for the matrix limit.
- `doc/DEVELOPING.md`: the two settings and the runner group step.

## Verification

- `node --test .github/scripts/tests/ci-routing.test.mjs
.github/scripts/tests/notify-pr-merge-review.test.mjs
.github/scripts/tests/ci-plan.test.mjs
.github/scripts/tests/ci-profile.test.mjs
.github/scripts/tests/run-quality-gates.test.mjs`: 200 tests pass.
- `actionlint` accepts the three workflows, including the `max-parallel`
expression.
- Not verified before merge: `pr.yml` calls `pr-trusted.yml@master`, and
the two short workflows run from master. The effect on queue time can
only be measured after merge.

## Risks

- A community pull request run takes longer when the hosted queue is
empty: about 45 minutes for the server matrix, compared with about 13
minutes. Today these runs take 150 to 200 minutes because of the queue.
- The AWS routing does nothing until a maintainer sets
`AWS_CI_SHORT_JOBS_ENABLED`. Before that, the `paperclip-ci-routing`
runner group must allow the two workflow references in
`doc/DEVELOPING.md`. If the variable is set first, the two jobs for
CODEOWNERS pull requests stay queued.
- The two jobs run only scripts from master and no pull request code.
The secrets they read do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 13:48:25 -06:00
DottaandClaude Opus 5.5 8ff45c0fa9 feat(connections): render MCP Apps views in the task chat, starting with Circleback (#21119)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents reach external apps through MCP connections. Paperclip
records each call and shows it as a tool row in the task chat.
> - MCP Apps (SEP-1865, extension `io.modelcontextprotocol/ui`, spec
2026-01-26) lets an MCP server attach an interactive `ui://` view to a
tool. Claude, Claude Code and VS Code render these views in chat.
Circleback uses them to show meeting cards.
> - Paperclip dropped the tool's `_meta.ui` at discovery and the
result's `_meta` at call time. Its gateway reported MCP App UI as
"unsupported". Circleback was reachable only through Composio, whose
toolkit has no `Display*` tools.
> - This pull request makes Paperclip an MCP Apps host. It also adds
Circleback as a native catalog connection.
> - The benefit is that a Circleback recap in a task shows the same
meeting cards as in Claude, and any other MCP server with views works
the same way.

## Linked Issues or Issue Description

**Subsystem affected**
Connections and apps (remote MCP gateway) and the task chat.

**Problem or motivation**
An agent that uses Circleback in Paperclip can only return text. The
same tools in Claude Code show interactive meeting cards
(`DisplayMeetings`). Paperclip discards the MCP Apps metadata, so it
cannot show them.

**Proposed solution**
Paperclip implements the host side of MCP Apps end to end. It keeps the
tool's view declaration and records each call that has a view. It
renders the view under the tool row in a sandboxed iframe that uses the
MCP Apps postMessage protocol. Circleback becomes a native catalog
connection, so its `Display*` tools are available.

**Alternatives considered**
Composio's `circleback_mcp` toolkit has 11 read tools and no `Display*`
tools, and its meta-tools return plain JSON, so it cannot carry views.
The spec's double-iframe sandbox proxy on a separate origin needs a
second origin that Paperclip does not have. A single opaque-origin
`srcdoc` frame with an injected CSP gives the same isolation.

**Roadmap alignment**
This extends the existing Apps catalog and MCP gateway. It adds no new
integration substrate (decision D4 in doc/connections/README.md).

**Additional context**
The standard, the design and the testing plan are in
`doc/connections/MCP-APPS.md`.

## What Changed

- **Shared:** `packages/shared/src/mcp-apps.ts` adds MCP Apps types,
`_meta.ui` and resource parsing (current and legacy keys), and a CSP
builder that keeps only well-formed `https` origins.
- **DB:** migration `0471` (`online_compatible`) adds
`tool_catalog_entries.ui_meta` and a new `tool_mcp_app_views` table.
- **Server, catalog:** discovery keeps each tool's `_meta.ui`. It is not
part of the version hash, so a view never quarantines a reviewed tool.
- **Server, gateway:**
- Paperclip advertises the MCP Apps client capability when it
initializes a session.
  - App-only tools (`visibility: ["app"]`) are hidden from agents.
- The result `_meta` is kept for the view and is not sent to the agent.
- Each successful call to a tool with a view records a view row.
Recording is best effort.
- The gateway's `initialize` now reports `paperclip/mcp-app-ui:
host_rendered`.
- The context protocol (`resources/read`) now uses the injected test
transport, like the other remote calls.
- **Server, routes:** `server/src/routes/mcp-apps.ts`
- `GET /api/issues/:issueId/mcp-app-views` lists a task's views. It
needs task read access.
- `GET /api/issues/:issueId/mcp-app-views/:viewId` returns the view
document, read live through the agent's connection.
- `POST /api/issues/:issueId/mcp-app-views/:viewId/tool-calls` runs the
view's `tools/call`. It needs `tools:use` and reaches only tools of the
same connection that allow `app`. It uses the connection test-call path,
so policy, Ask first and audit apply.
- **UI:** `ui/src/components/mcp-app/`
- The host protocol: `ui/initialize` with theme, style variables and
size limits; tool-input; tool-result; size-changed; `tools/call`;
`ui/open-link` (http and https only); teardown.
- An opaque-origin iframe (`allow-scripts`, never `allow-same-origin`)
with the CSP placed before the view's markup.
- The views of a run render with that run's reply, in the live turn and
in settled history. They are keyed by the reply's run, so history needs
no transcript.
- **Circleback:** a native catalog app (`mcp-oauth`, DCR, `user
user:write`) with:
  - the official logo;
- a reviewed risk classifier (Share* tools are writes, Delete* tools are
destructive, Display* tools are reads);
  - warnings in the sign-in step;
  - `doc/connections/CIRCLEBACK.md`.
  
Search now prefers a native app over an aggregator listing named "X
MCP".
- **Storybook:** `Tasks/MCP App views` shows the tool row with its view,
the ready state (Refresh calls a tool), loading, view unavailable, and a
tool call that needs approval.

## Verification

- `pnpm exec vitest run packages/shared/src/mcp-apps.test.ts
ui/src/components/mcp-app/`: 19 passed.
- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts`: 3
passed, on embedded Postgres. The test connects Circleback through the
catalog with DCR OAuth against a fake Circleback MCP server. It checks:
  - `ui_meta` is stored;
  - an app-only tool is hidden from the agent;
- an agent call records one view with arguments, structured content and
`_meta`;
  - a call without a view records nothing;
  - the routes serve the view's HTML and CSP;
  - the view's tool call works and a model-only tool is refused;
  - another task, another company and an agent actor are refused.
- Circleback connector: 13 focused files, 645 tests passed. These are
the app definitions, search, aggregator fallback, connection intents,
tool policy, `AppsConnect` and `Browse`.
- Typecheck passed for `@paperclipai/shared`, `@paperclipai/server` and
`@paperclipai/ui`. `node scripts/storybook-catalog.mjs` and the
migration compatibility check also passed.
- Storybook check in the browser:
- In both themes, the view runs the handshake, renders the meeting cards
from `structuredContent`, picks up the theme variables, and resizes to
its content.
  - Refresh sends `tools/call` and re-renders the cards.
- **Live Circleback end to end** on a local instance, with a real
Circleback account:
- Connected through the catalog entry with Circleback's dynamic client
registration and OAuth.
  - Discovery stored the real view declarations:
    - `DisplayMeetings` uses `ui://circleback/meetings.v2.html`.
    - `DisplayActionItems` and `DisplayCalendarEvents` also have views.
- `ListAssigneeOptions` and `SetCalendarEventRecording` are app-only
helpers with no view. They are hidden from agents.
  - A `claude_local` agent called `DisplayMeetings` on two tasks.
- The real 3.5 MB Circleback view rendered the meeting cards with the
agent's reply.
- Clicking a card opened that meeting in Circleback through
`ui/open-link`.
- The live run found two bugs, both fixed in this PR, each with a
regression test:
    - The 1 MB tool-result limit blocked the view document.
    - Views were hidden inside the folded tool history.

## Risks

- **New migration:** a new table and a nullable column. It is
`online_compatible`.
- **Existing connections:** tools get `ui_meta` on the next catalog
refresh. Until then, no view shows and the tool row is unchanged.
- **Untrusted provider HTML:** the view runs as provider HTML in an
opaque-origin sandbox. It cannot reach Paperclip's session, DOM or API.
Its network access is limited to the origins its server declared. A
view's tool calls use the viewer's permissions and the normal policy and
Ask-first path.
- **Agent tool lists:** app-only tools are removed from them. No current
provider marks tools app-only, so no listed tool disappears.
- **Search:** the change that prefers a native app over an "X MCP"
aggregator listing affects every app.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
subagents (one subagent authored the Circleback catalog connector).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:28:05 -06:00
DottaandClaude Opus 5.5 ef6d347f79 fix(runner): keep reattached runners answering after a restart (#21161)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute in a remote Runner that the server controls over
the durable PRP channel
> - After a server restart, the next server reattaches to the live
Runner and sends recovery barriers (`runner.drain`, `session.snapshot`)
> - These barriers timed out while the Runner stayed connected, and each
same-run retry failed the same way
> - After three attempts the run stopped with
`native_session_retry_exhausted` and needed a manual Retry
> - This pull request finds the two causes and fixes both in the
controller and the transport
> - The benefit is that restart recovery finishes on its own, without a
manual Retry

## Linked Issues or Issue Description

**What happened?**

After a server restart, a run reattached to its live remote Runner. The
Runner authenticated. Then a recovery command got no recorded answer for
30 s or more, although the Runner stayed connected. Errors on staging:

- `PRP command session.snapshot timed out (runner connected; pending
ahead 0; last runner event seq 869 55s ago)`
- `PRP command runner.drain timed out (runner connected; pending ahead 1
[semantic_tool.result] oldest 170s ...)`
- `runnerd exited while waiting for runner.drain`

Each reattach failed the same way. After three attempts the run ended
with "Workspace repair required" (`native_session_retry_exhausted`).

Root causes, proven with the real Runner and the fake Codex harness:

1. The Runner answered in about 40 ms. At reconnect it also sends the
event backlog that it collected while the server was down. The
controller handles inbound frames of a connection in order, and it
commits each event (accounting write and checkpoint flush) before the
next frame. So the answer waited behind the whole backlog. With 400
events at 100 ms each, the answer was handled after 44 s, and the 30 s
barrier had already failed. The `semantic_tool.result` case is the same:
its answer waited behind the backlog, and the next command is sent only
after that answer is handled. The Runner code was not the cause.
2. When a Runner did not answer during close, the close path queued
`runner.drain` and `runner.suspend`, kept the Runner alive for
reattachment, and saved them in the journal. The next attempt delivered
that old `runner.suspend` to the reattached Runner, and the Runner
suspended and exited.

**Expected behavior**

A reattach to a live Runner completes recovery. A Runner answer that is
queued behind earlier frames is not a timeout. A reattach never delivers
a terminal command from an earlier attempt.

**Steps to reproduce**

1. Start a Codex run in a remote Runner with a turn that streams output.
2. Restart the server. Let the Runner collect a few hundred events while
it is disconnected.
3. Let the next server reattach while event commits are slow, as on a
loaded database.
4. See `session.snapshot timed out (runner connected; pending ahead 0)`.
5. Alternatively, let a close path leave `runner.drain` and
`runner.suspend` pending, then reattach. See `runnerd exited while
waiting for runner.drain`.

**Paperclip version or commit**

`master` at 3fb35fcf13.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`).

## What Changed

- `DurablePrpControlPlane` records each Runner `command_result` frame
when it arrives, before the ordered frame chain handles it.
`queuedCommandResultProgressAt(commandId)` returns the latest time that
the chain made progress while that answer is queued.
- `waitForRunnerCommand` keeps waiting past its deadline while the
answer to the command, or to a pending command ahead of it, is queued
and the chain advanced in the last 30 s. When the chain stops for 30 s,
the barrier fails as before. A wait that starts after its deadline still
rejects a late ACK. Frame order does not change.
- When a barrier completes after such an extension, it credits the time
past the deadline (plus the 5 s post-stop grace) to the admission clock
(`RunnerStallLedger.credit`). Later barriers that share a cold admission
deadline (Pi, OpenCode, ACPX) keep their time.
`#waitForProviderIdentityWork` checks the ready state once before it
rejects an expired deadline.
- `DurablePrpControlPlane.withdrawUndeliveredTerminalCommands()` runs
only when the transport adopts a live Runner. The controller sends only
the first pending command, so no command after it reached the Runner,
and a live, serving Runner cannot have executed a suspend. The method
removes every pending `runner.suspend` and `runner.shutdown` from that
pending suffix. It keeps every other pending command, for example a
`semantic_tool.result` that a still-running tool callback queued after
the close, and renumbers the kept commands so the Runner's command
sequence stays contiguous. Pending drains stay, because they have no
effect.
- A generated command id now skips an id that a renumbered command
already holds.
- `doc/execution-semantics.md` describes both rules.

### Changed existing test

`runnerd-codex-transport-remote-retention.test.ts` asserted that the
reattaching controller delivers the failed close's `runner.suspend`, so
the adopted Runner suspends and exits, and a later attempt relaunches it
(#20923). On staging that path failed the attempt with `runnerd exited
while waiting for runner.drain` (run 3e28c2b0) and did not recover. This
PR requires that a reattach does not inherit that suspend. The test now
asserts that the adopted Runner stays alive, answers `thread/read`,
keeps a `ready` journal, gets no suspend, and no duplicate Runner is
launched. The retention part of the test (the close keeps the frozen
Runner alive) is unchanged.

## Verification

- Red before the fix, green after (real `paperclip-runnerd` and fake
Codex):
- `... past an event backlog ahead of its recovery snapshot result`
failed with the staging error `session.snapshot timed out (runner
connected; pending ahead 0; ...)`. Debug timing showed the Runner
answered in 40 ms and the controller handled the answer 44 s later.
- `... past a runner.suspend left by an earlier attempt's close path`
failed with the staging error `runnerd exited while waiting for
runner.drain`.
- `... past a runner.suspend with a tool result queued after it` failed
the same way with tail-only withdrawal (the Greptile case), and passes
now. The tool result is delivered once and all sequence numbers stay
contiguous.
- Unit tests for `waitForRunnerCommand`: it waits while a queued answer
advances, it times out when progress stops, and a later barrier that
shares the admission deadline still completes after an extended one.
- `... past a tool result its predecessor queued after the restart`
guards delivery of a `semantic_tool.result` queued between controller
generations.
- The existing `shares Pi's absolute cold deadline across recovery
barriers and rejects a late ACK` passes unchanged.
- `cd packages/paperclip-runner && pnpm exec vitest run
src/live/runnerd-codex-transport.test.ts
src/live/runnerd-codex-transport-remote-retention.test.ts
src/live/runnerd-codex-transport-reattach-carried-commands.test.ts`
passes (320 tests).
- Related suites pass: adoption, carried suspend, stale controller
journal, stall, deadlines,
`src/backends/harness-driver-backend.test.ts`, and all
`src/control-plane` tests (330 tests).
- `tsc --noEmit -p tsconfig.test.json` passes in
`packages/paperclip-runner`.
- `pnpm verify` asked for broader coverage (exit 2). Required CI
supplies it.

## Risks

- A barrier can now wait longer than its deadline when the Runner has
answered and the controller still commits a large backlog. The wait ends
30 s after progress stops. This trades a slower recovery for no failed
run.
- The withdrawal applies only to a live adopted Runner. If such a Runner
had somehow journaled a withdrawn suspend, the next command gets a
`controller_sequence_mismatch` rejection, and the Runner executes
nothing.
- Renumbering changes the `controllerSeq` of pending commands that never
reached the Runner. Their command ids do not change.
- Not in this PR: a fallback that stops a connected Runner that never
answers and resumes it from its journal. With these fixes, the observed
staging failures no longer reach that state.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool
use (shell, file editing, browser for read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:50:43 -06:00
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00
DottaandClaude Opus 5.5 90ae3c4337 fix(native): process a superseded export's late stop receipt and stop its sandbox when none arrives (#21143)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run can be retired as superseded while its workspace export
fails. Its files stay unsaved, and the run stays the writer of the
task's workspace.
> - #21136 records those files for the automatic workspace recovery.
That record needs the sandbox's stop-and-retain receipt.
> - The native run projection worker checked for that receipt for one
hour after the run ended. After that, it dropped the run. Nothing woke
it when the receipt arrived later.
> - If the receipt came late or never, the run kept the task's workspace
fenced. Every queued run on the task waited and then failed setup.
Nobody owned the problem.
> - This pull request makes the worker process a late receipt, stop and
keep the sandbox itself when no receipt arrives, and never drop the run
silently.
> - The benefit is that the task's queue moves again by itself, and two
writers never use one workspace.

## Linked Issues or Issue Description

Refs #21136 (review thread on
`native-superseded-workspace-export.ts:84`)

**What happened?**
`preserveSupersededWorkspaceExport` returned `recheckAt: null` when the
sandbox stop receipt had not arrived within `finishedAt + 1h`. The
projection worker then deleted its entry. A later stop receipt
(`settleStopOnlyCleanup`) woke only the workspace recovery queue, not
the projection queue. `preserveNativeFailedWorkspace` requires
`hasConfirmedSandboxStopAndRetain`, so no hold was recorded. The run
stayed the dirty writer, and queued runs kept failing setup.

**Expected behavior**
A late receipt is processed at once. When no receipt arrives, Paperclip
stops and keeps the sandbox itself and waits for the provider's
confirmation. The workspace stays held while the sandbox may still run.
When the provider confirms that the sandbox is gone, the task continues
from its last save. When the stop stays unconfirmed for hours, the task
shows one plain notice, and the check continues.

**Steps to reproduce**
1. Retire a native run as superseded after its workspace export fails
(the #21136 shape).
2. Let the run's sandbox stop not settle (for example, the server that
owned the release is replaced).
3. Wait more than one hour after the run ended.
4. See that the projection worker drops the run, and that a later stop
receipt records nothing. The workspace stays held.

## What Changed

- `sandbox-stop-and-retain.ts`: a confirmed stop-only receipt now also
notifies `DELIVERY_QUEUES.nativeRunProjection` (scoped to the company
and run). New `prepareUnconfirmedSourceSandboxStop` puts a
stop-and-retain intent on the source lease from its current status. It
skips a destroyed sandbox, a stop already in progress, and a cleanup in
flight.
- `native-superseded-workspace-export.ts`:
- Within one hour of the run ending, the worker checks every minute
(unchanged).
- After that hour, it requests the stop-and-retain itself. The existing
pending-cleanup sweep dispatches the stop and retries it until the
provider confirms. The worker checks again every 15 minutes, and the
receipt wakes it at once. The workspace stays held.
  - A sandbox that another lease holds is never stopped.
- When the provider confirms that the sandbox was destroyed, the files
cannot be saved. In one transaction (task, run, then workspace locks),
the worker sets aside only that run's workspace epoch, wakes queued-run
dispatch, and records `issue.previous_run_files_lost` on the task.
- When the stop is still unconfirmed six hours after the run ended (or
the sandbox is held by another lease), the task gets one
`issue.previous_run_files_waiting` activity entry. The worker keeps
checking. It never drops the run.
- A stop-only dispatch now settles when the provider confirms that the
exact sandbox is already gone. Before, the runtime accepted only a
`stopped` reply and threw on `destroyed`, so a gone sandbox kept the
stop pending forever.
- `sandbox-stop-and-retain.ts`: new `acceptedStopOnlyReceipt` accepts
`stopped`, or `destroyed` for a stop-and-retain intent (a native
workspace export resume still needs `stopped`). `settleStopOnlyCleanup`
stores a `destroyed` receipt as `released` / `success` /
`workspace_source_unavailable`, removes the stop intent, never writes a
retain receipt, and wakes the projection and workspace recovery queues.
- `environment-runtime.ts`: the plugin `environmentStopLease` and
built-in `stopLease` checks use `acceptedStopOnlyReceipt`.
- `workspace-recovery.ts`: when its own stop settles as `destroyed`, the
source is unavailable (the existing alternative-strategy path) instead
of a binding failure.
- Daytona plugin: stop of a sandbox that Daytona reports as not found
returns `destroyed` (destroy already did this) instead of throwing. boxd
already returns `destroyed`. Other sandbox plugins have no stop-only
method, so nothing changes for them.
- New shared `SANDBOX_ALLOCATION_HOLDER_STATUSES` (`active`, `retained`,
`pending_cleanup`). The stop-only cleanup ownership check and the
worker's "held by another lease" guard both use it, so a retained warm
sandbox is never stopped and the two checks cannot drift.
- `ui/src/lib/activity-format.ts`: plain labels for the two task
activity entries. They do not ask a person to fix anything.
- `doc/execution-semantics.md`: document the rule.
- Not changed: the shared owner prover still refuses to treat a stopped
instance on another host as proof that its child processes stopped.

## Verification

- `server/src/services/superseded-workspace-export-recovery.test.ts` (4
new tests, red/green):
- "checks again every minute while the run's own release may still stop
its sandbox": the recheck branch. Nothing is recorded or stopped, and
the workspace keeps its writer.
- "stops and keeps the sandbox itself when no stop receipt arrived, and
records the files once the late receipt lands": after the hour, the
lease gets a stop-only intent. A second pass changes nothing. The late
receipt wakes the projection queue. The next check records the hold, and
the system recovery frees the workspace. Fails on master (`expected {
…(19) } to match object { status: 'pending_cleanup' }`).
- "names the unconfirmed stop on the task once, and keeps checking, when
the provider never confirms it": the expiry branch. One notice, the
check continues, no hold, and the workspace stays held. Fails on master.
- "continues from the last save when the provider confirms the sandbox
is gone": epoch 1 is set aside, the workspace is free, the task says so,
and the result is not applied. Fails on master.
- Review-round tests (all 5 fail on the first head `a208bfa58f`, pass
now):
- `environment-runtime.test.ts` "settles a plugin stop whose provider
reports the sandbox gone (boxd, Daytona not found) through the cleanup
sweep": provider dispatch through `releaseRunLeases` and
`sweepPendingCleanupLeases`. The lease settles `released` with a
`destroyed` receipt and no retain receipt.
- `environment-runtime.test.ts` "settles a built-in stop whose provider
reports the sandbox gone, without claiming retained files": the built-in
`stopLease` path. No destroy call.
- `superseded-workspace-export-recovery.test.ts` "continues from the
last save when Paperclip's own stop finds the sandbox already gone": the
worker's stop request gets a `destroyed` reply, the projection queue
wakes, and the workspace is freed at epoch 2.
- `superseded-workspace-export-recovery.test.ts` "never stops a sandbox
a retained warm lease still holds, says so once and keeps checking": the
source lease is unchanged, a recheck is returned, and `sandbox_in_use`
is recorded.
- Daytona `plugin.test.ts` "explicit stop confirms a sandbox Daytona no
longer has as destroyed".
- `environment-runtime.test.ts` (207) and Daytona `plugin.test.ts` (421)
pass.
- Related suites pass: `native-run-projection-work`,
`retained-native-session-cleanup-work`, `workspace-recovery-stop-proof`,
`native-failed-workspace-recovery`, `legacy-workspace-restore-recovery`,
`environment-retained-orphan-sandbox`,
`workspace-recovery-retry-copyback`, `workspace-recovery-fallback` (398
tests), and `ui/src/lib/activity-format.test.ts`.
- `tsc --noEmit` for `server`.

## Risks

- Low to medium risk. The new behavior acts only on an ended superseded
run that failed in its workspace export, still holds a dirty workspace,
and has no retained-file hold.
- The worker now stops an ended run's sandbox one hour after the run
ended if no stop receipt arrived. The stop uses the existing stop-only
contract and keeps the sandbox. A sandbox held by another lease is never
stopped.
- A destroyed sandbox's unsaved files are not recoverable. The task
continues from its last save and says so.
- If a provider never confirms a stop, the workspace stays held (by
design: no second writer). The task shows one notice, and Paperclip
keeps retrying.
- Shared cleanup code now treats a confirmed `destroyed` stop-only reply
as settled. Every caller was checked: the cleanup sweep and run release
count it as settled; the workspace recovery treats the source as
unavailable; the fallback treats it as stopped (nothing can run there).
No caller reads it as retained files, because no retain receipt is
written. A native workspace export resume still requires `stopped`.
- Each extra delivery notification per settled stop-only cleanup is
cheap. The projection worker's scoped load finds nothing for runs that
do not match.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:54:47 -06:00
DottaandClaude Opus 5.5 dfc510b887 fix(server): hand new run starts to the replacement server during a deploy overlap (#21127)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud replaces the server process on each release. The old
server keeps serving until the new one passes health checks and takes
the traffic.
> - During that overlap the old server keeps starting new runs right up
to SIGTERM, although the new server can already start them.
> - SIGTERM then cuts off exactly those runs while they still prepare.
In one audited swap the old server started 30 runs after the new server
booted, and 23 of them ended `process_lost`.
> - Admission must never pause, so the fix must hand over who starts new
runs, not stop admission.
> - This pull request makes the new server record when it starts runs.
The old server then leaves new run starts to it, and starts them again
if the new server stops or dies.
> - The benefit is that a deploy no longer starts runs on a server that
is about to stop, and the company always has one server that starts
runs.

## Linked Issues or Issue Description

Refs #20852 (preserves preparing and launching turns that a deploy
interrupts; this PR reduces how many such turns the old server starts).
Related: #21046, #21107 (shared controller-instance owner proof), #20986
(queued-run dispatch signal).

**What existing behavior does this improve?**

The start of new runs while two servers overlap during a Cloud deploy.

**Subsystem affected**

Server run admission (`server/src/services/heartbeat/queue.ts`), the
controller instance table, and startup and shutdown in
`server/src/index.ts`.

**Current behavior**

The old server starts queued runs until it receives SIGTERM. Runs that
it starts in the last minutes of the overlap are interrupted while they
prepare. Nothing tells the old server that its replacement already
starts runs: `controller_instances.epoch` is written at process start,
and that happens minutes before the new server can start runs.

**Proposed behavior**

The new server records `dispatching_at` and a short
`dispatch_lease_expires_at` on its `controller_instances` row right
after its startup queued-run pass. The old server reads this through an
in-memory flag at run admission and leaves new run starts to the new
server. Runs it already started continue and are handed over at SIGTERM
as today. If the new server stops, drains or dies, the old server starts
runs again within seconds.

**Reason and benefit**

Fewer runs are cut off by each swap. There is never a moment without a
server that starts runs.

**Breaking changes**

None. The migration adds two nullable columns. Releases without this
code ignore the columns, and their rows never carry the marker, so a
mixed-version overlap behaves as today.

## What Changed

- `controller_instances`: new nullable columns `dispatching_at` and
`dispatch_lease_expires_at` (migration
`0471_controller_instances_dispatch_marker`, two `ADD COLUMN`
statements, no lock beyond the catalog update).
- New `server/src/services/run-start-handover.ts`. It is one worker on
the shared delivery coordinator (`DELIVERY_QUEUES.runStartHandover`).
Its renewal and recheck are coordinator deadlines, and a drain change or
expiry wakes it.
- Replacement side: `begin()` records the intent and writes the marker
(it creates the boot's row if the row is missing). If the server drains
at that moment, the worker writes the marker when the drain ends. The
worker then publishes a `run-start-handover` hint and renews the 20 s
lease every 5 s. It renews only while an older boot that is not stopped
or fenced may still serve, and for at most 30 min. All marker writes run
in one serialized chain. `release()` at shutdown lands after any renewal
already in flight. A clear that fails is retried, and shutdown waits for
the release. While the server cannot dispatch (drain, quiesce, admission
closed), the worker clears the lease.
- Old-server side: `isYielding()` is an in-memory read. The hint
refreshes it, and so does one read that a dispatch pass waits for (at
most 1 s) when the last read is more than 2 s old. A read that gets no
answer within 1 s never keeps the server yielding. The server admits,
and the yield also ends when the replacement's last seen lease runs out
on the old server's own clock. Reads are a single flight on the
dedicated connection. While it yields, the server rechecks every 3 s,
calls `onResume` when the marker is gone, and sends a coalesced
queued-run-dispatch hint for skipped starts.
- It yields only to a newer epoch with a live marker. It never yields to
an older boot, to itself, or to a row without the marker, and it does
not yield without a row of its own.
- **Database invariant:** every statement of the hand-over runs in a
small transaction on the dedicated one-connection controller-lease pool
(`controllerLeaseDb`) with a 1 s statement timeout. That covers reads,
marker writes, the `run-start-handover` notification (commit-bound, in
the marker write's own transaction) and the skipped-start hint. None of
it touches the shared pool or registers tracked database work. The pool
is shared only with legacy controller-lease renewals, and never with the
liveness pool that native lease renewals use. A hand-over statement
waits at most one legacy batch (10 s), inside the 15 s between a marker
renewal and the marker's expiry. It delays a legacy renewal (60 s lease,
renewed every 10 s) by at most 1 s. I did not piggyback on the
controller-instance lease renewal, because that renewal runs only while
the server holds work and only every 30 s.
- **Import audit of `run-start-handover.ts`:**
  - `drizzle-orm` `sql`: builds statements, no I/O.
- `controllerLeaseDb`: the dedicated connection, used by every statement
here.
- `notifyDatabaseWorkOnCommit` (new in `work-signals.ts`): runs
`pg_notify` only on the transaction it is given and registers no intent,
so the coordinator's `reconcileDatabaseWork` has nothing of ours to
check.
  - `logger`: no I/O to the database.
  - `controller-process-identity`: in-memory boot id and host hash.
- `delivery-work-coordinator`: type only. The worker is registered for
lifecycle and deadlines; its `run` awaits only the serialized marker
chain, and reads are not awaited.
  - `DELIVERY_QUEUES`: constants.
  - `task-admission` drain subscriptions: in-memory events.
- `heartbeat.ts`: run admission is also closed while the server yields.
A `runStartHandover` service option (default: the process's registered
hand-over) lets one process host two assembled services in tests.
`queue.ts`: `startNextQueuedRunForAgent` and
`resumeQueuedRunsAfterDrain` first call `refreshRunAdmission()`. That
call returns `null` (nothing to await) unless the hand-over read is
stale.
- `index.ts`: creates the hand-over only when the instance lease is on,
the deployment notification channel exists, and the scheduler is
enabled. It calls `begin()` after the RunDispatch startup pass and wakes
that queue. It releases the hand-over when run admission closes at
shutdown, and stops it before the lease stops.
- `deployment-notifications.ts`: keeps the LISTEN connection open while
a hand-over involves this server, so skipped starts and lease changes
reach the other server at once.
- `doc/execution-semantics.md`: new section "Run starts during a deploy
overlap". It covers the marker, who yields, failure handling, what
counts as starting a new run, and what a post-deploy audit should show.

**Marker contract for other components (for example the deployer):** a
replacement starts runs when its `controller_instances` row has an
`epoch` greater than the old boot's, and `dispatching_at is not null and
dispatch_lease_expires_at > now() and stopped_at is null and fenced_at
is null`. `dispatching_at` stays set after the lease lapses, as the time
the boot began starting runs.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-start-handover.test.ts`: assembled-app test
with two `heartbeatService`s, each with its own pool, listener,
coordinator and boot. A wake that the old server accepts stays queued
there. The replacement's real queued-run dispatch starts it, and claims
are observed per pool. After a replacement release, the old server
starts the run itself. Red: with the yield removed from
`isRunAdmissionClosed`, the old server claims the run.
- `cd server && npx vitest run
src/__tests__/run-start-handover.test.ts`: 19 tests pass against
embedded PostgreSQL. Each test uses two servers, each with its own pool,
boot id, LISTEN connection and delivery coordinator. The tests cover:
- the old server is ready and the new server starts; the new server sets
the marker; the old server yields; the queued run is claimed only by the
new server, and a skipped start wakes the new server's dispatch
- the old server learns from its own dispatch path without a
notification: a read older than 2 s is refreshed before the start
- the new server dies after it sets the marker: the old server resumes
and runs its queued-run pass
  - the new server shuts down: the old server resumes at once
- the new server drains: it releases the marker and takes it back
afterwards
- a replacement that starts up during a drain takes over when the drain
ends
  - a shutdown release during an in-flight renewal stays released
- a clear that fails at shutdown (statement timeout behind a row lock)
is retried by a later release
- an older server that outlives the lease TTL keeps yielding while the
replacement renews
- a yield that the old server cannot confirm within 1 s (its only
connection is busy) does not stop starts (red/green)
- a yield ends when the replacement's lease runs out, even while every
read waits for a connection (red/green)
- renewals blocked past the TTL hand starts back once, then the
replacement takes them again (bounded flapping)
- a clear that fails when the replacement starts draining is retried
well inside the lease (red/green)
- with both servers' shared pools blocked past the TTL, the marker is
written, renewed and released, and the old server yields and resumes,
all before the block ends (red when hand-over statements use the shared
pool)
  - an older-release row without the marker changes nothing
- a server never yields to an older boot or to itself, and does not
yield without a row of its own
  - a lone server writes once, never renews, and admits from memory
  - without a hand-over (self-hosted) there is nothing to await
- Red: on `origin/master` the test file cannot run, because the module
and the columns do not exist. The old server has no signal and keeps
starting the handed-over run.
- `cd server && npx vitest run
src/__tests__/deployment-notifications.test.ts
src/services/heartbeat/dispatch-work.test.ts
src/__tests__/delivery-work-coordinator.test.ts
src/__tests__/controller-instance-lease.test.ts src/shutdown.test.ts`:
all pass.
- `cd packages/db && pnpm check:migrations`: safety and compatibility
checks pass. `cd server && npx tsc --noEmit -p .`: clean.
- `pnpm verify` asks for broad verification because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

## Risks

- Run starts move from the old server to the new server during the
overlap. If the new server cannot dispatch, the old server resumes on
the lease clear (about 3 s) or the lease lapse (up to about 23 s after a
crash). If a hand-over read fails, the server keeps admitting.
- If the old server was not listening for notifications, it can start
runs for up to about 2 s after the marker (the age of its last read). On
a dispatching server, the pre-dispatch read adds one small bounded query
at most every 2 s.
- An older boot that has been idle for more than 30 min counts as gone.
The new server then does not renew, and the old server admits again as
today.
- The migration adds nullable columns only. The deployer can apply it
while the previous release serves.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:09:53 -06:00
DottaandClaude Fable 5.2 2174f2dfa5 perf(server): stop re-reading and re-decrypting unchanged checkpoint headers on every upgrade readiness probe (#21130)
Cloud repeats the upgrade readiness probe. Every probe loaded, decrypted and parsed the checkpoint header of each active native run and of each retained task session's last run, even when nothing had changed.

The statement that reads a run now also reads its header's row revision (xmin, ctid, updated_at, stored size) and returns the material only for a revision the process has not inspected. A bounded memo in the checkpoint owner keeps the four identifiers the answer compares. It reuses an answer only for the same binding, row revision and secrets key, loads the key again for each answer it serves, never remembers a header that failed to decrypt, and forgets a header the latest probe did not inspect.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 10:45:44 -06:00
DottaandClaude Opus 5.5 f2e449f8f6 fix(retry): continue an accepted Retry past chat input held because it predates the stop (#21129)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person's Retry on a connected task (for example a GitHub pull
request review) must start work. #21080 made a Retry whose chat replay
is refused after the click continue from the recovered saved work
> - After #21080 was deployed, accepted Retries on a busy GitHub review
agent still did not start
> - Those tasks had a saved chat request that arrived while the failed
run was still running. Admission holds such a request as
`message_predates_stop`, so it can never start alone
> - #21080 refused to continue while any saved chat input existed, so
this held input blocked the continuation. For a second group, the latest
run was an admission that the recovery hold cancelled before dispatch,
and the recovered saved work belongs to the earlier failed run
> - This pull request lets the continuation adopt input that was held
because it predates the stop, and lets the recovered run that held a
cancelled admission count as the saved work
> - The benefit is that an accepted Retry starts work on these tasks too

## Linked Issues or Issue Description

Follow-up to #21080. No public issue exists. This is the bug
description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

After #21080 was deployed, `POST /api/issues/:id/retry` returned 202 for
blocked tasks of a GitHub review agent (`paperclip_runner`), but no run
started. Each Retry receipt changed to `failed` about 5 seconds later
(on the 14:48 Retries, 5 seconds for one task and about 75 seconds for
two others), with no run and no `issue.retry_continued_from_saved_work`
activity. Every affected task had a saved chat request (a newer GitHub
review request) with an `issue.execution_waiting` record of reason
`message_predates_stop`. The request arrived while the failed run was
running, so admission holds it permanently. The interruption projection
kept pointing at that held request. In a second group of tasks, the
latest run was cancelled before dispatch with
`execution_reconciliation_required`, and the recovered saved work
belonged to the earlier failed run that held it.

Root cause: in `continueRefusedTaskChatRetry`
(`server/src/services/task-retry.ts`), any saved chat input from
`selectTaskRetryInput` stopped the continuation, so the receipt settled
`failed`. The held input can never start by itself, so nothing ran. The
recovered-work check also looked only at the latest run, so it missed
the recovered run that held a cancelled admission.

**Expected behavior**

An accepted Retry starts work. Saved input that admission holds because
it predates the stop joins the continuation, and the task continues from
the recovered saved work.

**Steps to reproduce**

1. Bind a task to a GitHub chat conversation. Let a native run fail with
`native_continuation_requires_reconciliation`, and recover its saved
work.
2. Save a chat request while that run is still running, so admission
holds it with `message_predates_stop`.
3. Click Retry while the chat identity lock is busy, so the source check
is deferred and then refused.
4. The route returns 202. The Retry receipt changes to `failed`, and no
run starts.

**Paperclip version or commit**

a65644d1e1 (the deployed #21080). Reproduced on ef2b53b675.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

`paperclip_runner` (native runs, and legacy admissions cancelled before
dispatch). The fix is in core Retry code and is not adapter-specific.

## What Changed

- `server/src/services/task-retry.ts` (`continueRefusedTaskChatRetry`):
- Saved chat input whose source comment was written no later than the
failed run's stop boundary (`userStopBoundaryAt`: the recorded user Stop
request, otherwise `finishedAt`) no longer stops the continuation. This
is the same boundary and timestamp that admission uses. Admission holds
that input as `message_predates_stop`. The continuation coalesces its
receipts into the Retry receipt (`coalescedIntoWakeupRequestId`), so the
task stops waiting on it. The receipt ids are recorded in the activity
details.
- Saved input created after the stop, and unadmitted chat input, still
settle the Retry as failed, as before.
- When the latest run is an admission cancelled before dispatch
(`execution_reconciliation_required`, never started), the recovered-work
check also accepts the run named by that run's
`executionWait.recoveryActionId`.
- `doc/execution-semantics.md`: both rules.
- Tests in `server/src/__tests__/chat-channels.integration.test.ts`: two
new variants of the #21080 regression test.
`person_with_input_before_stop` saves a chat request before the failed
run stops. `person_after_held_admission` adds a legacy admission that
the recovery hold cancelled. Both start one run that resumes the
recovered workspace (`resume_preserved_workspace`). The held-admission
fixture binds the task to the recovered workspace, as on the hosted
tasks. Refusal edges `input_after_stop` and `input_inside_user_stop`
settle the Retry failed with no run and leave the input deferred.

## Verification

- Red before the fix: both new variants fail. The Retry receipt is
`failed` (or stays held), and no run starts.
- Green after the fix: both variants pass, and the held input receipt is
coalesced into the Retry receipt.
- `npx vitest run src/__tests__/chat-channels.integration.test.ts -t
"[Rr]etry"` with `src/services/task-retry.test.ts` passes 369/369 on the
final commit.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Staging was read with GET requests only.

## Risks

- Behavior change: input that arrived during a failed run no longer
starts its own turn through the chat retry. It is adopted into the
saved-work continuation, and the comment stays in the task thread.
Before this change, that input could never start at all.
- Input that arrived after the stop keeps its own admission path.
- No migration. Receipts that already failed stay failed. The next click
takes the new path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:34:36 -06:00
DottaandClaude Opus 5.5 df89950eb6 fix(native): save or set aside a superseded run's unsaved workspace so its task queue moves (#21136)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result. Then a finalizer exports the
run's workspace files and applies the result to the task.
> - When a newer run already decided the task, the finalizer retires the
older result as superseded. Since #21106, this also ends the run.
> - If that retirement interrupts a failed workspace export, the run's
files stay unsaved. The run also stays the writer of the task's
workspace. Nothing records those files for recovery, so nothing ever
releases the workspace.
> - Every run queued on the task then waits behind a save that never
settles. After one hour, each queued run fails setup instead.
> - This pull request records those files for the existing automatic
workspace recovery. Recovery saves them from the retained sandbox, or,
when every bounded attempt fails, continues the task from its last save
with the files attached.
> - The benefit is that the task's queue moves again by itself, and the
run's unsaved work is saved or kept findable. The superseded result is
never applied.

## Linked Issues or Issue Description

Refs #21106

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. Its workspace export failed once with
`remote_workspace_archive_fs_eacces`. The finalizer then retired it as
`native_finalization_superseded`, because another run had already
decided the task. #21106 ended that run at 14:59:17Z (`failed`). The
task still could not move. The next queued run was cancelled with
`workspace_busy` ("ended, but its workspace save has not settled"). Its
retry re-armed every minute. Four more runs stayed queued. The task
showed no recovery action.

**Expected behavior**
The ended run's unsaved files are saved from its retained sandbox, and
the workspace is released. If they cannot be saved within the bounded
retries, the task continues from its last committed save, and the files
are attached to the task. The queued runs proceed in both cases. The
superseded result is never applied.

**Steps to reproduce**
1. Accept a native result for a run that writes a remote workspace.
2. Let the workspace export fail once, while a newer run has already
committed the task's decision.
3. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is `failed`.
4. See that the workspace still names the run as its dirty writer, and
that no retained-file recovery action exists.
5. Queue another run on the task. It defers with `workspace_busy`, and
after one hour it fails setup with "The workspace still has an
unfinished writer or save operation."

**Root cause**
`recordRetryableFailure` (`native-run-finalizer.ts`) retires a
superseded finalizer without preserving the export.
`preserveNativeFailedWorkspace` records a retained-file hold for an
accepted result only when the failure code is
`native_workspace_sync_out_retry_exhausted`. A
`native_finalization_superseded` coordinator never qualifies.
`readRemoteWorkspaceWriterHold` keeps an ended dirty writer as the
holder until its save settles. Without a hold, no worker ever settles
it.

**Paperclip version or commit**
Staging Core with #21106 (`81d2e4d8d9`). Master `ef2b53b675` has the
same behavior.

## What Changed

- `native-superseded-workspace-export.ts` (new): identifies a superseded
finalizer that failed in the workspace export. It selects ended runs
that still hold their workspace with unsaved files and no retained-file
hold. It records the hold through `preserveNativeFailedWorkspace`. It
checks again only while a just-ended run's sandbox stop receipt is still
arriving (at most one hour).
- `native-failed-workspace-recovery.ts`: `preserveNativeFailedWorkspace`
treats a superseded workspace export like an exhausted copy-back, with
the same proofs (run ended, no live controller, accepted result,
confirmed sandbox stop-and-retain, source still bound to the workspace).
It marks the hold `supersededWorkspaceExport`.
- `native-run-projection-work.ts`: the native run projection worker also
selects these runs. It loads them at startup and wakes when a finalizer
changes. Rows written by older releases are therefore settled after
deploy without a person.
- `workspace-recovery-fallback-discovery.ts` and
`workspace-recovery-fallback.ts`: a marked hold whose bounded re-export
is spent (at most 5 retries in 30 minutes) is eligible for the existing
continue-from-last-save fallback. That fallback sets aside only this
run's workspace epoch and keeps its location on the workspace record. It
attaches the readable files to the task unapplied. The task message says
Paperclip "could not save" the previous run's unsaved files.
- `preserved-result-finalization.ts`: a superseded result is never
resumed for application, even after its files are saved.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/services/superseded-workspace-export-recovery.test.ts`
(new). The first two tests fail on master and pass on this branch:
- "records the files for automatic recovery, saves them from the
retained sandbox and frees the workspace". It uses the staging row
shape. The projection worker records the hold, and a second pass changes
nothing. The workspace waits for that save. The system recovery saves
`saved.txt` into a committed snapshot. The workspace becomes free. The
finalizer, the result and the task's decision do not change, and the
result is not resumed. On master the worker selects nothing (`expected 0
to be 1`).
- "continues from the last save with the files attached when every
re-export fails". After an exhausted re-export budget, the hold is
resolved `restored`. The files are attached (`quarantine.state =
captured`). The workspace moves to epoch 2 with no writer, and the
abandoned epoch keeps its lease location. The message has no request for
a person to fix anything.
- "leaves a superseded result alone when it never failed in the
workspace export".
- Related suites pass: `native-run-projection-work`,
`native-finalization-recovery`, `workspace-recovery-fallback`,
`native-failed-workspace-recovery`, `workspace-recovery-retry-copyback`.
- `tsc --noEmit` for `server`.

## Risks

- Low to medium risk. The change acts only on runs whose finalizer is
already terminal and superseded, that failed in the workspace export,
and that still own a dirty workspace. All existing preservation proofs
still apply.
- The saved files of a superseded run become the workspace's next saved
version. The run's result (the task decision) is still never applied.
- When the export can never succeed, the task continues from its last
save. Files that cannot be read are not attached. The workspace record
keeps the set-aside epoch's location. Saving the readable part of a
partly unreadable workspace is not in scope.
- Runs that already failed setup before this deploys are not restarted
by this change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:20:54 -06:00
Nicky LeachandPaperclip 782d6bead4 Configure managed plugin Runner lifecycle (#21065)
## Thinking Path

> - Paperclip runs agents in managed environments.
> - An agent can use an explicit environment or the instance default.
> - A lifecycle plugin must receive the driver that provisions the
agent.
> - A managed plugin environment also needs to select the Runner process
lifecycle.
> - This change gives lifecycle selection and provisioning the same
environment lookup.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Managed plugin environment provisioning and lifecycle configuration.

**Current behavior**
Lifecycle calls lack the selected plugin driver. Managed plugin
configuration cannot select warm Runner execution.

**Proposed behavior**
Resolve the environment once per lifecycle operation. Pass the driver
key for the plugin that serves it. Parse and persist the managed Runner
lifecycle mode.

The portable ownership and workspace placement changes are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066) and
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Share agent environment resolution between required plugin selection
and the lifecycle RPC.
- Pass `environmentDriverKey` to the plugin that serves the selected
environment.
- Allow managed plugin environments to set `warm` or `per_turn` and
persist the selected mode.
- Advertise `pluginEnvironmentRunnerLifecycleMode` in the release
capability manifest so senders can check support before including the
field.

## Verification

- Built shared types, the plugin SDK, and Runner. Server typecheck and
SDK source-pin check passed for the initial layer.
- The capability contract test failed before the manifest fix and passed
after it. Independent verification passed all 160 tests across managed
configuration, plugin environment persistence, lifecycle routing, and
environment reconciliation.
- Routing coverage includes the instance default, an explicit second
driver from the same plugin, and an explicit sandbox.

- [Hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/43193311862)
passed on `f07b192c21`. The first attempt timed out in an unchanged
workspace-recovery fixture and then collided with a background recovery
action. The exact test and all 44 tests in its suite passed locally; the
isolated shard retry passed without code changes.

## Risks

- Existing managed configuration can omit the lifecycle mode. Senders
must check the target release for `pluginEnvironmentRunnerLifecycleMode`
before sending the new field. Publishing this signal does not update
external deployment tools.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 08:18:04 -08:00
DottaandClaude Fable 5.2 1af987690d perf(storage): hash and encrypt workspace snapshots off the server thread (#21138)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process serves every request and agent turn on one
thread. The same process saves each remote workspace as one encrypted
archive in object storage, and restores it before a turn
> - `server/src/services/remote-workspace-snapshot-storage.ts` hashed,
encrypted and decrypted every byte of those archives on that thread.
`server/src/storage/s3-multipart.ts` computed the Content-MD5 of each
uploaded part on that thread too
> - Hashes and ciphers are synchronous in Node. A CPU profile of a busy
hosted server showed this work on the main thread, which was at 93% of
one core
> - Each megabyte of a snapshot delays every request and turn that waits
behind it
> - This pull request moves those passes to a small, bounded set of
worker threads. The checks, their order, their failures and the stored
format do not change
> - The benefit is that a 1 GiB save uses about 0.7 s of the server's
thread instead of about 2.8 s, and an unrelated request during the save
is answered in 1.3 ms (p99) instead of 16.7 ms

## Linked Issues or Issue Description

No public issue exists. I searched open and closed pull requests for
`workspace snapshot hash encrypt`, `event loop` and `archive io`.
Related merged work: #19900 gave these stream passes 1 MiB buffers,
#20554 removed a read-back from migration, #20722 took runner artifact
hashes off the event loop, #20709 records long event-loop blocks. No
open pull request covers this change.

**What happened?**

A 30 second CPU profile of the main thread of a hosted server (commit
1d153f62b6, Node 24, main thread at 93% of one core) showed:

- the two stream transforms in the snapshot save at 3.9% and 1.6%
inclusive. The first hashes the archive (SHA-256). The second hashes the
ciphertext (SHA-256);
- native hash `update` at 5.4% self time for the whole process. About
1.0 s of it was under those two transforms and 0.25 s was under
`fileRangeMd5`;
- the AES-256-GCM cipher of the same pipeline, and Buffer allocation
under the file read streams. Garbage collection was 25% of the thread.

What the code did with each byte, on the server's thread:

| Path | Pass | Digest or cipher | Purpose |
| --- | --- | --- | --- |
| Save | read archive | SHA-256 of plaintext | bound into the wrapped
key, checked on each restore |
| Save | same read | AES-256-GCM | confidentiality and authenticity |
| Save | same read | SHA-256 of stored bytes | recorded on the snapshot
row, checked on each restore and read-back |
| Save | read each part (S3) | MD5 | `Content-MD5`: the object store
rejects a part it did not receive intact |
| Save, read-back callers | read the kept file | SHA-256 of plaintext |
proof that the published file is the saved archive |
| Restore | download | SHA-256 of stored bytes | integrity of the object
against the snapshot row |
| Restore | read ciphertext | AES-256-GCM | authenticity |
| Restore | read plaintext | SHA-256 of plaintext | integrity against
the digest in the wrapped key |
| Restore | read plaintext again | none (copy) | publication beside the
destination |

**Expected behavior**

Digest and cipher passes over a snapshot do not run on the thread that
serves requests. Every check stays, with the same failure.

**Steps to reproduce**

1. On the parent commit, run `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-workspace-snapshot-crypto.ts --mib=1024` (copy the
script from this branch).
2. Read `upload.mainThreadCpuMs`, `upload.mainThreadBusyMs` and
`upload.unrelatedRequest`. The benchmark uses the real save and restore
functions, the real S3 client, and a local object store that checks each
part's `Content-MD5`. A separate process requests a trivial route from
the measured server and times each reply with its own clock.

**Paperclip version or commit**

`master` at 9e910ff9ae. The profile is from 1d153f62b6.

**Deployment mode**

All modes that save remote workspaces. The profile is from a hosted
server with S3 storage.

**Node.js version**

Profile: Node 24. Local benchmark and tests: Node 26.4.0 on darwin
arm64.

## What Changed

- `server/src/storage/file-crypto-pool.ts` (new): runs file digests and
AES-256-GCM passes on worker threads.
- Bounds: at most 2 jobs run at once. Each worker has two reused 1 MiB
read buffers and one write in flight. A waiting job holds file names and
byte ranges, not file contents. Waiting jobs are served in order.
- Cancellation: a waiting job is cancelled at once. A running job stops
at its next chunk, closes its files, and then the caller sees the same
`AbortError` as before.
- Shutdown: an idle worker does not keep the process alive. A busy
worker does, until its job ends or its caller's signal aborts it.
- Key handling: the pool copies the data key for the worker, moves that
copy (it is not cloned), and the worker erases it. A copy that never
reaches a worker is erased on this thread. The caller keeps and erases
its own key as before.
- The worker program is sent as source text, as
`remote-workspace-snapshot-archive.ts` already does. It needs no extra
file in the build or the runtime bundle.
- Save (`uploadWorkspaceSnapshot`): the one read that digests the
archive, encrypts it and digests the stored bytes runs in a worker. The
identity check around that read (`archive_changed`) stays on this thread
and is unchanged.
- Part upload (`fileRangeMd5`): the `Content-MD5` of each part is
computed in a worker. The first round of parts is digested before any
part is sent, so the parts still leave together and a rate limit still
answers one burst (the existing throttle test covers this). Every digest
of that round has ended, also after a failure or a cancellation, before
the upload settles.
- Restore (`loadWorkspaceSnapshotArchive`): the download no longer
hashes on this thread. One worker digests the downloaded object while
another decrypts it into private scratch. The results are then checked
in the previous order: object size and digest, header, key, GCM tag.
- Restore, one pass less: the decrypted archive was read twice (digest,
then copy). It is now digested while it is copied beside the
destination, and linked to the destination only when the digest matches.
The digest covers exactly the published bytes.
- Read-back callers (`verifiedArchivePath`): the digest of the kept file
runs in a worker, through the same open descriptor. The file identity
checks are unchanged.
- `server/scripts/benchmark-workspace-snapshot-crypto.ts` (new): the
reproduction and benchmark.
- Docs: `doc/run-log-events.md` (what each `storage.snapshot.*` phase
now does) and `doc/DEVELOPING.md` (the worker threads and the
benchmark).

Module boundaries (`rules/module-boundaries.mdc`): the pool is in
`server/src/storage/` because both consumers are parts of the same
upload, and storage must not import from services. Its contract is file
paths, byte ranges, an algorithm, and AES-GCM parameters. The snapshot
format (magic bytes, additional data, key wrapping, the order of checks)
stays in the snapshot service. The part layout and `Content-MD5` stay in
the S3 module.

Not changed, and why:

- The stored format, the ledger, attempt journaling, atomic publication,
and `recoverWorkspaceSnapshotUpload` /
`recoverWorkspaceSnapshotArchive`.
- `verifyObject` (full read-back for a store that did not verify the
upload, and for a retried save) still hashes on this thread. Its bytes
arrive from the network on this thread. To move it, the object must
first be written to disk.
- Socket writes of part bodies stay on this thread. I measured 1 MiB
part-body read chunks in place of 64 KiB: main-thread time did not
change and the longest event-loop delay increased, so I did not keep it.
- `remote-workspace-transport.ts` also hashes an archive on this thread.
It is outside the snapshot storage path.

## Verification

Regression tests, red then green (`cd server && pnpm exec vitest run
src/services/remote-workspace-snapshot-storage.test.ts
src/storage/file-crypto-pool.test.ts`):

- Parent implementation with the tests of this branch: 3 tests fail.
- `keeps encrypting a save while the server thread is occupied by other
work`: `expected 4194323 to be 25166144`. The test blocks the thread
until the encrypted file is complete. On the parent the file stops at 4
MiB.
- `answers unrelated requests while a save is being encrypted, and
digests or encrypts none of it on the server thread`: `expected 8388627
to be less than 65536` (bytes hashed on the thread).
- `digests and decrypts none of a save or a restore on the server thread
when the store verified the upload`: `expected 33556772 to be less than
65536`.
- This branch: 58 of 58 and 6 of 6 pass.
- The tests use synchronization, not time. A named pipe keeps a save in
the encrypt step for as long as the test feeds it. Deadlines are 30 s
and matter only for a failure.
- Integrity tests that pass on both the parent implementation and this
branch, which shows that the failures are the same: a truncated object
(`object_hash_mismatch`), an oversized object (`object_size_mismatch`),
an archive that grows, shrinks or is rewritten in place after its
identity was read (`archive_changed`, before any upload), a ciphertext
that was changed with the recorded digest changed to match (GCM
failure), a plaintext that differs from the digest in the key
(`plaintext_hash_mismatch`, staged copy removed), and the order object,
header, key.
- Cancellation: during encryption the save rejects with `AbortError`,
and leaves no scratch file, no object and no recorded attempt. During
the store write it leaves no object.
- Part digests (`src/__tests__/storage-s3-multipart.test.ts`): `leaves
no part digest running or waiting when the source is cancelled|removed
during the first round`. Before the review fix in a354242865 it failed
with `{ running: 2, waiting: 0 }` at multipart cleanup.
- Pool tests: digests equal single-threaded digests for a file, a range
and a descriptor; encrypt and decrypt round trip; each damaged
ciphertext is reported; at most 2 jobs run and the others wait in order;
a waiting job and a running job are both cancelled.

Benchmark, same machine, 5 alternating pairs (parent implementation,
then this branch) for each size. Values are medians, with the range in
parentheses.

Command: `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-workspace-snapshot-crypto.ts --mib=1024` (and
`--mib=200`).

1 GiB archive:

| Measure | Save, before | Save, after | Restore, before | Restore,
after |
| --- | --- | --- | --- | --- |
| Main-thread CPU time | 2,834 ms (2,717 to 3,007) | 742 ms (658 to 881)
| 1,602 ms (1,378 to 1,837) | 510 ms (470 to 589) |
| Main-thread busy time (event-loop utilization) | 2,938 ms | 858 ms |
1,673 ms | 597 ms |
| Total process CPU time | 6,735 ms | 4,762 ms | 5,564 ms | 3,306 ms |
| Wall time | 3,914 ms (3,656 to 4,253) | 2,580 ms (2,507 to 3,200) |
2,502 ms (1,756 to 2,922) | 1,799 ms (1,508 to 2,492) |
| Longest event-loop delay | 12.6 ms | 6.0 ms | 10.4 ms | 5.4 ms |
| Event-loop delay p99 | 9.2 ms | 2.0 ms | 2.1 ms | 1.8 ms |
| Unrelated request p99 (p50 is 0.2 to 0.3 ms in all runs) | 16.7 ms |
1.3 ms | 4.7 ms | 1.3 ms |
| Unrelated request, slowest | 18.5 ms | 4.5 ms | 10.0 ms | 3.5 ms |
| Peak resident memory | 495 MiB | 408 MiB | 524 MiB | 476 MiB |

200 MiB archive:

| Measure | Save, before | Save, after | Restore, before | Restore,
after |
| --- | --- | --- | --- | --- |
| Main-thread CPU time | 571 ms | 181 ms | 378 ms | 141 ms |
| Main-thread busy time | 602 ms | 236 ms | 395 ms | 167 ms |
| Total process CPU time | 1,235 ms | 935 ms | 1,385 ms | 840 ms |
| Wall time | 842 ms | 679 ms | 538 ms | 382 ms |
| Longest event-loop delay | 12.4 ms | 4.0 ms | 7.3 ms | 2.9 ms |
| Unrelated request p99 | 16.5 ms | 2.0 ms | 1.9 ms | 1.9 ms |
| Peak resident memory | 437 MiB | 398 MiB | 510 MiB | 424 MiB |

How to read these numbers:

- The parent did not block the thread for seconds at one time. It worked
in 1 MiB slices, so the longest single delay was about 12 ms. The cost
was the sum: the thread was busy for about 75% of a 1 GiB save. That sum
is what the profile showed, and it is what decreases.
- A worker thread moves CPU time; it does not remove it. Total CPU time
also decreased here (about 29% for a save, 41% for a restore) because
the workers reuse two buffers in place of one new 1 MiB Buffer for each
chunk, and a restore reads the decrypted archive once in place of twice.
- Wall time decreased because the part digests no longer wait for the
thread that sends the parts, and a restore digests and decrypts at the
same time.
- What remains on the server's thread is network and file streaming:
socket writes of the parts, the download, and the database calls.
- Limits: the machine was shared with other work (1-minute load average
8 to 14 on 18 cores) and timings varied between runs, which is why the
pairs alternate. Earlier runs on a busier machine (load 15 to 28) showed
the same CPU-time ratios with wall times up to 4 times longer for both
implementations. The object store was on loopback without TLS.

Other checks:

- `cd server && pnpm exec vitest run src/storage
src/services/remote-workspace-snapshot-storage.test.ts
src/services/remote-workspace-snapshot-reuse.test.ts
src/__tests__/remote-workspace-file-resources.test.ts` and the 8
`src/__tests__/storage-s3-*.test.ts` files: 249 tests in 15 files pass.
- `tsc --noEmit` over the changed TypeScript files: clean. `pnpm
check:module-boundaries`: pass.
- The pool, bundled with the options of
`server/scripts/bundle-runtime.mjs` and run with plain `node`: encrypt,
decrypt and digest round trip, and the process exits.
- `pnpm verify` exits 2 (broader verification required, because of the
new file in `server/scripts`). Required CI supplies that coverage.
- Not verified: the hosted server after deploy, Linux x86 with Node 24
outside CI, a real object store with TLS, and archives larger than 1
GiB.

## Risks

- This is integrity and confidentiality code. The mitigation is that
each check and its failure is pinned by a test that passes on the parent
implementation and on this branch, and that the format is unchanged, so
objects saved before and after this change restore with either version.
- Order of work in a restore: the ciphertext is now decrypted into
private scratch while the object digest is computed, not after it. No
output is used before the digest, header, key and tag checks pass, and
the scratch directory is removed on every failure. A damaged object is
reported with the same error, but after the decrypt work.
- The staged copy beside the destination (`.workspace-snapshot-*.tmp`)
is now written before the plaintext digest is known, and after the GCM
tag is verified. It is removed on a mismatch. The destination itself is
linked only after the digest matches, as before.
- Resource trade-off: up to 2 more threads, each with about 2 MiB of
buffers and a small heap (64 MB limit), that stay for the life of the
process once used. In exchange the server's thread does about 70% less
work for each snapshot. With more than 2 snapshot passes at one time,
the others wait. A part digest can wait behind a long encryption.
- Run-log timing shifts: `storage.snapshot.digest` is now about 0 ms,
`storage.snapshot.verify_plaintext` includes the copy, and
`storage.snapshot.publish` is only the link. `doc/run-log-events.md`
describes this.
- A worker that stops unexpectedly fails its job with
`file_crypto_worker_exited`. The save or restore then fails as for any
other I/O error, and the next job starts a new worker.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code with
tool use (shell, file edits, local test and benchmark runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 10:11:37 -06:00
DottaandClaude Fable 5.2 5b698b68cf perf(run-events): append a native event in one transaction of two statements (#21131)
One appended native run event cost eight statements on the database
connection (BEGIN, SAVEPOINT, run lock, replay probe, sequence update,
insert, count/max read, COMMIT). The query builder built five of them
again for every event, and each event created two transaction objects.

A native source event now takes the run lock and then runs one statement
that finds a replayed event or reserves the sequence and inserts, and
reads the retained count and maximum for the acknowledgement. The lock
stays a statement of its own so the append reads a snapshot taken after
the lock was granted. The control-plane port passes its controller check
to the append instead of opening an outer transaction, so there is no
SAVEPOINT. The hot statements are fixed SQL text.

Events that wake a queue still register the wake before the write and
only when the event is new. Order, sequence numbers and durability do
not change: every event still commits before its append returns.

Local benchmark (server/scripts/benchmark-run-event-append.ts), one
native event: 8.3 -> 4.6 statements, 589 -> 222 KB allocated,
1.09 -> 0.53 ms CPU, 28 -> 11.5 garbage collections per 1,000 events.
One ordinary event: 105 -> 27 KB allocated.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 09:58:31 -06:00
DottaandPaperclip 5c953c1c1b fix(ui): keep chat pins first and allow drag reordering (#20830)
Keep pinned chat agents first in saved order. Add pointer and keyboard reordering with shared navigation pin controls. Preserve focus and hidden pin positions.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-12 09:49:17 -06:00
DottaandClaude Opus 5.5 ef2b53b675 fix(environments): keep resuming an idle sandbox whose resume timed out (#21113)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on the next wake
> - When the resume call times out, the run failed after 90 seconds with
`reusable_sandbox_resume_failed (cause=transport_timeout)`
> - The provider keeps starting the sandbox after the host call times
out, so the sandbox is usually ready a little later
> - The run gave up after one call, and the task then waited behind a
Retry button
> - This pull request keeps the same run retrying an idle sandbox whose
resume only timed out, up to the provider's start deadline
> - The benefit is that recovery works automatically and a sandbox with
unsaved work is never destroyed

## Linked Issues or Issue Description

Refs #20626, #20828, #21020

**What happened?**
Three Codex runs on Daytona failed before provider dispatch with:
`Reusable sandbox lease "<id>" could not be resumed
(cause=transport_timeout); the lease was preserved and no replacement
was created.` Each run had `executionRecovery: { kind: "bootstrap",
providerWorkStarted: false }`. Each run took 100 to 122 seconds: one
resume call used the whole 90-second budget, so no in-run retry was
possible. The preserved leases were idle. Their last runs had ended
about two hours before, with a confirmed stop-and-retain. The workspaces
were not safe to replace, so the run failed strictly. The automatic
bootstrap retry was then refused ("The automatic retry could not
revalidate the original user continuation."), and all three tasks ended
`blocked` behind a Retry.

**Expected behavior**
A resume that only timed out, before any provider work, on a sandbox
that no live run holds, continues in the same run. The run keeps the
same sandbox and does not create a replacement or destroy the sandbox.

**Steps to reproduce**
1. Let a task's reusable Daytona sandbox stop and stay idle for some
hours (it auto-archives after 60 minutes).
2. Retry the task. The workspace has unsaved work, so replacement is not
safe.
3. The `environmentResumeLease` call takes longer than 90 seconds. The
host call times out (`RPC call "environmentResumeLease" timed out after
90000ms`), the 90-second budget is spent, and the run fails with
`reusable_sandbox_resume_failed`.

**Paperclip version or commit**
Staging at `849f109f` (current `master` has the same code path).

**Deployment mode**
Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases
on.

**Agent adapter(s) involved**
`codex_local` (Codex) on a Daytona sandbox environment. The change is
adapter-independent.

**Root cause**
`server/src/services/environment-runtime.ts`, reusable resume loop: the
first `environmentResumeLease` call got the whole 90-second budget as
its RPC timeout. When that call timed out, the loop saw no time left and
threw. The 90-second budget assumes that the bounded bootstrap retry
resumes the same sandbox later. That retry is not always admitted. A
timeout is different from a refusal: the provider is still starting the
sandbox.

## What Changed

- `reusable-sandbox-resume-replacement.ts`: add
`reusableSandboxLeaseIsIdle` (lease is `released` or `retained` and no
other live run holds it). `reusableSandboxReplacementIsSafe` now uses
it, with the same rules as before. Add
`REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS` (330 seconds, the Daytona start
deadline plus the call margin).
- `environment-runtime.ts`: when the 90-second budget ends on a
`transport_timeout` and the preserved lease is idle, extend the deadline
one time, up to the provider start deadline (at most 330 seconds from
the first attempt). The run keeps retrying the same sandbox with
backoff. Each call still has at most 90 seconds. Refusals, a live owner,
and other failures keep the current behavior. The run never replaces or
destroys the sandbox because of this extension.
- `doc/execution-semantics.md`: document the idle-timeout extension.

## Verification

- Red before the fix, with the new tests in
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`
(staging shape: released lease, finished owner run, workspace with
unsaved work, 300-second provider deadline, host RPC timeout): 2 tests
failed with the staging error `could not be resumed
(cause=transport_timeout); the lease was preserved and no replacement
was created.`
- Green after the fix: 16/16 pass. The timed-out idle sandbox resumes in
the same run with no create or destroy call. If every call times out,
the run stops at or before 330 seconds, makes more than one call, and
keeps the lease `released`. A sandbox that a live run still holds gets
one call and fails at 90 seconds, as before.
- `pnpm exec vitest run src/__tests__/environment-runtime.test.ts
src/services/environment-runtime-reusable-lease-probe.test.ts
src/services/remote-workspace-transport.test.ts`: 378 pass.
- `tsc -p server/tsconfig.json --noEmit` passes.
- `pnpm verify` exits 2 because a doc is a global CI input. Required CI
supplies the broad coverage.
- Not run: a live Daytona resume on staging. No later run attempted the
three preserved leases, so it is inferred, not observed, that the
sandboxes were ready after the host call timed out.

## Risks

- A run can now wait up to 330 seconds in preparation before its first
turn, but only when the resume timed out on an idle sandbox. Before this
change, that run failed and the task waited for a person.
- A provider that is down (not slow) now takes up to 330 seconds to fail
instead of 90 seconds. The failure is the same
`reusable_sandbox_resume_failed`, and the lease is still preserved.
- No persisted fields change. Leases saved by older releases follow the
same rule.
- The refused bootstrap retry admission for task Retry continuations is
a separate gap. This pull request does not change it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:28:32 -06:00
DottaandClaude Opus 5.5 81d2e4d8d9 fix(native): end a run whose finalizer a newer run superseded, so it stops holding its task (#21106)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result, then a finalizer exports the
workspace and applies the result to the task.
> - When a newer run already committed the task's decision, the
finalizer retires the older result as `native_finalization_superseded`.
> - That retirement set the finalizer terminal, but it did not end the
run. The run stayed `running` and held the task and every run queued
behind it, with no way out except a person.
> - A stuck run must never hold a task and its queue forever.
> - This pull request ends the run when its finalizer is retired as
superseded, and it ends runs that an older release left in that state.
> - The benefit is that the task continues with its queued runs by
itself.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. The workspace export then failed
(`remote_workspace_archive_fs_eacces`). The task's last decision came
from another run that was created 42 seconds later but ran earlier. The
finalizer judged this run superseded: it set the finalizer to
`terminal_failure` with `native_finalization_superseded`. The run stayed
`running` with "Recovery needed". Over one hour later it still held the
task, and five queued runs waited behind it. A server upgrade at 12:02Z
did not change it.

**Expected behavior**
A run whose finalizer is retired ends. The task keeps the newer run's
decision, the retired result is never applied, and the queued runs
proceed.

**Steps to reproduce**
1. Accept a native result for a run.
2. Commit a status decision on the same task from a run that was created
later.
3. Let the older run's finalization fail once, so the finalizer records
the failure.
4. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is still `running`.

**Root cause**
`recordRetryableFailure`
(`server/src/services/native-runtime/native-run-finalizer.ts`) projects
a terminal run status only when `exhausted && !supersededByNewerRun`. A
superseded retirement therefore never ended the run. No other path ends
a run whose finalizer is already `terminal_failure`: the reconciler
skips `terminal_failure` rows.

**Paperclip version or commit**
Staging Core `3eca7a6112`. The behavior is the same on master
`482063aeca`.

## What Changed

- `native-run-finalizer.ts`: a superseded retirement now ends the run
with the same status mapping as an exhausted finalization (workspace
failure: `failed`; otherwise the provider's terminal state). It does not
change the task, does not apply the result, and does not keep the
workspace export lease. The run's error says in plain words that a newer
run already decided the task.
- `native-run-finalizer.ts`: add `endSupersededNativeFinalizationRuns`.
It ends a native run that is still `running` while its finalizer is
`terminal_failure` / `native_finalization_superseded`. This covers rows
that an older release wrote. It locks in the finalizer's order
(coordinator, task, run), and re-checks the row under the lock.
- `native-run-projection-work.ts`: the registered native run projection
worker (it loads at startup, pages through all candidates, and wakes on
finalization changes) also selects such runs and ends them with
`endSupersededNativeFinalizationRuns`.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/__tests__/native-finalization-recovery.test.ts`, red on
master and green on this branch:
- "retires an older failed finalizer when a newer run already committed
the issue": the retired run now ends (`succeeded`, `finishedAt` set).
Master: the run stays `running`.
- "ends a run an earlier release left running after retiring its
finalizer as superseded": the shape of the staging row (workspace export
failed, superseded finalizer, run `running`). The repair ends it as
`failed` with the plain message. The task, its decision and the
finalizer stay unchanged, and no status decision is written for the run.
A second pass changes nothing.
- `server/src/__tests__/native-run-projection-work.test.ts`, "ends, at
startup, a run an earlier release left running after retiring its
finalizer as superseded": the registered worker ends the staging-shaped
row at startup and settles its claimed wake. Red without the worker
change.
- Related suites pass: `native-finalization-recovery`,
`native-run-projection-work`, `native-finalization-reconciler-timing`,
`native-reconciliation-timing`, `native-workspace-finalizer`,
`native-workspace-finalizer-failure`, `native-run-finalizer-telemetry`.
- `tsc --noEmit` for `server`.

## Risks

- Low risk. The change only ends runs whose finalizer is already
terminal and superseded. It never applies a result and never changes the
task.
- A superseded run whose workspace export failed now releases its
sandbox lease through the normal settlement instead of waiting forever.
Its result was not applied, because a newer run decided the task.
- Not changed here: the superseded check compares run creation times. A
run created earlier but executed later is still judged superseded. This
PR makes that verdict end the run; it does not change the verdict.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:07:18 -06:00
DottaandClaude Opus 5.5 0a2f618cae fix(native): leave a run another live server is preparing alone, and re-adopt a run a shutdown handed off (#21107)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade replaces the server. The old and the new server serve
together for a few minutes, then the old server gets SIGTERM and hands
its native runs to the next controller.
> - #21046 taught the legacy reaper to leave a run with its owner while
the owner's server process is proven alive. Native runs have a similar
gap.
> - Runtime selection writes a native run's coordinator with no
controller and no lease, and stops the run lease. The executor claims
the coordinator about 20 seconds later. In that window the other
server's sweep sees no owner and marks the run "ownership could not be
verified".
> - When the old server then hands the run off before any Runner
checkpoint exists, no sweep looks at it again. Only the next server boot
does, 15 to 40 minutes later. The task and its queued runs wait.
> - This pull request leaves such a run with its live preparing server,
and lets the serving server re-adopt a handed-off run through the
ordinary restart claim.
> - The benefit is that a run started during an upgrade continues within
one sweep, with no false "blocked" mark and no wait for another upgrade.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, an upgrade started a new server at
12:19:57Z. The old server started a native run and selected its runtime
at 12:20:32Z. At 12:20:35Z the run logged "Native execution ownership
could not be verified; automatic recovery is blocked"
(`observed_owner_unverified`). The old server's executor claimed the run
at 12:20:51Z. The old server got SIGTERM and suspended the run at
12:20:54Z (`awaiting_runner_reattach`). After that the run showed
"Confirming execution" for 29 minutes, and two queued runs waited behind
it. At 12:49:45Z the next upgrade's startup recovery restarted its
bootstrap, and the run succeeded at 12:58Z.

**Expected behavior**
- A server does not mark a run unverified, or recover it, while another
live server is preparing it.
- A run that a graceful shutdown hands to the next controller is
re-adopted by the serving server at once, not at the next boot.

**Steps to reproduce**
1. Seed a native run whose coordinator is `observed` with no controller
and no lease, and whose `controllerProcess` names another server with a
live controller instance lease.
2. Run the recovery sweep on this server. See that the run gets
`native_execution_ownership_unverified`.
3. Claim the coordinator for the other server, then write what its
graceful shutdown writes (`restart_suspended`,
`awaiting_runner_reattach`, lapsed lease) and stop its instance lease.
Give the run a predecessor's Runner PID.
4. Run the sweep again. See that nothing claims the run.

**Root cause**
- `recoverActiveExecutions`
(`server/src/services/heartbeat/recovery.ts`) treats an `observed`
coordinator without a live controller lease as an owner it cannot
verify. Between runtime selection and the claim, the only owner on
record is the run's `controllerProcess`, and the sweep did not check it.
- `listExpiredNativeRunnerAuthorities` (`native-runner-checkpoint.ts`)
selects runs for the restart claim only when they have a controller
checkpoint, or no process identity at all. A run handed off just after
its claim has no checkpoint, and a follow-up carries its predecessor's
Runner PID. The sweep then skips `awaiting_runner_reattach` runs, so
only a server boot looked at them again.

**Paperclip version or commit**
Staging Core `814fe5d20f` (old server) and `3eca7a6112` (new server).
The behavior is the same on master `849f109f62`.

## What Changed

- `recovery.ts`: before it lists expired native runs, the sweep checks
runs in the runtime-selection window (coordinator `observed`, no
controller, no lease, not tracked here). When the run's
`controllerProcess` names another server process that the shared owner
prover proves alive, the sweep leaves the run with that server: it does
not recover it and does not mark it unverified. It re-checks within 30
seconds. After 5 minutes without a claim
(`LEGACY_LIVE_OWNER_SETTLE_GRACE_MS`), or once that server is gone, the
existing rules apply.
- `native-owner-stamps.ts`: add `preparingControllerOwnerStamp` for that
proof.
- `native-runner-checkpoint.ts`: `listExpiredNativeRunnerAuthorities`
also selects a run that a graceful shutdown handed off and whose lease
lapsed: recovery state `awaiting_runner_reattach` or
`awaiting_evidence`, with a `restart_suspended` history entry that no
restart claim followed. It stays selected through ownership waits (the
departing server can still be alive at the first look). `recovery.ts`
schedules a 30-second re-check for an ownership wait that has no retry
of its own. The ordinary restart claim then proves the takeover and
picks the disposition, exactly as the startup pass of a new server does.
A row that an older release already marked unverified is re-adopted too:
the claim clears the mark.
- `doc/execution-semantics.md`: document both rules.
- No schema change. Nothing changes on the path that starts a run or a
turn: the new proof runs only in the recovery sweep, and only for runs
in the window.

## Verification

- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`,
block "a run the old server started during an upgrade overlap". Red on
master, green on this branch:
- "leaves the run with its live preparing server, then re-adopts it
after that server hands it off": two concurrent sweeps leave the run
`running` with no error and do not touch the coordinator. After the old
server claims, hands the run off and stops, one sweep of the new server
re-adopts it and reaches provider dispatch. Master: the run is marked
`native_execution_ownership_unverified`, and after the handoff nothing
claims it.
- "keeps a handed-off run eligible while the departing server is still
alive, and re-adopts it once it exits": the first sweep waits
(`awaiting_evidence`), the sweep after the departing process exits
re-adopts the run. Red with a filter that matches only the last history
entry.
- "re-adopts a handed-off run that an older release already marked
ownership-unverified": the stuck staging shape. Master: nothing claims
it.
- Related suites pass: `heartbeat-native-restart-suspension` (all 109),
`services/heartbeat/recovery.test.ts` (all 103).
- `tsc --noEmit` for `server`.

## Risks

- A run that another live server prepares, but never claims, now stays
`running` for up to 5 minutes before the existing rules apply. That is
the same bound #21046 uses.
- The guard trusts the shared owner prover. It reads "alive" only from a
live controller instance lease or a live PID in the same PID space.
- More handed-off runs reach the restart claim earlier. The claim keeps
all its fencing and evidence checks; this change only makes the serving
server ask the same question a new boot asks.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:06:57 -06:00
DottaandClaude Opus 5.5 a65644d1e1 fix(retry): continue an accepted connected Retry from recovered saved work when its chat replay is refused later (#21080)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks from a connected chat (for example GitHub pull request
reviews) show Retry when a run fails
> - A person's Retry on such a task first asks the chat retry owner to
repeat the exact failed chat request. #21038 made a refusal at the click
continue the task from saved work instead
> - The chat owner can also defer its source check (a busy identity,
endpoint, or agent row lock) and refuse it a few seconds later. Then the
route has already answered 202, and the task Retry receipt was marked
failed with no run
> - The task stayed blocked, and its execution blocker told the person
to send a message, while the interruption projection still offered Retry
> - This pull request makes that late refusal take the same board
continuation as a refusal at the click, and makes the blocker offer
Retry for recovered saved work
> - The benefit is that an accepted Retry always starts work, and the
two projections agree

## Linked Issues or Issue Description

No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a hosted deploy, about 20 blocked tasks of one busy GitHub review
agent (`paperclip_runner`, native remote runs) did not restart after
Retry. `POST /api/issues/:id/retry` returned 202 with `accepted: true`,
a `chatActionId`, and `presentation.state: retry_scheduled`. No run was
created. The task Retry receipt changed to `failed` 5 to 10 seconds
later, and the task stayed blocked. The activity record of each click
(`chat.failed_run_retry_requested`) has `sourceAuthorizationPending:
true`, so the chat owner deferred its source check at the click. No chat
wake row was written, so the chat replay ended before admission. The
last run failed with `native_continuation_requires_reconciliation` /
`runner_remote_process_identity_unavailable`. Its saved work was
recovered (`workspaceRecovery.status: recovered`, remote lease destroyed
with a termination receipt). The projections disagreed: `interruption`
had `canRetry: true`, but `executionBlocker` had `canRetry: false` and
`nextAction: "Saved work recovered. Send a message to continue."`.

Root cause: when the chat retry action settles `failed`,
`releaseUnadmittedTaskRetryInput` in
`server/src/services/chat-channels/task-retry-input.ts` marked the
linked task Retry receipt `failed`. #21038 added a board-continuation
fallback only for a refusal inside the click transaction. A refusal
after a deferred source check (for example 8 lock-busy attempts on the
agent row, about 7 seconds) had no fallback. Separately,
`getExecutionBlocker` hid Retry for every chat-bound task (`canRetry &&
!chatBinding`), and `workspaceRecoveryDisplay` always said "Send a
message to continue" for recovered work.

**Expected behavior**

A Retry that is accepted starts work. If the exact chat replay is
refused after the click, the person's Retry continues the task from its
recovered saved work. The execution blocker and the interruption both
offer Retry, and neither tells the person to send a message to unblock a
failed run.

**Steps to reproduce**

1. Bind a task to a GitHub chat conversation. Make its latest native run
fail with `native_continuation_requires_reconciliation` and recover its
saved work.
2. Hold the chat identity advisory lock while you click Retry, so the
source check is deferred (`sourceAuthorizationPending: true`).
3. Hold a row lock on the agent while the chat retry processes, so the
deferred check fails 8 times.
4. The route returned 202. The chat action settles `failed`, the task
Retry receipt changes to `failed`, and no run starts.

**Paperclip version or commit**

`master` at a8580160 (the deployed revision). Reproduced on f30005440b.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

`paperclip_runner` (native remote runs). The fix is in core Retry code
and is not adapter-specific.

## What Changed

- `server/src/services/task-retry.ts`: new `settleRefusedTaskChatRetry`.
When a chat retry action settles `failed`, the linked task Retry receipt
becomes an ordinary board continuation if all of these are true: it is a
person's Retry (or an automatic Retry that the same person's click has
joined), its source run is still the latest run and its saved work is
recovered (system recovery `complete`, workspace recovery `recovered`),
the clicking person still has task access, the chat wake has no run, and
no saved or unadmitted chat input remains. Any other refusal still
settles the receipt as failed, so the existing source-edited, revoked,
and identity-expiry outcomes do not change. A paused agent does not fail
the receipt: it stays deferred and starts when dispatch allows. The
receipt drops `chatActionId`, stays `deferred_issue_execution`, notifies
the deferred-receipt worker, and records
`issue.retry_continued_from_saved_work`. Otherwise the receipt is failed
as before. `reconcileTaskChatRetryReceipts` uses the same check, so a
crash before settlement is also covered. Its failed write matches the
chat binding it read, so a stale pass cannot fail a receipt that
settlement already converted.
- The task Retry intent now records `initiatedBy` for an automatic
Retry, so an automatic Retry keeps the exact-replay requirement after
the click too.
- `server/src/services/chat-channels/task-retry-input.ts`:
`releaseUnadmittedTaskRetryInput` calls the task owner's settlement
instead of failing the receipt directly.
- `server/src/services/execution-blocker.ts` and
`workspace-recovery-receipt.ts`: for a failed or timed-out run whose
saved work is recovered, the blocker offers Retry also on a chat-bound
task. The text follows actual Retry eligibility: "Saved work recovered.
Retry continues from it." when Retry is offered, "Saved work recovered."
when it is not (for example cleanup quarantine), and "Send a message to
continue" only for a stopped run. Only the task blocker decides Retry.
The run-level projection (`execution-projection.ts`) has no task
context, so it says "Saved work recovered." and never claims Retry.
`execution-projection.ts` uses the same text.
- `ui/src/lib/activity-format.ts`: labels for the new activity.
- `doc/execution-semantics.md`: the late-refusal rule.
- Tests: a new real-service case in
`server/src/__tests__/chat-channels.integration.test.ts` that reproduces
the hosted state (remote saved workspace, destroyed lease receipt,
recovered recovery action, deferred source check, refused replay), for a
person's Retry and for an automatic Retry. A gate case per continuation
condition (unrecovered workspace and revoked access settle failed; a
paused agent stays deferred and runs after resume), a stale-reconcile
race case, and a joined-click case. Two existing text expectations are
updated.

## Verification

- Red before the fix: with only the source changes removed, the new test
fails on the projection (`canRetry: false`, "Send a message to
continue."). With only the Retry settlement removed, it fails with the
receipt `status: "failed"` and no run.
- Green after the fix: `npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "continues an
accepted GitHub task Retry from recovered saved work"` passes 2/2. The
person's Retry starts one queued run that resumes the preserved
workspace, and the automatic Retry still settles as failed.
- Focused suites pass: `task-retry`, `execution-projection`,
`execution-projection-recovery`, `workspace-recovery-retry-copyback`,
`chat-failed-run-retry-boundary`, `workspace-recovery-admission`,
`remote-workspace-recovery-receipt-fallback` (372/372), and other
recovery, blocker and continuation suites.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Staging was read with GET requests only. No POST was sent.

## Risks

- Behavior change: a person's Retry on a connected task that the chat
owner refuses after the click now starts a board continuation. It is the
same continuation #21038 already uses for a refusal at the click, with
the clicking person's authority, and dispatch keeps every normal gate.
- Receipts that already failed on staging stay failed. The next click on
those tasks takes the fixed path. No migration.
- The blocker now reports `canRetry: true` for chat-bound tasks only
when the failed run's saved work is verifiably recovered.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:57:42 -06:00
DottaandClaude Opus 5.5 cc6728e8de fix(recovery): continue upgrade-stopped Retry runs that the sweep refused or skipped (#21108)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Hosted servers are upgraded often. Each upgrade stops the old server
gracefully, and a run that is still preparing ends `interrupted`
(`server_shutdown_interrupted`) with bootstrap evidence that provider
work did not start.
> - #21078 let a run that a person started with Retry continue
automatically after such a stop. After it deployed, a read-only audit of
a staging server found four groups of Retry-started tasks that still did
not continue: some were refused on every sweep, some were never looked
at, and one Retry that a person pressed failed and was not retried.
> - Product rule: an upgrade never leaves a task that needs a manual
Retry, and a Retry a person presses must work. A refusal that cannot
succeed later must not repeat forever.
> - This pull request fixes each cause found, keeps every authorization
check of the continuation gate, and ends the refusal loop with one plain
escalation.
> - The benefit is that Retry-started tasks continue through upgrades,
including tasks earlier releases left waiting, with no manual step.

## Linked Issues or Issue Description

Refs #21078, #21046, #21019, #14987, #19384

**What happened?** On a hosted staging server after #21078 deployed
(read-only audit, run events and task activity):
1. A Retry-started task was refused every ~2 minutes ("The automatic
retry could not revalidate the original user continuation."). While
preparing, its Retry had switched to a fresh isolated workspace because
the saved files could not be restored (activity
`issue.fresh_workspace_selected` by the system, 40 minutes after the
Retry). That rewrote the Retry request's strategy from `continue` to
`fresh_isolated_workspace`, and `readTaskRetryRetryAuthority` accepts
only `continue`.
2. Two tasks whose Retry replaced a native session had no recovery event
for hours. The stranded-task sweep skipped them silently.
3. A task whose Retry selected a fresh workspace at admission
(`explicitUserContinuation.readiness`) was refused on purpose
("fresh-workspace Retries keep their own contracts"), yet that contract
had no path for a shutdown before provider work. The task stayed in
progress after every upgrade.
4. A person pressed Retry. The run failed before provider work with
`reusable_sandbox_resume_failed`, a transient failure that is already in
the bounded bootstrap retry lane. One second later the stranded-task
sweep escalated the task to `blocked`, which cleared the claim. Two
seconds after that, the run's own failure path scheduled its retry and
was refused because the claim no longer named the run.

**Expected behavior:** each of these continues automatically once, under
the person's original Retry. A refusal that cannot succeed later
escalates the task once with a working Retry.

**Root causes:**
- Cases 1 and 3: the continuation gate refused every fresh-workspace
Retry, whether the fresh workspace was chosen at admission or later in
preparation. That includes one that a shutdown stopped before it did
anything.
- Case 2: the sweep's generic skips run before the continuation. Two of
them can match only because of state that the replaced session left. An
unfinished session goal (`agent_task_sessions.goal_status`) is skipped
as "goal recovery owns it". A native finalization row is counted as a
live execution path by `hasActiveExecutionPath`. The person's Retry had
already started a fresh session, so neither owns the task. The audit
could not read the database, so which of the two applied on staging is
not proven; the tests cover both.
- Case 4: a race. The sweep does not wait while a terminal run's own
executor is still settling it, so it escalates before that executor
schedules the run's bounded retry.
- The refusal loop: after a refusal the sweep left the task
`in_progress` and tried again on the next pass.

**Paperclip version or commit:** master `482063aeca`.

## What Changed

- `explicit-native-continuation.ts`: `readTaskRetryRetryAuthority`
renews a fresh-workspace Retry **only** when the parent run was stopped
by a graceful shutdown before provider work
(`isShutdownInterruptedPreparation`). The admitted `readiness` must name
the same Retry and source. A strategy that preparation switched to fresh
is rebuilt exactly as preparation records it: this Retry, its source,
and the source's leases. An automatic retry of the fresh attempt
inherits the selection from its own receipt. The successor's receipt
records the selection under `automaticRetry.readiness`. The successor's
`explicitUserContinuation` keeps its plain shape.
- `fresh-workspace-continuation.ts`: `readFreshWorkspaceReadiness` reads
that receipt for an automatic continuation, so the successor prepares
the same fresh attempt. A remote fresh workspace is keyed by the Retry,
so it is reused rather than created again. `freshReadinessSchema` is
exported and reused.
- `recovery/service.ts`:
- **Shutdown-stopped Retry preparations go first:** at the start of the
stranded sweep, every `in_progress` task whose latest run is a legacy
`server_shutdown_interrupted` preparation (`providerWorkStarted: false`)
carrying `explicitUserContinuation` is offered to the scheduler. It is
skipped when the agent is not invokable, the task has a live execution
path, a board-owned recovery action or a pause hold, or the run is still
settling. The unfinished-goal skip does not apply here, because the
person's Retry started a fresh session.
- `hasActiveExecutionPath` ignores a native finalization that a person's
Retry already replaced (`failureDetail.replacementDenied =
explicit_user_continuation` on a terminal run).
- The sweep skips a task while its latest terminal run is still being
settled: a live execution in this process, or another server's unexpired
controller lease. It checks again when that lease expires.
- A refused continuation (`continuation_user_authorization_missing`)
escalates the task once to `blocked` with the plain notice "Retry needed
… Press Retry to continue." Before, the task stayed in progress and was
refused again on the next pass.
- `heartbeat.ts`: `scheduleRecoveryRetry` reports a refused continuation
to the sweep.
- `doc/execution-semantics.md`: describes these rules.

**Failure classes admitted for a Retry-started run (case 4):** no new
class. The run's own bounded retry lanes are unchanged.
`reusable_sandbox_resume_failed`, `remote_workspace_restore_failed`,
`plugin_worker_unavailable`, `environment_provider_unavailable`,
`repository_catalog_unavailable`, runner revision mismatch and the other
transient bootstrap failures were already retried before provider work.
The fix removes the race that took the claim away from that retry.

**Refusals that remain by design:**
- The task is reassigned, `done` or `cancelled`.
- Another run holds the claim, or newer or live work exists.
- The Retry actor, generation or run changed, or it is a chat or
queued-message Retry.
- The message was edited or deleted.
- The receipt is missing or revoked.
- The run had another interruption cause, or provider work started.
- A fresh-workspace Retry whose run *failed* (not a shutdown) keeps its
one-run contract.
- A readiness that names another Retry or another source.

Each of these now escalates the task once instead of refusing on every
sweep.

## Verification

- `server/src/services/task-retry.test.ts`, new tests. Red on
`origin/master` sources and green on this branch (checked by restoring
the master sources):
- "continues a fresh-workspace Retry that a shutdown stopped before
provider work":
- Runs twice: once with the fresh workspace selected at admission, once
with it selected at preparation.
- Two concurrent sweeps create one successor, which gets the claim, and
dispatch accepts it.
- `readFreshWorkspaceReadiness(successor)` returns the same fresh
attempt.
- A second upgrade stops the successor, and the next successor carries
the same attempt.
- "continues a shutdown-stopped Retry whose replaced session left state
behind": two cases, an unfinished session goal and a native finalization
that the Retry replaced.
- "escalates a refused shutdown continuation once instead of refusing it
on every sweep": two sweeps. The task ends `blocked` with one "Press
Retry to continue" notice and one refusal event.
- "leaves a failed Retry run to its own executor while that executor
still settles it":
- Production shape: `reusable_sandbox_resume_failed` and a live
controller lease from the server that ran the run.
    - The sweep leaves the task and its claim alone.
    - The run's bounded retry is then scheduled and dispatch accepts it.
- "keeps a fresh-workspace Retry's own contract when it does not match":
three refusal cases (other Retry, other source, failed rather than
stopped). Green on both.
- Existing suites pass:
- `task-retry`, `explicit-native-continuation`,
`heartbeat-retry-scheduling`, `heartbeat-process-recovery`,
`interrupted-preparation-hold-settle`,
`heartbeat-issue-liveness-escalation`, `issue-recovery-actions`,
`recovery-deadline-acceptance`, `legacy-continuation-authority`,
`workspace-recovery-admission`, `native-workspace-export-retry`,
`heartbeat-paused-assignee-new-task`, `issue-tree-pause-resume-run`,
`heartbeat-stale-queue-invalidation`. 1,619 tests passed, with one
failure that a change in this branch fixed; rerun of the affected
suites: 560/560.
- The one failure came from the first version of the settling check,
which also waited for this process's own lease after a shutdown drain.
The check now uses this process's live executions and other processes'
leases only.
- `tsc --noEmit` for `server`.
- Required CI covers the full suite.

**After deploy, the next audit should show:**
- Tasks stopped by a shutdown while their Retry was preparing get one
`scheduled_retry` successor on the first sweep. This includes tasks
whose Retry selected a fresh workspace and tasks whose Retry replaced a
native session.
- They show no further "could not revalidate" events.
- A Retry-started run that fails transiently before provider work gets
its bounded retry, and the task is not escalated in between.
- A task whose continuation is truly refused is `blocked` once with
"Retry needed".

## Risks

- **Behavior change:** the sweep now offers shutdown-stopped Retry or
message preparations to the scheduler before its generic skips. All
authorization checks still run at admission, under the task and run row
locks. The scheduler reuses an existing successor, so two servers create
one continuation. The tests run two sweeps at once.
- **Settling wait:** a task whose latest terminal run still has another
server's live controller lease waits until that lease expires (at most
90 s) before the sweep acts. Normal turn starts are unchanged, and
nothing is added before a turn starts.
- **Escalation on refusal:** a refused continuation now moves the task
to `blocked` with a Retry notice, where before it stayed `in_progress`
with no path. Pressing Retry is a new, separately admitted request.
- **Rows from older releases:** no schema change and no backfill. A
shutdown-stopped fresh-workspace Retry is continued once. An older
successor receipt without `automaticRetry.readiness` behaves as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, read-only API reads, Vitest on embedded PostgreSQL) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:41:25 -06:00
DottaandClaude Opus 5.5 1d153f62b6 feat(docker): build the image in the merge queue and promote it on master (#21096)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `docker.yml` builds, probes, gates and signs the standard Core image
for every master push. Paperclip Cloud deploys a Core commit only after
that image is published.
> - Each master push rebuilds the image from scratch: about 10.6 min for
amd64 and 7.8 min for arm64.
> - The owner approved a merge queue for Core. A merge queue tests the
exact commit that master will become, before the merge.
> - This pull request builds and probes the image for that exact commit
in the queue, with no registry credentials. The later master push pushes
those exact bytes instead of rebuilding. When no matching verified build
exists, master builds from source exactly as today.
> - The benefit is that the image legs on master drop from about 8–11
min to about 3–4 min, and nothing changes until the owner turns the
queue on.

## Linked Issues or Issue Description

Refs #21013 (it measured the image path and proposed this plan).

**What existing behavior does this improve?**
Every master push rebuilds the standard image, even when the same commit
could already have been built while it waited to merge.

**Subsystem affected**
`.github/workflows/docker.yml` (`build-and-push` only), new
`.github/workflows/docker-merge-group.yml`, new
`scripts/merge-group-image.mjs`.

**Current behavior**
`build-and-push` always runs the full BuildKit build on master: 639 s
(amd64) and 468 s (arm64) in run 43133035209.

**Proposed behavior**
On a `merge_group` event, build both architectures for the merge-group
commit, probe PID 1, and upload an OCI archive plus a record of commit,
tree, architecture and digest. On the master push for the same commit,
each leg pushes that archive by digest instead of building. Everything
after the leg (probe, exact-source gate, signed-index check,
attestation) is unchanged.

**Reason and benefit**
The build moves before the merge, where it runs in parallel with the
queue's other checks. The master leg only downloads and pushes.

**Breaking changes**
None. Without `merge_group` events, `find` returns no run and the leg
builds as before.

**Additional context**
This PR does not touch the exact-source gate jobs
(`source_verification`, `publication_verification`, `merge-and-push`),
`scripts/watch-release-source.mjs` or `cloud-readiness.yml`. Another
change owns them.

## What Changed

- **`docker-merge-group.yml` (new, `on: merge_group` only).** It runs
only for this repository's master queue. Workflow permissions are `{}`
and the job has `contents: read` only. There are no secrets, no registry
login, no `cache-to`, no `push: true`, no environments, and checkout
uses `persist-credentials: false`. It runs on GitHub-hosted runners, not
the paid AWS fleets. Steps: build `target: production` with the same
build args as master (`PAPERCLIP_BUILD_COMMIT` is the merge-group SHA,
which becomes the master SHA). Read the public registry cache. Write
`type=oci` to a tar. `docker load` it and run
`scripts/assert-orphan-reaping.sh`. Only then write `record.json` and
upload `merge-group-image-<arch>` (3-day retention).
- **`scripts/merge-group-image.mjs` (new).**
- `find` (master push only): selects a completed, successful run of this
workflow, with event `merge_group`, the same `head_sha`, repository and
head repository both `1170821064`, and a `gh-readonly-queue/master/`
head branch. It needs exactly one unexpired artifact for the
architecture.
- `record` (merge group only): reads the archive's `index.json`, checks
each blob's sha256 and that the image is `linux/<arch>`, and writes
`{sha, tree, arch, digest}`.
- `verify` (master push only): re-reads the archive the same way and
requires record SHA = `GITHUB_SHA`, record tree = `git rev-parse
HEAD^{tree}` of the master checkout, same architecture, and archive
digest = recorded digest.
- **`docker.yml` `build-and-push`.**
  - The leg gets `actions: read` so it can download the artifact.
- New steps run after the registry login: find, download, then promote
(`verify`, then `skopeo copy --all --preserve-digests oci-archive:…
docker://ghcr.io/…@<digest>`). The step sets its `digest` output only
after the push succeeds.
- All three new steps are `continue-on-error`. Buildx, labels, metadata
and "Build and push by digest" run when `steps.promote.outputs.digest ==
''`, so any failure builds from source.
- "Select image digest" feeds the existing PID 1 probe and digest
export.
- **Tests (`scripts/merge-group-image.test.mjs`, 8 tests):**
- Run selection rejects another commit, another event, another workflow,
a fork head, another repository, another queue base, and failed,
cancelled or running builds.
  - Artifact selection.
- Archive digest with and without provenance, a tampered blob, the wrong
architecture, and a real tar read through the system `tar`.
  - Record checks for commit, tree, architecture and digest mismatch.
- The CLI refuses `record` outside `merge_group` and `find`/`verify`
outside a canonical master push.
- Static checks of the merge-group workflow: credentials, triggers,
permissions and runners.
- Static checks of the `docker.yml` wiring: promotion is gated and falls
back, the output is set only after the push, and every build step has
the fallback condition.

### How promotion proves it publishes the verified tree

1. `find` accepts only a run whose `head_sha` equals `github.sha` of the
push, meaning the queue landed the commit it tested (unverified: see
"Merge method" in Risks).
2. `verify` compares the record's tree with the tree of the master
checkout, and its commit with `GITHUB_SHA`.
3. The archive is content-addressed. The digest is recomputed from the
archive blobs and must equal the recorded digest. `skopeo
--preserve-digests` to that digest reference fails rather than produce
other bytes.
4. The leg then runs the PID 1 probe on that registry digest.
`merge-and-push` still signs only an index whose Linux images are the
probed digests (#21013), after the unchanged exact-source gate.
5. The pipeline itself is trusted only as master already ran it. `find`
and `verify` run the copy of `scripts/merge-group-image.mjs` from the
push's `before` commit (`git show "$BEFORE:…"`), not the copy the push
brings. `verify` refuses promotion if anything in `before..HEAD` (every
commit of a multi-PR group, not only the tip) changes
`docker-merge-group.yml`, `docker.yml`, the promotion script or the PID
1 probe. Such a push builds from source. So does the first push after
this PR merges, because its `before` has no verifier.

## Verification

- `node --test scripts/merge-group-image.test.mjs`: 10 pass. One builds
a real git history in which a two-PR push changes the pipeline in its
first commit only, and checks that the range check catches it.
- `node --test` on the workflow and contract suites (cloud-readiness,
docker-canary-promotion, post-merge-runner-routing,
lockfile-refresh-workflows, release-verify-workflow, reuse-canary-image,
standard-image-contract, cloud-source-verification, merge-group-image):
492 pass.
- `vitest run server/src/__tests__/docker-build-stamp.test.ts
server/src/__tests__/container-init-reaping.test.ts`: 22 pass.
- `actionlint` on both workflows: no new findings (the 6 remaining are
pre-existing `ls` and word-splitting notes in `docker.yml`).
- Local: `skopeo 1.24.1 copy --all --preserve-digests` between OCI
layouts kept the index digest. It accepted a `docker://…@sha256:`
destination and failed only at the network step.

Cannot be exercised before the queue exists, because no `merge_group`
event fires:
- the merge-group run itself: the hosted-runner build time, disk use
with tar plus `docker load`, `docker load` of a BuildKit OCI tar, and a
~5 GB artifact upload;
- the cross-run artifact download on master;
- whether `skopeo` exists on the runs-on AMI and accepts digest
destinations in its installed version.

Each master-side failure falls back to the full build. A merge-group
failure only blocks the queue if its check is required, which is why
that check is required last in the steps below.

## Risks

- **Inert until enabled.** No `merge_group` events fire before the owner
adds a merge queue. On master, `find` lists runs (one API call per leg,
about 1 s), finds none, and the leg builds as today. The only always-on
change is the leg's `actions: read` permission.
- **Build cache freshness.** A promoted master push writes no
`buildcache-<arch>`, and merge-group builds read but never write it.
With every push promoted, the cache ages, so dependency or weekly
tool-layer changes rebuild in each merge-group run until a fallback
build refreshes the cache. A follow-up can refresh the cache from master
without blocking publication.
- **Publication time is still gate-bound.** The exact-source gate (about
13–14 min after push) still decides when the image publishes. See
"Expected timing".
- **Merge method (unverified).** Promotion needs the master commit to be
exactly the commit the queue tested, because the image stamps the commit
SHA (`PAPERCLIP_BUILD_COMMIT`, `build-info.json`, the provider pack
revision). Matching only the tree would publish an image stamped with
another SHA, so this PR does not relax the check. I expect the queue to
fast-forward master to its tested commit. GitHub's merge-queue page does
not state this, and it cannot be tested before the queue exists. If the
queue lands a different SHA, promotion never matches and every push
builds from source. That is safe but inert. Step 4 below checks this.
- **skopeo on the AWS fleet.** The promote step checks for `skopeo`
first and logs "Merge-queue promotion unavailable" if it is missing,
then builds from source. If that notice appears, add skopeo to the fleet
image.
- **Hosted runner capacity.** amd64 merge-group builds run on 4-core
hosted runners, colder and slower than the AWS fleet. That adds queue
latency, not master latency.

### Settings the owner must enable, in order

1. **Prerequisite, a code change, not a setting.** The `master` ruleset
(id 13619726) requires `ci / verify` and `ci / e2e`. Those come from
`pr.yml`, which triggers only on `pull_request`. A merge queue waits for
required checks on the `merge_group` event too, so every entry would
stall until its timeout. First decide, and merge in a separate PR, how
`ci / verify` and `ci / e2e` report on `merge_group`: a full CI run on
the merge-group commit, or a fast pass that relies on the PR-head
result. Today the ruleset does not require up-to-date branches either.
2. Merge this PR. Nothing changes yet.
3. In Settings → Rules → Rulesets → `master` (13619726), add **Require
merge queue**:
   - merge method Squash;
- build concurrency 5, minimum group size 1, maximum group size 5, wait
0 min;
   - "Require all queue entries to pass required checks";
   - status check timeout 60 min.
Do not add the image build to the required checks yet. Promotion is then
opportunistic: if the merge-group build has not finished when master is
pushed, master builds from source.
4. After the first queued merge, confirm promotion can match. The master
commit SHA must equal the `head_sha` of its `Docker merge-queue build`
run, and the next master `Docker` run's "Promote the merge-queue image"
step must succeed. If the SHAs differ, promotion stays inert under this
merge method. Stop here and report it.
5. After about a week of green `Docker merge-queue build` runs, add
`build (linux/amd64, ubuntu-latest, amd64)` and `build (linux/arm64,
ubuntu-24.04-arm, arm64)` (app: GitHub Actions) to the ruleset's
required status checks. Then every queued commit has a verified image
ready to promote.
6. The existing bypass actors (org admins, repository admins,
integration 5534676) can still merge outside the queue. Those pushes
have no merge-group run and build from source.

### Rollback

- Remove the image checks from the required status checks (step 5), or
remove **Require merge queue** (step 3). Master pushes then find no
merge-group run and build from source. Nothing to migrate.
- Code: revert this PR. Old merge-group artifacts expire after 3 days.
Promoted images are ordinary digests under
`ghcr.io/paperclipai/paperclip` and stay valid.

### Expected timing

- Image leg on master: about 3–4 min (download about 5 GB, push the new
layers by digest, PID 1 probe) instead of 10.6 min (amd64) and 7.8 min
(arm64).
- Push to published: still bounded by the exact-source gate on master,
about 14–15 min (gate plus about 45 s for `merge-and-push`). Today it is
the same bound, but with the legs as a second 11-min path and the
serialized-run queue in front.
- If the gate later accepts proof produced in the merge queue (owned by
the gate change), push to published would be about 4–5 min (leg
promotion plus `merge-and-push`).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only GitHub API and registry reads).
Default reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:10:11 -06:00
DottaandClaude Opus 5.5 f8e1fb0278 perf(docker): take the orphan-reaping pull and pack chmod off the image path (#21013)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `docker.yml` builds and signs the standard Core image for every
master push. Paperclip Cloud deploys a Core commit only after this image
is published and attested.
> - A master commit takes 29 min (p50) and 56 min (p90) to reach a
published image. Every minute here delays a staging fix.
> - The run logs show where the time goes. The exact-source gate (about
14 min) is the critical path today. The build legs finish about 3 min
before it. After the gate, `merge-and-push` spends about 95 s to pull
the 4.9 GB amd64 image for one probe.
> - This pull request moves that probe into the build legs, which run
before the gate. It also removes a duplicate 187 MB layer from the amd64
image.
> - The benefit is about 95–100 s less from gate to signed image on
every master commit, with the same proof. arm64 is now also probed.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The time from a commit landing on master to the signed standard image
(`ghcr.io/paperclipai/paperclip:sha-<full>` plus its attestation).

**Subsystem affected**
`.github/workflows/docker.yml`, `Dockerfile`,
`scripts/standard-image-contract.mjs`.

**Current behavior**
Measured on run 43133035209 (sha 42fd790e, total 999 s):

- Push 01:13:47. The slowest exact-source job, `General tests (server
(7/10))`, ends 01:27:08. `Cloud source verified v1` waits 20 s for a
runner and ends 01:27:44. The watcher sees it at its next 30 s poll,
01:27:58.
- `build-and-push (amd64)` ends 01:24:52 and arm64 ends 01:21:59. Both
have slack before the gate. Run 43134250912 shows the same shape: the
gate opened 01:44:37 and amd64 ended 01:41:53.
- `merge-and-push` starts 01:28:03 and pushes the manifest at 01:28:34.
Then "Verify PID 1 reaps orphaned processes" pulls the full amd64 image
(01:28:34 to 01:30:09, 95 s). "Verify the immutable standard image" runs
the same probe again on the same bytes (5 s). The attestation ends
01:30:22.

Last 60 master pushes (2026-11-11 20:17Z to 2026-11-12 03:21Z): 43
cancelled or superseded, 14 succeeded, 2 failed, 1 in progress. Push to
first published image that contains the commit: p50 29.4 min, p90 55.5
min, max 85.6 min. A successful run takes 22.5 min (p50) from push. The
median gap between pushes is 4.5 min.

Where the amd64 "Build and push by digest" step spends its 582 s:

| Time | Phase | What happens |
| --- | --- | --- |
| 0–8 s | Load | Dockerfile frontend, cache manifest import |
| 8–54 s | Pull cached layers (46 s) | `deps`, `rust-chef`,
`runner-deps`, `runner-build` and `base` are **all cache hits**. The
`build` stage starts `FROM runner-build` and copies `deps`, so BuildKit
downloads and extracts them: a 932 MB node_modules layer (13 s download,
24.5 s extract) and about 650 MB of Rust `target/` |
| 54–108 s | Per-commit build (54 s) | `COPY . .` 10 s, mtime
normalization 11 s, UI 5 s, plugin SDK 3 s, server tsc 18.5 s, runtime
bundle 5.6 s |
| 108–206 s | `runner-provider-pack` (98 s) | Rebuilt every commit: it
is `FROM build` and stamps the commit. pnpm install about 25 s, pack 9
s, **Kimi smoke 37 s (amd64 only)**, archive 28 s |
| parallel | `production` | Tool layer is a cache hit but pulls 1.11 GB
(11 s). `COPY --from=build /app` 24 s |
| 206–294 s | Pack copy and chmod | `COPY` pack 7.5 s. `chmod -R a+rX`
plus UID check **71 s** (arm64: 20.5 s) |
| 294–420 s | Image export (126 s) | 115 s compressing new layers:
`/app` 1.40 GB, pack 1.01 GB, pack archives 1.06 GB, chmod copy 0.19 GB
(compressed). Push 10 s |
| 420–582 s | Cache export (172 s) | `mode=max`: 152 s "preparing"
(compressing the per-commit layers of `build` and
`runner-provider-pack`, which no later commit can reuse), 20 s upload |

What rebuilds every commit, and why: only the `build` stage and
everything after it. `PAPERCLIP_BUILD_COMMIT` is already declared after
the dependency layers, `CLI_TOOLS_CACHE_EPOCH` is weekly, and dependency
install already copies manifests first. The commit ARG is used by the UI
build, the server stamp, and the provider pack, so those must rebuild. A
pnpm `--mount=type=cache` gives nothing here: the runners are ephemeral,
and cache mounts are not exported to the registry cache.

Why amd64 is slower than arm64: both legs run on runs-on AWS 32-core
instances (m6a.8xlarge and m6g.8xlarge), not on `ubuntu-latest`. The
difference is content, not CPU. amd64 has a larger provider pack with
the Kimi smoke test (37 s), a 187 MB chmod copy layer (arm64: 0 MB), a
4.9 GB image (arm64: 2.9 GB), and a 172 s cache export (arm64: 100 s).

**Proposed behavior**
Run the PID 1 probe in each build leg, on its own architecture, against
the digest that leg pushed. Before signing, require that every Linux
image in the standard index is one of those probed digests. Set the
provider pack modes in the stage that writes the pack.

**Reason and benefit**
The probe pull leaves the post-gate path. The leg has about 3 min of
slack before the gate. The chmod change removes about 50 s and 187 MB
from the amd64 leg, which offsets most of the probe pull that the leg
now does.

**Breaking changes**
None. Tags, digests, attestation subject, and signer identity are
unchanged.

**Additional context**
The concurrency proposal and the build-in-PR plan are below, in "Risks".
They are not implemented.

## What Changed

- `docker.yml` `build-and-push`: new step "Verify PID 1 reaps orphaned
processes". It runs `scripts/assert-orphan-reaping.sh` in
`ghcr.io/...@<pushed digest>` with `--platform` set to the leg platform.
It runs before "Export digest", so `merge-and-push` only receives probed
digests.
- `docker.yml` `merge-and-push`: removed the two `docker run` probes
(tag and digest). "Resolve standard image digest" now runs
`standard-image-contract.mjs --resolve "$GITHUB_SHA" --verified-digests
/tmp/digests`.
- `standard-image-contract.mjs`: `--verified-digests <dir>` reads each
probed leg digest from the registry, checks that its bytes hash to that
digest, and collects its Linux image digests. It fails unless every
Linux image in the standard index is in that set. Registry content is
addressed by digest, so this proves that the signed index names exactly
the probed bytes, without a second pull.
- `Dockerfile`: `chmod -R a+rX /provider-pack` now runs in the same
`RUN` that writes the pack. `COPY` keeps the modes. The production stage
keeps the readability check as UID 65534.
- Tests: new contract tests (membership, bare manifest legs, digest
mismatch, empty set, lookup by digest). New workflow test: the probe
runs between the build and "Export digest", and the publisher does not
`docker run`. Updated the provider-pack Dockerfile test.
- `doc/standard-image-contract.md`: describes the per-leg probe and the
membership check.

Expected savings, per master commit:

| Change | Saving | Confidence |
| --- | --- | --- |
| Probe moved out of `merge-and-push` | 95–100 s from gate to
attestation (measured: 95 s pull + 5 s second probe) | High |
| Pack chmod in its own stage | About 50 s and 187 MB in the amd64 leg
(71 s vs 20.5 s on arm64, where the chmod layer is empty) | Medium-high
on size, medium on seconds |
| Net amd64 leg | About +10 to +45 s (probe pull estimated at 60–95 s,
minus about 50 s). Still before the gate | Medium |

## Verification

- `node --test scripts/standard-image-contract.test.mjs`: 10 pass (2
new).
- `node --test` on
`.github/scripts/tests/{cloud-readiness,docker-canary-promotion,post-merge-runner-routing,lockfile-refresh-workflows}.test.mjs`,
`scripts/__tests__/release-verify-workflow.test.mjs`,
`scripts/reuse-canary-image.test.mjs`,
`scripts/standard-image-contract.test.mjs`,
`scripts/cloud-source-verification.test.mjs`: 484 pass.
- `vitest run server/src/__tests__/docker-build-stamp.test.ts
server/src/__tests__/container-init-reaping.test.ts`: 22 pass. Red
check: with the `origin/master` `Dockerfile` and `docker.yml`, the 2 new
or changed assertions fail.
- Real registry, read only: `node scripts/standard-image-contract.mjs
--resolve 42fd790e80 --verified-digests
<dir>` with the two leg digests from run 43133035209 returns the
published index digest. With only the arm64 leg digest, it fails:
"linux/amd64 image sha256:0f0642e6… did not pass the build-leg
orphan-reaping probe".
- `actionlint .github/workflows/docker.yml`: the same 6 shellcheck
findings as `origin/master`, none new.
- Shell syntax check (`sh -n`) of the changed pack `RUN`.

Not exercised: the workflow itself (it runs only on master pushes, and I
did not dispatch it). No local `docker build`: the Docker daemon is not
running on this machine, and a full build does not fit a few minutes.
The leg probe pull time is an estimate from the publisher runner.

## Risks

- The leg now pulls its own image (4.9 GB amd64). AWS legs have about
175 GB free. Hosted tag builds run "Free runner disk" first, and the old
publisher pulled the same image on a hosted runner for tag builds.
- If the registry returns a leg index in an unexpected form, the resolve
step fails closed and no master image is signed. The real-registry check
above used today's exact shapes (OCI index with one platform image and
one attestation manifest per leg).
- Tag builds (`v*`, `nightly/v*`, `beta/v*`) keep the probe, now per
leg. They have no standard-image signing step, and the manifest list is
made only from probed digests.

### Deliberately not changed

- **The exact-source gate.** It is a deliberate "never publish an
unverified image" rule (`watch-release-source.mjs --require-proof` and
`--check-proof`, `doc/cloud-build-readiness.md`). I kept it. The owner
levers are: the slowest `cloud-readiness` shard (`server (7/10)`, 12
min, sets the gate), the 20 s runner wait for `Cloud source verified
v1`, and the watcher's 30 s poll (15 s average delay). I did not shorten
the poll because the org API rate limit was exhausted today.
- **Cache export (172 s amd64, 100 s arm64).** `mode=max` compresses
about 2 GB of per-commit pack layers every build. A fix: a separate
cache-only build of a small aggregate stage (`deps`, `runner-build`, the
production tool layer) with `mode=max`, and the main build with
`cache-from` only. Not done here: the legs are not on the master
critical path today, and I cannot measure it without a master run. It is
worth doing for tag builds or if the gate gets faster.
- **Cached layer pull (46 s).** The `build` stage starts `FROM
runner-build`, so it downloads about 650 MB of Rust `target/` that it
deletes at the end. The server build's `prepare:runner-vendor` uses that
target, so this needs a separate change.
- **Provider pack per commit (98 s amd64).** It rebuilds because it is
`FROM build` and stamps the commit. Its 1.06 GB transport archives also
duplicate the 1.01 GB pack inside the image. Keying the pack on its own
inputs is a product-contract change.

### Concurrency proposal (not implemented)

Today one run holds `docker-${{ github.ref }}` for 17–23 min. Most of
that is waiting for the gate, not building. A newer commit waits for
that run before its build starts (run 43134250912 waited 6.4 min).
Proposal:

1. Remove the workflow-level group. Add job-level groups:
`build-and-push` uses `docker-build-${{ github.ref }}-${{ matrix.arch
}}` with `cancel-in-progress: false`. At most one build per architecture
runs and the newest commit replaces the pending one. So not every commit
is built.
2. `merge-and-push` uses `docker-publish-${{ github.ref }}` with
`cancel-in-progress: false`. It skips publication when a newer master
commit already has a signed image, so the image order stays monotonic.
3. A superseded run must not keep its source watcher alive, or the API
load grows with each push. Fold the early watcher into the gate job, or
end it when its build job is cancelled.

Effect: the newest commit starts building at push, so its image
publishes at about gate time plus 45 s, instead of after the previous
run ends. That removes the 0–17 min queue wait, which is most of the gap
between the p50 of 29 min and the gate time of about 14 min.

### Plan: build in PR CI and promote on merge (not in this PR)

What must change:
1. Turn on the GitHub merge queue for `master`. The queue tests a
`gh-readonly-queue/...` commit and then fast-forwards `master` to that
exact commit, so the SHA stamped in the image (`PAPERCLIP_BUILD_COMMIT`,
`build-info.json`, the provider pack `runnerSourceRevision`, the UI
bundle) is the final master SHA.
2. On `merge_group`: build both architectures with **no secrets and no
registry credentials**, because the code under test is not yet on
master. Write each image as an OCI layout artifact. A second job that
runs no PR code pushes those artifacts by digest to a staging
repository, keyed by the merge-group SHA.
3. Run `cloud-readiness` on `merge_group` for the same SHA, so
exact-source proof exists before merge.
4. On the master push, `docker.yml` looks up the staged digests for
`github.sha`, checks that the tree and SHA match, retags them as
`sha-<full>`, and signs with the same identity
(`docker.yml@refs/heads/master`). If no staged image exists, it falls
back to the current build.

Guarantees to keep: never publish or sign an image without exact-source
proof for that SHA. The signer identity and the attested digest stay the
same. No secrets in any job that runs non-CODEOWNER code. The PID 1
probe runs on both architectures before signing. Superseded queue
entries do not publish.

Expected saving: from master push to signed image, about 1–2 min
(lookup, retag, attest) instead of 29 min p50 and 56 min p90 today. The
work moves before the merge, so the merge itself waits for the queue
(about the current gate time of 14 min). A staging fix then reaches
Cloud about 25 min sooner at the median.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used via Claude Code
with tool use (shell, file edits, read-only GitHub API and registry
reads). Default reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:29:11 -06:00
DottaandClaude Opus 5.5 533f135780 fix(native): send a re-wake whose crashed predecessor's warm Runner cannot be adopted to a fresh sandbox instead of failing it (#21001)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a warm remote sandbox leaves its Runner `ready` in
the sandbox. The next run of the task adopts that Runner live, after it
proves that the previous owner finished
> - When the server is killed after a run parked on a question, the warm
owner lived only in the killed server's memory. The Runner in the
sandbox was never suspended
> - A re-wake on the replacement server resumed that sandbox and tried
live adoption. A proof failed, and the new run failed with
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate` and needed a Retry, although
it did no provider work
> - #20863 already sends such a run to a fresh sandbox when the
predecessor's Runner cannot be retired, but the live-adoption path of a
resumed sandbox (`remote-warm-session-recovery.ts`) was not connected to
it, and the failed run's inherited Runner identity blocked the
"workspace unchanged" proof that the fallback needs
> - This pull request connects the two: a refusal of live adoption after
the predecessor is durably terminal with an accepted result goes to the
#20863 fresh-sandbox path, and an inherited Runner identity no longer
counts as a process of the follow-up
> - The benefit is that a re-wake seconds after a crash restart
continues without a Retry

## Linked Issues or Issue Description

Refs #20863, #20844, #20928, #20979, #20953, #20972

**What happened?**
Restart-chaos wave `w-20261112T012841-s7000` (master `4b24352712`,
volume-free mode, Daytona warm, Codex, 10 tasks, K1 and K15 with a fresh
home and overlap), task t08 (W2Q):
- Run `58f57bf1` asked its question and succeeded at 01:29:56. Its warm
owner stayed in the server's memory.
- K1 killed the server at 01:30:06. The replacement was ready at
01:30:15.
- A board comment woke the task at 01:30:17. Run `8f812df6` started at
01:30:19 and resumed the same sandbox (`run.startup.reuse`
`verified_reuse`).
- The first live-adoption inspection passed. The second check
(`revalidate`, in backend setup, before the Runner attach) refused at
01:30:21 (stack: `remote-warm-session-recovery.ts:217` from
`revalidate`). The run failed `native_session_interrupted` with
`INSTRUCTION_COLLECTION_UNAVAILABLE`.
- The failed run kept its full save because the unchanged-files proof
refused with `process_recorded`: the run carried its predecessor's
Runner process identity. Retry (`6968dc0f`) worked.

**Expected behavior**
The predecessor finished with an accepted result before the kill, and
the new run did no provider work. The task must continue without a
Retry.

**Steps to reproduce**
1. Finish a warm remote run that parks on a question.
2. Kill the server (SIGKILL) and start a replacement.
3. Wake the task within seconds. The new run resumes the warm sandbox.
4. Any live-adoption proof fails, and the run fails terminally with
`prior_owner_terminal_state_indeterminate`.

**Which proof failed**
The old refusal had no reason. The wave evidence proves that it was
`revalidate`, after a passed inspection, and not the SIGTERM of the next
K15 fault (that server logged "shutdown signal received" only at
01:31:09). `revalidate` checks the predecessor's coordinator row, the
current run's coordinator row and lease, and the task. The evidence
cannot tell which one changed in the first seconds after the restart.
This pull request names the failed proof
(`warm_recovery_refused=<reason>`), so the next occurrence identifies
it. The fix covers the class: every refusal that concerns the
predecessor or the sandbox.

## What Changed

- `server/src/services/native-runtime/remote-warm-session-recovery.ts`:
- Each refusal names its proof from a closed list:
`identity_incomplete`, `target_not_resumed_sandbox`, `owner_unsettled`,
`execution_mismatch`, `checkpoint_unsettled`, `journal_unsettled`,
`controller_not_stopped`, `prior_lease_mismatch`, `owner_changed`,
`current_lease_changed`, `current_run_not_admitted`,
`current_owner_changed`.
- After the predecessor's run is `succeeded` and its coordinator is
`committed` with a result and no lease, a refusal about the predecessor,
its journal or the sandbox throws
`NativePredecessorRetirementUnprovenError` (predecessor run, sandbox).
The existing #20863 path then marks the sandbox `replaceBeforeReuse` and
re-queues the run in the uncharged `interrupted_preparation` lane. The
fresh sandbox is used only when the workspace save is clean and the
harness backup is verified (unchanged #20863 safety rules); otherwise
the sandbox is resumed as before.
- A refusal because the new run is no longer admitted (stopped,
reassigned, paused, company inactive) or no longer owned by this
controller stays an ordinary failure, so a fallback cannot become a path
around Stop or reassignment.
- `server/src/services/remote-workspace-unchanged-proof.ts`: a run whose
recorded Runner process identity is exactly a settled predecessor's
identity of the same session (`carriesInheritedRunnerIdentity`, from
#20979), with no `native.process_identity_recorded` event of its own, is
no longer refused with `process_recorded`. A provider session of its own
(`sessionIdAfter`) and every other check still refuse.
- `doc/execution-semantics.md`: document both rules.

## Verification

- Red on `origin/master` (`8818402e7a`):
- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts
-t "crash restart"`: `AssertionError: expected Error:
runner_state_identity_mismatch: pr… to be an instance of
NativePredecessorRetirementUnprovenError`.
- `npx vitest run
src/services/remote-workspace-unchanged-release.test.ts -t "follow-up
that carries"`: `expected false to be true` for the inherited-identity
case; the own-identity and own-record cases passed (still refused).
- Green with the fix: `native-controller-recovery.integration`,
`remote-workspace-unchanged-release`, `native-recovered-owner-followup`
and `heartbeat-native-predispatch-retry` pass (149 tests). Also run:
`native-session-executor.test`, `explicit-native-continuation.test`,
`task-workspace-binding.test`,
`native-runner-restart-recovery.integration.test` (see the PR comment
for the result). `tsc --noEmit` for the server passes.
- The new integration case is the staging shape: a predecessor that
parked on a question (unanswered `semantic_tool.input` before its
terminal), a replacement controller, a resumed sandbox. It checks the
journal refusal, a predecessor row taken by another worker between
inspection and revalidation, a changed current lease, and that a
cancelled current run is not sent to the fallback.
- Chaos: the before/after rerun of seed 7000 is in a PR comment.

## Risks

- The fallback starts the next attempt on a fresh sandbox instead of
reusing the warm Runner. That costs one cold start, only after live
adoption is refused.
- The unchanged-files proof accepts an inherited Runner identity. The
identity must match a settled predecessor of the same session exactly
(PID, group and start time), and the run must have no own identity
record, no execution event, no runner checkpoint, no workspace operation
and no result. These are the same facts #20979 uses for restart
recovery.
- Error text of this refusal gains `; warm_recovery_refused=<reason>`.
Callers match the `runner_state_identity_mismatch: ` prefix only.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:08:03 -06:00
DottaandClaude Opus 5.5 c4927e1ec2 fix(recovery): continue a Retry-started run that a server shutdown stopped while preparing (#21078)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade stops the old server gracefully. A legacy run that is
still preparing ends `interrupted` with `server_shutdown_interrupted`.
The new server must continue it automatically (#21019, #21046).
> - A run that a person started with Retry (or a message) carries
`explicitUserContinuation`. Its automatic retries need their own receipt
from `admitExplicitContinuationRetry` (#14987, #19384).
> - That admission refused every shutdown-interrupted Retry run. The
task then stayed in progress with no run, and the stranded-task sweep
refused it again about every 2.5 minutes, forever. A person had to press
Retry.
> - An upgrade is routine. It must never leave a task that needs a
manual Retry.
> - This pull request lets the person's original authorization cover a
continuation after a shutdown that stopped the run before provider work
started. All other checks of the gate stay.
> - The benefit is that Retry-started tasks continue through an upgrade,
and tasks that earlier releases left waiting continue on their own after
deploy.

## Linked Issues or Issue Description

Refs #21019, #21046, #14987, #19384

**What happened?**
On a hosted staging server, each automatic upgrade stopped legacy runs
that were still preparing (`executionRecovery {kind: bootstrap,
providerWorkStarted: false}`). When the run came from a person's Retry
(wake reason `task_retry`), the continuation was refused with "The
automatic retry could not revalidate the original user continuation."
(`continuation_user_authorization_missing`). The task stayed
`in_progress` with no run and no next retry. Runs from older upgrades
were refused again by each sweep. A run from the same upgrade that did
not come from a Retry got its continuation in the
`interrupted_preparation` lane.

**Expected behavior**
A Retry-started run that a graceful shutdown stopped before provider
work continues automatically, once, under the person's original Retry.
Tasks that an earlier release already refused continue after deploy with
no manual step.

**Root cause**
Three conditions refused it (`server/src/services/heartbeat/retries.ts`,
`server/src/services/explicit-native-continuation.ts`):
1. The scheduler admits an explicit continuation only for the
`transient_failure`, `server_restart`, preparation-wait and
agent-directory lanes. A shutdown-interrupted preparation with unspent
failure budget uses `interrupted_preparation`, so it was refused before
admission ran.
2. `admitExplicitContinuationRetry` accepted only a `failed` or
`timed_out` parent. These parents are `interrupted`.
3. It required `task.executionRunId` to be the parent. The refusal
released the claim (`releaseIssueExecutionAndPromote`), so every later
sweep was refused by this check too.

**Paperclip version or commit**
master `f30005440b`.

## What Changed

- `retries.ts`: an explicit continuation may also use the
`interrupted_preparation` lane. Lane choice is unchanged (see below).
- `explicit-native-continuation.ts`: `admitExplicitContinuationRetry`
also accepts a parent that is `interrupted` by a graceful shutdown
(`isServerReplacementInterruption`, so `server_shutdown_interrupted`)
with bootstrap evidence `providerWorkStarted: false`
(`isShutdownInterruptedPreparation`).
- For such a parent only, a released claim (`executionRunId` null) is
accepted. Admission restores the claim to the parent under the task
lock, as the shutdown path leaves it, so the scheduler passes the claim
to the successor and dispatch accepts it. If the parent validation then
refuses, admission releases the claim again in the same transaction. A
claim that another run holds still refuses.
- `doc/execution-semantics.md`: describe the rule and the conditions
that still refuse. Also apply the sentence that #19384 left pending (a
plain task Retry is renewable).
- No schema change, no backfill. Nothing changes on the path that starts
a turn.

**Which lane a graceful-shutdown legacy preparation uses (after
#21046).** `scheduleBoundedRetryForRun` sets `interruptedPreparation`
for an `interrupted` run with the bootstrap evidence.
`usesServerRestartLane(run, true)` returns false while the failure
budget is unspent, so the run uses `interrupted_preparation`: it does
not spend the failure budget, the first continuation is due at once, and
it is bounded by `INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`. With a
spent failure budget it uses `server_restart`. This is the intended
design (#21046 comment on `usesServerRestartLane`), so no lane change.
The `server_restart` lane is for `process_lost` replaced preparations
and for runs past dispatch.

**Conditions that still refuse** (each has a test or an existing test):
- The task has another assignee, or is `done` or `cancelled`.
- Another run holds the task's claim.
- Newer work on the task (a run created after the parent, a retry child,
or a live run).
- The Retry request changed: other actor, not admitted, other
generation, bound to another run, a chat Retry, a fresh-workspace Retry.
A message continuation with an edited or deleted message.
- A missing or revoked receipt (dispatch validation of the parent
fails).
- An execution blocker on the task, or a live adapter controller for the
parent.
- An `interrupted` parent with another cause (for example an operator
interrupt), or with no bootstrap evidence, or with `providerWorkStarted:
true`. A parent that failed after provider work keeps the existing rules
(`failed`/`timed_out` only).
- Native runtime parents (unchanged, see below).

**Native runtime.** A native `paperclip_runner` run is not interrupted
by a graceful shutdown: `drainRunningRunsForShutdown` suspends it and
the next server reattaches it (`restart_suspended`). It does not reach
this gate. Native replacement paths (`native-safe-replacement`,
`native-provider-capacity-retry`) remove `explicitUserContinuation` from
the successor and use their own contract. So the native path does not
have this refusal, and this pull request does not change it.

## Verification

- `server/src/services/task-retry.test.ts`, new tests. Red on
`origin/master` (no successor), green on this branch:
- "continues a Retry-admitted run that a graceful shutdown stopped while
it was preparing": production shape. A legacy run admitted by a task
Retry, `preparing`, owned by this server, with bootstrap evidence.
`drainRunningRunsForShutdown("SIGTERM")` interrupts it with
`server_shutdown_interrupted` and schedules one
`interrupted_preparation` successor (`failureRetries: 0`,
`interruptedPreparations: 1`) with its own receipt. The claim passes to
the successor and dispatch accepts it. A concurrent stranded sweep and a
second schedule call reuse the same successor.
- "continues a task an earlier release left waiting after refusing its
shutdown continuation" (a direct Retry run, and an automatic retry of a
Retry run): the row and task as an older release left them (claim
released). Two concurrent stranded sweeps schedule exactly one
successor; the claim passes to it; dispatch accepts it.
- "does not continue a shutdown-interrupted Retry run without its
original authority" for: provider work started, no bootstrap proof,
another interruption cause, claim held by another run, reassigned,
superseded, other Retry actor, cancelled task. No successor, no receipt,
and the claim is left as it was.
- I confirmed that each source change is needed: without the lane change
both main tests fail; without the released-claim rule the older-release
test fails.
- Suites: `task-retry`, `explicit-native-continuation`,
`legacy-explicit-continuation-recovery`, `quarantined-task-retry`
(587/587); `heartbeat-process-recovery`, `heartbeat-retry-scheduling`
(520/520).
- `tsc --noEmit` for `server`.
- `pnpm verify` selects a broad run (exit 2) because `retries.ts` is
central; required CI covers that.

**After deploy, the next audit should show:** a Retry-started run that a
shutdown stops while preparing gets a `scheduled_retry` successor with
`scheduledRetryReason: interrupted_preparation` (or `server_restart`
with a spent failure budget) and no
`continuation_user_authorization_missing` event. The tasks that earlier
upgrades left waiting get one such successor on the first stranded sweep
of the new server, with the task claim on the successor.

## Risks

- Behavior change: a shutdown-interrupted preparation of a Retry or
message turn now continues automatically, within the existing uncharged
and bounded lanes. The run did no provider work, so there is nothing to
reconcile.
- Rows saved by older releases: a refused row whose claim was released
is continued once. If another run took the claim, or newer work exists,
it stays refused.
- Two servers at once: admission runs in the scheduler transaction under
the task and run row locks, and the scheduler reuses an existing
successor (`retryOfRunId`). The tests run two sweeps concurrently and
get one successor. An old server that still refuses cannot clear the
claim after the new server moved it to the successor (its release
matches only the stopped run).
- The claim restore writes `executionRunId` for a terminal run inside
the scheduling transaction. If admission refuses after the restore, it
releases the claim again before the transaction commits.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, Vitest on embedded PostgreSQL) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:02:56 -06:00
DottaandClaude Opus 5.5 f30005440b fix(recovery): reconnect a live remote Runner with its admitted skills instead of re-downloading them (#21064)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner sandbox. A server upgrade
detaches each running turn, and the new server must reconnect to its
Runner (restart recovery).
> - A reconnect runs the full turn preparation before it contacts the
Runner. On a new server, skill preparation downloads every saved skill
version again, under one lock for the whole company.
> - After an upgrade, many reconnects wait behind each other's
downloads. The Runner stops answering after about 2 minutes, so a late
reconnect loses the turn.
> - This pull request lets a reconnect to the same run's live remote
Runner use the exact saved skill versions, without downloading them and
without the lock.
> - The benefit is that skill preparation no longer adds to the
reconnect delay, and it does not grow with the number of reconnects.
Normal turn starts do not change.

## Linked Issues or Issue Description

**What happened?**
After an automatic staging upgrade, runs that the new server reached
late failed. The measured time from detach to reconnect was 114–152 s
for the failed runs. Turn preparation took 30–130 s, and skill
preparation alone took 46 s on one run.

**Expected behavior**
A reconnect to a live Runner reaches it quickly. Its speed must not
depend on how many other runs prepare skills at the same time.

**Steps to reproduce**
1. Run a remote Runner turn for each of many tasks in one company.
2. Upgrade the server while those turns run, so the new server starts
with an empty skill cache.
3. See that each reconnect waits in
`heartbeat.prepare.configuration_skills` for the company skill cache
lock and downloads the skills again.

**Paperclip version or commit**
master at 4228bcf89c

**Additional context**
Root cause: `withRuntimeSkillEntries` holds the company-wide skill cache
lock (a Postgres advisory lock) while it materializes each saved skill
version. A new server has an empty skill cache, so each reconnect
downloads its skills again, one run at a time. The Runner already has
these files, and the reconnect path does not read them.

## What Changed

- `company-skills.ts`: new `admittedVersionsOnly` option. It still
checks that each saved version exists and that the agent can read the
skill. It does not materialize the files or take the company skill cache
lock.
- `heartbeat/live-runner-adoption.ts`: `adoptsLiveRemoteRunner` is true
only for a `reattach_remote_runner` claim for this run, with the claim's
own lease owner, on a native run.
- `heartbeat/execution-configuration.ts`: uses the option only when all
of these are true: the reconnect gate is true, the run has its saved
native input with a runtime context, and the run has its saved skill
versions. In that case it times the phase as
`configuration_skills_adopted` and adds one lifecycle run event
(`skillPreparation: "reused_admitted"`).
- `heartbeat.ts`: passes the gate to execution configuration.
- `native-runtime/runtime-context.ts` and `native-session-executor.ts`:
on the same reconnect to a remote Runner filesystem, the Runner
controller describes the Runner's skills by their admitted digests. It
rehydrates the pinned skill files (which takes the same lock) only if it
must launch a Runner process and stage them. All other starts rehydrate
as before.
- `doc/execution-semantics.md`: describes the new behavior.

Not in this PR (by design): a separate "dispatch-only" reconnect path,
and a new startup scheduling lane. The native executor needs about 40
values from preparation (runner environment, MCP token, managed AI
credentials, the reattached execution target, run output). Connector and
instruction preparation also feed the session fingerprint. That cut is
not mechanical, so this PR removes the largest measured single cost
only.

Other `heartbeat.prepare.*` costs that stay on the reconnect path (use
post-upgrade timings to choose the next cut):
- `configuration_secrets`, `configuration_connector_assignments`,
`configuration_connection_instructions`,
`configuration_connector_skills`/`_delivery`,
`configuration_agent_identity`, `configuration_ai_runtime`
- `environment_selection`, `workspace` (retained lease resume and
realization), `session`, `task_context`, `run_identity`
- `github_environment`, `github_launchers`, `run_output`,
`runtime_services`, `instructions`, `runtime_selection` (including
`runtime_selection.skill_assets` and `runtime_selection.workspace_sync`)
- The wait from detach to "run started" (queue and controller-preparing
waits) is outside `heartbeat.prepare.*`.

## Verification

- New test `reconnects to a live remote Runner with its admitted skills
without re-downloading them or waiting on the cache`
(`server/src/__tests__/heartbeat-execution-configuration.test.ts`):
- It admits a run, deletes its skill cache (like a new server), and
holds the company cache lock.
- The reconnect preparation finishes while the lock is held. It returns
the same key, runtime name and version. The session fingerprint is
identical. The cache is not created again.
- Normal preparation of the same saved selection still waits for the
lock and materializes the files ("normal start unchanged").
- Red on master source: `A live Runner reattach must not queue behind
skill downloads: expected null not to be null`. Green with this change.
- New test in `managed-skill-content.test.ts`: with the cache lock held
and an empty cache, `prepareRuntimeContextSource` for a reconnect
returns at once with the admitted context. Its staging step and a normal
start still wait for the lock and then rehydrate the same digests.
- New `server/src/services/heartbeat/live-runner-adoption.test.ts`: the
gate is false for a normal start, a scheduled resume, a dead-Runner
resume, an incomplete bootstrap, another run's claim, a different lease
owner, and a legacy run.
- `npx vitest run
src/__tests__/heartbeat-execution-configuration.test.ts
src/__tests__/skill-version-cache.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/heartbeat/live-runner-adoption.test.ts`: 289 passed, 14
skipped. After the controller change: restart-recovery, diskless
restart, runtime skill preservation, managed skill content, session
executor and execution configuration suites: 1686 passed, 14 skipped.
- `tsc --noEmit -p server/tsconfig.json`: pass.
- `pnpm verify` asks for broader coverage (exit 2). Required CI supplies
it.

After deploy, a reconnect run shows
`heartbeat.prepare.configuration_skills_adopted` and the
`reused_admitted` event, and no `skills.cache.acquire` or
`skills.version.materialize` phases.

## Risks

- Low. The option applies only to a same-run reconnect of a retained
remote Runner that has a saved runtime context and saved skill versions.
Every other start keeps the current path.
- Access revocation and deleted versions still fail the reconnect, as
before. A failure to download the files on this server no longer fails a
live reconnect, because the Runner already holds them.
- Nothing new is saved, so rows from older releases, two live servers,
and a deploy during a reconnect behave as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code search, editing, shell, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:58:01 -06:00
DottaandClaude Fable 5.2 8e956da1cf fix(native): keep a live reattached Runner when its restart recovery read times out (#21053)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes (Daytona) use a detached Runner. A
server replacement detaches the controller and leaves the Runner alive.
The new server reattaches to it and continues the same turn
> - On staging, about half of the in-flight runs at each upgrade lost
their turn. The new controller reattached to the live Runner (same
process verified), its recovery read timed out (`PRP command
session.snapshot timed out`), and the failure cleanup killed the Runner
> - The next same-run reattachment then found `process_not_running`.
#21039 continues these runs in a fresh session, but the in-flight turn
is lost and its outcome is unknown
> - The cause of the kill is in the Codex driver's `recoverSession`: its
cleanup closed the transport without retention. For a reattached Runner
that means SIGKILL after the close deadline. #21023 already keeps the
Runner for a failed open session, but not for a failed recovery
> - This pull request keeps the live Runner when a recovery fails or is
aborted on its way to it. Identity refusals and protocol integrity
faults still stop it
> - The benefit is that a slow reattachment no longer destroys a turn
that was still running

## Linked Issues or Issue Description

Refs #21023, #20923, #21039, #21052

**What happened?**
Staging replaced its server at about 07:10Z with 17 remote native runs
in flight. For each, the old server logged "Server shutdown suspended
native controller ownership without cancelling provider work". The new
server reattached. Example run: the Runner process (same pid and start
time) was verified at 07:12:37 and the transport authenticated at
07:12:32. At 07:13:28 the recovery failed with
`native_session_recovery_failed: ... PRP command session.snapshot timed
out`, and a same-run resume was saved. At 07:14:12 the same-run
reattachment found `process_verification` / `process_not_running`, with
the Runner journal still `ready` and no exit record from the Runner
itself. The run failed with `runner_remote_process_identity_unavailable`
/ `native_continuation_requires_reconciliation`. 90 of 93 such failures
in two hours of upgrades had this sequence.

**What did you expect to happen?**
A recovery that times out against a live Runner is a transient failure.
The same-run reattachment should find the Runner alive and adopt it
again. Killing the Runner is only for a deliberate stop, an identity
refusal, or an integrity fault.

**Root cause**

`packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts`
`recoverSession`: the `catch` block and the abort handler call
`bootstrapCancellation(...).close()`, which calls `transport.close()`
without `retainUnsettledRemoteRunner`. In `runnerd-codex-transport.ts`
`#closeOnce`, an unresponsive adopted Runner that is not retained gets
`SIGKILL` after the close deadline. The Codex driver serves remote
Codex, OpenCode and ACPX runs, which matches the failures seen on
staging for all three driver kinds.

## What Changed

- `codex-app-server-driver-impl.ts`: `recoverSession` classifies a
thrown recovery failure. A transient one (`provider_transport_failed`, a
`PRP command ... timed out`, or the transport closing while waiting) now
detaches this controller with the transport's restart detach
(`detachControllerForRestart`). That path queues no `turn.stop` and no
`runner.suspend` and sends no signal, the same as a server restart. The
Runner and its in-flight turn stay for the same run's next reattachment.
An abort (a Stop or the recovery deadline), any message that names a
mismatch, a different session, identity, authority, integrity or
incompatibility, any unclassified failure, any returned identity
refusal, and a protocol integrity fault still close and stop the Runner.
`openSession` is unchanged.
- `codex-app-server-driver.recovery.test.ts`: the staging message
(`codex_history_read_failed: ... provider_transport_failed: PRP command
session.snapshot timed out`), and a plain snapshot timeout now detach
and never close. An abort of a hung read still closes and stops the
Runner. A thrown different-session refusal,
`native_adopted_provider_identity_mismatch`, an identity refusal that
also mentions a timeout, and an unclassified failure still close without
retention. The existing integrity-fault cases check that the close does
not retain the Runner.
- `doc/execution-semantics.md`: describes the rule.

## Verification

- Red on `origin/master` with only the new tests: `cd
packages/paperclip-runner && npx vitest run
src/drivers/codex/codex-app-server-driver.recovery.test.ts` fails 2 of
69 (the two detach cases; master calls `close()` and stops the Runner).
- Green with the fix: 69 of 69.
- `npx vitest run src/drivers/codex src/native-session-runtime.test.ts
src/backends src/live/runnerd-codex-transport-remote-retention.test.ts
--maxWorkers=4` passed except two timing tests (Codex command
resolution, background cleanup containment) that pass alone and do not
touch this code.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes.
- `detachControllerForRestart` is the existing restart path. It stops
the pump, disconnects the Runner, and drains the controller journal. It
queues no command and never signals the process.
- Not run: a live Daytona restart. The next post-upgrade audit should
show fewer `runner_remote_process_identity_unavailable` failures after a
`session.snapshot timed out` resume. Instead, the run log should show
"controller authority detached for restart; durable runner left alive"
and then a successful same-run reattachment.

## Risks

- Medium. A Runner that the recovery could not reach now stays alive,
with its turn, for the same-run reattachment. This is the state a server
restart already leaves. The same-run reattachment adopts it, or a Stop
cleanup or saved-work recovery stops it.
- If the Runner stays unresponsive across every same-run attempt, the
run no longer gets the `process_not_running` proof that #21039 uses for
its automatic fresh-session continuation. The run then keeps its visible
Retry. The staging data shows that reattachments reaching the Runner
sooner succeed, so a later attempt is expected to succeed. This is not
proven against a live sandbox.
- An abort of the recovery (a Stop or the recovery deadline) is
unchanged: it closes and stops the Runner.
`plugin-runner-release-resume.integration.test.ts` (14 of 14) covers a
Stop during an adopted resume.
- No turn is replayed and no second Runner is launched. Adopting an
existing Runner never launches a new one.
- Previous-release story: no persisted state changes. The decision is
made in the process that runs the recovery, so a deploy during a run
changes nothing.
- No wait is added before a normal turn start. Only the recovery failure
path changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, with tool use (shell, file edits,
test runs, read-only staging API reads). Model ID as attributed in the
commit trailer: Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 02:59:46 -06:00
DottaandClaude Opus 5.5 424e2c6488 fix(recovery): keep live owners' runs through an upgrade and continue replaced preparations in the restart lane (#21046)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade replaces the server. The old and the new server serve
together for several minutes. A legacy run proves that its owner is
alive by renewing a 60-second lease on its `heartbeat_runs` row.
> - #21019 added an uncharged `server_restart` retry lane for runs that
a restart or replacement ended. On the first upgrade that shipped it, no
run used that lane. The new server also failed many runs as
`process_lost` while the old server that owned them was still serving.
> - Each of these runs then spent one failure retry. A run that was
already the last automatic retry was held for manual reconciliation, so
the task needed a person.
> - An upgrade is routine. It must not cost a task its work or a retry,
and it must not leave a task waiting for a manual Retry.
> - This pull request sends a run that a replacement stopped during
preparation to the `server_restart` lane. It also stops the reaper from
failing a run while the server process that owns it is proven alive.
> - The benefit is that a task continues by itself through an upgrade,
also when the database is slow during the swap.

## Linked Issues or Issue Description

Refs #21019, #21042

**What happened?**
A hosted staging server was replaced by an automatic upgrade. The new
server started at 06:18:46Z, became ready at 06:26:17Z, and the old
server received SIGTERM at 06:27:34Z. Between 06:23Z and 06:26Z the new
server failed 44 runs as `process_lost` ("Process lost -- server may
have restarted; retrying once",
`processLossDiagnostic.controllerReplaced: true`). The old server still
owned these runs and was still serving. Every reaped run was a Runner
run in `executionStage: preparing` with no process recorded. Each
successor was scheduled with `scheduledRetryReason: transient_failure`
and `executionRetryAccounting.failureRetries: 1`. No run on the server
has `retryReason: server_restart` or
`executionRetryAccounting.serverRestarts`.

The run evidence also shows why the leases lapsed. The old server's
lease renewals waited 19 to 50 seconds on the database
(`controllerLeaseLoss.renewalPendingMs`) while its timers fired on time
(`deadlineLateMs` 0 to 17 ms). #21042 fixes that renewal stall. This
pull request fixes how the reaper and the retry scheduler treat these
runs.

**Expected behavior**
- A server never fails a run as lost while the server process that owns
it is alive and serving.
- A run that a replacement really ended continues once, automatically,
without spending the failure budget.

**Steps to reproduce**
1. Seed a legacy `paperclip_runner` run in `executionStage: preparing`,
owned by another server boot, with a lapsed run lease.
2. Run `reapOrphanedRuns()` on the new server.
3. See that the successor is `transient_failure` with `failureRetries:
1`. When the chain already spent its failure budget, see that no
successor is created and the run is held for reconciliation.
4. Give the owner a live controller instance lease (the old server is
still serving) and run `reapOrphanedRuns()` again: the run is failed as
`process_lost` anyway.

**Root cause**
- `usesServerRestartLane` (`server/src/services/heartbeat/retries.ts`)
left every run with the bootstrap evidence `providerWorkStarted: false`
to the `interrupted_preparation` lane. But `scheduleBoundedRetryForRun`
gives that lane only to an `interrupted` run (or a native preparation).
The reaper writes the same evidence on a `failed`/`process_lost` run
that it stops during preparation. That run fell through to the charged
`transient_failure` lane. The #21019 tests used runs in `dispatching`,
so they did not catch it.
- `legacyExecutionNeedsReconciliation` held every run whose failure
budget was spent before it checked the bootstrap evidence. A run that a
replacement stopped before dispatch did no provider work, but it was
held for reconciliation, and the task waited for a person.
- The orphan reaper (`recoverActiveExecutions`,
`server/src/services/heartbeat/recovery.ts`) treated a lapsed run lease
as proof that the owner was gone. It checked the shared owner prover
only for "gone", never for "alive". During the swap the old server's
renewals were slow, so its leases lapsed while it was alive.

**Paperclip version or commit**
Staging Core `51d77648bc` (first build with #21019). The behavior is the
same on master `148d2b629c`.

## What Changed

- `retries.ts`: `usesServerRestartLane(run, preparationLaneTakesRun?)`
keeps the `interrupted_preparation` exception only when that lane really
takes the run. `scheduleBoundedRetryForRun` passes its own verdict.
Without one, only an `interrupted` run qualifies. A `process_lost` run
that a replacement stopped during preparation now continues in the
`server_restart` lane.
- `legacy-execution-recovery.ts`: a replaced preparation
(`isReplacedPreparation`) does not need reconciliation, also when its
failure budget is spent. The `server_restart` bound limits it.
- `recovery.ts` (orphan reaper):
- When a run's lease lapsed but the shared owner prover proves its owner
process alive (for example, the owner's controller instance lease is
live), the reaper leaves the run with that owner. The owner stops and
settles the run itself (`controller_lease_lost`, uncharged
`interrupted_preparation` lane). The reaper recovers the run once the
owner is gone or stopped. It also recovers the run when the owner has
not settled it within `LEGACY_LIVE_OWNER_SETTLE_GRACE_MS` (5 minutes)
after the lease lapsed.
- A setup that a replacement stopped before dispatch uses the
`server_restart` bound for its retry decision, whatever the adapter.
- `legacy-controller-lease.ts`: add `proveLegacyController(run)`, which
returns the shared prover's verdict. `legacyControllerProvenGone` uses
it, and the reaper computes the proof once per run.
- `server-replacement-interruption.ts`: add
`isReplacedPreparation(run)`: bootstrap evidence `providerWorkStarted:
false`, a recorded controller process stamp, and a server-replacement
interruption. A row saved by an older release without the stamp keeps
the charged lane (the existing documented rule).
- `legacy-execution-recovery.ts` (`settleInterruptedNativeBootstrap`): a
reconciliation hold that an older release saved on a replaced
preparation is settled once its owner is proven gone, even when the
failure budget is spent. The run continues in the `server_restart` lane.
- `legacy-controller-lease.ts`, `recovery/ownership-registry.ts`: the
`legacy_interrupted_preparation_hold` kind judges a finished run's hold
by its recorded controller process
(`finishedLegacyControllerOwnerStamp`). Revocation by the orphan reaper
swaps in a random boot id, so before this change a reaped preparation's
hold had owner "none" and was never settled.
- `recovery/service.ts`, `heartbeat.ts`: the stranded-task sweep reports
exhaustion from the scheduler's own `retry_exhausted` result, so a spent
`interrupted_preparation` lane blocks the task even when the synchronous
budget check reads an unspent restart bound.
- `doc/execution-semantics.md`: document these rules.
- No schema change. Nothing changes on the path that starts a run or a
turn (no new waits and no file work).

## Verification

- Red on `origin/master`, green on this branch
(`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs
ended by a server restart or replacement"):
- "continues a first attempt / a run that was already the last automatic
retry that a replaced server left preparing, in the restart lane". This
is the production shape: a `paperclip_runner` run in `preparing`, two
servers sweep at the same time. Master: `transient_failure` with
`failureRetries: 1`, and no successor for the spent chain. Now: one
`server_restart` continuation, due at once, with the failure count
unchanged.
- "leaves a preparing/dispatching run whose lease lapsed with its owner,
then recovers it once the owner stops". The owner's controller instance
lease is live and two servers sweep. Master: `process_lost` at once.
Now: the run stays `running` with its owner. After the owner stops, it
becomes `process_lost` and a `server_restart` continuation is scheduled.
- "recovers the run when its live owner has not settled it within the
grace".
- `interrupted-preparation-hold-settle.test.ts`: "releases an older
release's hold on a preparation a server replacement stopped with its
failure budget spent" for an unrevoked and a reaper-revoked lease (red
without the settlement and owner-stamp changes) and "blocks the task
when a replaced preparation's lane is spent even though the restart
bound is not" (red without the sweep change).
- Related suites: heartbeat process recovery, retry scheduling, orphaned
active lease sweep, recovery deadline (bootstrap and ordinary
bootstrap), legacy continuation authority, undispatched execution wait,
agent status recovery, legacy workspace restore recovery, legacy
execution recovery, legacy controller lease, server replacement
interruption, heartbeat recovery, provider failure classification,
reconciliation run read, and wake-queue policy.
- `tsc --noEmit` for `server`.

## Risks

- A run whose owner is alive but no longer drives it now stays `running`
for up to 5 minutes after its lease lapsed, and then the reaper recovers
it. The owner's own sweep still recovers a run that the owner itself
lost the handle for (this boot is never deferred).
- The guard trusts the shared owner prover. That prover reads "alive"
only from a live or grace controller instance lease, or from a live PID
in the same PID space. A server whose instance lease cannot renew for 60
seconds stops itself, so a stuck owner cannot hold runs forever.
- More runs continue in the uncharged lanes. The `server_restart` bound
(6 in 60 minutes) still stops a loop, and the stop is shown on the task.
- Rows saved by older releases: a `process_lost` row with the bootstrap
evidence and an old-boot owner now continues in the `server_restart`
lane instead of being held. These runs did no provider work.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only log and API inspection) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 02:18:39 -06:00
DottaandClaude Fable 5.2 fd5ee7e385 fix(native): continue a routine assignment or chat message whose remote Runner an upgrade lost mid-turn (#21052)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes (Daytona) use a detached Runner.
When the server is replaced during an upgrade, the new server reattaches
to that Runner
> - On staging, upgrades every 15–40 minutes lose the in-flight Runner.
The same-run reattachment then finds `process_verification` /
`process_not_running`, the run fails with
`runner_remote_process_identity_unavailable`, and the task is blocked
> - #21039 made these runs continue once in a fresh session (the turn is
recorded as having an unknown outcome and is not replayed). It covers
user wakes and automatic retries (`retryOfRunId`) only
> - A routine's system assignment (`issue_assigned`, requested by the
system) and an external chat message wake have no `retryOfRunId`. They
got `not_user_request` and stayed blocked for a person
> - This pull request treats those two wakes as owed task work, so they
continue the same way once the Runner is proven gone
> - The benefit is that an upgrade no longer leaves these tasks waiting
for a Retry click

## Linked Issues or Issue Description

Refs #21039, #21023, #20923

**What happened?**
Staging replaced its server at about 07:10Z. 17 remote native runs were
in flight. Each new controller reached its Runner 114–152 s after
detach. Recovery failed with "PRP command session.snapshot timed out".
The same-run reattachment then found the Runner process gone, and each
run failed with `native_continuation_requires_reconciliation`. #21039
continued 14 of the 17 automatically. The other 3 were a routine's
assignment (two runs) and an external chat message. They have no
`retryOfRunId`, so `requestInterruptedNativeTurnRetry` returned
`not_user_request`. Those tasks are still blocked with no retry.

**What did you expect to happen?**
The Runner is proven gone, so nothing can still run the turn. The run
should continue once in a fresh session, the same as an automatic retry
of owed work.

**Steps to reproduce**
1. Start a remote native run from a routine assignment, or from an
external chat message.
2. Replace the server while the provider turn is active, so that the
reattachment finds `process_not_running`.
3. The run fails and the task stays blocked. No automatic Retry is
requested.

## What Changed

- `server/src/services/heartbeat/restart-interrupted-turn.ts`: after the
Runner is proven gone, a system wake also continues when it is a system
assignment (`issue_assigned`), or when it is a comment or external chat
message wake whose message is still on the task (not deleted, not
written by a run). It runs under the run's responsible user, at most
once per chain of runs, like an automatic retry already does. A linked
person's external chat message continues under that person when the
message is still on the task. Monitors and other system wakes are
unchanged.
- The same file and `server/src/app.ts`: the hook now passes the chat
service's Retry callback (`prepareFailedChatRunRetry`, registered at app
startup next to the budget-resume callback). A task in a connected
conversation whose endpoint restricts external execution then continues
through the chat Retry path. Before, `requestTaskRetry` refused it
(`retry_refused`) and the task stayed blocked.
- `server/src/services/runner-lost-mid-turn-continuation.test.ts`:
production-shape cases for a system assignment, an unlinked external
chat sender, and a linked person's chat message. Negative cases: a
deleted message, a message written by a run, a deleted linked message,
and a system assignment whose Runner was not proven gone.
- `doc/execution-semantics.md`: describes the rule.

## Verification

- Red on `origin/master` with only the new tests: `cd server && npx
vitest run src/services/runner-lost-mid-turn-continuation.test.ts` fails
4 of 16. The three new continuation cases return `{ requested: false }`.
The deleted linked-message case returns `not_user_request` instead of
`message_unavailable`.
- Green with the fix: 16 of 16. Two more cases bind the task to a real
restricted Slack endpoint and conversation. With the chat callback
registered, the run continues and the Retry carries the chat action (18
of 18). Without the callback, the result is `retry_refused`, which was
the behavior before for every connected conversation.
- Related suites pass: `runner-lost-mid-turn-continuation.test.ts`,
`heartbeat-native-remote-failure-retention.test.ts`,
`workspace-recovery-retry-copyback.test.ts`,
`governed-wait-runner-loss.test.ts` (116 tests).
- `cd server && npx tsc --noEmit` passes.
- Production evidence was read from staging run events through the
read-only API: 3 of 17 losses at the 07:10Z swap took this path. After
this change the next post-upgrade audit should show an automatic
`task_retry` for every `native_continuation_requires_reconciliation` run
whose refusal is `process_not_running`, including routine assignments
and chat messages.

## Risks

- Low. The change applies only after the Runner process is proven gone
(`process_verification` / `process_not_running`), so no Runner can still
run the turn. The turn is not replayed. The fresh session continues over
the saved work, as in #21039.
- The task can contain partial tool work from the lost turn. This is the
same accepted risk as #21039.
- Previous-release story: no persisted shape changes. The rule reads the
wake's reason and comment id, which earlier releases already write. A
run an older release already settled keeps its visible Retry.
- This does not stop the Runner loss itself. That is a separate fix.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, with tool use (shell, file edits,
test runs, read-only staging API reads). Model ID as attributed in the
commit trailer: Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 02:11:28 -06:00
DottaandClaude Opus 5.5 4e519bd428 fix(chat): send writes with an empty responseText and point card declines to resolve-from-comment (#21050)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents call the Paperclip API through `call_api` and find
routes with `search_api`
> - #21035 added a pre-send refusal for writes that carry
`responseText`, and new summaries for the question cancel and withdraw
routes
> - Full chat cohort 5 (OpenCode with DeepSeek v4.1 flash) showed that
both changes went too far
> - The refusal blocked a valid write that had an empty `responseText`,
and the summaries hid the route that records a person's decline
> - This pull request narrows the refusal and corrects the guidance text
> - The benefit is that valid writes go through on the first try, and
agents settle a declined card instead of leaving it pending

## Linked Issues or Issue Description

Refs #21035, #21002

**What happened?**
- inbox-search.restore-archive: DeepSeek sent `responseText: {}` with a
valid body on `POST /api/issues/{id}/inbox-archive`. The refusal from
#21035 rejected 24 POSTs in 3 minutes. The model did not drop the key,
and the run was cancelled at the 240s window. In cohort 4 the same POSTs
went through on the first try.
- questions-and-approvals.reject-and-revise: the person said "Do not
publish it; just make a draft I can review." The CEO quoted the cancel
summary ("an agent withdraws its own pending card with POST
.../withdraw"), saw that the person created the publish card, and left
the card pending. The plan decision did not settle. In cohorts 3 and 4
the CEO declined the card with `resolve-from-comment` (resolver policy
`anyone`).

**Expected behavior**
- A write with an empty or default `responseText` is sent once. A write
whose `responseText` sets a window is still refused before it is sent.
- The guidance says that withdraw is only for the agent's own obsolete
card, and that a person's explicit decline or answer goes through
`resolve-from-comment` when the card's resolver policy allows.

**Steps to reproduce**
Run the two chat cases above with OpenCode and DeepSeek v4.1 flash on
master.

## What Changed

- `call_api` (`assertRunnerApiDispatchable`): the check reads the raw
`responseText` value. `{}` or `{ offsetBytes: 0 }` on a write is dropped
and the write is sent. A `responseText` with `limitBytes` or a later
offset is still refused. `saveResponse: false` on a write was already
accepted; a test now covers it.
- OpenAPI: the cancel and withdraw summaries say that withdraw is only
for the agent's own obsolete card, and that a decline or answer on the
person's explicit instruction uses `POST .../resolve-from-comment` when
the resolver policy allows. The `resolve-from-comment` summary says it
also records a decline of a card the person created.
- `AGENT_CHAT_DIRECTIVE`: a person's explicit decline in chat answers a
pending confirmation with reject, whoever created the card. Withdraw
never records the person's decision.
- `doc/runner-api-tools.md`: describes the narrowed refusal.

## Verification

- `server/src/services/native-runtime/runner-api.test.ts`: new test
sends `POST /api/issues/{id}/inbox-archive` with a valid body and
`responseText: {}`, `{ offsetBytes: 0 }`, `saveResponse: false`, or
both. The request is sent exactly once. Red before the fix (3 of 4 cases
refused), green after. The refusal test for a `responseText` with
`limitBytes` stays.
- `server/src/__tests__/openapi-routes.test.ts`: new test for the
cancel, withdraw and resolve-from-comment summaries. Red before, green
after.
- `server/src/__tests__/agent-conversations.test.ts`: new directive test
for the decline rule. Red before, green after.
- Also passed: `runner-api.integration.test.ts`,
`native-case-tools.integration.test.ts`, server `tsc --noEmit`. SDK
public input pins: `--check` reports no change.
- Paid run (OpenCode + DeepSeek v4.1 flash) on this branch for
inbox-search.restore-archive, questions-and-approvals.reject-and-revise,
gateways-profiles.rotate-token and budgets-costs.repair-cost-recording:
4 of 4 cells passed on 3f20fb2bcc (run 43177772183); details in a PR
comment.

## Risks

- A write with a populated `responseText` is still refused, as in
#21035. Only the empty and default shapes change. A dropped empty
`responseText` changes nothing, because the server already removed it
from writes before sending.
- The directive grows by two sentences, which adds a small number of
prompt tokens per chat turn.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 02:09:14 -06:00
DottaandClaude Opus 5.5 3c058c37e6 fix(retry): continue a connected task from saved work when its exact chat request cannot be repeated (#21038)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks that come from a connected chat (for example GitHub pull
request reviews) show a Retry button when a run fails
> - The task projection offers Retry, but the Retry route sends a
connected task to the chat retry owner, which only replays the exact
failed chat request
> - When that exact request cannot be replayed (the attempt already
saved a result, its output was published, the latest run did not start
from a chat message, or the requester's access changed), the route
returns 409 and tells the person to check the connection by hand
> - A Retry button that is shown must work, and recovery must not ask
the person to inspect anything
> - This pull request makes a person's Retry continue the task from its
saved work when the exact chat replay is refused, the same way a new
message on the task would
> - The benefit is that the projection and the route agree, and blocked
connected tasks become retryable after deploy with no migration

## Linked Issues or Issue Description

No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a hosted deploy, five blocked tasks that came from a connected GitHub
conversation showed Retry (`interruption.canRetry: true`). Each click on
Retry returned 409 `chat_failed_run_retry_not_authorized`. The message
was "The exact failed chat request can no longer be retried safely.
Check its current connection, access, and recovery state." or "This
attempt already recorded a result. Send a new message on the task to
continue from the saved work." The latest runs were legacy sandbox setup
failures, native runs that needed reconciliation, a run that started
from an issue monitor instead of a chat message, and a native run that
already saved its result. Root cause: `admitTaskRetry` in
`server/src/services/task-retry.ts` called the chat retry owner for
every connected task and passed its exact-replay refusal straight to the
person. The projection cannot predict that refusal.

**Expected behavior**

A shown Retry works. If the exact chat request cannot be repeated, Retry
continues the task from its saved work, the same way a new message on
the task would. Users do not get operator instructions.

**Steps to reproduce**

1. Bind a task to a restricted chat conversation.
2. Make its latest run fail so that the chat retry owner refuses the
exact replay. For example, use a run whose source is not a chat message,
or a native run with an accepted result.
3. Click Retry (`POST /api/issues/:id/retry`).
4. The route returns 409 `chat_failed_run_retry_not_authorized`.

**Paperclip version or commit**

`master` at 5a7378af5 (the deployed revision), reproduced on 51d77648bc.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

Not adapter-specific (core bug). Seen with `paperclip_runner` legacy and
native runs.

## What Changed

- `server/src/services/task-retry.ts`: a person's Retry runs the exact
chat replay inside a savepoint. On a
`chat_failed_run_retry_not_authorized` refusal, the savepoint rolls back
and the same Retry receipt stays an ordinary board continuation. The
clicking person is the responsible user, the run starts a fresh session
(`forceFreshSession`) from saved work, and no chat action, chat
authority, or output replay is created. Retry that Paperclip requests by
itself (`initiatedBy: restart_recovery | provider_limit_reset`) keeps
the exact-replay requirement. Other errors (403, lock contention,
database errors) still propagate.
- `server/src/services/chat-channels/failed-run-retry.ts`: plain refusal
wording. Runs saved by an earlier release with the old diagnostic are
still recognized by the pre-provider check.
- Tests: new red/green cases in `server/src/services/task-retry.test.ts`
(four refusal reasons on a chat task whose latest run came from an issue
monitor, plus automatic Retry still refused). Updated two real-service
route cases in `server/src/__tests__/chat-channels.integration.test.ts`
(accepted native result on an inherited GitHub task, and an unknown chat
source) from expecting 409 to expecting a board continuation without a
chat replay.
- A person's Retry on that person's own saved board turn (`source:
issue.comment.reopen`) that cannot be certified for replay also
continues from saved work, without replaying the saved message.
Automatic Retry still gets 409 `saved_board_message_not_replayable`.
- A person's Retry when no chat retry owner is wired also continues from
saved work. Automatic Retry still gets 409.
- The fallback is not used while the conversation holds saved or
unadmitted chat input for the task. That input keeps its own admission
path and the existing refusal.
- Docs: `doc/execution-semantics.md` and `doc/connections/GITHUB.md`.

### 409 paths in the task Retry route after this change

| Path | Who sees it | Result |
| --- | --- | --- |
| `chat_failed_run_retry_not_authorized` from the chat retry owner |
Person | Continues from saved work (board continuation) |
| `saved_board_message_not_replayable` | Person | Continues from saved
work |
| No chat retry owner wired | Person | Continues from saved work |
| Any of the three above while saved or unadmitted chat input is pending
| Person | 409. The pending input continues through its own admission
and delivery. |
| Any of the three above | Automatic Retry (restart recovery, provider
limit reset) | 409, recorded by the hook. No button. |
| `task_retry_busy` (chat authorization lock held) | Person | 409 with
`retryable: true` after 10 internal attempts. A new click works. |
| Idempotency key reused for a different task or user | Client error |
409. The UI makes a new key for each click. |

## Verification

- Red before the fix: `npx vitest run src/services/task-retry.test.ts -t
"connected task"` failed 4/4 with `Error: refused` (409). The board-turn
and missing-owner cases failed with "This saved message cannot be
retried safely..." and "Retry is unavailable for this connected
conversation." `src/__tests__/chat-channels.integration.test.ts -t
"continues an inherited GitHub task"` failed with `expected 202
"Accepted", got 409 "Conflict"`.
- Green after the fix: `src/services/task-retry.test.ts`,
`src/__tests__/durable-chat-wakeup.test.ts`,
`src/services/budget-resume-continuation.test.ts`: 160/160 passed.
- Full `src/__tests__/chat-channels.integration.test.ts`: 1949/1951
passed. The 2 local failures (`binds Slack tools...`, `materializes
direct external-chat finals...`) read `.claude/skills/slack/SKILL.md`
from the parent checkout of a nested local worktree (ENOENT). They do
not touch Retry and depend on the local checkout layout.
- `npx tsc --noEmit -p server/tsconfig.json`: passed.
- `pnpm verify` exited 2 (a doc file is a global CI input), so required
CI supplies broad coverage.
- Staging was inspected with GET requests only. No POST was sent to
staging.

## Risks

- Behavior change: a person's Retry on a connected task can now start a
board continuation where it was refused before. It uses the same
authority as a new message from that person, and dispatch keeps all
normal gates (task access, executor containment, recovery holds,
unadmitted chat input).
- The continuation does not publish the earlier result again and does
not reuse the chat requester's authority.
- No migration. Existing blocked tasks become retryable at the next
click.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 01:27:54 -06:00
DottaandClaude Opus 5.5 0dfbbdeb31 fix(chat): route company creation, decline retries and question cancellation correctly (#21035)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents follow the chat directive and discover API routes with
search_api and call_api
> - Full chat cohort 4 (OpenCode with DeepSeek v4.1 flash) showed four
guidance gaps
> - The missing-name rule overrode the board-only company-creation
boundary; a repeated request after a decline did not post a new approval
card; search_api showed the board-only question cancel route as open to
agents (and withdraw as board-only); write calls carried `responseText`
beside or instead of `body`
> - This pull request fixes the guidance, the route authorization
metadata and the pre-send refusal
> - The benefit is that agents take the supported path the first time
instead of asking for the wrong input, refusing a valid retry, or
calling a route that returns 403

## Linked Issues or Issue Description

**What happened?**
- onboarding.new-company: asked to "set up a new company for my website
business", the CEO cited the missing-name rule (#21007) and asked for a
business name with a question card. Creating a company is a board
action, so the agent should send the person to the creation flow.
- app-permissions.ask-before-action: the person declined a write card
and then repeated the same request. The CEO asked the person to tell it
to resubmit instead of posting a new approval card. The gateway already
allows a fresh request after a new human message
(`isNewHumanIntentAfterToolRejection`), but the decline text said "Do
not retry the same call".
- questions-and-approvals.multi-part-question: search_api listed `POST
/api/issues/{id}/interactions/{interactionId}/cancel` as
`board_or_agent`. The route rejects agents. The agent route,
`/withdraw`, was listed as board-only although its route admits the
card's creator agent and the task's assignee.
- Several cases: write calls through call_api carried `responseText`
with no body (for example "Received arguments: operationId, pathParams,
responseText").

**Expected behavior**
The agent links the company-creation flow; it posts a fresh approval
card for a repeated request; search_api shows the true authorization of
cancel and withdraw; a write with `responseText` is refused before it is
sent, with the same guidance #21002 gave `saveResponse`.

**Steps to reproduce**
Run the four chat cases above with OpenCode and DeepSeek v4.1 flash on
master.

Refs #21002, #21007

## What Changed

- `AGENT_CHAT_DIRECTIVE`: the missing-name rule applies to work the
agent creates itself and never comes before a board-only boundary. For a
new company the agent links [Create organization](/onboarding) (no
company prefix) and repeats the business details. A repeated request
after a declined approval card is an explicit retry: make the same call
so a fresh card is posted. A clarification question card puts candidate
links in its description, because labels show as plain text.
- Decline instructions in `tool-action-delivery.ts` and
`tool-gateway.ts` say that a later request from the person for the same
action is an explicit retry.
- OpenAPI: `POST /api/issues/{id}/interactions/{interactionId}/cancel`
is board-only and its summary names the `/withdraw` route for agents.
`/withdraw` is no longer marked board-only, which matches its route.
- `call_api`: a new write that carries `responseText` is refused before
it is sent; the message says `responseText` never carries request fields
and names the allowed body fields. Older saved calls still validate for
recovery. `doc/runner-api-tools.md` is updated.

## Verification

- `server/src/__tests__/agent-conversations.test.ts`: new directive
tests for the board-only boundary order, the decline retry and card
links.
- `server/src/__tests__/openapi-routes.test.ts`: cancel is `board`,
withdraw is `board_or_agent`, the cancel summary names withdraw.
- `server/src/services/native-runtime/runner-api.test.ts`: a write with
`responseText` is refused before sending (three window shapes), the same
write without it is sent once, and saved calls still validate.
- `tool-gateway.test.ts`, `tool-gateway-service.test.ts`,
`issue-thread-interaction-routes.test.ts`,
`tool-action-replay-intent.test.ts`: passed. `server` `tsc --noEmit`:
passed.
- Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) together with the
eval PR: see the PR comment for results.

## Risks

- A model that used to send a write with `responseText` and a valid body
now gets one refusal and must resend without it. Nothing is sent before
the refusal, so there is no duplicate side effect.
- The directive grows by a few sentences, which adds a small number of
prompt tokens per chat turn.
- Agents that read `/withdraw` as board-only before now see that they
can withdraw their own card; the route already enforced who may.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:27:35 -06:00
DottaandClaude Opus 5.5 aa94939988 fix(native): continue a run whose remote Runner died mid-turn after a restart, without replaying the turn (#21039)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a server restart, recovery reattaches to that
Runner on the run's own sandbox lease
> - On staging, a deploy detached the controller and left the Runner
alive. The restart's reattachment then found the Runner process gone
with a provider turn in flight. The run failed with
`runner_remote_process_identity_unavailable` and the task waited for a
person
> - The run log said `startup reuse: environment (fresh) created`, so
the sandbox looked re-created. It was not. The reattachment resumed the
exact retained lease. The receipt reported the lease's original creation
> - `requestInterruptedNativeTurnRetry` refused every unsettled turn
(`provider_turn_unsettled`), and refused every system wake unless the
journal proved that no turn was issued. So this run got no automatic
continuation
> - This pull request continues such a run once in a fresh provider
session when the refused reattachment proved the Runner process gone. It
records that the turn's outcome is unknown and never replays the turn.
It also makes the environment receipt tell the truth
> - The benefit is that a restart that kills a Runner in the middle of a
turn no longer needs a Retry click

## Linked Issues or Issue Description

Refs #20853, #20923, #21023, #20278, #20627, #20828

**What happened?**
A staging deploy landed in the middle of a remote native Codex run on
Daytona. Shutdown detached the controller and logged "durable runner
left alive". The new server reattached through the run's original lease
(same lease row, same provider sandbox id, same directory). Its process
check then found the Runner gone: `process_verification` /
`process_not_running`, `runnerLifecycle: ready`, `runnerSettled: true`,
last diagnostic "WebSocket peer closed the connection", empty stderr.
The provider checkpoint still had an active turn. The run failed with
`runner_remote_process_identity_unavailable` /
`native_continuation_requires_reconciliation`, and the task went to
`blocked`.

**Root cause**
- `server/src/services/heartbeat/restart-interrupted-turn.ts`
(`requestInterruptedNativeTurnRetry`): an unsettled last turn always
returned `provider_turn_unsettled`, also when the refusal proved the
Runner process gone. A system wake (here `transient_failure_retry`)
continued only with `providerTurnIssued: false`.
- The sandbox was not re-created. The environment and lease listings
show one lease for the run, never replaced, and the final workspace
snapshot used the same provider sandbox. `environmentStartupReuse` read
`sandboxLeaseAcquisition: created` from the lease's first acquisition,
so the reattachment's receipt said `fresh / created`. The 46 s acquire
was the provider resume of that exact lease. The evidence available to
GET requests cannot say what stopped the Runner process (the resume
result is not kept). This pull request does not change that path.
- #21023 named this run as not covered. #20853 resumes a Runner only at
a settled, idle `ready` cut. Here the provider checkpoint has an active
turn, so #20853 stays fail-closed, as it should.

**Expected behavior**
The run continues once by itself in a fresh provider session over the
saved work. The turn whose outcome is unknown is recorded and is not
replayed.

## What Changed

- `restart-interrupted-turn.ts`: read the last
`native.remote_runner_recovery_failed` refusal once. When it proves the
Runner process gone, an unsettled last turn no longer blocks the
automatic Retry. Before it requests the Retry, the server records
`native.interrupted_turn_outcome_unknown` once (turn id, last turn
event, `replayed: false`, closed `runnerExit` fields). The Retry is the
existing one: saved-work recovery and stop proof first, then a fresh
provider session (`forceFreshSession: true`), once per chain.
- The same file: a system wake whose run is an automatic retry of owed
work (`retryOfRunId` set) also continues under the run's responsible
user once the refusal proved the Runner gone. Other system wakes (for
example a monitor) keep their policy.
- `packages/adapter-utils/src/startup-reuse.ts` and
`native-session-executor.ts`: a `reattach_remote_runner` recovery
reports its `environment` receipt as `reused / verified_reuse`. That
path only resumes the run's exact retained lease and never acquires one.
- Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`.

Previous-release story: no persisted shape changes. The rule reads the
closed `remoteRunnerRecovery` fields that the previous release already
writes; the new test uses the staging refusal payload as written. A run
that a previous release already failed this way continues on the next
completion or restart hook call only; runs already settled stay as they
are and keep their Retry. The new event is additive.

## Verification

- Red on `origin/master` (51d77648bc), with only the new tests: `cd
server && npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts` fails 4 of 6.
The three continuation cases return `{ requested: false, reason:
"provider_turn_unsettled" }`, and the monitor case returns that reason
instead of `not_user_request`.
- Green with the fix: 6 of 6. The cases cover a user's assignment
through the restart claim, the staging shape (system
`transient_failure_retry` with `retryOfRunId`), and a same-run retry.
Each one checks one recorded unknown-outcome event, an idempotent second
call, saved-work recovery first, one queued successor with
`forceFreshSession: true`, and one Retry intent. Negative cases: a
refusal that did not prove the process gone, no refusal, and a monitor
wake. Each keeps the visible Retry and records nothing.
- `cd packages/adapter-utils && npx vitest run
src/startup-reuse.test.ts`: the new receipt case fails before (`fresh /
created`) and passes after.
- Focused suites pass: `workspace-recovery-retry-copyback.test.ts`,
`governed-wait-runner-loss.test.ts`,
`heartbeat-native-remote-failure-retention.test.ts`,
`native-runner-restart-recovery.integration.test.ts` (302 passed, 14
skipped), and `native-session-executor.test.ts -t "startup
reuse|restart|reattach"` (149 passed).
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/adapter-utils typecheck` pass.
- `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global
CI input. CI supplies that coverage.
- Not run: a live Daytona restart. The change is in the server's
recovery decision; the tests use the fixture's sandbox confinement.

## Risks

- Medium. A run whose Runner died in the middle of a turn now continues
by itself. The turn's outcome is unknown, so tool work from that turn
can be partly done. The successor is a fresh session over the saved
workspace; it does not replay the turn. Saved-work recovery stops and
keeps the sandbox before the successor starts, so the old provider
cannot still run.
- Without positive proof that the Runner process is gone, an in-flight
turn keeps the visible Retry, as before.
- An automatic retry of owed work now continues after a lost Runner.
Monitor and other system wakes without `retryOfRunId` do not. The bound
is still one automatic Retry per chain.
- The environment receipt changes only for the reattachment path. No
execution decision reads it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:13:25 -06:00
DottaandClaude Opus 5.5 51d77648bc fix(native): keep a remote Runner alive when a failed turn start or event stream closes the run (#21023)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 stopped a failed close from force-stopping an unreachable
remote Runner. It asks for retention only when the session was not
cancelled
> - The runtime cancels the session itself when a turn start or the
event stream fails ("Native session event consumption failed."). That
cleanup sets the same flag as a user Stop
> - So after a failed turn start, the close still sent `SIGKILL` to the
Runner, and the same-run reattachment failed with
`runner_remote_process_identity_unavailable`. The task waited for Retry
> - This pull request separates failure cleanup from Stop. Failure
cleanup still revokes the turn's output, but the close keeps the Runner
for the same-run reattachment
> - The benefit is that these runs continue without Retry. Stop and
protocol integrity faults still stop the Runner

## Linked Issues or Issue Description

Refs #20923, #20853

**What happened?**
After the staging redeploy, five remote native runs failed with error
`runner_remote_process_identity_unavailable` and error code
`native_continuation_requires_reconciliation`. In four of them, the
first attempt started after the deploy. Its `turn/start` failed with
`runnerd did not report the provider turn identity`. The run log then
showed `runner did not prove durable suspension before checkpoint` and
`runner shutdown failed: Error: Durable recovery runner timed out.` The
diagnostic `remote runner left alive unsettled for same-run
reattachment` from #20923 did not appear. One second later the Runner
exited (`runner_remote_process_exited lifecycle=ready`). The same-run
reattachment 20–40 s later found `process_verification` /
`process_not_running` with `runnerSettled: false`.

**What did you expect to happen?**
The failed close keeps the Runner alive, as #20923 intended. The
same-run reattachment adopts it, and the run continues without Retry.

**Steps to reproduce**
1. Start a remote native Codex run.
2. Make `turn/start` fail (for example the Runner does not report the
provider turn identity in time), or let the event stream fail or time
out.
3. The runtime cancels the session as cleanup. `HarnessNativeSession`
treats that as a Stop, so `close` asks for `retainUnsettledRemoteRunner:
false`, and the transport kills the Runner.

**Root cause**
`packages/paperclip-runner/src/native-session-runtime.ts`
`attemptOptionalSessionCancellation` calls `session.cancel`.
`HarnessNativeSession.cancel`
(`packages/paperclip-runner/src/backends/harness-driver-backend.ts`)
sets `#explicitlyCancelled`, and `close` used
`retainUnsettledRemoteRunner: !this.#explicitlyCancelled`.

The fifth run is different. A server shutdown detached its controller
and left the Runner alive. Then the sandbox environment was acquired
fresh (45.9 s) before the reattachment, and the Runner was gone in the
middle of a turn. This pull request does not change that path. See
Risks.

## What Changed

- `NativeSession.cancel` accepts an optional `failureCleanup` field.
- `attemptOptionalSessionCancellation` passes `failureCleanup: true`,
except for a `NativeSessionProtocolIntegrityError`.
- `HarnessNativeSession` tracks a Stop separately from output
revocation. `cancel` with `failureCleanup` revokes the turn's output
only. `cancel` without it, and `revokeTurnPublication`, record a Stop.
`close` asks for retention when no Stop was recorded and no protocol
integrity fault occurred.
- Tests: three runtime cases with the real Codex backend (turn start
fails, event stream times out, Stop then failure) and a new backend
case.
- Docs: `doc/execution-semantics.md` describes the rule.

## Verification

- Red, on `master` with only the new tests: `cd
packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts -t "remote Runner retention after a
failed run"` fails 2 of 3 (`expected false to be true`): the close asked
to kill the Runner after the turn start failed and after the event
stream timed out. The Stop case passed.
- Green with the fix: the same command passes 3 of 3.
`src/backends/harness-driver-backend.test.ts -t "retain an unsettled"`
passes 3 of 3, including the failure-cleanup case. That case also checks
that the turn's result stays unpublished.
- `npx vitest run src/native-session-runtime.test.ts src/backends
src/drivers/codex
src/live/runnerd-codex-transport-remote-retention.test.ts
--maxWorkers=4`: 31 files, 814 tests pass.
`codex-protocol-integrity.test.ts` confirms that an integrity fault
still stops the Runner.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` is
clean.
- `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global
CI input. CI supplies that coverage.
- Not run: a live Daytona run. The change is in the controller library
that runs on the server, so it needs no new Runner image.

## Risks

- Low to medium. After a failed turn start or event stream, an
unreachable remote Runner now stays alive instead of being killed. This
is the state #20923 already accepted for other failed closes. The
session stays quarantined, and the same-run reattachment or a Stop
cleanup owns the Runner.
- Stop is unchanged. A server Stop calls `cancel` without
`failureCleanup`, so the close still stops the Runner, also when the
stopped run later fails. A governed wait uses `revokeTurnPublication`,
which also records a Stop.
- A protocol integrity fault needs an operator and is not resumed on the
same run, so it still stops the Runner.
- Previous-release story: this change adds and changes no persisted
state. A Runner that an older release already killed keeps its current
named refusal and Retry. A deploy during a run changes nothing, because
the decision is made in the closing process.
- Not covered: a Runner that dies with its sandbox while the controller
is detached for a restart, in the middle of a turn. Its journal is a
settled `ready` cut with an active turn. #20853 resumes only idle
`ready` cuts. This needs a separate fix.
- No wait is added before `turn.start`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:21:05 -06:00
cf1def0ca6 feat(inbox): rebuild compact cards and share task review controls (#20961)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - The inbox collects decisions, requests, and updates for the people
in a company.
> - People need a clear personal queue and a consistent way to answer
each request.
> - Repeated headings, inconsistent card actions, and separate review
flows make that work harder.
> - This pull request connects the inbox UI to the routing foundation in
#20959.
> - Compact cards use shared task controls and open supporting work in a
side panel.
> - People can answer, claim, transfer, archive, and inspect outcomes
from one place.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The inbox cards, personal views, shared task controls, and decision
history.

**Subsystem affected**

UI. This PR targets master and uses the routing foundation merged in
#20959.

**Current behavior**

The inbox repeats headings and counts. Cards have different layouts and
action placement. The requester can repeat the signed-in person's name.
Review and question controls differ between the inbox and task thread.

**Proposed behavior**

Open Assigned to you by default. Show one count line above the cards.
Use a compact card layout with consistent actions and supporting
details. Show the requester's avatar and name only when they differ from
the signed-in person. Share question controls, including a radio option
for Other that reveals a text field when selected.

**Reason and benefit**

People can scan their queue and complete requests without losing the
surrounding task or creating separate drafts for the same review.

**Breaking changes**

The default inbox view becomes personal. The all-assignee and shared
views remain available. Card close menus distinguish personal archive
and snooze from shared withdrawal.

**Additional context**

Refs #20959, #19363, and #19520. Refs #20268 for related
actionable-request work. This stack adds personal routing and shared
controls. ROADMAP.md includes Multiple Human Users and Agent Reviews and
Approvals.

## What Changed

- Add personal, unassigned, company, all-assignee, and archived inbox
views. Add claim, transfer, personal close actions, receipts, and
history controls.
- Use compact decision cards with a single count line, one-pixel
borders, and equal space above, below, and to the right of each primary
action.
- Open artifacts and synchronized confirmation/approval controls in a
shared side panel. Share their revision drafts and action state with the
task thread. Other requests keep one editable form on the inbox card
while reviewing an artifact.
- Make Other match the question options and reveal its field only when
selected. Apply this to inbox and task question cards.
- Hide self-requester labels, space agent avatars, and use the standard
canceled status icon for withdrawn requests.
- Use connector logos in connector cards. Keep the connector promotion
background and sidebar border continuous.
- Add company and project decision-owner controls and document the
updated inbox behavior.

## Verification

- The complete [GitHub Actions CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/43141559641)
passes on `8d81a75e49b518fff6b6ff4c23dd67979b0b97ee` (Linux/Node 24:
typecheck, build, general and serialized tests, Runner support, browser
shards, and aggregate gates). Greptile is 5/5 on that exact head with
zero open findings, and the security scan passes.
- Local UI typechecking passes. Eight focused suites pass 382 tests: 236
inbox/review/question/connector tests plus all 146 composer tests.
Regressions passed red to green for attachment links, prompt rendering,
single-form artifact review, connector scope copy, accessible headings,
and the composer test harness's missing theme provider. The final commit
changes only that test harness; production code is unchanged from
`59d015833f`.
- Browser acceptance on the production implementation confirms preserved
selected answers with one editable question form, accessible Markdown
prompts, inert image references, accurate connector scope copy, native
PDF downloads, image sidebar previews, and Open file requesting the
attachment endpoint while preserving the inbox URL. Request reads and
file responses were intercepted; no question answers were written. The
preview is healthy on the final head.
- Visual and interaction checks cover Other radio behavior in both
themes and on mobile, the loaded Google Calendar logo, and 20px above,
below, and to the right of the primary action across all 23 visible
cards. The affected desktop/mobile navigation and sidebar specs pass
locally and in the final CI browser matrix.
- Real-editor browser checks confirm Ctrl/Cmd+Enter cannot bypass
another person's confirmation. Cancel preserves the draft and unlocks
silently; Continue submits once. Approval write failures remain visible.
- `pnpm verify` returns exit 2 because DESIGN.md requires broader
coverage; the final full CI matrix supplies every requested gate. Maps
and Storybook catalog structural checks pass. The routing foundation
#20959 merged after passing 249 database tests, server typechecking,
full CI, Greptile 5/5, and security scanning. Its backend and SDK
registry are preserved.
- Demo data is retained for testing; database suites and browser specs
use separate disposable databases.

## Risks

- The inbox uses the routing contracts merged in #20959. Personal
filters and history affect shared task entry points; the affected
controls are included in local regression coverage and full CI.
- Personal defaults and close-menu wording change inbox behavior.
Existing resolver permissions still control shared actions.
- Shared revision state and live invalidation affect two surfaces.
Focused tests cover draft synchronization, action locks, and incoming
updates.
- Visual changes use the existing design tokens and component
primitives.

## Model Used

OpenAI GPT-6 through Codex. Used repository tools, code execution, and
browser checks. This session does not expose a more specific runtime
model ID or its context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 21:05:00 -08:00
DottaandClaude Opus 5.5 156453fe7c fix(recovery): continue runs that a server restart or replacement ended, without spending the failure budget (#21019)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A deploy replaces the server. The old server stops its running
legacy runs at SIGTERM (`server_shutdown_interrupted`). The new server
finds runs whose owner died without that drain (`process_lost`, "server
may have restarted").
> - Each of these runs got its follow-up from the bounded transient
retry. That retry spends the failure budget (2 attempts) and waits 30 to
60 seconds.
> - So the first deploy spent one failure retry. A run that was already
the last automatic retry got no follow-up at all. The task then rested
on a manual Retry.
> - A deploy is routine. An interruption by the server is not a failed
attempt, and it must never leave a task waiting for a person.
> - This pull request moves these runs to their own `server_restart`
retry lane. The lane never spends the failure budget, starts the first
continuation at once, and has its own bound.
> - The benefit is that a task continues by itself through several
deploys a day, and a real loop still stops and shows why.

## Linked Issues or Issue Description

**What happened?**
A zero-downtime deploy replaced a hosted server. In the 11 minutes
around the swap, 11 legacy runs ended with `server_shutdown_interrupted`
and 7 ended with `process_lost`. At least 7 tasks were left with no
follow-up run and no scheduled retry, so each needed a manual Retry. Two
of the stranded runs were already automatic retries.

**Expected behavior**
Each interrupted run continues once, automatically, on the surviving
server, with its original wake.

**Paperclip version or commit**
Core commit 7d9973405c. The behavior is
the same on master bf18b348c5.

**Steps to reproduce**
1. Start a legacy run with a conversation adapter (for example
`codex_local`) as the second automatic retry of its chain (failure
budget spent).
2. Send SIGTERM to the server (`drainRunningRunsForShutdown`), or let a
new server boot find the run without its owner (`reapOrphanedRuns`).
3. See that no successor run is created. The scheduler logs "Bounded
retry exhausted".

**Root cause**
- `enqueueProcessLossRetry`
(`server/src/services/heartbeat/recovery.ts`) calls
`scheduleBoundedRetryForRun`. That function put these runs in the
`transient_failure` lane, with `maxAttempts = 2` counted from
`executionRetryAccounting.failureRetries`
(`server/src/services/heartbeat/retries.ts`). A first attempt got a
charged retry 30 to 60 seconds later. A run with 2 spent failure retries
got `retry_exhausted`.
- The orphan reaper (`recoverActiveExecutions`) skipped the retry when
`failedRunRetryBudgetSpent(run)` was true. It also skipped a
conversation run that had no local child process recorded (for example a
run that was dispatching), because `shouldRetry` needed a local child, a
preparing stage, or a monitor wake.
- The stranded-task sweeper then saw the spent budget and moved the task
to `blocked`.

## What Changed

- `execution-recovery-attempt.ts`: add the `server_restart` retry lane.
It is an uncharged lane. Its counter (`serverRestarts`,
`serverRestartWindowStartedAt`) is saved in `executionRetryAccounting`,
so it is visible on each run. The bound is 6 continuations in one
60-minute window. A later interruption after the window starts a new
window.
- New `server-replacement-interruption.ts`:
`isServerReplacementInterruption(run)` names a legacy
`server_shutdown_interrupted` run, or a `process_lost` run whose owner
was another server boot. It reads the new
`processLossDiagnostic.controllerReplaced` verdict. For rows saved by an
older release, it reads the owner boot id or `runPredatesObserver`.
- `retries.ts`: `scheduleBoundedRetryForRun` uses the `server_restart`
lane for these runs (`usesServerRestartLane`). The first continuation is
due at once; later ones in the window back off like the
interrupted-preparation lane. `failedRunRetryBudgetSpent` reports the
restart bound for these runs, so the sweeper escalates only when that
bound trips. An interrupted preparation keeps its own lane while its
chain has failure budget left.
- `recovery.ts` (orphan reaper): record `controllerReplaced`. A
conversation run that a restart ended is eligible for continuation even
without a local child process, and it checks the restart bound, not the
failure budget.
- `wake-queue/adapters/postgres.ts`: carry the restart counter to
successors that the wake queue creates.
- No schema change. No change to how runs start (no new waits before a
turn).

Behavior that did not change:
- At most one successor per run: every path still uses the predecessor
row lock and the `retry_of_run_id` check in
`scheduleBoundedRetryForRun`. Two servers alive at once reuse the same
successor.
- A non-conversation adapter that reached dispatch stays held for
reconciliation, because a replay can repeat its actions.
- Native runtime runs keep their own recovery.
- A server that began shutdown already starts no queued run
(`closeProcessRunAdmission` in `server/src/index.ts`, checked in
`heartbeat/queue.ts`). Its queued and newly scheduled runs stay in the
database for the new server.

## Verification

- Red on `origin/master`, green on this branch
(`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs
ended by a server restart or replacement"):
- Graceful shutdown, first attempt: master scheduled a charged
`transient_failure` retry. Now one immediate `server_restart`
continuation.
- Graceful shutdown, already the last retry: master scheduled nothing.
Now one continuation; the failure count stays 2.
- Owner lost at boot (`process_lost`), first attempt and
already-retried: same results, with two servers running the boot sweep
at the same time. Only one successor.
- Old server exited after it committed the interruption and before it
scheduled the continuation: the new server's sweep continues the run.
- Run saved by an older release (no `controllerReplaced`, spent failure
budget): the sweep continues it instead of moving the task to `blocked`.
- The restart bound trips after 6 in one window, records `retryReason:
server_restart` on the run, and the sweeper escalates. The same count in
a closed window continues.
- A run that failed by itself (`adapter_failed`) with a spent failure
budget still gets no retry.
- Each continuation's execution continuation reports the original wake
(`trigger.retriedReason = issue_monitor_due`).
- Unit tests:
`server/src/services/server-replacement-interruption.test.ts`.
- Six earlier tests asserted escalation after "three deploy restarts"
spent the failure budget. They now spend the restart bound instead, so
they still prove escalation when the bound trips.
- Existing test `server/src/services/heartbeat/queue.test.ts` covers
"starts no run once shutdown closed run admission".
- `pnpm verify`: see the PR checks.

## Risks

- A run that a restart ends now continues more often (up to 6 times per
hour per chain, not 2 in total). A run that itself crashes the server is
still stopped by the bound, and the stop is shown on the task.
- Tasks that an older release already moved to `blocked` stay blocked.
Tasks that are still `in_progress` or `todo` with a stranded run
continue on the next sweep.
- A `process_lost` row from an older release whose owner was the same
boot that later restarted is read as a restart. That case is a lost
in-memory handle, which is also not a failure of the agent's work.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:56:49 -06:00
DottaandClaude Opus 5.5 8c761abae3 fix(environments): wait for a re-activating sandbox provider before refusing a lease resume (#21020)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a controller restart, a run whose remote Runner kept running
reattaches. First it resumes its exact sandbox lease through the sandbox
provider plugin (`resumeRunLease`).
> - #20982 made a plugin whose worker `initialize` timed out retry its
activation, so the plugin comes back seconds later instead of staying
`error`.
> - Chaos cell CR6 (#20983) shows the gap that remains. The reattaching
run asks for its lease during those seconds and gets
`PluginWorkerUnavailableError`. The setup retry covers that error only
for legacy runs, so the native restart-recovery run fails and needs
Retry.
> - This pull request makes the exact-lease resume wait, for a bounded
time, for the worker of a plugin that is `ready` (re-activating) or
recovering from activation timeouts.
> - The benefit is that a restarted run rides through a short plugin
re-activation instead of failing into Retry.

## Linked Issues or Issue Description

Refs #20982, #20983

**What happened?**
CR6 after #20982 (`w-20261112T041333-s9531`) passed in part. The plugin
re-activated 5 s after its `initialize` timeout, but both tasks finished
with `R3:retry_needed`. Run `a6b3f0f6` resumed its lease at 22:15:35,
inside that window. It failed with `setup_failed` /
`plugin_worker_unavailable` and then `agent_stop_unconfirmed`, because
`run-completion.ts` retries `PluginWorkerUnavailableError` only for
legacy runs (`runtimeMode === "legacy" && nativeSessionId === null &&
runnerInstanceId === null`).

**Expected behavior**
The resume waits for the plugin's worker while the plugin re-activates.
It resumes the same lease when the worker comes back. It still refuses,
as `PluginWorkerUnavailableError`, after the bound.

**Steps to reproduce**
1. Run chaos cell CR6: a K1 restart, then a stall of the replacement's
plugin worker during activation.
2. The reattaching run fails and needs Retry, although the plugin
recovers within seconds.

**Paperclip version or commit**
master at f1aef0cb58.

## What Changed

- `environment-runtime.ts`, `resumeRunLease`: the worker is not running,
but the plugin is `ready` or its activation-timeout recovery is in
flight. In that case the resume polls `isRunning`, every
`pluginWorkerReadyPollMs`, for up to `pluginWorkerResumeWaitMs` (default
60 s). That bound covers the first two activation retries (5 s, then 15
s). When the worker runs, the normal exact resume goes on. After the
bound it refuses with `PluginWorkerUnavailableError`, as before.
- Nothing changes when the worker is running: no wait, no added latency
on the normal path. A plugin that is `disabled`, `uninstalled`, or
`error` for a reason other than a timeout is not waited on.
- `doc/plugins/PLUGIN_SPEC.md` §12 describes the wait.

## Verification

- Red on master: `npx vitest run
src/__tests__/environment-runtime.test.ts -t "re-activating plugin's
worker comes back"` (in `server/`) fails with
`PluginWorkerUnavailableError`.
- Green: the test passes. The worker comes up on the fourth `isRunning`
check, and the lease is resumed with one `environmentResumeLease` call.
- The existing refusal test, now with a 50 ms bound, still refuses with
no provider call.
- `environment-runtime.test.ts` passes (205 tests), and `tsc --noEmit`
passes for `server`.

## Risks

- A restart-recovery run whose provider plugin never comes back now
waits up to 60 s more before it fails. It fails the same way as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:49:23 -06:00
DottaandClaude Opus 5.5 a9a08d8169 fix(plugins): retry a plugin activation whose worker initialize timed out (#20982)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins, sandbox providers such as Daytona among them, run as worker
processes that the server activates at boot. Activation sends the worker
an `initialize` RPC with a 15-second timeout.
> - When `initialize` timed out, the loader marked the plugin `error`
for the life of the server process. Only the next boot or an operator
retried it.
> - For a sandbox provider, that means every Daytona run on that server
fails. New runs fail with `lease_acquire_failed … plugin is currently
error`. Reattachments fail with "The exact sandbox lease cannot be
resumed by its verified provider". Retry hits the same dead plugin.
> - A worker that is slow or stalled for a moment at boot is transient,
not a broken plugin.
> - This pull request retries an activation whose `initialize` timed
out, with a bounded backoff. Meanwhile the server keeps serving, and
runs that need the plugin see a restarting worker, which setup retries.
Deterministic failures stay a permanent error.
> - The benefit is that a short plugin stall at boot no longer disables
a provider until the next deploy.

## Linked Issues or Issue Description

Refs #20966, #20857

**What happened?**
Chaos wave `w-20261111T232850-s9521` (cell CR4, first version) stopped
the replacement server's Daytona plugin worker while the server was
activating it. The worker's `initialize` RPC timed out after 15 s
(`INITIALIZE_TIMEOUT_MS`). `plugin-loader.ts` then called
`lifecycleManager.markError`, and the plugin stayed `error`. Task t03's
reattachment failed in `resumeRunLease`, and its Retry did not start a
new run. Task t04 failed 18 s after launch with `lease_acquire_failed`
("…plugin is currently error"), before its own fault.

**Expected behavior**
An activation timeout is retried, and the plugin ends `ready` when the
worker answers. Runs that need the plugin meanwhile are retried within
their bounded setup budget. A bad manifest, a worker that exits during
`initialize`, or a version mismatch is still a permanent error.

**Steps to reproduce**
1. Start the server.
2. Stop (SIGSTOP) a plugin worker for more than 15 s while the server
activates it.
3. Resume the worker. The plugin stays `error` until the next boot.

**Paperclip version or commit**
master at dfd2a6efca.

## What Changed

- `plugin-worker-manager.ts`: the "Worker initialize failed" error keeps
its cause, the `JsonRpcCallError` with `PLUGIN_RPC_ERROR_CODES.TIMEOUT`.
- `plugin-loader.ts`, `activatePlugin`:
- An activation whose error chain has the TIMEOUT code is retried after
5, 15, 45 and 120 s (`activationTimeoutRetryDelaysMs`).
- Before each retry, the loader tears down the partial runtime and keeps
the plugin `ready`. It sets `lastError` to "Activation timed out;
retrying in Ns (retry n of 4): …" so operators can see the state.
- A retry stops when the plugin is no longer `ready` or its worker
already runs. Shutdown clears pending retries.
- After the last retry, the loader calls `markError` with "Activation
timed out after 4 retries: …". Any other failure calls `markError` with
"Activation failed: …" at once, as before.
- `pluginLoader.requestActivationTimeoutRecovery(plugin)`: re-activates
a plugin in `error` whose `lastError` starts with "Activation timed
out", at most once per minute.
- `app.ts` exposes it through
`workerManager.requestActivationTimeoutRecovery`.
- `environment-runtime.ts`:
- Lease acquisition, for both pinned and key-resolved providers, calls
`requestActivationTimeoutRecovery` for such a plugin and reports
`worker_unavailable` (`PluginWorkerUnavailableError`, which setup
retries) instead of the terminal "plugin is currently error".
- `resumeRunLease` reports a stopped worker as
`PluginWorkerUnavailableError` when the plugin is `ready`, or is
re-activating from a timeout, instead of the generic "cannot be resumed
by its verified provider".
- This covers every plugin kind: activation retry is in the shared
loader. The on-demand path is in the sandbox-provider resolution, which
is the path a run waits on.
- `doc/plugins/PLUGIN_SPEC.md` §12 describes the rule.

## Verification

- Red on master:
`server/src/__tests__/plugin-loader-activation-timeout.test.ts` uses a
real PostgreSQL plugin registry and a bundled image fixture, and all 3
of its tests fail on master. On master, "initialize times out once, then
succeeds" stays `error` and the worker starts only once.
- Green, all 3 pass:
- A timeout followed by success ends `ready`, with a running worker and
no `markError`.
- Timeouts past the bound end `error` with the "Activation timed out
after 2 retries" message, and an on-demand request re-activates the
plugin to `ready`.
- A worker crash during `initialize` is a permanent error, with no retry
and no on-demand re-activation.
- `environment-runtime.test.ts` has new tests:
- A sandbox plugin errored by activation timeouts gives
`PluginWorkerUnavailableError` and requests re-activation. When no
re-activation is in flight, the error stays terminal.
- `resumeRunLease` on a stopped worker of a `ready` or re-activating
plugin gives `PluginWorkerUnavailableError` and sends no provider call.
- These files pass: `environment-runtime`,
`environment-run-orchestrator`, `plugin-loader-error-retry`,
`plugin-environment-driver-ready-recovery`,
`bundled-plugin-registration`, `plugin-lifecycle-restart`,
`plugin-lifecycle-delivery` and `plugin-dev-watcher`.
- `tsc --noEmit` passes for `server`.
- Chaos: cell CR6 (a plugin stalled during activation at boot) runs
before and after this change. The results are in a comment.

## Risks

- A plugin whose `initialize` keeps timing out now takes about 3 minutes
to reach `error`, instead of 15 s. During that time runs that need it
see a restarting worker and are retried within their setup budget.
- On-demand re-activation is limited to one attempt a minute per plugin,
so a stuck plugin cannot crash-loop.
- Nothing changes for deterministic activation failures.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 21:18:47 -06:00
DottaandClaude Opus 5.5 50b79d711a fix(runner): keep call_api writes carrying their body (#21002)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agents use the `call_api` tool when no dedicated tool covers
an API operation
> - #20606 added an optional `saveResponse` flag to the shared
`call_api` schema for GET reads
> - After that change, OpenCode agents with DeepSeek v4.1 flash often
sent write calls without `body`
> - Body-less POST calls reached the routes and failed with 400; agents
then repeated the same call many times
> - This pull request moves the save option to its own read-only tool
and refuses body-less writes before they are sent
> - The benefit is that the write schema returns to its earlier shape,
and a missed `body` gets one clear, actionable error

## Linked Issues or Issue Description

Refs #20606

**What happened?**

In a full chat-eval cohort on master (OpenCode, DeepSeek v4.1 flash),
write calls through `call_api` arrived without `body`. The model's own
reasoning planned a body, but the emitted calls had only `operationId`,
`pathParams`, `query` and `saveResponse`. About 100 refused PATCH calls
listed exactly those arguments (the cohort before #20606 had none with
`saveResponse`). 72 POST calls reached the server with no body and got
400 (0 in the earlier cohort). One case sent the same body-less resolve
call 21 times.

**Expected behavior**

Write calls carry their request fields in `body`. When a body is
missing, the agent gets one clear error before anything is sent, and it
can correct the call.

**Steps to reproduce**

Run the OpenCode chat cases that resolve approvals, edit queued
messages, delegate with context, edit task properties or start preview
services on master after #20606. Count `call_api` writes without `body`
and POST 400 responses.

**Additional context**

The tool schema that OpenCode sends to the model is intact: `body` is
present. A short-context replay with the same schema did not reproduce
the omission (direct provider calls and local OpenCode runs: every write
carried `body`, with or without `saveResponse`). The omission appears in
long sessions, and it correlates with the added read-only flag. This
change removes that flag from the write schema and adds a safety net for
the remaining cases.

## What Changed

- Remove `saveResponse` from the advertised `call_api` schema and
description. The description now says that request fields go only in
`body`, and the `body` description says when it is required.
- Add a separate read-only `save_api_response` tool (`operationId`,
`pathParams`, `query`). It accepts only GET operation IDs, has no body
field, and uses the existing `saveResponse` export path with the same
access, sensitivity, budget and integrity rules.
- Refuse a write before dispatch when its body is missing: PATCH with a
required body (as before), and POST or PUT whose JSON request schema
names required fields. The error names the operation, the allowed
fields, the received arguments, the shape to resend, and that `query`,
`responseText` and `saveResponse` never carry request fields.
- Keep server-side acceptance of `saveResponse` on `call_api`, so saved
calls and receipts from earlier runs stay valid for recovery.
- Regenerate the runner contract, catalog, seeded fixture and manifest.
Update `doc/runner-api-tools.md`.

## Verification

- Red at master sources: the 3 replayed recorded writes (POST
resolve-from-comment, PATCH issue, POST comment, each with `query: {}`
and `saveResponse: false` and no body) were sent to the route instead of
being refused, and the schema test found no `save_api_response` tool. 4
tests failed.
- Green with this change:
`server/src/services/native-runtime/runner-api.test.ts` 2105/2105
passed; `runner-api.integration.test.ts` 61/61 passed;
`runner-bridge-tools.test.ts`, `task-document-key-contract.test.ts` and
`runner-api-artifacts.test.ts` passed; runner
`protocol-action-contracts.test.ts` 70/70 passed; server `tsc --noEmit`
passed.
- `runner-api-artifacts.integration.test.ts` now saves through
`save_api_response` and checks that a write operation ID or a body is
refused. Its download step needs Linux; it fails on macOS at master too,
so CI covers it.
- Paid E2E, OpenCode + DeepSeek v4.1 flash, [run
43138806990](https://github.com/paperclipai/paperclip/actions/runs/43138806990)
on this fix: 5/5 affected cases passed
(questions-and-approvals.reject-and-revise,
steering.edit-queued-message, task-coordination.delegate-with-context,
tasks.edit-properties, workspaces.preview-service). Body-less write
refusals went from 30 to 0, and body-less POST 400s went from 48 to 0,
compared with the earlier cohort on the same cases. The approval resolve
call was sent once (it was sent 24 times before). Every logged write
carried a body; the remaining non-2xx writes are the cases' intended
stale-edit and failing-fixture responses.
- Run-history redaction also covers `save_api_response` arguments; a
regression test fails on the earlier source and passes now.

## Risks

- Agents that learned `call_api` + `saveResponse: true` in a running
session will see a schema refusal and must use `save_api_response`.
Saved calls keep working for recovery.
- The new POST/PUT refusal applies only when the JSON request schema
names required fields, so writes whose body is optional are still sent.
- One more small read tool is advertised when API tools are enabled.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool
use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:46:14 -06:00
DottaandClaude Opus 5.5 faa78dd818 fix(agents): keep a managed instructions bundle managed when a restarted volume-free server saves another setting (#20997)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent settings are saved through `PATCH /api/agents/:id`. On a
volume-free server, a save is refused when the agent's instructions are
external, because files outside the managed root do not survive a
restart
> - A managed bundle saves its instruction paths as absolute paths under
the home of the server that wrote them. A replacement volume-free server
can start with a new home
> - After such a restart, any `adapterConfig` change re-derived the
bundle mode from the saved `instructionsFilePath`. That path is outside
the new managed root, so the bundle was read as `external` and the save
failed with 422 "Volume-free servers require managed agent instructions"
> - This pull request keeps a managed bundle managed when the save keeps
its saved `instructionsFilePath`, and rewrites its paths under the
current managed root
> - The benefit is that a board user can change one setting (for example
`idleTimeoutMs`) after a restart, and a real switch to external
instructions is still refused

## Linked Issues or Issue Description

**What happened?**
In restart-chaos wave `w-20261112T012841-s7000` (volume-free mode,
Daytona, Codex, restarts K1 and K15 with a fresh server home), the board
sent `PATCH /api/agents/:id` with
`{"adapterConfig":{"idleTimeoutMs":301001}}`. All 3 tasks of the W2C
workload got 422 "Volume-free servers require managed agent
instructions".

**Expected behavior**
A partial `adapterConfig` change of an agent with managed instructions
succeeds on a restarted volume-free server.

**Steps to reproduce**
1. Create an agent with a managed instructions bundle on a volume-free
server.
2. Restart the server with a new `PAPERCLIP_HOME`.
3. Send `PATCH /api/agents/:id` with
`{"adapterConfig":{"idleTimeoutMs":301001}}`.
4. The server returns 422.

**Why Codex looked different from Claude**
It is not the adapter. The Claude wave that passed
(`w-20261112T010036-s6801`) had no faults, so its PATCH ran on the
server that wrote the bundle. The Codex wave sent its PATCH after K1 and
K15 restarts with a new home. The new test fails the same way for a
Claude agent after a fresh-home restart, and passes for both adapters on
the original server.

**The PATCH contract**
`adapterConfig` merges into the saved config unless the request sets
`replaceAdapterConfig: true` (validator in
`packages/shared/src/validators/agent.ts`, route in
`server/src/routes/agents.ts`). The UI full-config save sends
`replaceAdapterConfig: true` with the whole config. The merge was
already correct. The fault was in
`syncInstructionsBundleConfigFromFilePath`, which the route applies to
the effective (merged) config: it classified the bundle only from the
absolute file path and the current managed root. The contract does not
change.

## What Changed

- `server/src/services/agent-instructions.ts`
(`syncInstructionsBundleConfigFromFilePath`): when the saved bundle is
managed and the effective config keeps the saved `instructionsFilePath`,
keep it managed with the root that `deriveBundleState` already resolves
(the current server's managed root) and the saved entry file. A
different `instructionsFilePath` is classified as before.
- `server/src/__tests__/agent-volume-free-config-patch.test.ts`: route
tests with embedded Postgres for a Codex agent and a Claude agent: a
one-setting change after a fresh-home restart, the same change on the
original server, and a real switch to an external path, which is still
refused.
- `doc/DEVELOPING.md`: describe how a volume-free server keeps a managed
bundle across a new home.

## Verification

- Red on `origin/master` (`9dfdeaad86`), `cd server && npx vitest run
src/__tests__/agent-volume-free-config-patch.test.ts`:
- `× changes one codex setting after a fresh-home restart and keeps
managed instructions`
- `× changes one claude setting after a fresh-home restart and keeps
managed instructions`
- `AssertionError: {"error":"Volume-free servers require managed agent
instructions"}: expected 422 to be 200`
- The 4 other cases passed (the original server, and the refused
external switch).
- Green with the fix: the same file passes 6/6, together with
`agent-instructions-service`, `agent-instructions-routes`,
`ephemeral-server-contract` and `agent-configuration-save` (65 tests).
- The restart-chaos W2C workload (`tests/restart-chaos/rewake.ts`)
already sends this PATCH after K1/K15 restarts and caught the fault as
`T:config_changed_before_wake`. No new chaos cell is necessary.

## Risks

- Low risk. The change applies only when the saved bundle is managed and
the save keeps the saved `instructionsFilePath`. Then the paths are
rewritten under the current managed root, which is the root that bundle
reads already use.
- The volume-free guard is unchanged. A save that sets an
`instructionsFilePath` outside the managed root is still refused (tested
for both adapters).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:06:51 -06:00