Commit Graph
3069 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 c6c7a61e2c fix(retry): release a Retry held on a resolved decision (#21267)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails, the task shows a Retry button. Admission
can hold an accepted Retry while a person must answer a pending approval
or question (`decision_pending`).
> - The hold is saved on the deferred Retry receipt. Nothing cleared
that saved reason when the person answered, so the task kept saying
"Retry is waiting for a pending approval or question" when nothing was
left to answer.
> - The deferred-receipt worker drops a refused receipt from its
schedule. No producer hinted it when the decision resolved, so the held
Retry did not start until some unrelated event rescanned the company.
> - A Retry was also held by a confirmation that was already open when
the failed turn was admitted. That turn had already run past the
confirmation, so the Retry added a gate that the original turn did not
have.
> - A person's comment on the task started a continuation, but the held
Retry receipt stayed deferred next to it.
> - This pull request makes a resolved decision release the held Retry
at once, keeps the projection from showing a resolved decision as the
wait, stops holding a Retry on a decision that predates the failed turn,
and settles a held Retry when a comment or answer continues the same
failed run.
> - The benefit is that Retry always works: a person never waits on a
decision that is already resolved.

## Linked Issues or Issue Description

**What happened?**

A board user pressed Retry on blocked tasks. No run started. The task
said "Retry is waiting for a pending approval or question to be
resolved. Your message is saved." In one task, the only interaction was
a `request_confirmation` that the previous successful run created. The
user's later comment started the turn that failed. The confirmation
stayed pending, so every Retry attached to the same held receipt. After
the user rejected the confirmation and sent a new comment, the task
still showed the same wait. No run started until a server restart
rescanned the deferred receipts about three minutes later. The task
projection said `canRetry: true` with the `decision_pending` wait.

**Expected behavior**

An accepted Retry is never held on a decision that is already resolved.
Accepting or rejecting the decision releases a held Retry right away. A
person's comment on the blocked task starts work, and the held Retry is
settled. The task projection never shows a resolved decision as the
Retry wait.

**Steps to reproduce**

1. Let an agent run on a remote runner create a `request_confirmation`
for the task.
2. Let the runner be lost so the run fails with
`native_continuation_requires_reconciliation`, and let saved work
recovery complete.
3. Press Retry. The Retry receipt is saved with `executionWait.reason =
decision_pending`.
4. Reject or accept the confirmation.
5. Observe that the receipt keeps `decision_pending`, the task keeps
showing the wait, and the Retry does not start until an unrelated
rescan.

**Paperclip version or commit**

Staging Core `57aeda190b`. Fixed on top of `origin/master` `53608bd1d6`.

**Deployment mode**

Hosted cloud (staging), remote sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (native runtime).

## What Changed

- `hasPendingContinuationDecision` (`explicit-native-continuation.ts`)
is now the single check for a decision that holds a person's
continuation. Admission and the task projection both use it.
- For a Retry, a pending interaction that was created before the failed
source run was created does not hold the Retry. That turn was already
admitted while the decision was open. Decisions that the failed run
created (for example a plan review) still hold the Retry.
- `releaseDecisionHeldExecution` (`deferred-execution-wait.ts`) removes
the saved `decision_pending` wait from deferred receipts on the task and
hints each receipt to the deferred-receipt worker. It runs in the
resolution transaction of every interaction resolution path (through
`notifyChatConfirmationResolution`) and in question answers.
- The task projection reads the live decision state. A saved
`decision_pending` wait whose decision is resolved is not shown. This
also repairs receipts saved before this change.
- When a comment or an interaction answer continues a failed run,
`retireTaskRetriesContinuedBy` (`task-retry.ts`) settles held Retry
receipts for the same source run as `task_retry_superseded`.

## Verification

- Red/green: `cd server && npx vitest run
src/services/governed-wait-runner-loss.test.ts`. The five new tests fail
on `origin/master` sources and pass with this change (17/17 pass):
  - releases the held Retry as soon as the person chooses accept
  - releases the held Retry as soon as the person chooses reject
- never shows a resolved confirmation as the Retry's wait, and both
projections offer the same Retry
- a person's comment starts work and settles the held Retry once nothing
is left to answer
- starts a Retry at once when the confirmation was already open before
the failed turn started
- The existing tests that keep a Retry or a message on hold while a plan
review, governed tool action, secret proposal, or connection
authorization is pending still pass.
- Related suites: deferred execution wait, explicit native continuation,
interaction resolution and service, task interruption presentation and
query, task retry, deferred receipt work, interaction routes.

## Risks

- Medium-low. A Retry now starts when the only pending interaction is
older than the failed run. That turn already ran with the interaction
open, so the Retry gets no new authority. Governed tool actions still
need their own approval before the tool runs.
- Interaction resolution now also updates deferred receipts on the same
task. The update is limited to receipts with a saved `decision_pending`
wait. A receipt that is still blocked by another pending decision is
held again on its next admission.
- Issue-linked approvals (`approvals` table) do not yet hint the held
receipt when they resolve. The projection no longer shows them as the
wait after they resolve.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:58:48 -06:00
DottaandClaude Opus 5.5 8a0f95849d perf(native-runtime): start the first turn without re-copying launch files, and never fail a run on a slow copy (#21287)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode in a remote sandbox must have a durable,
verified copy of the provider launch files before their first turn (the
rule from #19622)
> - Today each new run copies these files out of the sandbox, uploads
them and verifies them again, even when the company already has a ready
artifact with the same files
> - The whole copy runs under one five-minute clock. When eight runs
start together on small sandboxes, the copies compete, and slow copies
fail the run before its first turn
> - This makes cold starts slow (p50 273 s from run start to turn
submitted) and makes half of the first runs fail in a burst
> - This pull request reuses a ready artifact with the same digests,
shares one in-flight copy per company and manifest, and bounds each
network step instead of the whole copy
> - The benefit is that first turns start in about one minute on a cold
burst and in about 20 s when the artifact is already ready, and a slow
copy no longer fails the run

## Linked Issues or Issue Description

Refs #19622

**What happened?**
When 8 OpenCode runs started together on 1 vCPU Daytona sandboxes, 4 of
8 first runs failed before their first turn. Each run copied, uploaded
and verified the same provider launch files. The single 300 s clock over
the whole copy expired. Run start to turn submitted was p50 273 s and
p95 314 s.

**Expected behavior**
A run starts its first turn without a new copy when a ready, verified
artifact with the same files exists. A slow copy does not fail the run.

**Steps to reproduce**
1. Use a company with remote OpenCode agents on Daytona sandboxes with
durable storage.
2. Start 8 runs at the same time on new sandboxes.
3. See the time before the first turn and the failed runs.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- `retainReadyNativeProviderArtifacts` (`native-provider-artifacts.ts`):
reuse a ready, verified company artifact with the same paths and SHA-256
digests through a lookup only. No file is copied, uploaded or verified
again. The lookup (`readyNativeProviderArtifactCandidates`) matches
target, paths and digests in Postgres with jsonb containment on the
company's ready rows (existing partial index), newest first, limit 4. It
does not load or hash other retained versions. No migration.
- `shareNativeProviderArtifactCapture`
(`native-provider-artifact-capture.ts`): one in-flight capture per
company and manifest digest per server. Other runs wait for it, then
reuse its result.
- The single 300 s clock over the whole capture is replaced by a bound
for each network step (upload, verifying download): max(120 s, size at
256 KiB/s). A timeout has the typed cause
`native_provider_artifacts_transfer_timeout`. The capture makes up to 3
attempts from the local copy.
- Stop now interrupts the capture and the wait for a shared capture
(`ActiveNativeSession.admissionAbort`).
- `doc/opencode-cold-recovery.md` describes the new behavior.
- Tests: new `native-provider-artifact-gate.test.ts` (7 tests), 5 new
tests in `native-harness-backup-storage.integration.test.ts`, and new
executor variants `opencode-new-durable-slow-capture` and
`opencode-new-durable-ready` in `native-session-executor.test.ts`.

## Verification

Local (this branch, on master 9b613562b7):

- `npx vitest run
src/services/native-runtime/native-provider-artifact-gate.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
3 files, 1511 passed, 7 skipped.
- `npx tsc --noEmit -p server`: pass.
- Lookup scale: the integration test "loads only the ready artifact
whose launch files match, however many other versions the company keeps"
adds 200 ready non-matching versions. It returns only the matching
artifact. Without the SQL filter it fails (4 rows returned).
- Red/green: with master's `native-session-executor.ts` and the new
tests, `opencode-new-durable-slow-capture` fails with `TimeoutError: The
operation was aborted due to timeout`, and `opencode-new-durable-ready`
fails because the ready artifact is not looked up. With this change,
both pass.

Daytona (same commit content, cherry-picked onto the baseline commit
84a1ee3e11 with #21262; image `sha256:9461cc75…`; 1 vCPU sandboxes;
OpenCode):

| Wave | Shape | First runs OK | Run start → turn submitted | Other |
|---|---|---|---|---|
| Baseline `w-20261113T011027-s1702` | 8 runs started together | 4/8 (4
failed) | p50 273 s / p95 314 s | "only a getter" TypeError ×8,
`bootstrap_not_proven_incomplete` ×8 |
| Fix `w-20261113T031045-s1702` | same seed and shape | 8/8 | p50 66.0 s
/ p95 85.7 s | 8/8 tasks hands-off, 0 Retry banners; first agent event
p50 72.9 s / p95 90.0 s; exactly 1 upload, 7 runs waited on it (~65 s
shared capture); all failure codes 0 |
| Fix `w-20261113T034033-s1702` | 16 tasks, cap 8 | 16/16 | p50 19.1 s /
p95 24.3 s | second batch (artifact already ready, cold sandboxes) gate
time 24, 25, 26, 27, 31, 31, 45, 119 ms; first agent event p50 25.9 s |

## Risks

- The rule from #19622 still holds: a durable, verified artifact exists
before the first turn. Reuse only accepts a ready, verified artifact of
the same company with the same paths and digests.
- The first run per company and manifest per server still pays the full
transfer.
- Two servers can still upload duplicate artifacts for the same company
and manifest. Sharing is per server.
- No test drives Stop through the executor during a capture. Signal
handling is unit-tested only.
- A capture that fails all 3 attempts still ends blocked with a working
Retry. A follow-up PR (pre-turn failures continue automatically) removes
that.
- The documented five-minute deadline per capture is replaced by bounds
per network step. A very large artifact on a slow link can take longer
than five minutes in total, but each step still has a bound.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Default context
window; extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:48:55 -06:00
DottaandClaude Opus 5.5 0c04ff76eb fix(native-runtime): a follow-up that arrives as a warm session closes continues the task (#21279)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent session on a remote sandbox stays warm after a run,
and an idle timeout closes it after about 5 minutes
> - When a person's follow-up comment arrived just as that idle close
began, the new run failed with `native_session_supervisor_busy`
> - The idle close of the finished owner could also fail its final
checkpoint with `native_runner_checkpoint_owner_lost`, because the task
already pointed at the queued follow-up run
> - Then the retry was held for workspace repair and the task stayed
blocked; the environment could not be deleted (409) because its warm
runner could not be stopped
> - This pull request makes the follow-up wait for the idle close that
is in flight, and lets the finished owner save its final checkpoint
while its successor still waits in the queue
> - The benefit is that a follow-up that lands at the warm session's
idle close continues the task with no person involved

## Linked Issues or Issue Description

Refs #18011 (the checkpoint ownership fence this change extends).

**What happened?**

A follow-up comment arrived 295–340 s after the previous run ended,
while the warm session's idle close was in progress. The new run failed
with `native_session_supervisor_busy`. In another case, the idle close
failed its final checkpoint with `native_runner_checkpoint_owner_lost`,
the next run failed
`native_instruction_materialization_handoff_unavailable`, and the retry
was cancelled with `execution_reconciliation_required`. The task stayed
blocked. A hosted instance showed the same class as "previous run's
session on this sandbox could not be retired".

**Expected behavior**

The follow-up run waits for the idle close (inside the existing bounded
release wait) and then starts. The finished owner saves its final
checkpoint and closes. The task completes hands-off.

**Steps to reproduce**

1. Run an opencode agent on a Daytona sandbox with a warm session (idle
timeout 300 s).
2. Let a run finish.
3. Post a comment on the task 298–306 s after the run ended.

**Paperclip version or commit**

master `444dc18d98`.

**Deployment mode**

Local server with Daytona remote sandboxes (restart-chaos harness, light
lane). The same class was seen on a hosted staging instance.

## What Changed

- `claimWarmNativeInstructionCopy` now also waits when an idle close is
in flight for the session scope (`closingWarmNativeSessions`). The wait
uses the same deadline as the existing release wait
(`warmSessionReleaseWait`), so it adds no new wait and nothing before
`turn.start` on the normal path.
- `validateNativeRunnerCheckpointOwner` accepts the finished owner's
final checkpoint (accepted result, no new work) when
`issues.executionRunId` points at a successor run that is `queued` or
`scheduled_retry` for the same company and task and has no
`native_run_finalizations` row.
- Tests: a server test for a follow-up that arrives while the idle
timeout closes the warm session; four new scenarios (`queued_successor`,
`scheduled_retry_successor` × `runner.suspend`/`turn.stop`) and four new
negatives (`queued_wrong_issue`, `queued_new_work` × both commands) in
the early-retirement checkpoint matrix.
- Restart-chaos harness:
`RESTART_CHAOS_MIX_FOLLOW_UP_AFTER_END_MS=<min>-<max>` and
`RESTART_CHAOS_MIX_FOLLOW_UP_COUNT` post the staging-mix follow-up a set
delay after the task's last run ended, so a cell can aim the follow-up
at the warm idle close. The restart-chaos README describes both
variables. A failed post is retried with the same `clientRequestId`, the
grade check `after_end_follow_ups_served` needs an agent reply after
each post and the ledger marker, and the task budget includes the
planned waits.
- Harness: the wave database cleanup
(`tests/runner-e2e/controller-database.ts`) no longer waits forever when
a group signal (for example `timeout` or Ctrl-C) already stopped the
embedded Postgres. Before, `embedded-postgres` `stop()` waited for an
exit event that had already fired, and an interrupted wave hung after
the Daytona R6 sweep.

## Verification

- Fix 1, red without the product hunk: `native-session-executor.test.ts
-t "idle timeout is closing"` fails with `promise rejected "Error:
native_session_supervisor_busy"`. Green with the fix: 1 passed.
- Fix 2, red without the product hunk:
`native-runner-restart-recovery.integration.test.ts` fails 4 of 208
(`queued_successor` and `scheduled_retry_successor` ×
`runner.suspend`/`turn.stop`) with
`native_runner_checkpoint_owner_lost`. Green with the fix: 208 passed,
14 skipped. The two new negative cases pass in both states.
- Harness hang fix, red without the fix: the new
`controller-database.test.ts` case ("finishes cleanup when the database
already stopped because a group signal reached it") times out on cleanup
(`expected 'hung' to be 'cleaned'`). Green with the fix: 5 passed, 1
skipped.
- After the rebase onto master `7154030ef6`, both server test files
together: 1639 passed, 21 skipped.
- Restart-chaos harness tests (`staging-mix`, `staging-mix-retry`,
`observer`): 75 passed, including a test for the after-end retry, the
`after_end_follow_ups_served` grade check and the added budget. `tsc
--noEmit` passes for `server`, `tests/runner-e2e` and
`tests/restart-chaos`.
- `pnpm verify` exits 2 because `tests/restart-chaos/observer.ts` is a
global CI input. Required CI supplies the broad coverage.
- Daytona cell on this branch (4 opencode tasks, warm sandboxes, 1 vCPU,
follow-up comment posted 300.8–308.2 s after each task's last run ended,
image built for master `444dc18d98`): 4/4 tasks done hands-off, 8/8 runs
succeeded, 0 Retry banners, 0 `owner_lost` / `supervisor_busy` / "could
not be retired" / `execution_reconciliation_required` lines in the
server log, environment delete clean, warm-shutdown grade pass, 0
sandboxes left. The harness exited by itself after 1076 s.
- Same cell before the fix (master, 8 tasks): 1/8 tasks blocked
(`native_session_supervisor_busy` →
`native_runner_checkpoint_owner_lost` →
`execution_reconciliation_required`). With fix 1 only: 1/8 still blocked
(`owner_lost` on the idle close). With both fixes: 8/8 done and 0 failed
runs (cut by the 20-minute cap before cleanup), then 4/4 above.
- The cell grade `T:completion_once` fails on all 4 tasks by design: the
timed follow-up comment arrives after the task is done, so the agent
posts a second completion comment.

## Risks

- Fix 1 can delay a follow-up by up to the existing warm release wait
(30 s) while an idle close finishes. Before this change, that run
failed. If the close fails or the wait expires, the claim returns null
and the run uses a fresh instruction copy, as before.
- Fix 2 widens the checkpoint ownership fence from #18011. It does not
reopen what #18011 closed:
- The exemption applies only to the finished owner's final save: the
checkpoint must hold an accepted result and no new work, so no provider
turn can be granted through it.
- The successor must still be unclaimed (`queued` or `scheduled_retry`)
and must have no native coordinator (`native_run_finalizations`) row. No
controller has started it, so nothing can have consumed or depended on
the predecessor's checkpoint. Once the successor is claimed, the
existing preparing/claimed rules from #18011 apply unchanged.
  - The successor must belong to the same company and the same task.
- Cases that must still reject, with tests in the same matrix:
- `queued_wrong_issue`: a queued successor for a different task is
rejected with `native_runner_checkpoint_owner_lost` and the stored
checkpoint does not change.
- `queued_new_work`: a `turn.start` (new work) while the successor is
queued is rejected with `native_runner_checkpoint_owner_lost` and the
stored checkpoint does not change.
- All earlier rejections stay: unclaimed native coordinator,
dispatching, expired or foreign preparation, foreign native preparation,
wrong task, new work, unaccepted result, replaced controller.
- Not changed: an owner whose close fails still blocks environment
delete (409). That is a separate product decision.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Earlier commits on
this branch were written by the same model in an earlier session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:42:50 -06:00
Awssandcommitperclip[bot] af1c685f5f fix(secrets): return 422 for secret proposals missing a required field (#15348)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can propose a company secret or a secret binding through
the API.
> - The proposal route reads the request body and passes it to the
service.
> - The service trimmed required string fields before it checked them.
> - A request that omitted name or justification threw a TypeError.
> - The route then returned 500 instead of a validation error.
> - This pull request checks the required fields first and returns 422.
> - The benefit is that a client can tell a bad request from a server
fault.

## Linked Issues or Issue Description

Closes #15334

## What Changed

- Check `name` and `justification` in `createSecret` before `trim`, and
return 422 when either is missing.
- Check `justification` in `createBinding` the same way.
- Keep the existing empty-value and malformed-value checks and the
optional `key` field unchanged.
- Add a regression test that covers the missing-field cases for both
proposal kinds and keeps the complete-body case at 201.

No documentation change is needed. The OpenAPI document already lists
422 for this route, so the server now matches the documented contract.

## Verification

- Run the route suite. It passes.

`./node_modules/.bin/vitest run
server/src/__tests__/secret-proposals-routes.test.ts` gives 26 passed.

- Revert only the service change and run the new test. It fails with
`expected 500 to be 422`, which proves the test catches the bug. Restore
the change and it passes again.

- Run the server typecheck. It passes.

`pnpm --filter @paperclipai/server run typecheck` exits 0 with no
errors.

- A `POST /api/agents/me/secret-proposals` that omits `justification`
returns 422 after the fix and returned 500 before it. A complete body
still returns 201.

## Risks

- Low risk. The change adds two field checks before an existing `trim`
call. Valid requests still return 201. A missing required field now
returns 422, which the route already documents.

## Model Used

Claude Opus 4.8 (`claude-opus-4-8`), with extended thinking and tool
use. I chose the issue, set the evidence bar, directed each step, and
reviewed the code, the tests, and the results before this PR. The model
traced the route and the service, wrote the fix and the regression test,
and ran the checks under that direction.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
2026-11-12 22:31:42 -06:00
Nicky LeachandClaude Opus 5.5 717aae060a fix(environments): let same-run recovery create a replacement once its sandbox is proven stopped (#21223)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute on environment leases, and `createOwnedSandbox`
records each sandbox creation so that a lost reply never creates a
second sandbox
> - The record key is company, environment, run and request key. The
plugin request key is the environment fingerprint, so it is the same for
every attempt of one run
> - A same-run recovery can need a new environment after the first lease
is closed with successful cleanup and a termination receipt
> - The old record is still `attached` to the closed lease, so the
recovery fails with "Sandbox creation has already transferred ownership"
before the provider is called
> - This pull request lets the creation fence accept the same stop proof
that recovery already requires. A creation record whose sandbox is
proven stopped becomes resolved, and a resolved record no longer holds
its key
> - The benefit is that same-run recovery gets one replacement sandbox,
while unconfirmed cleanup still blocks a second sandbox

## Linked Issues or Issue Description

Fixes #21214

Refs #20755. That pull request fixes the same failure with a different
rule. It retries under a new key generation (`<key>#2`, `#3`, ...) when
the recorded lease has ended, and it accepts any ended status, including
`failed`, without a termination receipt. A lease whose cleanup is not
confirmed can then produce a second sandbox while the first one can
still run. This pull request keeps the original key and requires the
same stop proof that Core uses elsewhere
(`hasRemoteTerminationReceipt`). It does not add a key generation.

## What Changed

- `server/src/services/sandbox-creation-operations.ts`: when
`createOwnedSandbox` finds the key held by an `attached` operation whose
lease is not active, it checks the same stop proof that same-run
recovery requires before it acquires:
`hasRemoteTerminationReceipt(resolveRemoteTerminationLease(lease))`, for
the sandbox that this operation created (`creationProviderLeaseId`, or
the lease's `providerLeaseId`).
- With that proof, it runs the same admission transaction one more time.
That transaction first marks the old operation `resolved` (`WHERE id AND
state = 'attached' AND lease_id`), and then inserts the new operation.
  - Without that proof, it throws exactly as before.
- The check covers every source of a stop receipt: `releaseLease`, the
Runner `complete` operation (which writes receipts with direct SQL for
reusable leases), stop-and-retain settlement, and the receipt that
`resolveRemoteTerminationLease` derives for an expired reusable lease.
- A task replacement can leave an earlier operation attached to a lease
that now holds another sandbox. That operation is not proven and keeps
its key.
- Concurrent recoveries serialize on the row update. The second one
updates nothing, finds the first one's unresolved operation through the
partial index, and replays or refuses as before. There is one provider
call.
- Migration `0475_sandbox_creation_owner_unresolved.sql` recreates
`sandbox_creation_operations_owner_idx` under the same name as a partial
unique index (`state <> 'resolved'`). The drop and the create are in one
migration transaction. The Drizzle schema matches.
- The conflict lookup in `createOwnedSandbox` ignores resolved rows. An
active attached lease is still replayed. Pending and
`needs_reconciliation` rows still refuse. `releaseLease` does not
change.
- `doc/DATABASE.md` and `doc/idle-environment-cleanup.md` describe the
operation lifecycle.

## Verification

- `pnpm --filter @paperclipai/db generate`
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/server typecheck`
- `vitest run src/migration-snapshot-drift.test.ts` (db)
- `vitest run src/__tests__/sandbox-creation-operations.test.ts`: 48/48.
With the service change reverted, the 5 replacement tests fail.
- The 9 other server test files that read `sandbox_creation_operations`,
plus `environment-runtime.test.ts`: 564/564
- `pnpm check:migrations` passes

New tests in `sandbox-creation-operations.test.ts`:

- A stop receipt from `releaseLease` (sandbox driver, and the plugin
driver through `acquireRunLease` in the #21214 sequence), from the
`complete` operation's direct write on a retained reusable lease, and
derived only from the destroyed successor of an expired reusable lease
each let the next creation resolve the old operation and create exactly
one replacement. A retry replays that replacement. These tests fail
before the fix.
- Failed cleanup, success without a receipt, a receipt for another lease
or another provider task, a retained lease, and pending cleanup each
still throw "transferred ownership" with no provider call.
- An earlier task's operation that a task replacement left attached
keeps its key.
- Two concurrent recoveries make one provider call. This test fails
before the fix.
- Reapplying migration 0418 keeps the partial index.

I did not run `pnpm verify` or the full suite locally. CI runs them.

## Risks

- Migration: the drop and the rebuild of the index take an exclusive
lock on `sandbox_creation_operations`. The table has one small row for
each sandbox creation, so the lock is short.
- Rolling deploy: the migration is marked `online_compatible`. The
previous release uses an untargeted `ON CONFLICT DO NOTHING`, which also
matches a partial unique index. Its conflict lookup has no state filter,
so it can find a resolved row and throw. That fails closed and never
creates a second sandbox.
- A legacy runner-task operation without `creationProviderLeaseId`
matches any lease it is attached to. Runner-task operations record this
field before they attach, so this is only a theoretical gap.
- An operation stays `attached` until the next creation with the same
key checks it. This is the same as before when no recovery comes.
- Stop-and-retain settlement is not tested directly. The shared receipt
check covers it.
- The fence is unchanged for every case without a stop receipt.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents) and extended thinking. One sub-agent
wrote the change. A second session reviewed it before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:24:34 -08:00
DottaandClaude Opus 5.5 4dc5492c2d fix(recovery): retry a failed Retry run's sandbox resume failure when the sweep reaches it first (#21124)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on its next run
> - When the resume fails before provider work
(`reusable_sandbox_resume_failed`), the run gets a bounded automatic
retry
> - On a hosted staging server, 8 of 14 such failures in 10 hours did
not get that retry. Each task ended `blocked` and needed a person to
press Retry
> - In all 8, the stranded-task sweep blocked the task between the
failed write and the run's own retry scheduling. The block cleared the
claim that the retry needed
> - This pull request makes the sweep give such a run to the same
bounded retry scheduler, not escalate it
> - The benefit is that a transient sandbox resume failure no longer
costs the task, also when settlement is slow or the server stops during
an upgrade

## Linked Issues or Issue Description

Refs #21108, #21113, #21078

**What happened?**
A person pressed Retry on a task. The run tried to resume the task's
saved Daytona sandbox before provider work. The resume failed
(`cause=transport_timeout`, or `provider_unavailable status=503`), and
the run was written `failed` with `reusable_sandbox_resume_failed`. Then
the stranded-task sweep moved the task to `blocked` and cleared its
claim. This happened 1 to 43 seconds after the failed write. The run's
executor scheduled its bounded retry 2 to 81 seconds after the failed
write, and the gate refused it ("The automatic retry could not
revalidate the original user continuation."). Read-only audit of 14
failures (all legacy runtime, `executionRecovery: bootstrap,
providerWorkStarted: false`): 6 got their retry, and the 5 most recent
ones recovered. 8 were refused this way and wait for a person. In one
example the run was written failed at 14:23:31, the task was blocked at
14:24:14, and the retry was refused at 14:24:52. The server was being
replaced at that time.

**Expected behavior**
A transient preparation failure before provider work continues with its
bounded retry. The sweep must not take the claim away from that retry.
When the retry budget is spent, or the continuation is refused, the task
escalates once, as before.

**Steps to reproduce**
1. Press Retry on a task whose reusable sandbox resume fails before
provider work.
2. Let the run's executor be slow to settle the failure (more than the
remaining controller lease, at most 60 seconds), or stop its server
after the failed write.
3. Let the stranded-task sweep run on another server. The task becomes
`blocked`, and no retry is scheduled.

**Root cause**
`classifyContinuationFailure` lists `reusable_sandbox_resume_failed` and
the other transient preparation codes in
`NON_RETRYABLE_CONTINUATION_ERROR_CODES`. The sweep assumes that the
executor already scheduled the dedicated bounded retry, so it escalates
at once and never asks the scheduler. #21108 added a skip while the
executor still settles the run. That skip uses a live local execution or
an unexpired controller lease. The lease stops renewing when the run is
written terminal (renewal requires `status = 'running'`). So the skip
covers at most about 60 seconds after the last renewal, and it covers
nothing when the executor's server stops during settlement.

**Paperclip version or commit**
Staging at `a65644d1e1`. Current `master` has the same code path.

**Deployment mode**
Hosted (cloud), two servers alive during each upgrade.

**Agent adapter(s) involved**
`codex_local` on a Daytona sandbox environment. The change is
adapter-independent.

## What Changed

- `server/src/services/heartbeat/retries.ts`: add
`failedPreparationOwnsBoundedRetry(run)`. It is the persisted form of
the run-completion decision: a failed legacy run with no native session,
bootstrap evidence that provider work did not start, and a transient
preparation code. A restore failure counts only when its owner marked it
retryable. Codes that an accepted interaction retries itself are
excluded.
- `server/src/services/recovery/service.ts`: when the latest run is such
a run, the sweep does not escalate it as non-retryable. It goes to
`scheduleRecoveryRetry`, which is the run's own bounded scheduler. The
scheduler reuses a successor that already exists, so the executor and
the sweep create one successor. The task escalates when the scheduler
reports exhaustion or a refused continuation. When the scheduler
declines the retry for another reason (for example a permanent refusal),
the task escalates once as a non-retryable failure, as before. The
#21108 settling skip still runs first.
- `server/src/services/heartbeat.ts`: pass the predicate to the recovery
service.
- `doc/execution-semantics.md`: document the rule.

## Verification

- New tests in `server/src/services/task-retry.test.ts`, in the staging
shape: a Retry-admitted run failed with
`reusable_sandbox_resume_failed`, bootstrap evidence, and an expired
controller lease from another server.
- "retries a failed Retry run's transient preparation failure when the
sweep reaches it first" (`slow_executor`, `executor_gone`). Red before
the fix: the task became `blocked`, as on staging. Green after the fix:
the task stays `in_progress` with one `scheduled_retry` successor
(`transient_failure`, attempt 1), and dispatch accepts it. For
`slow_executor`, the executor's own scheduling then reuses the same
successor, and there is still one successor.
- "still escalates a failed Retry run's transient preparation failure
once its bounded retries are spent": the task becomes `blocked` with no
successor.
- "escalates a failed Retry run's transient preparation failure once
when the scheduler declines it permanently" (review finding): red before
the follow-up commit (the task stayed `in_progress` with no run), green
after (the task is `blocked` once, and a second sweep adds no notice).
- `pnpm exec vitest run src/services/task-retry.test.ts`: 114/114 pass.
- `pnpm exec vitest run src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/heartbeat-process-recovery.test.ts` (with task-retry): 631
of 633 passed. The 2 failures were 15-second timeouts under parallel
load in native Board-wait tests that this change does not touch. They
passed when run alone.
- `tsc -p server/tsconfig.json --noEmit` passes.
- Not run: a live staging reproduction. The staging evidence is
read-only (run events, task activity, run records).

## Risks

- The sweep can now schedule the bounded retry before the executor does.
The scheduler locks the task and the run and reuses an existing
successor, so there is never a second successor. All continuation gate
checks still apply.
- The sweep acts only after the #21108 settling skip lapses, so a
healthy run is not affected. No wait is added before a normal turn
start.
- No schema change. Rows from older releases follow the same rule
because the predicate reads only persisted fields.
- Out of scope: the provider-side causes (Daytona 503 at 14:21Z, resume
calls that time out). #21113 handles the in-run wait for idle transport
timeouts.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, read-only API reads, Vitest on embedded PostgreSQL) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:20:55 -06:00
Nicky LeachandClaude Opus 5.5 afe86e9ded fix(runtime): record a plugin task's stop receipt when its stop is proven (#21237)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin environment runs each Paperclip Runner turn as a provider
task, and a follow-up run can continue the same Claude/ACPX conversation
from the saved checkpoint
> - Core authorizes that continuation only when the predecessor run's
remote execution has stopped (`remoteExecutionHasStopped`), so two
Runners never write one checkpoint
> - The predecessor's own settlement already proves the stop: the
provider reports `executionStopped`, and Core then completes the task
> - On a per-turn (ephemeral) lease, Core records the stop receipt only
when it releases the lease, and that release can be held for a long time
(for example by a pending workspace export)
> - In that window, an ordinary follow-up fails with `adapter_failed:
Plugin Runner retained session is not authorized`, even though the stop
is already proven
> - This pull request records the stop receipt when the stop is proven,
and the stop check accepts it before the release
> - The benefit is that follow-ups continue as soon as the predecessor
is proven stopped, and the release gates keep their meaning

## Linked Issues or Issue Description

Fixes #21211

Refs #21214 and #21223. Those cover the creation fence for same-run
recovery, which still requires a released lease.

Production evidence for this change: in both observed failures, the
predecessor was a per-turn plugin run. Its `task.settle` completed with
all provider calls successful. Core then did not release its lease for
30 minutes or more, and no release call was made in that time. The
follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint`
→ `resolvePluginTaskResume`) within seconds of admission.

## What Changed

- `server/src/services/environment-task-runtime.ts`
- After a recorded provider stop, `complete` writes the bound stop
receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the
current lease for every lease policy. Before, it did this only for
`reuse_by_environment` leases. Reusable tasks still confirm every owner
and still move retained owners to released.
- New work on a lease (`submit`, `copilot_metadata`, first portable
resume) removes the stop receipt in the same write that clears
`runnerTaskExecutionStopped`. A replacement physical task does not copy
the old receipt.
- `server/src/services/remote-execution-termination.ts`
- New `hasBoundRemoteStopReceipt` checks only the receipt binding
(company, run, lease, provider and physical task).
`hasRemoteTerminationReceipt` uses it and keeps its released-only
semantics. Resource cleanup gates do not change.
- `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a
lease that is released with its receipt, or that is still `active` (not
released) with a bound stop receipt. A lease whose release started and
did not settle does not count. After release starts, the release owns
the receipt.
  - New `remoteExecutionHasReleased` gives the release-based answer.
- `server/src/services/environment-run-orchestrator.ts`: same-run plugin
recovery uses `remoteExecutionHasReleased`. The same-run resume check
after it requires released receipts, so its behavior does not change.
- `resolvePluginTaskResume` keeps its checks and their order. Its
refusal now names the first failing check: `Plugin Runner retained
session is not authorized (reason: <code>)`, for example
`predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The
old message stays as the prefix, and no ids go into it. The
stale-availability recheck in `environment-task-runtime.ts` names its
refusal the same way. Production telemetry for #21211 could not tell
which check fired. The next occurrence will show it.
- `doc/execution-semantics.md` and
`doc/testing-native-remote-continuity.md` describe the stop receipt and
when it is recorded.

Caller audit for the changed meaning of `remoteExecutionHasStopped`:

| Caller | Needs | Decision |
| --- | --- | --- |
| `agent-instruction-working-copies.ts` | stop | Accepts. Directory
cleanup still requires a released "destroyed" receipt. |
| `environment-run-orchestrator.ts` | release | Moved to
`remoteExecutionHasReleased`. |
| `plugin-task-resume.ts` | stop | Accepts. This is the fix. |
| `native-runtime/native-runner-ownership.ts` | stop | Accepts.
`hasPendingRunCleanup` still counts the active lease as pending. |
| `native-runtime/preserved-native-result-finalization.ts` | stop |
Accepts. Workspace recovery has its own gate. |
| `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts.
`hasPendingRunCleanup` still holds active leases. |
| `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop
acknowledgement and quarantined ownership retirement. Holds and blockers
still apply on the normal path. |
| `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors
still fail it. |

## Verification

From `server/`, after `node scripts/agent-preflight.mjs`:

- `npx vitest run
src/__tests__/environment-task-authority.integration.test.ts`: 10/10
pass. The 2 new tests fail without the fix.
- The plugin task and environment runtime suites
(`environment-task-authority.integration`,
`plugin-runner-task.integration`, `plugin-warm-tasks.integration`,
`environment-task-runtime`, `plugin-task-source-recovery.integration`,
`environment-runtime`): 6 files, 366 pass.
- The audited caller suites (`agent-instruction-working-copies`, case
handoff legacy and native, `environment-run-orchestrator`,
`heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`,
`native-remote-retry-admission`, `preserved-native-response-wait`): 9
files, 806 pass.
- `pnpm --filter @paperclipai/server typecheck`: pass.

The new integration tests cover:

- `complete` writes a bound stop receipt on a still-active ephemeral
lease. `remoteExecutionHasStopped` is true and
`hasRemoteTerminationReceipt` is still false.
- `resolvePluginTaskResume` (inspect and launch) authorizes a
follow-up's new task, and the real host `read_state` checkpoint read
succeeds, while the predecessor's release is held.
- Refusals: no provider stop, a receipt for another provider task, an
unstopped sibling lease, and a release in progress.
- New work on the lease removes the receipt. Release overwrites the
receipt with its own receipt. Warm owners keep their behavior.

I did not add an executor-level test. `native-session-executor.test.ts`
mocks `resolvePluginTaskResume`, so the integration test runs the real
checkpoint read path instead. I did not run the full suite locally. CI
runs it.

## Risks

- `remoteExecutionHasStopped` now returns true earlier for a run whose
leases are still held. Each caller is audited above. Callers that need
released resources use the release-based predicates.
- The stop receipt can now be on an `active` lease. The SQL readers of
`remoteExecutionTermination` (cost accounting, execution blockers,
idle-work proof, orphan settlement) also require a released status.
`pinFailedObservedWorkspace` treats an active ephemeral lease with a
receipt as not pinnable. That path applies only to the remote-workspace
flow and falls back to `preserveNativeFailedWorkspace`.
- Leases that settled before this change have no early receipt. They
continue to wait for their release.
- Nothing compares the refusal message exactly. Matchers use substrings,
and run failure classification does not use the message.
- A follow-up still fails closed if the predecessor's stop was never
proven. This change does not add a recoverable wait.
- The reason the release was held in production is not addressed here.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents, Honeycomb trace queries) and extended
thinking. One sub-agent wrote the change. A second session reviewed it
before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:20:45 -08:00
DottaandClaude Opus 5.5 497ccba8f1 fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner that the server
controls over the durable PRP channel
> - When a run is interrupted, a same-run retry reattaches the live
Runner and reads its session (`session.snapshot`) before it continues
> - On staging, the reattached Runner authenticated and restored its
session, and then did not answer the read
> - The recovery kept that Runner, so each later attempt adopted the
same Runner and failed the same way until the attempt budget ended the
run
> - This pull request makes the next attempt replace that Runner with a
fresh Runner over its saved journal, and makes a failed Codex history
read say what failed
> - The benefit is that an interrupted Codex run continues without a
manual Retry, and its provider turn is not sent again

## Linked Issues or Issue Description

Refs #21161, Refs #21053, Refs #20853

**What happened?**

A Codex agent on Daytona ran recovery runs (wake
`issue_recovery_action_restored`). In one hour, 11 of about 35 finished
runs failed. All failed the same way:

- `native_session_recovery_failed: HarnessCapabilityUnavailableError:
read is unavailable: Error: runnerd exited while waiting for
session.snapshot` (8 runs)
- `native_session_recovery_failed: HarnessReconciliationError:
codex_history_read_failed: thread/turns/list requires supported
paginated history; Error: provider_transport_failed: PRP command
session.snapshot timed out (runner connected; pending ahead 0; last
runner event seq 338 76s ago)` (3 runs)

The run log of each run shows one sequence:

1. Attempt 1 fails with `runnerd did not report the provider turn
identity`. The close leaves the Runner alive and unsettled for the
same-run reattachment.
2. Attempt 2 adopts that Runner (same PID). The Runner authenticates.
The log shows `runnerd restored its durable PRP session and provider
thread`. Then the driver's recovery read (`thread/read` or
`thread/turns/list`, which both send `session.snapshot`) gets no answer.
The Runner still answers WebSocket pings, but it sends no command result
and almost no events (`pending ahead 0`, last event 76 s to 391 s
earlier). The recovery treats this as transient and detaches. The Runner
stays alive.
3. Attempt 3 adopts the same Runner again and fails the same way. In
some runs a protocol fault (`thread_binding_mismatch`) closes the
transport during the read, so the message is `runnerd exited while
waiting for session.snapshot`.
4. The provider attempt budget ends the run.

#21161 did not help. Its wait extension applies only while a queued
answer is being committed. Here the Runner sent no answer at all.

The `paginated history` text was also wrong. `codex-history.ts` called
every failed history read unsupported, also a transport timeout.

**Expected behavior**

A Runner that left a recovery read unanswered is not adopted again. The
next attempt starts a fresh Runner from the saved journal, resumes the
provider session from the saved checkpoint, and does not send the
provider turn again. A failed history read reports the real failure. A
Codex build that refuses paginated history falls back to one complete
thread read.

**Steps to reproduce**

1. Start a Codex run in a remote (Daytona) Runner.
2. Let the first attempt fail at turn start, so that the close keeps the
Runner alive.
3. Let the reattached Runner stop answering commands after it
authenticates (for example, stop the Runner process after its first
answer).
4. See `PRP command session.snapshot timed out (runner connected;
pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3,
with the same Runner PID.

**Paperclip version or commit**

Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on
`master` after `53608bd1d6`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `nativeRecoveryCommandUnanswered` (executor) finds a
`native_session_recovery_failed` whose Runner did not answer a recovery
command (`PRP command <type> timed out` or `runnerd exited while waiting
for <type>`). Identity, authority, integrity and incompatibility
refusals do not match.
- When such a failure ends a `reattach_remote_runner` attempt on a
sandbox target, the failure detail records `adoptedRunnerUnanswered:
true`.
- `claimNativeRestartRecoveries` copies it to the next claim as
`replaceUnansweredRunner: true`, in the same way as
`exitedBeforeSessionOpen`.
- `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps
all identity checks. When the old Runner is still alive, it returns no
adopted process, and the attempt launches a fresh Runner from the saved
journal. The existing launch fence stops the old `paperclip-runnerd` on
that state folder and its whole lineage, and fails the launch closed
when it cannot prove them gone. A Runner that already exited mid-turn
keeps the existing fail-closed path, because its provider or tool
processes can survive where the fence cannot find them. The task then
continues in a fresh session from the saved work. The attempt budget is
unchanged.
- `codex-history.ts`: a history read that fails without a provider
answer is now `codex_history_read_failed: <method> read failed;
<error>`. A provider refusal of the method (`-32601` or `-32600`) on the
first page falls back to one `thread/read` with `includeTurns: true`.
That fallback fails closed when the thread differs, or when turns, items
or turn identities are missing. A refusal on a later page still fails
closed.
- `doc/execution-semantics.md` describes both rules. The launch-fence
code comment now names the replacement case.

### Changed existing test

`codex-history.test.ts` "does not turn an interrupted page or
unsupported method into idle history" used a plain `Error("method not
found")` on page 2 and expected the `supported paginated history` text.
A plain error is now a failed read, not an unsupported method. The test
now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2
and still expects a fail-closed rejection. A new test covers the
failed-read message.

## Verification

- Red before the fix, green after:
- `server/src/services/native-runtime/native-session-executor.test.ts` ›
"a same-run reattachment whose adopted Runner never answered its
recovery read": the replacement and staging-message tests failed on the
old code (the Runner was adopted again, the three staging messages were
not recognized). Two further tests ("never resumes beside an exited
unanswered Runner's lineage") failed on the first revision of this PR,
which relaunched beside an exited mid-turn Runner, and pass now. All 15
pass.
- `native-runner-restart-recovery.integration.test.ts` › "asks the
same-run reattachment to replace a Runner that left its recovery read
unanswered" (embedded Postgres): the claim had no
`replaceUnansweredRunner` on the old code. Now passes.
- `codex-history.test.ts`: 7 new tests failed on the old code. All 16
pass.
- Real `paperclip-runnerd` and fake Codex:
`runnerd-codex-transport-remote-retention.test.ts` › "continues the same
provider thread from a fresh Runner over the journal of a retained
Runner that stopped answering, without replaying its turn". It runs the
staging sequence: turn started, close keeps the stopped Runner, a second
controller adopts it, the Runner stops answering and the controller
detaches, the Runner and provider are killed (the launch fence), and a
fresh Runner over the same journal returns the same thread. The fake
Codex call log shows one `thread/start` and one `turn/start`.
- Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus
the remote-retention, adoption and cold-restore transport suites (442
tests). Two suites timed out under parallel load and pass alone. Server:
`native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation` (1714 passed, 21 skipped).
- `pnpm typecheck` in `server` (all layers) and `tsc` for
`paperclip-runner` source and tests pass.
- Not run locally: the container-gated remote Runner suites
(`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the
full suite.

## Risks

- Medium. The replacement relaunches a Runner over a journal that is not
settled, but only while the old Runner is alive. Safety comes from three
facts: the launch fence stops the old Runner and its lineage with proof
or fails the launch; the Runner restores its own durable journal (as
local `resume_dead_runner` does); and the real-runnerd test shows that
the turn is not sent again. An exited Runner is never relaunched beside
its possible survivors.
- If the previous close left a `runner.suspend` pending, a fresh Runner
can execute it and exit. The attempt then fails as it does today. The
usual detach path after an unanswered read sends no suspend.
- The replacement uses the existing attempt budget. It does not add
attempts.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events
and logs), extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:06:24 -06:00
Nicky LeachandClaude Fable 5.2 9b613562b7 feat(docker): split the Runner image into stable layers and add a slim variant (#21229)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Part 2 of this stack builds the Runner alone as the image
`runner-artifacts`, in one layer of about 850 MB
> - Most of those bytes are provider binaries and Node. They change only
with a provider version or the Node pin, but one layer means that every
Runner source change makes all 850 MB new
> - About half of the image is the pack's transport archives. A consumer
that copies the pack as files does not need them
> - This pull request splits the image into layers by how often their
content changes, and adds a slim variant without the transport archives
> - The benefit is that a new image shares about 93% of its bytes with
the one before it, and a consumer that needs no transport archives pulls
half as much

## Linked Issues or Issue Description

Refs #21204 (part 2 of this stack) and #21167 (provider pack layers). No
issue exists for this change.

**Problem or motivation**

The image from part 2 is one layer. A registry and a builder store and
move layers, not files. A change to one Runner source file therefore
makes a new 850 MB layer, although 790 MB of its content is the same as
before. The image also holds the provider pack two times: as files, and
as transport archives of those files. Some consumers need only the
files.

**Proposed solution**

Write the image as four layers: the pack's large dependency files,
everything else, the transport archives of those large files, and the
other transport content. Make the two large layers byte-identical from
build to build. Add the target `runner-artifacts-slim`, which is the
first two layers. The full image is the slim image plus the two
transport layers, as the same layer blobs. Check all of this on the
built images.

**Alternatives considered**

- Two unrelated images. Then the registry stores the shared content two
times unless the layers happen to match, and nothing ties the two
together.
- Split by path only, with all files of 1 MiB or more in the first
layer. Six of those files are the Runner's own bundles under `dist`.
They change with the Runner's source, so the first layer would change
with it too.
- Leave the transport archives as they were written. Their bytes then
depended on timing (see "What Changed"), and the archive layer never
repeated.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 3 of 4. Each part is on top of the one before it:

1. #21228 (merged): an optional field in the provider pack manifest
2. #21204: build and check the Runner image, with no publication
3. #21229: layers by change rate, and the slim variant (this pull
request)
4. #21230: publish both variants from `master`

## What Changed

- `Dockerfile`: two final stages. `runner-artifacts-slim` is two `COPY
--link` layers. `runner-artifacts` starts from it and adds two more. The
build stage sorts its output into the four layers. Every path is where
it was in part 2: consumers copy by path and see no difference.

  | Layer | Content | Changes with |
  | --- | --- | --- |
| 1 | The pack's dependency files of 1 MiB or more: provider binaries
and Node | A provider version or the Node pin |
| 2 | Everything else, with `runner-image.json` | Every Runner source
change |
| 3 (full only) | The transport archive of each file in layer 1 | The
same as layer 1 |
| 4 (full only) | The other transport archives, the overlay archive and
the receipt | The same as layer 2 |

- Stable bytes. The files of layers 1 and 3 get timestamp 0, and the
large dependency files get it before their transport archives are
written. A file under the pack's `dist` is the Runner's own output, so
it and its archive are in layers 2 and 4.
- `scripts/create-provider-pack-archive.mjs`, two separate commits.
**This script also writes the application image's archives.**
- The same file now gives the same archive every time. The script piped
tar into Node's zstd compressor. When the pipe ended while the
compressor still had its last chunk queued, Node finished the zstd
stream two times, and the archive got an empty frame of nine bytes at
its end. That depended on timing. In the pinned Linux base, 37 of 60
runs of the old code had the frame. The compressor is now ended only
after it has taken every chunk.
- Archive N holds file N of the receipt, and the receipt listed the
files in path order. `dist` sorts before `node_modules`, so one more
large file in the Runner's own output gave every dependency file another
archive name, and layer 3 became a new blob with the same contents. The
receipt now lists the files outside `dist` first. The readers
(`provider-pack-layers.ts`, `retained-provider-pack-source.ts`) pair
file N with `N.tar.zst` and do not use the path order, so they need no
change. For the application image, the archive numbers change one time,
and the bytes of each archive do not change.
- `runner-image.json` is the same file in both variants. Its new field
`fullImageOnly` names the two paths that only the full image has.
`paths` did not change.
- `scripts/runner-image-layout.mjs check` takes `--variant slim|full`.
It is required and has no default. A slim tree that holds a transport
path fails. A full tree without them fails.
- `scripts/runner-image-oci.mjs` (new, 585 lines). It is not an input of
the image, because the image build does not run it.
- `unpack <oci archive> --out <tree>` writes the tree of an image from
the layers of its archive, in order. It checks every blob against its
digest. It refuses a whiteout, an absolute path, a path with `..`, a
path that is written two times, an entry behind a link, and every entry
that is not a file, a directory or a link.
- `check-layers --slim <oci archive> --full <oci archive>` checks the
table above on the built images. It reads the receipt to decide which
archive belongs in which layer. It fails when the full image does not
start with the slim image's layer blobs in the same order, when a path
that is not a directory is in two layers, and when a shared layer holds
transport content under any name or directory.
- The script reads the archive and its layers itself. It does not use
the system's `tar`, so every platform reads them the same way. It finds
a member only under the name that buildx gives it.
- `.github/workflows/runner-image.yml`: each leg builds each variant one
time, as an OCI archive and nothing else. It unpacks both archives, runs
`check` on each tree with its variant, runs `check-layers`, and runs the
smoke matrix on the slim tree. The tree that is checked is therefore the
bytes of the image, not a second export of the same build. Every step
that builds has a time limit. The workflow still has no registry access.
- Why one export: in the first runs of this pull request, the slim build
stalled on all four legs. Each build command had two exports, a tree and
an archive. The slim command, with every stage from the cache, wrote its
archive and then made no progress on the tree for 50 minutes. A test now
holds the rule of one export and a time limit for each build command.
- The smoke matrix runs on the slim tree. The daemon is in a layer that
both variants share, so one run covers both.
- `server/src/__tests__/runner-image-identity.test.ts`: the test that
holds the rules for the Runner stages now finds a stage wherever
BuildKit starts one. It fails for `FROM x as name`, a flag on `FROM`, a
lowercase or indented `COPY`, an `ADD` or `ONBUILD`, a `COPY` with
another flag, and a `COPY` that continues on the next line. Nine samples
show each case.
- Two comments are corrected. The comment on
`PAPERCLIP_RUNNER_INPUTS_DIGEST` in `build-provider-pack.mjs` said that
no check reads the field. The `Dockerfile` comment on the publication
argument named a workflow "that builds the image for publication".
- `doc/runner-image-contract.md` (159 lines): the two variants, the
section "Layers", how the checks get their tree, and the new checks.
`doc/DOCKER.md` names the slim target.

Script sizes: `runner-image-layout.mjs` 969 lines,
`runner-image-oci.mjs` 585, `runner-image-identity.mjs` 166,
`create-provider-pack-archive.mjs` 114.

## Verification

Image identity at this commit:
`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`.

Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and
zstd, the worktree mounted read-only):

- `runner-image-identity.test.ts` (41), `runner-image-layout.test.ts`
(100), `runner-image-oci.test.ts` (50), `runner-image-workflow.test.ts`
(8), `provider-pack-archive.test.ts` (47, none skipped),
`retained-provider-pack-source.test.ts` (19),
`remote-provider-pack-verification.test.ts`,
`docker-image-layer-reuse.test.ts`, `docker-build-stamp.test.ts`: 293
tests pass in 9 files.
- The same files pass on macOS. The tests of `check-layers` failed on
Linux in the first runs of this pull request: their fixtures named the
archive members `./index.json`, which bsdtar finds and GNU tar does not.
The fixtures now name the members as buildx does.

`linux/arm64`, on a local Docker host, with the new export path:

- Each variant builds as an OCI archive only: the full target in 156
seconds without a cache, the slim target in 5 seconds from the cache.
- `unpack` writes the full image (10,519 entries that are not
directories) in 4 seconds, and the slim image in 4 seconds.
- `check <tree> --variant <variant> --expect-inputs-digest <digest>
--expect-source-revision <sha> --expect-architecture arm64` passes on
each unpacked tree. `runner-image.json` is the same file in both.
- `check-layers --slim <slim> --full <full>` passes.
- The smoke matrix passes on all five base images, from the unpacked
slim tree.

| Layer | Digest | Bytes | In |
| --- | --- | --- | --- |
| 1 | `sha256:3f21b15b3a99…70ac1c` | 379,609,589 | slim and full |
| 2 | `sha256:d5567acc977f…b4f12a` | 43,906,956 | slim and full |
| 3 | `sha256:6a340088ca2c…6ba5b5` | 410,938,259 | full |
| 4 | `sha256:789e97f5db88…3eab44` | 15,656,249 | full |

- The slim image is 423,516,545 bytes in two layers. The full image is
850,111,053 bytes in four.

Stability, with three builds that share no cache for the build step:

| Build | Source revision | Runner build output | Layer 1 | Layer 2 |
Layer 3 | Layer 4 |
| --- | --- | --- | --- | --- | --- | --- |
| A | `aaaa…` | this commit: 6 files of 1 MiB or more under `dist` |
`3f21b15b…` | `d5567acc…` | `6a340088…` | `789e97f5…` |
| B | `bbbb…` | this commit | `3f21b15b…` | `a2f8a4fe…` | `6a340088…` |
`4e6c492d…` |
| D | `dddd…` | one new source file that adds two large files under
`dist` | `3f21b15b…` | `33b0bdaf…` | `6a340088…` | `0ea322bf…` |

- Layers 1 and 3 are byte-identical in all three: 790,547,848 of
850,111,053 bytes, 93%. Build D has two more large files under `dist`,
eight in place of six, and another image identity. Its dependency files
keep the archive numbers 0 to 20. With the earlier numbering, those
numbers moved by two, and layer 3 was a new blob with the same contents.
- Layer 1 has the same digest as before this round. Layer 3 has a new
digest one time, because the archives have new numbers.

Other checks:

- `tests/runner-e2e` workflow tests: 39 pass. `node --test
.github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image
contract scripts: 33 pass. `check-docker-deps-stage.mjs`,
`check-node-version-policy.mjs` and `docker buildx build --check` report
nothing.

GitHub, at this head:

- The `Runner image` workflow passes on both architectures with the new
export path: `Build and check (amd64)` in 14m41s and `Build and check
(arm64)` in 8m41s. Each leg builds both variants, checks both layouts
and the layer rule, and runs the smoke matrix. The first runs of this
pull request stalled in the slim export before any check.

Never run:

- `native-session-executor.test.ts`. It does not load on this host, with
or without this change, because `@paperclipai/plugin-sdk` is not built
here. It imports the reader of the retained pack and no archive number.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The two changes to `create-provider-pack-archive.mjs` change the
application image's transport archives: about half of them lose nine
bytes, and all of them get new numbers one time. An archive's content
does not change. A reader pairs file N of a receipt with archive N of
the same build.
- The large dependency files in the pack, and in their transport
archives, now have the modification time 0 (1970). Before, they had the
build time. Nothing in this repository reads those times.
- `unpack` is a reader of tar and gzip that this repository owns. It
reads the layers that BuildKit, GNU tar and bsdtar write (tested with
each), and it fails for a header it does not know.
- The check on the built images, not the recipe, is what proves the
layer rule; it runs in each leg of the workflow.
- `check` now needs `--variant`. A caller that does not pass it fails
with a clear message. No caller exists outside this repository yet.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 19:52:02 -08:00
DottaandClaude Opus 5.5 3b4c3265ed fix(recovery): offer Try again on a model-unavailable failure and recheck the model first (#21244)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native run fails, the task thread shows a failure marker, and
some markers offer Try again
> - A `native_provider_model_rejected` failure blocks the task for the
person and shows "Model unavailable", but it has no Try again control
> - Recovery must just work: after the person fixes the model, they need
one obvious control that continues the task, and that control must not
start a run that is sure to fail
> - This pull request shows Try again on the "Model unavailable" marker
and makes the retry recheck the selected model before it starts a run
> - The benefit is that the person fixes the model, presses Try again,
and the task completes, and a too-early press explains which setting to
change instead of failing again

## Linked Issue or Issue Description

Refs #21210

**What happened?**
Paid run 43321773136, `recovery.retry-exhausted` (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`). The run failed with
`native_provider_model_rejected`. The task stayed blocked and the thread
showed "Model unavailable", but no Try again control appeared
(`task-chat-run-failed-try-again` not found in 30 s). Only the "Run
failed" and "Usage limit reached" markers offer Try again.

**Expected behavior**
The "Model unavailable" marker offers Try again. Pressing it checks the
selected model first. If the provider-rejected model is still selected,
no run starts, and the person sees which setting to change with a link
to it. After the person changes the model, Try again starts one retry of
the original run.

**Steps to reproduce**
1. Set an agent's model to an id the provider rejects.
2. Assign it a task. The run fails with
`native_provider_model_rejected`.
3. Open the task. Before this change, the marker has no Try again.

## What Changed

- `server/src/services/native-runtime/model-rejected-retry.ts`:
`rejectedModelRetryRefusal` reads the model the failed run sent
(`runnerProfileJson.nativeExecutionInput.provider.model`) and the model
a new run would use (the task's model override for its assignee, else
the agent's model). If they are the same, it returns a 409 refusal with
code `native_provider_model_still_unavailable`, the setting to change
(`agent_model` or `task_model_override`) and the rejected model.
- `server/src/routes/agents.ts`: the exact failed-run retry (`POST
/agents/:id/wakeup`, `reason: retry_failed_run`) applies this check
before it queues anything.
- `ui/src/components/task-chat/adjacent-errors.ts`,
`TaskChatThreadView.tsx`: one `isRetryableFailureMarker` rule, now
including "Model unavailable".
- `ui/src/lib/model-retry-refusal.ts`, `ui/src/pages/IssueDetail.tsx`: a
refused retry shows "Model still unavailable" with the server's message
and an "Open <agent> model settings" link to `/agents/<agent>/runtime`.
A task override refusal names the override in the message.
- Eval `recovery.retry-exhausted` (oracle `runtime.retry-exhausted.v7`):
the rejected model stays selected after the failure. The person presses
Try again (409, toast names the model, its link opens the agent's model
settings, no run starts in 20 s, task stays blocked), restores the
working model, presses Try again (202), and the one retry of the
original run uses the working model, saves the brief and completes the
task. 3 runs: failed 1, succeeded 2. The budget-hold and two follow-up
chat phases are removed; the case now tests the model recheck.
`recovery.yaml` context and expected outcome updated, recorded in
`chat-full-catalog-testops-manifest.json` `changedFields`, pinned in its
test, descriptors and inventory digest updated.

## Verification

- RED on master: `TaskChatThread.test.tsx` "offers Try again on a
model-unavailable failure and retries that run" fails (no control).
`agent-live-run-routes.test.ts` "refuses a retry while the rejected
model is still selected and starts no run" fails (the route queues the
run).
- GREEN: both pass. `agent-live-run-routes.test.ts` 101/101.
- `server/src/services/native-runtime/model-rejected-retry.test.ts`
(embedded PostgreSQL), 6/6: refuses while the agent still selects the
rejected model (names the agent setting); names the task override;
allows the retry when the agent or the override now selects another
model, or the run failed for another reason; ignores another company's
run.
- `ui/src/lib/model-retry-refusal.test.ts`: names the setting and links
to `/agents/morgan/runtime`; no link for a task override; other errors
fall through to the generic toast.
- Server `tsc --noEmit` clean. UI `tsc` clean apart from the
pre-existing generated storybook catalog. `node
scripts/check-token-gates.mjs`: clean.
- Eval: `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog tests/runner-e2e/chat-catalog-domains
tests/runner-e2e/chat-*.test.ts`: 339 files pass. New calibration
rejects an accepted early retry, a run started while the model is
rejected, a refusal without the model, setting link or blocked task, a
retry that used the old model, and duplicate or missing retries. `pnpm
test:e2e:runner:typecheck`: pass. Manifest test 10/10.
`source-sdk-public-inputs --check`: `changed: false`.
- Paid run: Runner Full-Stack E2E 43338695850 on throwaway branch
`chat-eval/qa-mr-paid`. Result to follow in a comment.

## Risks

- The recheck compares the selected model with the model the provider
rejected. It does not call the provider. A model the provider rejected
stays refused until the person selects another one; a different but also
unsupported model starts a run that fails with the same marker again.
- The refusal message is a toast (15 s). The marker detail still says to
choose a supported model.
- No persisted shape changes; runs saved by older builds carry the same
`runnerProfileJson` field.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:42:52 -06:00
DottaandClaude Opus 5.5 7154030ef6 fix(heartbeat): draining/yielding servers claim no queued run, and closed tasks get no system continuation (#21258)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - During a release swap, the outgoing server drains. It must not start
new runs, because its own exit would cut them off.
> - The drain gate (run admission closed at SIGTERM) and the hand-over
gate (#21127, yield to a replacement that already dispatches) were
checked only by the queued-run dispatch passes.
> - Other start paths (executeRun of a queued run, delivered
continuations, immediate wake starts) went straight to `claimQueuedRun`,
so an outgoing server could still start runs after traffic moved.
> - Also, a system continuation queued while its task was open started
after another run had finished the task, and took an extra turn on a
done task.
> - This pull request makes `claimQueuedRun`, which every start path
uses, honour the drain and hand-over gate, and cancel a system
continuation whose task closed while it was queued.
> - The benefit is fewer runs cut off by a swap, and no extra turns on
finished tasks.

## Linked Issues or Issue Description

Refs #21127, #21135

**What happened?**
1. An outgoing server kept starting runs minutes after traffic moved to
its replacement. Each such run was cut off when the old server exited.
The run start went through `executeRun` of a queued run, which called
`claimQueuedRun` without the drain or hand-over check.
2. In release-swap waves, a continuation (execution reconciliation
delivery, invariant or recovery continuation, blocker-stall wake) was
queued while its task was open. Another run then finished the task. The
continuation still started and took an extra turn on a done task.

**Expected behavior**
A draining or yielding server leaves a queued run queued for the serving
server, on every start path. A system continuation does not start on a
task that was closed while it was queued. A person's message on a done
task still runs.

**Steps to reproduce**
1. Close run admission (drain) or set the hand-over yield on a server.
2. Start a queued run through `executeRun` (for example a delivered
continuation).
3. See the run claimed and started on the draining server.

## What Changed

- `claimQueuedRun` (`server/src/services/heartbeat/queue.ts`) refreshes
and checks the run-admission gate first. When admission is closed
(draining or yielding), it leaves the run queued and returns. The
dispatch passes already did this check; now every start path does.
- `claimQueuedRun` re-reads the task for a system continuation
(`issue_recovery_action_restored`, `issue_continuation_needed`,
`issue_assignment_recovery`, blocker-stall wake). If the task is `done`
or `cancelled`, it cancels the run as `issue_closed_before_start`, skips
its wake, and releases the task. Other wake reasons, including a
person's message, are not affected.
- That cancel never consumes a person's message. Admission can attach
deferred comments to a queued system continuation without changing its
wake reason. The cancel locks the run's wake and the run (the order
admission's coalescing uses), reads the run's current comment ids, and
if any is a person's comment that no run of this agent has received yet
(read-only use of `deliveredQueuedCommentIds`, which includes the
history-read rule), the run is not cancelled: it is claimed as that
person's follow-up, which may reopen the task through the normal path. A
comment attached concurrently is either seen under the lock or, after
the cancel commits, saved behind the cancelled run by admission.
- Admission is checked again inside each claim transaction (ordinary
claim, queued-comment claim, native review claim), right after the write
that moves the run to `running`. If admission closed during the awaited
checks, the transaction rolls back: the run stays queued with no
`startedAt`, its wake and the task's execution lock are untouched, and
nothing is charged.
- Every cancellation `claimQueuedRun` makes from the caller's copy of
the run is now a compare-and-set from `queued`
(`setRunStatusFromLive(..., ["queued"])` through a new
`cancelRunIfStillQueued` helper): the closed-task cancel, the
refused-interaction cancel, the missing-@mention cancel, the daily-cap
cancel and the blocked-dependency cancel. A run another dispatcher
already started or finished keeps its status and result, and no wake or
release follows. The queue module no longer receives the unconditional
`setRunStatus`.
- The closed-task cancel takes its decision under the wake and run
locks, then cancels through `setRunStatusFromLive` fenced on the row
version (`xmin`) the decision read. That saves the same never-started
proof (`executionRecovery: { kind: "bootstrap", providerWorkStarted:
false }`) as every other queued-run cancel, so task release promotes a
person's message saved behind it. A message attached after the decision
changes the run, the cancel matches nothing and the decision is taken
again. `HeartbeatRecoveryDependencies.setRunStatusFromLive`
(`recovery.ts`) now types its existing `cancellationCondition` parameter
as `SQL`.
- The dispatch ranking in `startNextQueuedRunForAgent` is not changed.

### What I checked in `claimQueuedRun` for writes from the caller's copy

- Cancellations through `setRunStatus`: all five now conditional on
`queued` (above).
- The final claims (ordinary, queued-comment, native review) already
lock and re-read the run and update `where status = 'queued'`.
- The case-handoff cancel and the chat-control stop already lock the run
and update `where status = 'queued'`.
- The budget-stamp update already writes `where status = run.status`
(`queued`).
- `cancelRunInternal` (agent removed, agent not invokable, board-owned
budget stop, pause hold, budget cancel) re-reads the run and only
cancels a non-terminal status, so it cannot overwrite a finished result.
It can still stop a run another server has just started; that is a
policy stop (agent removed, budget, hold) that would apply to that run
anyway, and it is unchanged here.
- `cancelStaleQueuedRun` already takes `expectedStatus: "queued"`.

## Verification

Red-to-green tests in `server/src/services/heartbeat/queue.test.ts`:
- "claims no queued run once run admission is closed, whatever path asks
(draining or yielding server)": red before the gate was added to
`claimQueuedRun`, green after.
- "does not start a %s run whose task another run already finished":
`issue_continuation_needed` and `issue_recovery_action_restored` are
cancelled as `issue_closed_before_start`; `issue_commented` (a person's
message, carrying its `wakeCommentId`) is claimed and the stored run is
`running`.
- "answers a person's message attached to a system continuation on a
closed task once": still undelivered, the continuation is claimed and is
the only unfinished run (red when the cancel ignores comment ids, green
after); already received by the closing run, the continuation is
cancelled as `issue_closed_before_start` and no other run exists.
- "keeps the result of a continuation another dispatcher already started
and finished": the stale queued copy is passed after the run succeeded;
the stored run stays `succeeded` with its result, and no wake or release
follows. Red before the compare-and-set (the run was overwritten to
`cancelled`), green after.
- "returns the claim to the queue when run admission closes while the
claim is being checked": admission closes inside an awaited check; the
run stays `queued` with no `startedAt`, the task's execution lock and
the wake are unchanged. Red before the in-transaction re-check, green
after.

Red-to-green test through the real release path, in
`server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts`:
- "delivers a person's message saved behind a system continuation
cancelled at claim because its task closed": a never-started legacy
continuation on a done task, with a person's deferred message behind it.
Red before the fix: the cancelled run had `resultJson: null`, with no
never-started proof. Green after: the proof is present, the wake is
skipped, and the message is promoted and runs once.

Test expectations changed because they asserted the old behaviour:
- `server/src/services/agent-interaction-delivery.test.ts` "cancels
refused queued runs through the public queue claim without dispatch":
now expects the cancel through `setRunStatusFromLive(runId, "cancelled",
["queued"], …)` instead of `setRunStatus`.
- `server/src/services/heartbeat/queue.test.ts` (refused-assignment held
case): asserts `setRunStatusFromLive` is not called, instead of
`setRunStatus`.
- The `issue_commented` case of "does not start a %s run…" now asserts
the run is claimed and `running` (it previously asserted only that the
error code was not `issue_closed_before_start`).

Local command, on this branch based on current master (includes #21234;
the dispatch ranking it changed is untouched here):
- `cd server && pnpm exec vitest run
src/services/heartbeat/queue.test.ts
src/__tests__/recovery-wake-herd.test.ts`: 102 tests passed. After the
proof fix, `queue.test.ts` again passed 97/97, and the new stale-queue
test passed. (The file's `afterAll` database cleanup hook times out on
this loaded machine on master too; it is not related to this change.)

Why the local release-swap wave does not show the yield fix:
- The "yield to the replacement" signal exists only when the run-start
handover is created (`server/src/index.ts`, about line 1320). That
happens only when a direct deployment listen URL exists, that is, with
`PAPERCLIP_EPHEMERAL_SERVER=true` and `PAPERCLIP_DATABASE_LISTEN_URL`
(the volume-free configuration).
- The local short form of the wave leaves `--ephemeral-server` out, so
the handover is disabled and `isRunAdmissionClosed()` becomes true only
at SIGTERM. The wave therefore cannot exercise the yield.
- The evidence for this fix is the unit test above and the reasoning
that every start path now reaches the same gate in `claimQueuedRun`.
Hosted confirmation must come from a volume-free instance.
- The closed-task fix was found in release-swap waves (an extra turn on
a done task in two tasks).

## Risks

- `claimQueuedRun` can now await an admission refresh. The refresh
returns at once when the cached state is fresh, so the normal path adds
no wait before `turn.start`.
- A system continuation for a closed task is cancelled. This matches
admission, which already refuses such wakes for a closed task.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:36:05 -06:00
DottaandClaude Opus 5.5 d169dd5a90 fix(presentation): offer Retry only when no automatic continuation or scheduled retry is pending (#21264, includes #21200)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a run fails with a transient error, Paperclip schedules the
next attempt by itself and shows "retrying automatically" with a
countdown.
> - The task presentation also offered the Retry button next to that
countdown.
> - The system owns the next attempt while the run is `scheduled_retry`,
so the button is misleading: it suggests that a person must act.
> - This pull request stops offering Retry while a scheduled retry is
pending. Retry returns when the scheduled retry is past due with nothing
pending, or when the retries are exhausted.
> - The benefit is one clear message: Paperclip retries by itself, and
Retry appears only when it is the way forward.

## Linked Issues or Issue Description

**What happened?**
A legacy run in status `scheduled_retry` (for example reason
`transient_failure`, which is not a shared-resource wait and has no
system recovery) projected `state: "retry_scheduled"` with `canRetry:
true`. The UI then showed the Retry button beside "retrying
automatically". The cause is in `projectTaskInterruptionState` in
`server/src/services/task-interruption-presentation.ts`: the `retryAt >
now` branch set `canRetry: !preparationPending && !nativeResumeAt`,
which ignores that the run itself is the scheduled attempt.

**Expected behavior**
While a run is `scheduled_retry` and its retry time is in the future,
the task shows the countdown and offers no Retry. When the scheduled
retry is past due with nothing pending, or the retries are exhausted,
Retry is offered.

**Steps to reproduce**
1. Let a task's run fail with a transient failure that schedules an
automatic retry.
2. Open the task before the retry time.
3. The task shows "retrying automatically" and the Retry button.

**What a Retry click does today in this state**
`requestTaskRetry` (`server/src/services/task-retry.ts`) treats a latest
run in `scheduled_retry` as busy. It accepts the click (202), records
the request as `coalesced` onto the scheduled run and returns
`scheduledRetryRunId`. It does not start a second attempt next to the
scheduled one. The button is therefore misleading but not unsafe, and
this pull request changes only what the task shows.

## What Changed

- `server/src/services/task-interruption-presentation.ts`: in the
`retry_scheduled` branch, `canRetry` is now also false when `run.status
=== "scheduled_retry"`.
- `server/src/services/task-interruption-presentation.test.ts`: three
tests for a scheduled transient-failure retry.
- This branch is based on current `master` and includes #21200, which
has not merged yet and changes the same file. The diff of this pull
request against #21200 is one line and the tests.

## Verification

- `cd server && npx vitest run
src/services/task-interruption-presentation.test.ts -t "automatic
scheduled retry"`
- "shows a pending transient-failure retry as scheduled without a
competing Retry": red on `master` (`canRetry: true`), green with this
change.
- "offers Retry once the scheduled retry is past due with nothing else
pending": green before and after.
- "offers Retry once the scheduled retries are exhausted": green before
and after.
- Locally: `task-interruption-presentation.test.ts`,
`task-interruption-query.test.ts` and
`task-interruption-automatic-path.test.ts` pass (107 tests). Server
typecheck: 0 errors. The wider suites that read the presentation run in
CI; they were not repeated locally because the machine was under memory
pressure.

## Risks

- Low risk. The change affects presentation only. The Retry route still
accepts a click and coalesces it onto the scheduled run, as before.
- If a scheduled retry never dispatches, the task offers Retry again
once the retry time passes, as before this change.
- No schema, API or UI change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in
Claude Code with tool use (shell, file edits, test execution). Extended
reasoning was available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:14:41 -06:00
DottaandClaude Opus 5.5 6cb0dd0c6b fix(recovery): continue twice-interrupted tasks and waiting follow-ups after a release swap (#21250)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is cut off by a restart or a release swap, the
fresh-continuation fallback continues the task in a new run.
> - The continuation goes through normal admission, or through a
person's saved follow-up when one is waiting.
> - In release-swap waves, three gaps left tasks stopped with no run:
admission refused an isolated continuation of a task with no project, a
person's saved follow-up behind the replaced run was never promoted, and
the fallback relied on a saved follow-up that the cut-off run had
already read.
> - This pull request closes those three gaps.
> - The benefit is that a twice-interrupted task, and a task with a
waiting follow-up, continue by themselves.

## Linked Issues or Issue Description

Refs #21135, #21222

**What happened?**
1. A task with no project was interrupted twice. The second continuation
used a fresh isolated workspace, which clears the task's workspace
binding. The admission preflight still required that binding for an
isolated task with no project, so it refused the continuation as
`workspace_worktree_requires_project` about every 15 s. Each refusal
re-blocked the task and added a comment (329 to 428 skipped wakes per
wave). The runtime itself supports this case: it prepares a new
workspace (`create_new`).
2. The fallback continued a task with delivery `saved_input` (a person's
deferred follow-up is the next turn). The stranded-queue pass refused to
promote that follow-up, because the replaced native run's finalization
was `terminal_failure` (restart recovery ended it), not `committed`. The
release path would also have re-blocked the task with a reconciliation
incident. The follow-up never ran.
3. The fallback chose `saved_input` because a deferred user wake
existed, but the cut-off run had already taken that message into its
turn. Promotion settled the message as delivered, and no run followed.
The task showed "working".

**Expected behavior**
An interrupted task continues by itself in each case, with no person
action and no repeated comments.

**Steps to reproduce**
1. Run a release-swap wave with projectless tasks in isolated mode and
person follow-ups during runs.
2. Interrupt the same task twice, or interrupt a run that has a deferred
follow-up behind it.
3. See the task stay blocked or "working" with no run.

## What Changed

- Admission (`server/src/services/execution-workspace-policy.ts`,
`server/src/services/heartbeat/queue.ts`,
`server/src/services/fresh-continuation-fallback.ts`):
`isUnrunnableWorktreeCombo` accepts `freshIsolatedContinuation`. The
admission preflight sets it from the new
`isFreshIsolatedContinuationWake`, which returns at once (no query)
unless the wake context carries `freshContinuationIsolation`. This does
not touch the dispatch ranking.
- Follow-up promotion
(`server/src/modules/wake-queue/adapters/postgres.ts`): the
stranded-queue pass and the release path skip the native finalization
gate and the reconciliation incident for a run that the fallback
superseded (`agentRunSupersededByFreshContinuation`).
- Saved-input choice
(`server/src/services/fresh-continuation-fallback.ts`): a deferred user
wake counts as saved input only when it has no comment ids, or still has
a message that the agent has not received
(`undeliveredLegacyUserCommentIds`, called, not changed). Otherwise the
fallback records its own continuation.

- Review round 1: (a) whenever the old writer is not proven stopped
(`fresh_isolated_workspace`), the fallback records its own isolated
continuation first, also when the task had a binding; saved input
follows it on the new workspace. (b) Release skips the manual-admission
hold and the pending-cleanup check for a run the fallback replaced with
an isolated continuation (`fresh_isolated_workspace` or `remote_epoch`,
via `freshContinuationIsolatesRun`); other replacements still wait.
- Review round 2 (the isolated continuation must run before the waiting
follow-up): while the fallback's `fresh_isolated_workspace` continuation
is still pending delivery, release leaves the person's follow-up
deferred. Otherwise the follow-up would become the task's next run
without the isolation marker, and delivery would count it as the
continuation and skip the isolated run. The continuation's own release
promotes the follow-up behind it. `freshContinuationIsolatesRun` now
also reports `continuationPending`.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `server/src/__tests__/execution-workspace-policy.test.ts`: admission
keeps a fresh isolated continuation of a projectless task runnable.
- `server/src/modules/wake-queue/adapters/postgres.test.ts`
"restart-ended native run": saved input behind a run the fallback
replaced is promoted.
- `server/src/services/fresh-continuation-fallback.test.ts` "relies on
saved input only while it still holds an undelivered message" (taken:
true gives `recorded`, taken: false gives `saved_input`).
- `postgres.test.ts` "promotes saved input behind an exhausted restart
hold ... (released, continuation pending)": the follow-up stays deferred
while the isolated continuation is pending, then is promoted when the
continuation's run is released. It was red before the fix (`expected
'promoted' to be 'released'`).
- Local, round 2: `cd server && npx vitest run
src/modules/wake-queue/adapters/postgres.test.ts
src/services/fresh-continuation-fallback.test.ts`: 2 files, 177 tests
passed.

Local command, on this branch based on current master (includes #21222
and #21234):
- `cd server && npx vitest run
src/services/fresh-continuation-fallback.test.ts
src/__tests__/execution-workspace-policy.test.ts
src/modules/wake-queue/adapters/postgres.test.ts`: 3 files, 221 tests
passed (re-run after rebasing on #21234).

Local release-swap cells:
- L1 (6 tasks, 3 swaps): 5 fresh isolated continuations delivered,
including a second one on the same task; 0
`workspace_worktree_requires_project` rows (before: 329 to 428 skipped
wakes per wave).
- R2 (6 tasks, 3 swaps): a task interrupted three times had its second
and third continuations delivered. The same cell found gap 3 (a
follow-up already read by the cut-off run), which the last commit fixes.
- Hour-long release-swap wave (w-20261112T233630-s7201) on the branch
that holds these fixes and the related recovery fixes: 11/11 tasks
finished hands-off (master: 9/11). The lost-continuation and
follow-up-promotion fixes were visibly working in it.

Review-round tests: "moves the task off the old writer's workspace and
runs the isolated continuation before saved input" (changed expectation:
`recorded` instead of `saved_input`; red on the previous head);
"promotes saved input behind an exhausted restart hold once a %s
continuation replaced the run" (isolated: promoted, red before;
`writer_ended`: held). 3 files, 224 tests passed.

Conflict note: the last commit was resolved onto current master. It no
longer imports `taskHistoryReadsForRun`, which belongs to a separate
opening-note change.

## Risks

- An isolated projectless continuation now passes admission. It runs in
a new workspace that the run prepares, so it never shares a workspace
with the cut-off run.
- A superseded run's follow-up is promoted without its native
finalization committed. The superseded run is fenced (its writes are
refused), so there is still one writer.
- No migration. No schema change. The normal admission path adds no
query.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:10:47 -06:00
DottaandClaude Opus 5.5 db802d1230 fix(heartbeat): deliver a queued comment again unless the reading run cites it as handled (#21243)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can add a comment to a task while the agent's run on that
task is still working
> - #21175 made a `get_task_history` read of a queued comment's current
revision count as delivery
> - A read is not handling. In `agent-chat.multiple-handoffs`, Morgan's
running turn read the "Launch sooner" comment, set it aside, and saved
"Launch later". No continuation ran, so the instruction was lost.
> - This pull request counts a read as delivery only when the run's
accepted result cites the comment as handled. Otherwise one continuation
still delivers it, and that continuation is told the comment may already
be applied.
> - The benefit is that a person's comment is never silently lost, and
the #21175 protection against repeated completed work stays in place

## Linked Issue or Issue Description

Refs #21175
Refs #21203
Refs #21210

**What happened?**
Paid run 43321773136 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`), case
`agent-chat.multiple-handoffs`. Morgan's running turn called
`get_task_history` and received the CEO's queued "Launch sooner" comment
at its current revision (`task_history.read` at seq 75). The model
judged the comment as an unauthenticated agent instruction and saved
"Launch later" from the brief. Its accepted result did not mention the
comment. `deliveredQueuedCommentIds` counted the read as delivery, the
release drain dropped the held comment, and no continuation ran.

**Expected behavior**
A read counts as delivery only when the run records the comment as
handled. A comment that was read but not recorded as handled is
delivered once in a new turn. The new turn's prompt says that an earlier
turn already read the comment and may have applied it.

**Steps to reproduce**
1. Start a native run on a task.
2. Queue a comment behind the run.
3. Let the run read `get_task_history` and finish without applying the
comment.
4. Before this change, no continuation runs and the comment is lost.

## What Changed

- `server/src/services/queued-comment-delivery.ts`: the history-read
delivery rule now also requires the reading run's `run.result.accepted`
event to cite `comment:<id>` (top-level `evidence[].ref` or a
criterion's `evidenceRefs`, prefix match so an annotated ref counts).
The read revision, succeeded-run and finished-after-revision rules are
unchanged. All consumers (release drain, stale-continuation save,
ownership reconciler, first-turn queued input, remote-Stop resume) use
this one rule.
- New `readByEarlierTurnCommentIds`, `commentsArrivedDuringRun` and
`taskHistoryHandlingGuidance`.
- `get_task_history` (both tool authorities) returns
`arrivedDuringThisTurn` (the `comment:<id>` refs of comments other
authors posted or edited after the run started) and a `handling`
instruction to cite each applied comment in `paperclip_finish` /
`paperclip_block` evidence. The comment list itself is unchanged. A
failure to compute the list returns no guidance, so the comment stays
owed.
- `buildPaperclipWakePayload` marks a wake comment `readByEarlierTurn`
when another finished run of the agent read its current revision.
`renderPaperclipWakePrompt` adds one line after that comment: check the
current task state first, and do not repeat work that is already done.
- `doc/run-log-events.md`: the delivery rule and the new fields.
- Evals: `agent-chat.multiple-handoffs` (oracle
`runtime-multiple-handoffs.v6`) and `steering.edit-queued-message`
(oracle `edit-queued-message.structural.v6`) accept in-turn consumption
only with a current-revision read and a citation. A read without a
citation must get exactly one continuation whose saved wake marks
`readByEarlierTurn`, and the final saved revision must hold the change
("Launch sooner"; blue). `runtime-queue.ts` reads citations from the
run's `run.result.accepted` event. No scenario YAML changed.

## Verification

- RED on master `b1ef1f3022`,
`server/src/__tests__/heartbeat-held-comment-wake.test.ts`: 10 of 41
fail. The key one: "settles a held message as soon as its run closes the
task (read mid-turn: true, cited: false)" delivered 0, expected 1.
- GREEN: 41/41. The three required shapes:
- read, not cited: one continuation, and its wake marks
`readByEarlierTurn` (`...read mid-turn: true, cited: false` and the
stale-queued-run variant).
  - read and cited: no continuation (`...cited: true`).
  - never read: one continuation, as before.
  - Edited after the read, or a failed reader: still owed.
- `server/src/services/queued-comment-history-read.test.ts` (remote-Stop
resume): succeeded+cited not owed; succeeded+uncited owed; failed+cited
owed. 4/4.
- `packages/adapter-utils/src/server-utils.test.ts`: the prompt line
renders only for `readByEarlierTurn`.
- Neighbouring suites: `muse-control`, `dot-runner`,
`paperclip-runner-tool-authority`, `paperclip-control-plane-port`,
`muse-native-follow-up`, `runner-prp-coordinator`,
`native-session-handoff`, `execution-continuation-history-read`,
`heartbeat-comment-wake-batching`, `ownership-reconciliation-report`,
`run-preparation`: all pass. Server and adapter-utils `tsc --noEmit`:
clean.
- Eval graders: `npx vitest run --config
tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog
tests/runner-e2e/chat-catalog-domains`: 283 files pass. New calibration:
the observed lost-instruction case (read, not cited, no continuation)
fails; read-not-cited with one warned continuation passes; an unwarned
continuation, a missing final change, and a cited read plus a
continuation fail.
- `pnpm test:e2e:runner:typecheck`: pass. `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10.
`node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
`changed: false`.
- Paid run: Runner Full-Stack E2E 43338695850 on throwaway branch
`chat-eval/qa-mr-paid` (this change plus the model-retry fix on master
19f8ed1ea2). Result to follow in a comment.

## Risks

- A run that applies a queued comment but does not cite it gets one more
turn on that task. That turn is told the comment may already be applied.
In the restart-chaos t005 shape (#21175) this can reopen a done task
when the agent does not cite the comment. This is the chosen tradeoff: a
lost comment is worse than one extra checked turn.
- Each `get_task_history` call adds one small indexed read of the run's
`started_at`.
- Upgrade-safe: runs saved by older builds have no citation, so a
read-only delivery from before this change stays owed and is delivered
once more. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:07:08 -06:00
DottaandClaude Opus 5.5 04b50e6007 fix(native-runtime): a timed-out step records its real cause instead of leaving the run unrecoverable (#21262)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs start a Runner and a provider session, then send the
first turn
> - When a startup step fails, the native session executor redacts the
error, classifies it and records the failure
> - A timed-out step fails with a `DOMException` (`TimeoutError` or
`AbortError`). Its `message` property has only a getter
> - The redaction assigned to `error.message`. The assignment threw a
`TypeError`, so the classifier did not run and the real cause was lost
> - The run then fell into a restart-recovery chain that ended in
"requires reconciliation" wording for an operator
> - This pull request makes the redaction safe for errors with a
read-only message
> - The benefit is that a timed-out step records its real cause and
settles through the normal failure path

## Linked Issues or Issue Description

**What happened?**

In a local chaos cell with 8 opencode tasks on Daytona sandboxes, four
first runs failed before their first turn. A step before the first turn
ran under a five-minute `AbortSignal.timeout`. The step took longer than
five minutes. The run log shows `TypeError: Cannot set property message
of which has only a getter` at the startup failure handler, then
`native_finalization_missing`. The run never recorded the timeout.
Recovery then restarted the bootstrap, saw "runnerd exited while waiting
for run.prepare", and ended with `runner_state_identity_mismatch:
bootstrap_not_proven_incomplete` and
`native_continuation_requires_reconciliation`.

**Expected behavior**

A failed startup step records its own cause. The failure handler
classifies it and settles the run. It does not throw a second error that
hides the first one.

**Steps to reproduce**

Make any step before the first turn reject with
`AbortSignal.timeout(n).reason`. Before this change,
`executePaperclipNativeSession` rejects with the `TypeError` above and
the coordinator row keeps no failure detail. The new integration test
does this with the embedded database.

## What Changed

- Add `redactNativeFailureError` in
`server/src/services/native-runtime/native-failure-redaction.ts`. It
redacts the message and stack in place when it can. When the message is
read-only, it returns a new `Error` with the redacted message, the same
`name` and the redacted stack.
- Use it in the startup failure handler of
`executePaperclipNativeSession` in place of the direct assignment.
- Add unit tests for a real `TimeoutError`, a read-only `AbortError`
that holds a secret, and an ordinary typed error.
- Add an integration test: a step before the first turn rejects with a
real `TimeoutError`. The run settles through the classifier.
`failureDetail.message` is "The operation was aborted due to timeout".
The run error code is not `native_continuation_requires_reconciliation`.

## Verification

- Red on master: `native-failure-redaction.test.ts` failed 2 of 3 with
`TypeError: Cannot set property message of which has only a getter`. The
new integration test in `native-startup-retry.integration.test.ts`
failed: the call rejected with that `TypeError` and not with the
`TimeoutError`.
- Green with this change:
- `pnpm --filter @paperclipai/server exec vitest run --config
vitest.config.ts
src/services/native-runtime/native-failure-redaction.test.ts
src/services/native-runtime/native-startup-retry.integration.test.ts`
- `src/services/native-runtime/native-session-executor.test.ts`: 1431
passed, 7 skipped.
  - `pnpm verify` for the affected files.

## Risks

- Low risk. The change only alters behavior when the error message
cannot be assigned. Before, that case threw a `TypeError` and skipped
the failure handler.
- The replacement error keeps the `name` but is a plain `Error`, so
`instanceof DOMException` is false after redaction. No code in the
handler checks for `DOMException`.
- After this PR, a capture timeout before the first turn still ends
blocked. The run records the real cause, but it shows generic wording
and waits for a person to press Retry. The reason: a generic timeout has
no typed cause, so the classifier records it as
`native_session_interrupted` and treats it as ambiguous state. Follow-up
PRs give the capture step a typed cause and per-step bounds, and retry a
failure that provably happened before any prompt automatically.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs). Extended reasoning
was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:01:04 -06:00
DottaandClaude Opus 5.5 1c33275ada fix(heartbeat): start the new owner after reassignment away from a queued creator run (#21256)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A lead agent often creates a task for itself and then reassigns it
to another agent
> - The creator is busy in its own chat run, so the new task's
assignment run stays queued and does not take the task's execution lock
> - #21030 releases the outgoing assignee's never-started run on
reassignment, but only when that run holds the execution lock
> - Here the run did not hold the lock, so it stayed queued. The new
owner's wake then adopted it as the active run and was deferred behind
it
> - When the queue later withheld the stale run as
`issue_assignee_changed`, nothing promoted the deferred wake, and the
stranded-wake sweep ignores assignment wakes. The new owner never ran
> - This pull request closes all three gaps
> - The benefit is that a reassigned task starts its new owner at once,
and recovery still starts it if a release is missed

## Linked Issues or Issue Description

Refs #21030

**What happened?**
In the paid chat evals `onboarding.first-work` and
`questions-and-approvals.duplicate-answer` (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master b1ef1f3022), the
CEO created a task for itself while its chat run was active, then
reassigned it to Morgan. The CEO's queued assignment run was withheld at
the execution gate with "issue assignee changed before the queued run
could start; the new owner will be woken instead". Morgan's wake stayed
`deferred_issue_execution` until teardown, about 4.5 minutes later.
Morgan never ran.

**Steps to reproduce**
1. Give an agent a running run on another task.
2. Create a task assigned to that agent, so its assignment run is queued
without the execution lock.
3. Reassign the task to a second agent.
4. The second agent's wake is deferred behind the stale run and is not
promoted when that run is withheld.

**Expected behavior**
The stale run is released and the new owner gets exactly one run for the
task, promptly.

## What Changed

- `server/src/routes/issues.ts`: on reassignment, when no live run holds
the task's lock, also release the outgoing assignee's never-started
`issue_assigned` run for that task. This uses the existing
`issue_reassigned` stop from #21030. The run is read through a new
`heartbeat.getQueuedAssignmentRunForIssue` accessor in
`heartbeat/run-state.ts`; route test mocks of the heartbeat service gain
that method.
- `server/src/services/heartbeat/queue.ts`: when the queue gate
withholds a queued run as `issue_assignee_changed`, it now releases the
task and promotes the new owner's deferred wake. The promoted run starts
after the claim lock is released (`deferredPostCommitEffects`), like the
neighboring cancellation branches. A failed promotion is logged and left
to recovery.
- `server/src/modules/run-dispatch/adapters/postgres.ts`: the same
cancellation hints each deferred wake of the task to the
deferred-receipt worker in the same transaction, so recovery runs even
if the process stops before promotion.
- `server/src/services/heartbeat/queue.ts`, `deferred-receipt-work.ts`,
`deferred-execution-notifications.ts`: the stranded-wake sweep now
includes deferred `issue_assigned` wakes of the current assignee. It
promotes them only when no queued, running or retrying run remains for
the task, through the normal release admission.
- `server/src/__tests__/heartbeat-reassignment-deferred-wake.test.ts`:
the cohort's state through the real wake path. One case covers the gate
withholding the stale run. One case covers a wake left deferred after
the blocking run finished. Both expect exactly one run by the new owner,
executed.
- `server/src/__tests__/issue-reassignment-queued-run.test.ts`: the
#21030 case now also runs when the queued run does not hold the lock,
and expects exactly one new-owner run.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-reassignment-deferred-wake.test.ts
src/__tests__/issue-reassignment-queued-run.test.ts` passes. On master,
both heartbeat cases fail (the wake stays `deferred_issue_execution`)
and the no-lock route case fails (the stale run stays queued). With only
the gate change removed, the gate case fails and the recovery case still
passes.
- Focused suites: `deferred-execution-receipt-work`,
`heartbeat-lock-release-on-reassignment`,
`heartbeat-stale-queue-invalidation`,
`issue-agent-mutation-ownership-routes`, `src/modules/run-dispatch`,
`src/modules/wake-queue` pass.
- `cd server && npx tsc --noEmit` passes.

## Risks

- Behavior change: reassignment now cancels the outgoing assignee's
queued `issue_assigned` run for the task even when it does not hold the
lock. The queue gate already cancelled that run later as
`issue_assignee_changed`; now the new owner does not wait for it.
- The stranded-wake sweep now also reads deferred assignment wakes. It
skips any wake whose task still has a live run, so a wake waiting behind
a real owner is not promoted early.
- Upgrade safety: no schema or saved-state shape changes. Wakes already
saved as `deferred_issue_execution` by the previous release are promoted
by the sweep after deploy.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:36:05 -06:00
DottaandClaude Opus 5.5 500bc694b5 fix(recovery): keep a draining server's lease alive and end a crashed server's runs in about 2 minutes (#21238)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A release swap runs two servers for a short overlap. The old server
drains, and the new server takes over the runs.
> - Each server proves that it is alive with a controller-instance
lease. Recovery uses that lease to decide whether a run's controller is
gone.
> - In local swap cells, a healthy old server exited during the overlap,
because its lease renewal waited for a connection on the busy main pool.
Every run it drove was interrupted.
> - When a controller crashed, its runs waited for another server boot,
or for a timer that restarted on every look. Tasks stood still for 21 to
28 minutes.
> - This pull request renews the lease on its own connection, and lets
the live server end a crashed controller's runs within a bounded time.
> - The benefit is that a swap does not kill a healthy server, and a
crashed server's runs end in about 2 minutes, so their tasks continue by
themselves.

## Linked Issues or Issue Description

Refs #21135, #21222

**What happened?**
1. During a two-server release swap (local cells, load 14 to 34), the
outgoing server exited with "controller instance lease lost while
holding work". Five lease renewals in a row timed out after 10 s. The 10
s timeout included the wait for a connection on the shared main pool,
which the traffic switch and the replacement's startup pass kept busy.
2. When the old server was killed in the middle of the overlap, its runs
stayed in progress for 21 to 28 minutes. The live server did not
re-check a crashed controller's runs: only a new boot did. The
unverified-controller wait started when a server first looked at the
run, not when the controller left, and the successor's own wait record
counted as the dead owner's sign of life.

**Expected behavior**
A healthy server keeps its lease during a swap. When a controller is
gone (its lease lapsed and it is fenced or stopped), the live server
ends that controller's runs within a bounded time, without another
server boot, so restart recovery continues the tasks.

**Steps to reproduce**
1. Run two servers against one database with a release swap and a 200 s
overlap.
2. Kill the old server 40 s into the overlap (SIGKILL).
3. See that its runs stay in progress for more than 20 minutes, until
another server starts.

## What Changed

- Lease renewal on its own connection (`packages/db/src/client.ts`,
`packages/db/src/index.ts`,
`server/src/services/controller-instance-lease.ts`):
- New `controllerInstanceLeaseDb(db)`: one private 1-connection pool
(`application_name` `paperclip-controller-instance-lease`), made the
same way as the existing `controllerLeaseDb` and closed by
`closeLivenessDb`.
  - The renewal is one statement on that connection, as before.
- The connection sends `statement_timeout` = 8 s (below the 10 s client
timeout) as a startup parameter (new
`DatabaseClientOptions.statementTimeoutMs`, used only by this pool). The
server cancels a slow renewal, so it never holds the single connection
that the next renewal needs, and there is no extra round trip per
renewal. (An earlier revision set it per transaction; that added three
round trips to every renewal and broke the idle-server receipt test.)
- If a connection pooler refuses the startup parameter ("unsupported
startup parameter"), the lease logs a warning and renews on a
replacement connection without it.
- A crashed controller's runs
(`server/src/services/recovery/ownership-registry.ts`,
`server/src/index.ts`,
`server/src/services/native-runtime/native-restart-recovery.ts`):
- The ownership reconciler on the live server hands runs whose
controller is proven gone to restart recovery. It does not wait for
another server boot.
- The unverified-controller wait is bounded from the moment the
controller's lease lapsed, not from the first look.
- When the controller boot is `stopped` or `fenced`, the run ends at
once.
- A successor's wait record no longer refreshes the dead owner's sign of
life.

Safety of ending a run when its controller boot is fenced or stopped:
- Only `stopped` (written by the boot's own graceful stop) and `fenced`
(a newer epoch fenced it after its lease expired plus 30 s grace) count.
`live` and `grace` never count, so a controller that is alive and
renewing is never treated as gone. `expired` alone does not count
either.
- A boot that cannot renew stops itself at its self-fence window (about
its lease expiry), and a boot that finds itself fenced exits at once. So
when another server can fence it, the old process has already stopped
driving runs. The lease change above removes the main cause of healthy
boots failing to renew.
- If the old process is still alive (partition or stall) and its Runner
continues: the old run is fenced. Its agent token is refused for writes
(`agent_run_superseded`), its remote workspace epoch is abandoned or the
continuation gets a new workspace, and the native coordinator records
`replacementDenied`. There are never two writers in one workspace.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `server/src/__tests__/controller-instance-lease.test.ts` "own
connection": red (10004 ms timeout while the main pool is held) to
green. Full file green (26).
- `server/src/__tests__/controller-instance-lease.test.ts` "has the
server cancel a renewal blocked on a row lock, and renews on the same
connection next time": red without the connection-level timeout (the
renewal runs to the 10 s client timeout and stays active), green with it
(cancelled at about 8 s, no active lease statement left, the next
renewal succeeds in under 2 s on the same backend).
- `packages/db/src/client-options.test.ts`: the statement timeout is
sent as a startup parameter.
- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`:
"lapsed longer ago" / "end of the bound" (wait counted from the lease
lapse), the live-server re-check of a crashed controller, the
fenced/stopped boot ending the run at once, and the successor's wait
record. Full file green.

Changed existing expectation (it asserted the old wait for a stopped or
fenced controller):
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
"keeps foreign local Runner and provider ownership after its controller
is stopped/fenced": the disposition is now `blocked`
(`controller_process_space_unverified`) instead of `awaiting_evidence`,
the run is `failed` with `native_restart_recovery_blocked`, and the
finalization is `terminal_failure` with `executorStopEvidence:
"unverified"`. The safety checks stay: no local PID probe, no claim, no
stamp, no local stop proof, the run keeps its process identity and
session checkpoint, and no result is written.

Local commands, on this branch based on current master:
- `cd server && npx vitest run
src/__tests__/controller-instance-lease.test.ts
src/__tests__/heartbeat-native-restart-suspension.test.ts`: 2 files, 140
tests passed.
- `npx vitest run src/__tests__/controller-instance-lease.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
225 passed, 14 skipped (real-process cases that need CI).
- `pnpm exec vitest run src/__tests__/maintenance-server-routes.test.ts
-t "releases an ordinary held task Retry"`: passed (idle receipt: one
lease statement after the work ends).

Local swap cells (two servers, embedded Postgres, real agents, old
server SIGKILLed 40 s into a 200 s overlap):
- Before: runs of the killed server stayed in progress 21 to 28 minutes.
- N1 (2 tasks): 2/2 finished hands-off. Kill to run end 5 min 31 s (the
successor's wait write still moved the timer; fixed by the last commit).
- R1 (6 tasks, 3 adapters, subtasks, follow-ups, 2 swaps): 4/4
hands-off, 0 banners, 0 duplicates, 0 invariant violations. Kill to run
end 3 min 39 s.
- N2 (same seed as N1, all fixes): 2/2 hands-off. Kill to run end 2 min
01 s; fresh runs started at 2 min 32 s and 3 min 02 s. The floor is the
lease TTL (90 s) plus the fence grace (30 s).
- Hour-long release-swap wave (w-20261112T233630-s7201) on the branch
that holds this fix and the related recovery fixes: 11/11 tasks finished
hands-off (master: 9/11). The lease and crashed-controller fixes were
visibly working in it.

## Risks

- A pooled database URL whose pooler refuses `statement_timeout` as a
startup parameter: the first renewal fails over to a connection without
it (logged), so the lease keeps working without the server-side cancel.
- One extra database connection per server, held while the server runs.
It is idle except for one renewal every few seconds.
- A run whose controller boot is fenced or stopped now ends at once
instead of after the bound. See the safety notes above: only a boot that
is not alive can be in these states.
- No migration. No schema change. Nothing changes before `turn.start` on
the normal path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:11:27 -06:00
DottaandClaude Opus 5.5 9025597ce4 fix(feedback): keep run ids and dates intact when redacting phone numbers (#21251)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People can export a feedback bundle that holds run records for
review
> - The bundle is redacted before export, and one rule removes phone
numbers
> - The phone rule also matched two all-digit groups inside a run UUID
> - A run id such as `9224d1c8-1dbf-4314-9924-e16fcf3ad1fa` became
`9224d1c8-1dbf-[REDACTED_PHONE]-e16fcf3ad1fa`, so the bundle no longer
named its run
> - This pull request keeps UUIDs and ISO dates out of the phone rule
> - The benefit is that exported ids and dates stay correct and real
phone numbers stay redacted

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `cli-operations.feedback-bundle`, the exported
bundle's `envelope.targetId` was
`9224d1c8-1dbf-[REDACTED_PHONE]-e16fcf3ad1fa`. The phone rule
`(?<!\w)(?:\+?\d[\d ()-]{7,}\d)(?!\w)` matches two all-digit UUID groups
joined by `-`. About 2 to 3 percent of random UUIDs have that shape. The
same rule also matched a bare date such as `2026-11-12`.

**Steps to reproduce**
1. Call `sanitizeFeedbackText` with text that contains
`9224d1c8-1dbf-4314-9924-e16fcf3ad1fa`.
2. Read the output.

**Expected behavior**
The UUID stays unchanged. A real phone number such as `+1 (415)
555-1234` is still redacted.

## What Changed

- `server/src/services/feedback-redaction.ts`: the phone rule now runs
only on the text between public identifiers. UUIDs and ISO-8601 dates
and timestamps are copied through unchanged. Other rules are unchanged.
- `server/src/__tests__/feedback-redaction.test.ts`: a regression test
with the `-4314-9924-` UUID, an all-digit UUID, a date and a timestamp,
next to a real phone number that must stay redacted.

## Verification

- `cd server && npx vitest run src/__tests__/feedback-redaction.test.ts`
passes (13 tests). The new test fails on master: the UUID and the date
are replaced with `[REDACTED_PHONE]`.

## Risks

- Low risk. A phone number written in the exact shape of a UUID or an
ISO date is no longer redacted. Those shapes are not phone numbers.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:06:39 -06:00
DottaandClaude Opus 5.5 ddc0aacad8 test(recovery): pin the daily-limit fallback test to the current UTC day (#21236)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The fresh-continuation fallback continues interrupted tasks. Its
tests run on every pull request.
> - One test checks that the agent's daily run limit keeps a
continuation pending.
> - The test failed on unrelated pull requests just after 00:00 UTC. It
passed the rest of the day.
> - The daily run limit counts runs started in the current UTC day. The
test's source run started 11 minutes before now, so just after midnight
it counted as yesterday's run.
> - This pull request pins the test's runs to the current UTC day.
> - The benefit is that the test gives the same result at every hour,
and CI and the release source check do not fail after midnight UTC.

## Linked Issues or Issue Description

**What happened?**
`server/src/services/fresh-continuation-fallback.test.ts > keeps the
continuation pending while the agent's daily run limit declines it, then
delivers it` failed in "General tests (server (11/12))" on two unrelated
pull requests at about 00:10 UTC, with `expected [ { …(81) } ] to have a
length of +0 but got 1`.

**Expected behavior**
The test passes at every hour of the day.

**Steps to reproduce**
1. Run the test between 00:00 and 00:11 UTC (or move the source run's
`startedAt` to before today's UTC midnight, which is the same state).
2. See that the daily limit does not decline the continuation, and a
successor run is created.

The product code is correct: `dailyRunConditions` in
`server/src/services/heartbeat/timer-policy.ts` counts runs whose
`startedAt` is in the current UTC day. Only the test depended on the
clock.

## What Changed

- The test now sets the source run's `startedAt` to one minute before
today's UTC midnight, and adds a succeeded run started now on another
task. That run uses the agent's single daily run.
- Test file only. No product change.

## Verification

- RED: with only the source run moved before today's UTC midnight (the
state from 00:00 to 00:11 UTC), the test fails with the same message as
CI: `expected [ { …(81) } ] to have a length of +0 but got 1`.
- GREEN: with the run started today added, the test passes.
- `cd server && npx vitest run
src/services/fresh-continuation-fallback.test.ts`: 53 passed (on current
master).

## Risks

- Low risk. Test file only.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
behavior change)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:51:51 -06:00
DottaandClaude Opus 5.5 ade7e0643f fix(recovery): pace recovery continuations and start person-initiated work first (#21234)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Execution recovery continues tasks that a restart or a lost run left
unfinished
> - Recent recovery changes restored many held tasks at the same time,
and each restored task got a queued run on its agent
> - The agent's queue starts runs oldest first, so a person's Retry or
chat message waited behind every older recovery run
> - On staging, one agent had 131 queued runs, and person Retries waited
more than 50 minutes
> - This pull request paces how many recovery continuations enter an
agent's queue, and it starts person-initiated work before system
recovery work
> - The benefit is that a burst of recovery cannot starve people, and
every recovery run still runs

## Linked Issues or Issue Description

Refs #21222, #21139, #21135

**What happened?**

Staging deployed #21139 and #21135 at 22:53Z. Within one hour, the
company had 201 queued runs and 78 running runs. 122 queued runs had the
recovery wake reason `issue_recovery_action_restored`. Each one was on a
different task. They all came from the reconciled-execution delivery
pass (`source: execution.reconciled`), at a rate of 5 to 15 per minute
from 22:57Z to 23:51Z. One agent had 131 queued runs. The oldest queued
run had waited since 22:53Z. A person's Retry at 23:14Z was still queued
4 minutes later. Six person Retries on that agent, queued from 23:08Z to
23:14Z, were still queued at 00:04Z, more than 50 minutes behind
recovery runs. At 00:04Z, that agent's newest start was a recovery run
created at 22:57Z. In the last 400 runs, 25 succeeded.

**Expected behavior**

A person's Retry, chat message, or assignment starts before system
recovery work on the same agent. Recovery work still drains at a steady
rate. All recovery runs run eventually, and none is dropped. A task
whose continuation waits shows that the task continues automatically. It
does not show that the task is idle, and it does not ask for Retry.

**Steps to reproduce**

1. Give an agent a concurrency limit of 1.
2. Queue many recovery continuations for the agent
(`issue_recovery_action_restored`), for example after a server restart
with many held tasks.
3. Click Retry on another task of the same agent.
4. See that the Retry stays queued until all older recovery runs start.

**Paperclip version or commit**

Staging build 2026.1106.0+2187.git.57aeda190

**Deployment mode**

Hosted cloud (staging)

**Agent adapter(s) involved**

paperclip_runner (native and legacy runtimes). The problem is in the
shared run queue, so it applies to all adapters.

## What Changed

- New module
`server/src/services/heartbeat/queued-run-dispatch-order.ts` decides the
order in which an agent's queued runs start. Person-initiated runs start
first: `task_retry`, external chat messages, and any wake that a user
requested. Other work comes next, together with recovery runs for
interruptions that ended in the last 30 minutes. Old-backlog recovery
runs come last. Inside each class, the existing order applies:
readiness, then priority, then age. Runs whose dependencies are not
ready stay last.
- Recovery runs get a bounded share. When the agent started 3 other runs
after the oldest queued recovery run arrived and none of them was
recovery work, the next slot goes to a recovery run. The share applies
across the whole batch when one dispatch fills many slots. Thus people
cannot starve recovery.
- `startNextQueuedRunForAgent` in
`server/src/services/heartbeat/queue.ts` uses the new order. It changes
only the ranking. It does not change `claimQueuedRun`, the admission
preflight, or the promoter.
- `deliverReconciledExecutions` in
`server/src/services/execution-recovery-resolution.ts` paces delivery.
It leaves a continuation pending when the agent already has a queued
recovery run, or when the agent's queue is at or above its concurrency
limit. One pass delivers at most 5 continuations per company and at most
1 per agent. The pass reads pending continuations in pages and leaves
out agents and companies that it already handled, so continuations for
busy agents cannot hide continuations for other agents. A continuation
that has waited 5 minutes or more is admitted even when ordinary work
keeps the queue full, and it goes ahead of fresher continuations,
longest wait first. Otherwise the most recent interruption goes first.
The existing 15-second continuation schedule delivers the remaining
items. This schedule runs only while pending continuations exist, so an
idle server does not poll.
- The assigned-work sweep
(`server/src/services/recovery/assigned-work-sweep.ts`) counts a pending
continuation as a pending wake for its task. Thus the invariant does not
see a held-back task as stranded, and it does not enqueue a second wake.

## Verification

- `pnpm exec vitest run
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/recovery-wake-herd.test.ts` (in `server/`): 17 tests pass.
- Red/green: with the source changes reverted to master, all tests in
`recovery-wake-herd.test.ts` fail. Each review fix also has a test that
fails on the head before that fix. With this change, they pass.
- `recovery-wake-herd.test.ts` covers these cases with embedded Postgres
and the real heartbeat queue:
- Six recovery wakes are queued, then a person's request. The first
dispatch starts the person's run. All six recovery runs start later.
- Four pending continuations for one agent: one pass delivers one. The
others stay pending and are delivered on later passes. The continuation
schedule stays armed while items are pending and returns null after all
items are delivered.
- A continuation stays pending while the agent's queue is at its
concurrency limit.
- The per-company cap applies, and the most recent interruption is
delivered first.
- A continuation that has waited past the bound is admitted while
ordinary work keeps the queue full.
- New interruptions arrive for the same agent on every 15-second pass,
and the older continuation is still admitted within the bound.
- Continuations for four busy agents (32 rows, more than one page) do
not hide an idle agent's continuation.
- A held-back continuation counts as a `queued_wake` pending intent for
the assigned-work invariant, and the task shows `working` with
`canRetry: false`.
- `queued-run-dispatch-order.test.ts` simulates dispatch: a Retry starts
ahead of 40 recovery wakes, and all 40 still run. With a person Retry
arriving at every step, every window of 4 starts includes 1 recovery
run. A fresh interruption's continuation is not put behind old-backlog
recovery runs. When one dispatch fills 8 slots, the order is PPPRPPPR (P
is a person run, R is a recovery run).
- Related suites pass: fresh-continuation fallback, issue recovery
actions, completed-turn cleanup hold, issue monitor scheduler, workspace
recovery fallback, assigned-work invariant, execution control work,
execution recovery resolution, and the heartbeat queue.
- `node scripts/typecheck-layers.mjs` (in `server/`) passes.

## Risks

- Behavior change: when a person's run waits behind recovery runs, it
now starts first. Recovery runs start a little later, but they still get
at least 1 of every 4 starts while people send work.
- Delivery of continuations is slower during a burst: at most 1 per
agent and 5 per company every 15 seconds. This is intentional. A
continuation that waits shows as continuing automatically, and the
invariant does not repair it twice.
- Ranking adds up to three small indexed reads when an agent has more
than one queued run. The reads run only when the queue has recovery runs
or wake requests to classify.
- No schema or migration changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, and read-only staging
API requests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:30:56 -06:00
DottaandClaude Opus 5.5 2d1c2db64a feat(mcp): host the MCP Skills extension for connected apps (#21219)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use connected apps through MCP. Paperclip already hosts MCP
Apps views and speaks stateless MCP 2026-07-28.
> - The MCP Skills extension (SEP-2640, Final) lets an app publish
skills (instructions and files) next to its tools. Hugging Face already
serves 25.
> - Paperclip did not read them, so agents missed each app's own
workflow guidance.
> - Skills are a prompt-injection surface. The spec makes manifest
binding, verification, origin tagging and approval revocation host
requirements.
> - This pull request makes Paperclip a Skills host: discover, review,
verify, and deliver app skills to the agents that use the app.
> - The benefit is that connected apps can teach agents their workflows,
under board control.

## Linked Issues or Issue Description

**Problem**
Connected MCP servers can publish skills with the
`io.modelcontextprotocol/skills` extension. Paperclip ignored them. A
board could not see, review or approve them, and agents never received
them.

**Proposed solution**
Implement the host side of SEP-2640. List skills during catalog refresh,
record them per connection and company, bind board approval to the full
file manifest, and load approved skills into agent runs with verified,
lazily fetched content.

**Why it matters**
Apps ship workflow guidance with their tools. Without the extension,
each team must copy that guidance into company skills by hand, and it
goes stale.

Refs #21119, Refs #21140 (the MCP Apps and stateless MCP client work
this builds on).

## What Changed

- **Client capability.** Every stateless request declares
`io.modelcontextprotocol/skills`. Legacy handshake sessions do not. The
capabilities that `server/discover` advertises are remembered beside the
protocol era.
- **Discovery.** Catalog refresh calls `skills/list` (paged) only when
the server declared the extension. A listing inside its `ttlMs` is
reused. A skills failure never fails the tool refresh. Nothing runs
while a run starts.
- **Schema.** New `tool_mcp_skills` (company, connection, skill URI;
frontmatter, sorted manifest, manifest digest, review state) and
`tool_mcp_skill_files` (verified bytes keyed by connection, URI and
digest). Both cascade with the connection.
- **Governance.** New skills wait for review. Approval binds to the
order-free digest of every `{uri, digest, size}`; a request for any
other digest gets 409. A changed manifest or frontmatter revokes
approval and logs `tool_mcp_skill.approval_revoked`. Dynamic, oversized
and invalid skills are declined with a reason. Approve and decline use
the company skill policy (`skills.install`, `external_package`,
`mcp-connection:<id>`).
- **Run time.** Approved skills of connections the agent can use load in
parallel with other skill preparation, inside a 3 s budget. `SKILL.md`
is read with the agent's grant and checked for size, SHA-256 and
frontmatter. A failure calls `skills/get` and takes the skill out of
use. Each skill becomes a per-run bundle on the connector skill path,
named in its app's namespace, tagged with its origin. Paperclip writes
the frontmatter: `allowed-tools` only if the board approved it.
- **Supporting files.** `GET
/api/agents/me/mcp-skills/:skillId/files?path=` serves manifest-listed
files to an agent in an active run, through the skill's own connection.
The sandbox bridge allows the route.
- **Board API and UI.** `GET/POST /api/companies/:companyId/mcp-skills…`
(list, read instructions, approve, decline). The Skills page shows
"Skills from connected apps" with app, state, size, requested tools and
plain-language notes. It renders nothing when no app publishes skills.
- **Docs.** `doc/connections/MCP-SKILLS.md`, linked from
`doc/connections/README.md`.
- **Shared frontmatter parser.** It now decodes YAML single-quoted
scalars. Before, 5 of Hugging Face's 25 skills kept their quotes and
would fail the frontmatter check; now all 25 match. The generated runner
copy is regenerated.

File size notes: `server/src/services/tool-gateway.ts` grows by about
200 lines. The run-time reads must use its grant resolution and
`callAssignedConnectionProtocol`, like MCP Apps view reads. Splitting
the gateway is a separate refactor. `CompanySkills.tsx` and
`CompanySkills.production.tsx` gain 2–4 lines each; the section lives in
its own component.

## Verification

- `server/src/__tests__/mcp-skills.test.ts` (15 tests) uses a fake
stateless skills server and embedded Postgres. It covers: discovery only
after the server declares the extension; the capability in `_meta`;
`ttlMs` reuse; lazy fetch (no reads at listing or approval; `SKILL.md`
at load; supporting files on read); a stale-digest approval refused; a
reorder that keeps approval; a changed or added file that revokes it;
dynamic skills declined; a same-size tampered `SKILL.md` rejected,
refreshed with `skills/get`, and taken out of use; mismatched
frontmatter rejected; reads outside the manifest refused; the same URI
and name from two servers kept apart; company scoping; a slow app
skipped within the budget and cached later; HTTP authorization (board
only, agent needs an active run, run header must match) and activity
logging.
- Red/green: with the digest check disabled, the tamper test fails; with
the manifest comparison disabled, the revocation test fails. Both pass
with the code.
- `server/src/__tests__/mcp-protocol-versions.test.ts`,
`mcp-apps.test.ts`, `connection-instructions.test.ts`,
`tool-gateway-discovery-http.test.ts`: 47 pass (two capability
assertions updated for the new extension).
- `packages/shared/src/frontmatter.test.ts` adds a single-quoted scalar
case; the runner contract test passes (23 tests).
- `ui/src/components/skills/ConnectedAppSkills.test.tsx` (3) and
`ui/src/pages/CompanySkills.test.tsx` (44) pass.
`packages/adapter-utils/src/sandbox-callback-bridge.test.ts` (89)
passes.
- `pnpm --filter @paperclipai/server typecheck` passes. UI `tsc` passes
except the known missing generated Storybook catalog files.
- `pnpm verify` returned exit 2 (doc change is a global CI input);
required CI supplies broad coverage.
- Live check against `https://huggingface.co/mcp` without credentials,
through Paperclip's own services and the real egress guard: 25 skills
discovered as pending; three approved; cold load 135 ms, cached load 13
ms; a supporting file read through the manifest; the bundle tagged with
its origin.
- Storybook: `Agents and reusable capabilities/Skills and Skill
Studio/Connected app skills` (needs review, all states, tool request,
name collision, reading instructions).

## Risks

- **Migration:** two new tables only. No changes to existing tables.
Safe with the previous release running.
- **Run start:** companies without an approved app skill pay one indexed
read. With approved skills the wait is capped at 3 s; late or failing
skills are left out with a `skills.mcp_unavailable` run warning and
never fail the run.
- **Agent harness enforcement:** Paperclip cannot gate each
code-execution tool call an agent makes while following an app's skill.
Approval is explicit and per skill, the approval note says agents may
run commands the skill describes, and `allowed-tools` is never granted
without a separate tick.
- **Shared-home adapters** receive app skills like company skills. A
revoked skill leaves on the next skill sync.
- Not in this PR: per-connection opt-in for dynamic skills; serving
Paperclip's own skills from its public MCP endpoint;
`resources/directory/read`; app skills on an agent's skills tab.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs, live HTTP checks).
Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:18:06 -06:00
DottaandClaude Fable 5.2 826d41e646 feat(ui): redesign Settings → Secrets around what people manage (#21225)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents need API keys and tokens. People manage them in Settings →
Secrets.
> - The page showed every credential in one table. Tokens that Paperclip
saved for connections, chat channels and model sign-ins sat beside the
few secrets a person added, many with record IDs as names.
> - Each row repeated an always-true status, a version and a key. New
secret opened on two stacked mode switches and two storage pickers.
> - A person could not see which agents use a secret without opening it,
and had to open it again after creation to give an agent access.
> - This pull request redesigns the page, the New secret dialog, the
details panel, the Vaults tab, the import dialog, the Proposals tab and
My secrets. It keeps the existing data and API calls.
> - The benefit is a list that shows what a person manages, a create
flow with two fields, and a secret that an agent can use as soon as it
is saved.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The Settings → Secrets page, its New secret dialog, the secret details
panel, the Vaults tab, the import-from-vault dialog, the Proposals tab
and the My secrets tab.

**Subsystem affected**

UI: `ui/src/pages/Secrets.tsx` and `ui/src/pages/secrets/*`.

**Current behavior**

- One table lists all secrets. On a busy organization most rows are
credentials that Paperclip saved itself, and many have record IDs as
names.
- Each row shows Active, a version, a chip and the key. The row does not
show which agents use the secret.
- A three-line help banner is always visible.
- New secret starts with two mode switches (Organization / Each user,
Managed value / External reference), then a provider picker and a vault
picker. The Create button is disabled without a reason.
- The details panel has Details, Usage and Access events tabs.
- The Vaults tab has one section per provider, with empty placeholder
boxes and a status picker in the form.

**Proposed behavior**

- The list shows the secrets that the team added. Each row shows who
provides the value (Organization, or Each user with a count) and which
agents use it.
- Credentials that Paperclip saved collapse into "Managed by Paperclip".
They are grouped by origin and have readable names.
- One Filter menu covers provided-by, storage and status. Active filters
show as chips. Archived secrets stay hidden until a person asks for
them.
- New secret has a name, one explicit Organization / Each user choice, a
masked value, an optional list of agents, and "Stored in" only when more
than one storage place exists. Create stays enabled and says what is
missing. A `NAME=value` pasted into Name fills both fields.
- The details panel is one scrolling view: agents, other places the
secret is set, details, and recent activity.
- The Vaults tab is one flat list with health and secret counts. The
connect dialog has two required fields. Paperclip checks a new vault
immediately after it is saved.

**Reason and benefit**

A person can find their own secrets, see if a secret is used, and create
a secret that an agent can use in one step.

**Breaking changes**

None. The API gains one optional query parameter (`order` on the board
proposals list). The default order is unchanged. See Risks for the UI
behaviors that changed.

## What Changed

- **List.** Rewrote the Secrets tab in `Secrets.tsx`. Added
`secret-list-parts.tsx` for the Provided by and Used by cells, the help
popover and the managed groups. The Folders / Flat toggle is removed.
Folders are always on and search covers all folders.
- **Managed grouping.** Added `secret-origin.ts`. It infers the origin
of a secret from the name and key conventions of the server modules that
create secrets (`tool_app.*`, `tool_oauth.*`, `slack_manager.*`,
`chat.*`, `ai_*`, `routine-*`, `agent-observability.*`). It is used for
presentation only.
- **Agent usage.** Added `agent-secret-access.ts`. It holds the grant
and revoke logic that was inside the details panel, and builds one usage
index from the agents list for all rows.
- **New secret.** Rewrote the dialog. Added `buildCreateVaultOptions`
and `parseEnvAssignment`. Agents selected in the dialog get an
environment variable binding after the secret is saved. A failed grant
shows an error and keeps the secret.
- **Details.** Replaced the tabs with sections. "Also set in" lists the
bindings that are not agents. Activity shows agent names and the last
five entries, with "Show all activity".
- **Vaults.** Flat list with a built-in Paperclip row, a Default badge
that follows `getSelectableProviderConfig`, Check again, Enable, and an
Add vault menu. The dialog has Name and AWS region, with naming and tags
in a disclosure. The Status picker is removed.
- **Import, Proposals, My secrets.** Restyled to the same design.
Proposals adds a Recently decided list. The board list endpoint `GET
/api/companies/{companyId}/secret-proposals` takes a new optional
`order=resolved` to sort by decision time.
- **Copy.** "Provider vault" is "vault". "Local encrypted" is
"Paperclip" or "Paperclip-encrypted". Toasts and block reasons use plain
words.
- **Stories.** `secrets.stories.tsx` has new stories for the managed
section, a folder, New secret, the details panel and Vaults. The
Storybook usage fixture now returns the `target` field that the API
returns.
- **Tests.** Updated the page, import, proposals and My secrets tests.
Added `secret-origin.test.ts`. Updated the runner E2E vault-reference
journey and the Storybook visual render-state copy for the new labels.
- **Docs.** `Secrets → Provider vaults` is now `Secrets → Vaults` in
three documents.

`Secrets.tsx` was 4,855 lines and is 4,951 lines. The redesign edits the
page in place to keep its queries, mutations and deep links. Splitting
the file is a separate change.

## Verification

- `cd ui && node ../scripts/run-tsc.mjs -b`: no errors.
- `cd ui && pnpm vitest run src/pages/secrets src/pages/Secrets.test.ts
src/pages/Secrets.render.test.tsx
src/components/AgentSecretAccessEditor.test.tsx
src/components/AgentConfigForm src/components/IssueThreadInteractionCard
storybook/map-tooling.test.ts`: 14 files, 467 tests passed.
- `cd server && pnpm vitest run
src/__tests__/secret-proposals-routes.test.ts -t "decision time"`:
passed.
- `pnpm vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog-domains/external-vault-reference
chat-catalog-domains/external-personal-secret-menu
chat-catalog-domains/external-personal-secret-entry-dialog
chat-catalog-domains/external-personal-secret.test`: 32 passed, 2
skipped.
- `node scripts/check-token-gates.mjs`, `node scripts/check-maps.mjs`,
`node scripts/check-module-boundaries.mjs`, `node
scripts/storybook-catalog.mjs`: clean.
- `pnpm verify`: exit 2. It asks for broader coverage because a runner
E2E source file changed. Required CI supplies that coverage.
- All 22 Secrets stories render and their play functions pass in
headless Chromium against a local Storybook.
- End to end, on an isolated local instance (`paperclipai worktree init
--empty`, one company, two agents), with a headless Chromium script. 16
of 16 steps passed, with no page errors:
  - Empty state. Create explains missing fields.
- Create an organization secret and give an agent access. The agent's
saved config has the `secret_ref` binding.
- Paste `NAME=value`. Create an each-user secret. Add a value in My
secrets. The list shows "1 of 1 added".
  - Create a folder, create a secret in it, search across folders.
- In details: add an agent, replace the value, remove the agent. The
agent's saved config changes each time.
  - Disable, archive, and the Archived filter.
  - Managed secrets leave the team list and show under their groups.
- Add a Paperclip-encrypted vault and an AWS vault. New secret then
shows "Stored in".
- The import dialog explains a denied vault. The Proposals tab shows its
empty state. The list at 390 px width.
- Not run: the paid runner E2E journeys. I updated
`external-vault-reference.ts` for the new labels and could not execute
it. A live AWS import, and a proposal from a real agent run, were not
exercised.

## Risks

- **Origin grouping is a heuristic.** It reads naming conventions that
live in server modules. If a module changes its naming, its secrets show
in the team list again. Nothing is hidden or blocked by it. A recorded
origin on each secret is the durable fix.
- **Removed UI.** The Folders / Flat toggle, the "View references"
dialog, the vault Status picker, and creating draft GCP or HashiCorp
vault entries from the Add vault menu. Existing draft vaults still show
and can be edited. A disabled vault now has an Enable action.
- **Default status filter.** The list now shows active and disabled
secrets. Before, it showed active secrets only.
- **Built-in storage in "Stored in".** A provider's built-in storage is
not offered once that provider has a usable vault. The form already
replaced that choice with the vault before this change.
- **Agent access at creation.** Access is a second request for each
agent after the secret is saved. If one fails, the secret exists without
that access and an error says so.
- **Automation contracts.** Labels that agents and the runner E2E
journeys depend on are kept: the key in each My secrets row, "Set
value", "Set my value", "Update my value", "Save value", and the import
dialog test IDs. The import entry point moved into the "More secret
actions" menu and the journey is updated for it.
- **Value masking** uses `-webkit-text-security`. A browser without it
shows the value as typed.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) through Claude Code, with tool
use, code execution and browser automation. Three delegated agents on
the same model restyled the import dialog, the Proposals tab and My
secrets.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 19:17:41 -06:00
DottaandClaude Fable 5.2 2de5d97958 fix(startup): report ready without waiting for backlog first passes (#21247)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A hosted instance upgrades by starting a new server beside the old
server. The deployer sends traffic to the new server when `GET
/api/ready` returns 200.
> - `/api/ready` returns 200 only after the boot sequence in
`server/src/index.ts` sets the phase to "ready".
> - The boot sequence awaited the first pass of many delivery queues.
Each pass grows with saved backlog or waits on outside services.
> - On a busy instance, these passes took more than the deploy's
health-check window. The deployer killed two new servers in a row before
they became ready, including a rollback to an image that had started in
86 s one hour before.
> - This pull request moves those first passes off the readiness path.
Readiness now waits only on named, bounded preconditions, and each step
reports its duration.
> - The benefit: an upgrade or rollback becomes ready in boot time, not
in backlog time. A slow start also names its slow step in one log line.

## Linked Issues or Issue Description

**What happened?**
On the hosted staging instance, two replacement servers did not report
`/api/ready` and were killed after about 9 minutes. The first was a new
image and the second was a rollback to the old image. At that time the
instance had about 200 queued runs, about 80 running runs, a slow
database, and many continuation runs. The failed server logged "managed
environments ensured from managed config" (`index.ts` line 1284 on
master). It never logged "worktree run-execution cutoff state" (line
1726), which the server always logs when the heartbeat service exists.
Between those two lines, boot awaited only these steps: `tool_reviews`
(four tool-review queue first passes), `gateway_sessions`,
`delivery_workers` (budget enforcement, cost accounting, fast responses,
chat completions, connection continuations, tool action receipts and
question responses), the GitHub connection event and continuity workers,
and `terminalWorkspaceWorker.ready`. Each step is a delivery-coordinator
`register(...).ready`, and that promise resolves only after the queue's
whole first pass. No queue except `mcp-client-mirror` (also failing on
the healthy server) logged an error, so the cause was slow passes and
not failures. The log had no per-step markers, so it does not show which
of these awaits was still running at the kill.

**Expected behavior**
Readiness means "this server can serve requests and take over". It must
not wait on work that grows with backlog or that depends on an outside
service. Queues own that work and retry it on their own schedule after
readiness.

**Steps to reproduce**
1. Start the server while the first pass of any of the queues above does
not finish. The new test in `server-startup-feedback-export.test.ts`
does this for every queue.
2. On master, `startServer()` does not resolve and the phase stays
"recovering".

## What Changed

- `server/src/index.ts`: These first passes now start in the background
as tracked scheduler work and do not hold readiness: tool reviews,
gateway sessions, delivery workers, GitHub connection workers, the
terminal workspace sweep, environment lease cleanup (scheduler-disabled
path) and scheduled database backups. Shutdown and idle sleep still wait
for them through `trackHeartbeatSchedulerWork`, and each queue keeps its
retry schedule.
- `server/src/index.ts`: The restart backup after an idle wake (a
`pg_dump` whose time grows with the database) holds idle eligibility
(`markIdleStartupComplete`), as before. It no longer holds readiness.
- `server/src/index.ts`: Every remaining await on the boot path is a
named `startupTimeline.precondition(...)`: `listen`,
`runtime_control_operations`, `execution_policy`,
`managed_environments`, `worktree_run_execution`,
`scheduling_suppression`, `native_recovery`, `external_adapters` and
`cloud_configuration`. These preconditions do not change.
- `server/src/startup-timeline.ts`: Adds `precondition(step, work)` and
`background(step, work)`. A precondition that is still running after 30
s is named in the warning "startup step is still holding readiness",
which repeats every 30 s. A background step logs a warning once at 30 s
and logs its duration when it finishes late. A background step that
fails is logged with its step name. The line "startup timeline at
readiness" now includes `readinessStepsMs`, `backgroundStepsMs` and
`backgroundStepsOutstanding`. All timers are single unref'd timeouts and
the module adds no queries.
- `server/src/__tests__/startup-readiness-boundary.test.ts`: This guard
test parses `index.ts` with `parseAst` from `vite`. The server already
depends on `vite`, so this PR adds no dependency and no lockfile change.
The test fails when an await on the boot path (between the phases
"recovering" and "ready") is not a named precondition, or when the
precondition names do not match a reviewed list that gives the reason
for each name.
- `server/src/services/terminal-workspace-work.ts`: Updates a comment.
The first sweep no longer holds readiness, and it still skips
compaction.

## Verification

- Red/green regression test: `server-startup-feedback-export.test.ts`,
"reports ready while backlog queues' first passes never finish". This
test makes every first pass above never finish. With master `index.ts`
it fails with `expected 'still starting' to be 'ready'` for both
scheduler settings. With this change it passes.
- Guard test: with master `index.ts`,
`startup-readiness-boundary.test.ts` fails and lists 18 unnamed awaits,
including `timeStartupRecovery("delivery_workers")` and
`terminalWorkspaceWorker.ready`. With this change it passes.
- `cd server && npx vitest run
src/__tests__/server-startup-feedback-export.test.ts
src/startup-timeline.test.ts
src/__tests__/startup-readiness-boundary.test.ts
src/startup-recovery-timing.test.ts
src/__tests__/terminal-workspace-work.test.ts`: all tests pass.
- `cd server && pnpm typecheck`: passes.
- `pnpm verify` exited with code 2 on the first revision (a lockfile
change that this PR no longer makes). Required CI supplies the broad
coverage.
- After deploy, run `grep "startup timeline at readiness"` on a server
start. The line shows each precondition's duration and the background
passes that are still running.

## Risks

- The listed first passes now run at the same time as native run
recovery and the requests that the server serves, not before them.
Requests were already served during recovery. Commit-woken passes of the
same queues, and the fresh queued-run dispatch, already ran at the same
time as native recovery. The coordinator still serializes the passes of
each queue.
- `/api/ready` can now return 200 while a queue still has backlog. That
is the intended meaning, and it matches the queued-run and recovery
passes that already run after readiness (#21014).
- Native run recovery still holds readiness, which is correct because
adoption must happen before takeover. If that recovery is slow, the
server now logs its step name every 30 s.
- Low risk otherwise: no schema or API change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, model ID `claude-opus-5-5` (commit
trailer: Claude Fable 5.2). Tool use and code execution were enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 19:16:23 -06:00
DottaandClaude Opus 5.5 b0233b5b33 fix(native): count a parent run's read of a finished child as delivery of its result (#21227)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task gets a wake when a child task completes. If the parent
is running, that wake waits behind the run.
> - At finalization, a parent's done claim stays open
(`native_child_completion_pending`) while such a wake is still owed, so
the parent never finishes without the child result.
> - The server had no record of what a running parent actually read. A
parent that read its finished children mid-turn and then finished the
task still kept the wake owed.
> - The task stayed `in_progress`, the wake started a second parent
turn, and the agent wrote a second completion summary.
> - This pull request records a run's read of a finished child and
counts that read as delivery of the child result.
> - The benefit is one completion turn when the parent already read its
children, while unread child results still get a fresh turn.

## Linked Issues or Issue Description

Refs #21189 (child results saved before the run started). Refs #21175
(the same pattern for comments: `task_history.read`). Refs #15372 (the
pending child-completion guard).

**What happened?**

Restart-chaos wave `w-20261112T223803-s7101`, on a master build that
includes #21189, task t006 (OpenCode, native runner, staging-mix parent
with two subtasks):

- Parent run `bfe9941c` started at 22:55:56.
- Its children CHA-11 and CHA-12 finished at 23:07:53 and 23:11:38.
- The child-completion intent and its deferred dispatch row were saved
at 23:11:38, after the run started. So the #21189 rule counted them as
new input.
- At 23:26:41 the run's reasoning says "Both subtask ledgers exist …
Both subtasks are done". The run then saved the CHILD hashes and posted
DONE at 23:33:39.
- At finalization, the done claim was committed as
`native_child_completion_pending`. The deferred wake then started run
`40928116`, which wrote a second completion summary.

Root cause: `hasPendingNativeChildCompletion()`
(`server/src/services/native-runtime/native-child-completion-delivery.ts`)
proved delivery only by wake binding and timestamps. No durable record
showed that the finishing run had read the completed child.

**Expected behavior**

A finishing run's recorded read of a completed child counts as that
child result being delivered. A read while the child was open, a read by
another run, or a failed event write does not count. An unread child
result still keeps the task open and gets one fresh parent turn.

**Steps to reproduce**

1. Start a native parent run.
2. Complete a child while the run is active. The child-completion wake
waits behind the run.
3. Let the run read the child (`get_task_context`, or `GET
/api/issues/:childId`), then claim `done`.
4. Finalize and release the run. On `master`, the decision is
`native_child_completion_pending` and the wake becomes a second run.

The same wave's t001 grade ("duplicate completion") is not this bug.
That run only said it *will* post the DONE marker later, and the grader
counts any mention. Its `subtasks_once` failure comes from the
restart-recovery continuation. This PR does not change either one.

## What Changed

- `native-child-completion-delivery.ts`:
- New `recordChildTaskRead()` writes a `child_task.read` run-log event.
The payload is `{ parentIssueId, children: { id: { status, updatedAt } }
}`, and it lists only children in `done`.
- `hasPendingNativeChildCompletion()` no longer counts a
child-completion wake as pending when the finishing run recorded a read
of that wake's completed child in `done` at or after the child's
completion decision (`status_decisions.created_at`, or the wake's
`createdAt` when the decision row is missing).
  - The #21189 start-time rule is unchanged.
- `paperclip-runner-tool-authority.ts`: `get_task_context` records the
finished child tasks it returns. A failed write is logged and leaves the
result owed.
- `routes/issues.ts`: `GET /issues/:id`, `GET /issues/:id/documents` and
`GET /issues/:id/documents/:key` record the read when the request has
all of these:
  - an agent actor with a run token
  - a child issue in `done`

This runs after the existing access and company checks. So a board user,
another company's agent, or a read of an open task writes nothing. A
qualifying read adds one statement: the existing
`appendHeartbeatRunEvent` fast path, a single `update … returning` /
`insert` CTE bound to the run's id, company and agent. Every other read
adds no query.
- `doc/run-log-events.md`: documents `child_task.read` and how delivery
uses it.
- Tests: a finalization-to-release regression in
`heartbeat-process-recovery.test.ts` and a new
`native-child-task-read-routes.test.ts`.

## Verification

- RED on `master`: in `heartbeat-process-recovery.test.ts` "counts a
child result as read by the finishing parent run (saved after the run
started, read: after the child finished)", the run reads the child
through the real `get_task_context`, then finalization and the
wake-queue release run.
- On master: `native_child_completion_pending`, task `in_progress`, and
the wake is promoted into a second run, the same as t006.
- With this change: `done`, the wake is cancelled, and there is one run.
- The other cases in the same test behave the same before and after this
change:
  - saved before the run started: done
  - not read: pending, fresh turn with the child results
  - read while the child was open: pending
- `native-child-task-read-routes.test.ts` (5 tests):
- An agent read of a done child through each of the three routes records
the read and clears the pending result.
  - An open child records nothing.
  - A board read records nothing.
  - Another company's agent gets no 200 and records nothing.
- Neighbouring suites after merging `origin/master`:
`heartbeat-process-recovery`, `muse-control`,
`heartbeat-held-comment-wake`, `native-status-arbiter-corpus`,
`heartbeat-dependency-scheduling`, `paperclip-runner-tool-authority`,
`agent-task-hierarchy-depth-routes`, and the two new test files: 8
files, 982/982 tests pass.
- `pnpm --filter @paperclipai/server typecheck` passes.

**Not verified**

- No chaos wave has run on this change yet.
- In t006, the parent's tool calls fell in the 4,220 run events that the
evidence omitted. Its reasoning text proves the read, but the exact tool
(`get_task_context`, `get_task`, `read_task_document` or `call_api`) is
not visible.
- A read through `search_tasks`, or through a shell command that calls
the API without the run token, is not recorded. Such a result stays owed
and gets one fresh turn, which is the old behavior.

## Risks

- A recorded read of a done child now lets the parent finish without
another turn. If the agent read the child but ignored it, no second turn
reminds it. A read while the child was still open never counts.
- Each qualifying agent read of a done child appends one run-log row.
Board reads and reads of open tasks write nothing.
- No migration. Wakes saved by older builds work unchanged. Without a
recorded read, the old behavior applies.
- This is a run-log change only (instance database). It adds no
Telemetry or OpenTelemetry export.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:07:31 -06:00
Devin FoleyandPaperclip ad320aeb71 fix(native): wait for a dated provider limit reset and keep it out of Sentry (#19775)
Keep native quota cooldowns tied to valid dated reset messages. Parse explicit years and strict month names while preserving already-expired dates. Keep typed native quota and rate-limit failures on their tasks in unexpected Sentry reporting mode. Preserve setup, process, malformed and all-mode reporting.

Verified current-head CI, Greptile 5/5 at Medium impact, local parser/report red-green tests and PostgreSQL integration coverage.

Co-Authored-By: Devin Foley <devin@devinfoley.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-12 17:06:22 -08:00
Nicky LeachandClaude Fable 5.2 2dcfdabc79 feat(runner): let a provider pack record the Runner image it was built in (#21228)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - A provider pack has a manifest, `provider-pack.json`. It records the
commit the pack was built from as `payload.runnerSourceRevision`
> - The next pull requests build the Runner alone as an image that has
an identity of its own: a digest of the Runner's inputs
> - A pack from that image must be able to say which image it came from.
The commit does not say that, because many commits have the same inputs
> - This pull request adds one optional field to the pack manifest for
that digest
> - The benefit is that a reader of a pack can find the exact image that
built it. Packs that are built today do not change

## Linked Issues or Issue Description

Refs #21204 (the Runner image, part 2 of this stack). No issue exists
for this change.

**Problem or motivation**

The provider pack manifest records a source revision. It has no place
for the identity of a Runner image. The Runner image in #21204 is named
by a digest of its inputs, and one image serves every commit that has
those inputs. Without a field in the manifest, a pack cannot name the
image that built it.

**Proposed solution**

Add the optional field `payload.runnerInputsDigest`. The pack builder
writes it only when the environment variable
`PAPERCLIP_RUNNER_INPUTS_DIGEST` is set. The value must be 64 lowercase
hexadecimal characters. The pack builder fails for another value.
Nothing sets the variable in this pull request.

**Alternatives considered**

- Put the digest in a file beside the pack. Then the pack digest does
not cover it, and a copy of the pack loses it.
- Reuse `runnerSourceRevision`. That field is a commit, and code
compares it with other commits.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 1 of 4. Each part is on top of the one before it:

1. `runner/pack-inputs-digest`: an optional field in the provider pack
manifest (this pull request)
2. `runner/runner-image-build` (#21204): build and check the Runner
image, with no publication
3. `runner/runner-image-layers`: layers by change rate, and the slim
variant
4. `runner/runner-image-publish`: publish both variants from `master`

## What Changed

- `packages/paperclip-runner/scripts/build-provider-pack.mjs`: reads
`PAPERCLIP_RUNNER_INPUTS_DIGEST`. When it is set, the script checks its
form and writes `payload.runnerInputsDigest` after
`runnerSourceRevision`. When it is not set, the manifest is the same as
before, byte for byte.
-
`server/src/services/native-runtime/remote-provider-pack-verification.ts`:
the manifest type has the optional field `runnerInputsDigest`. No check
reads it.
-
`server/src/services/native-runtime/remote-provider-pack-verification.test.ts`:
one test. A sealed manifest with the field passes verification. A
manifest whose field was changed after it was sealed fails, because the
pack digest covers the field.

The change is 20 added lines in 3 files.

## Verification

- `cd server && node node_modules/vitest/vitest.mjs run
src/services/native-runtime/remote-provider-pack-verification.test.ts`:
19 tests pass.
- Part 2 builds a pack with the variable set and a pack without it, on
`linux/arm64`. The first has the field and the second does not.

Never run:

- The full test suite and the full typecheck. CI must supply them.

## Risks

- Low risk. The field is optional and additive, and nothing sets the
variable yet.
- The pack digest covers the payload. A pack that has the field
therefore has another digest than the same pack without it. No pack has
the field until part 2 builds one.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 16:58:14 -08:00
DottaandClaude Opus 5.5 19f8ed1ea2 fix(recovery): continue only recent interruptions from the held backlog, newest first, paced (#21222)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a server restarts or a deploy swaps servers, agent runs are
interrupted. The fresh-continuation fallback (#21135) continues each
interrupted task in a new run from its saved work.
> - The fallback also has a backlog pass. It continues tasks that older
releases left held by a recovery record.
> - The backlog pass had no age limit and no pacing. It took the oldest
records first, 25 per pass.
> - On the first hosted release with the fallback, it continued 176
tasks in 30 minutes. 74 of them had a source run that ended more than 7
days ago. The oldest ended four weeks ago. More than 100 runs were
queued, and one agent had about 130 runs.
> - The user-facing goal of the fallback is "an interruption is
invisible". It is not "run last week's failures again".
> - This pull request limits the backlog pass to recent interruptions,
newest first, at a paced rate.
> - The benefit is that a release no longer starts a herd of old,
unwanted runs, and recent interrupted work still continues by itself.

## Linked Issues or Issue Description

Refs #21135

**What happened?**
A hosted instance took its first release with the fresh-continuation
fallback. Within 30 minutes, the backlog pass continued 176 held tasks.
By source run age: 26 under 1 day, 76 from 1 to 7 days, 74 over 7 days.
The source end codes included `adapter_failed` (27),
`runner_remote_provider_artifact_incompatible` (14),
`native_provider_terminal_failed` (14) and `setup_failed` (8). These are
configuration or provider failures, not interruptions. The pass kept
firing at 12 to 36 continuations per 5 minutes, and the queued backlog
grew past 100 runs.

**Expected behavior**
The backlog pass continues a task only when the task was recently
interrupted by a restart, a deploy, a lost lease, a lost process or a
refused recovery. Old records and configuration or provider failures
stay as they are, for a person's Retry. The pass starts new runs at a
controlled rate, newest first.

**Steps to reproduce**
1. Have held recovery records that older releases saved, with source
runs that ended days or weeks ago, some with `adapter_failed`.
2. Deploy a release that contains #21135.
3. See that the disposition worker continues all of them, oldest first,
25 per pass, and queues a run for each.

## What Changed

- `continueHeldExecutions` and `heldExecutionsAwaitingFreshContinuation`
(server/src/services/fresh-continuation-fallback.ts):
- Continue a held record only if its source run ended within
`BACKLOG_CONTINUATION_MAX_AGE_MS` (24 hours) AND its error code is in
`BACKLOG_CONTINUATION_SOURCE_CODES` (restart, deploy, lease,
process-loss and recovery-refusal codes).
- Do not touch any other held record. Do not settle it and do not modify
it. A log line counts these records, at most once every 10 minutes.
  - Order by the source run's end time, newest first.
- Continue at most 5 records per pass and 10 per rolling minute per
server.
- The rule I chose, and why: 24 hours and interruption codes only. A
person who is waiting for a task notices within a day. A run that failed
for configuration or provider reasons fails again the same way, so a
person must change something and press Retry. The 7-day rule of the
assigned-work sweep is too wide for this pass: 76 of the 176 continued
tasks were from 1 to 7 days old.
- The live path does not change. Fresh interruptions come through
restart recovery and the disposition pass (`recovery_refused`), and
through continuation wakes. They are not paced and have no age rule.
- Upgrade safety: records that a release already continued carry
`freshContinuation` and are excluded as before. Queued continuation runs
are not changed.
- The 10-per-minute limit is per server. During a deploy overlap, two
servers can each continue 10 per minute. This is acceptable for a short
overlap, and it keeps this change free of shared state.

## Verification

- `cd server && npx vitest run
src/services/fresh-continuation-fallback.test.ts`: 53 passed.
- New test "continues only recent interruptions from the held backlog,
newest first, and leaves the rest untouched (staging herd)". It uses
mixed-age and mixed-cause records: 1 h and 6 h restart codes, 3-day and
28-day restart codes, and 2 h
`runner_remote_provider_artifact_incompatible` and `adapter_failed`.
Only the two recent restart-caused records continue, the 1 h record
first. The four other records keep the same status, evidence and
`updated_at` across repeated passes.
- New test "paces the held backlog to ten continuations a minute per
server": 12 eligible records give 10 continuations in the first minute
and the remaining 2 in the next minute.
- RED on origin/master: both new tests fail (`expected [] to have a
length of 1`; `expected 12 to be 10`).
- The existing tests for the live path (for example "continues tasks
that older releases left blocked by these codes", with a source run that
ended one minute ago) still pass.
- `pnpm --filter @paperclipai/server run typecheck`: clean.

## Risks

- A held record from 1 to 7 days ago, caused by an interruption, no
longer continues by itself. It stays as it was before #21135: the task
shows its recovery state and a person can press Retry.
- A record whose cause is not in the list (a future interruption code)
is not continued by the backlog pass. The live path still continues
fresh interruptions for any code that is not a person's.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 18:36:22 -06:00
DottaandClaude Opus 5.5 334b2ae066 fix(recovery): continue a task blocked behind its workspace's ended writer once the hold releases (#21221)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote task workspaces have one writer run at a time, and an ended
writer keeps the workspace until its files are saved or set aside
> - When a task's next runs fail setup on that hold ("The workspace
still has an unfinished writer or save operation."), stranded-work
recovery blocks the task with a board escalation
> - The retained-file recovery from #21136 later released such a hold on
staging (it continued from the last save), but nothing settled the board
escalation, so the task stayed Blocked until a person pressed Retry
> - Recovery must work without a person: the block belonged to the hold,
and the hold is gone
> - This pull request adds a pass to the active recovery-action sweep
that settles that escalation once the workspace's writer hold is free
> - The benefit is that such tasks continue by themselves with one plain
notice and one run, with no Retry

## Linked Issues or Issue Description

Refs #21136
Refs #21143

**What happened?**
On staging, an ended run held its task's own remote workspace after its
superseded workspace export failed. The task's queued runs failed setup
with the unfinished-writer error, and stranded-work recovery blocked the
task with a board-owned escalation. After #21136 was deployed, its
retained-file recovery recorded the hold 77 seconds after server start
and, 19 minutes later, continued the task from its last save and
released the workspace. The task still stayed Blocked behind the board
escalation, and no run was queued. A board user had to press Retry.

**What did you expect to happen?**
Once the hold that caused the setup failures is released, the task
returns to `todo` and its assignee continues it, with no Retry.

**Steps to reproduce**
1. Give a task a remote workspace whose writer is an ended run with
unsaved files.
2. Let the task's next run fail setup with the unfinished-writer error,
and let stranded-work recovery block the task.
3. Release the hold (save the files, or continue from the last save).
4. Run the active recovery-action sweep. Before this change the task
stays Blocked with an active board escalation.

**Why the merged fix did not unblock the task**
The retained-file fallback only releases a Blocked status that its own
hold set (`holdOwnsBlockedStatus`). Here stranded-work recovery set the
block with a different, board-owned escalation, and a late status
decision 1.4 seconds later kept that block. So the fallback correctly
did not take the status, and no code settled the board escalation after
the hold was gone.

## What Changed

- Add `continueTasksAfterWorkspaceWriterReleased`
(`server/src/services/recovery/released-writer-escalation.ts`). It
selects board-owned `stranded_assigned_issue` escalations whose latest
run failed setup with the unfinished-writer error before any provider
work, by the current assignee. In one transaction it checks that the
block belongs to the escalation, that no newer run exists for the task,
that dependencies allow work, and that the workspace's writer hold is
now free. Then it resolves the escalation with a named outcome, returns
the task to `todo`, records one plain activity notice, and records a
pending wake.
- A later status decision that only kept the Blocked status it found
(from `blocked` to `blocked`) does not own the block. A decision that
chose the block still keeps the task blocked.
- Run the pass in `reconcileActiveRecoveryActions`, next to the legacy
rehome pass, and include its next wake retry time in the pass deadline.
- Generalize the legacy rehome wake delivery to
`deliverPendingRecoveryWakes`, keyed by the evidence key, so both passes
share one crash-safe, backed-off wake delivery. The rehome wake keeps
its fresh session. The new wake resumes normally.
- Keep the company selected by the scheduled sweep while a
released-writer wake is owed (`recoveryActionSweepCondition`).
- Document the rule in `doc/execution-semantics.md`.

## Verification

- `cd server && npx vitest run
src/__tests__/released-writer-escalation.test.ts
src/__tests__/legacy-workspace-rehome.test.ts
src/__tests__/issue-blocker-stall-wakeups.test.ts
src/__tests__/issue-blocker-stall-waiter-statement.test.ts
--maxWorkers=2`: all pass.
- Red/green: with the sweep hook removed, "continues the task once the
hold is released" fails (the task stays `blocked`). With the hook, it
passes.
- The new test seeds the staging row shape: a native writer run ended
with `native_finalization_superseded` still writing the task's own dirty
workspace, three legacy setup failures with the unfinished-writer error
and `{ kind: "bootstrap", providerWorkStarted: false }`, a board
escalation with `latestRunErrorCode: "setup_failed"`, the block 4.4
seconds after the last failure, and a later `blocked` to `blocked`
status decision.
- The tests also cover: no change while the writer still holds the
workspace, one wake only, a pending wake after a crash, a refused wake
that backs off, a later decision that chose the block, a dependency that
forbids work, a newer run, and another setup error.
- `cd server && npx tsc --noEmit -p tsconfig.json`: passes.

## Risks

- Low risk. The pass only acts on board escalations whose evidence is
exactly the unfinished-writer setup failure, and only when workspace
reservation would now admit a new writer. It never changes the
workspace, never applies a result, and never releases a writer fence.
- The task gets one normal wake through the issue-state guard (`todo`,
same assignee), so it cannot start twice.
- The pass adds one bounded query per sweep of a company with active
recovery actions.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only browser API reads of staging) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:58:06 -06:00
DottaandClaude Opus 5.5 b1ef1f3022 fix(recovery): settle a run's late exhaustion hold into its recorded fresh continuation (#21217)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run cannot be resumed, #21135 continues the task in one fresh
run from its saved work, with no person action
> - In Runner Full-Stack E2E 43321773136, two tasks whose run lost its
Runner three times recorded that continuation ("Unfinished run continues
in a fresh run from saved work", `delivery: recorded`), but no
continuation run started in the next 5 minutes
> - The native executor saves its exhaustion hold
(`native_session_retry_exhausted`, board-owned) in the same failure
path. When that hold lands after the decision, the decision is
idempotent per run and returns `already_continued`, so nothing settles
the new hold. The hold then blocks the continuation's delivery.
> - This pull request settles such later reconciliation holds of the
same run into the recorded continuation
> - The benefit is that the recorded continuation starts, and the task
does not wait on a person for an infrastructure fault

## Linked Issues or Issue Description

Refs #21135

**What happened?**
Runner Full-Stack E2E 43321773136, cases `recovery.explain-stall` and
`recovery.source-action` (OpenCode, native Runner): the server logged
the fresh-continuation decision at 22:51:30 and 22:51:55. Then the same
run logged "native result could not be marked workspace_failed" and
"heartbeat execution failed" (`native_session_retry_exhausted`). No
continuation run was created in the following 5 minutes. The ownership
pass kept reporting one `execution_reconciliation_hold` in progress.

**Expected behavior**
One fresh continuation starts from saved work after an infrastructure
loss, whatever order the failure path saves its records in.

**Steps to reproduce**
1. Record a fresh continuation for a failed native run.
2. Save an active `native_session_retry_exhausted` hold for the same run
afterwards.
3. Run the disposition and continuation delivery workers. Before this
change, no successor run is created.

## What Changed

- `server/src/services/fresh-continuation-fallback.ts`: when a
continuation is already recorded for the run, the decision settles the
run's other owned reconciliation holds (blocking, not a human gate) into
it: resolved, `restored`, system-owned, the person-facing text,
`coalescedIntoRecoveryActionId` and the primary's delivery state. A task
those holds blocked returns to `todo`. The continuation delivery worker
is notified.
- `server/src/services/fresh-continuation-fallback.test.ts`: regression
test for a late exhaustion hold.

## Verification

- RED before the change: `npx vitest run
src/services/fresh-continuation-fallback.test.ts -t "exhaustion hold
lands after"` failed with `expected [] to have a length of 1 but got
+0`.
- GREEN after: the full file passes (52/52).
`issue-recovery-actions.test.ts` and `execution-control-work.test.ts`
also pass (191 tests in the three files). Server `tsc --noEmit` is
clean.
- The E2E recovery cases in #21210 check the continuation starts and
finishes on the next cohort.

## Risks

- Low. Only holds of the same run that the fallback already owns
(reconciliation causes, blocking, no person gate) are settled, and only
after a continuation exists for that run. Human gates and workspace
restore holds are untouched.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:46:21 -06:00
DottaandClaude Opus 5.5 79334f58ab test(runtime): stop one slow plugin Runner release/resume case from failing the cases after it (#21213)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The plugin Runner release/resume integration test proves that a
released Runner can be resumed by the next run
> - The test failed on unrelated PR #21171 in the "Verify Paperclip
Runner (vitest native server)" lane
> - One slow case caused 5 more cases to fail, and the PR retry could
not run
> - #21199 fixed the same class of failure in the plugin Runner eviction
test
> - This pull request applies the same isolation to the release/resume
test, and also isolates its shared spies
> - The benefit is that one slow case no longer fails the cases after it

## Linked Issues or Issue Description

Refs #21199

**What happened?**

`server/src/services/native-runtime/plugin-runner-release-resume.integration.test.ts`
failed on unrelated PR #21171. "completes codex runs through a retained
cached process" timed out at 60s. Then 5 more cases failed, some at 60s
and some in under 3s with `RangeError: Maximum call stack size
exceeded`. Every retry failed on `duplicate key value violates unique
constraint "issues_identifier_idx"`.

Root cause: every case changed process-wide state and restored it in its
own `finally` block. That state was the default TLS trust,
`https.globalAgent`, a `vi.spyOn` on
`PaperclipRunnerToolAuthority.prototype.definitions`, and a `vi.spyOn`
on `inspectRemoteWarmSessionRecovery`. Vitest continues to the next case
when a case times out, but the timed-out function continues to run.
While it ran, the next case read the timed-out case's spy as the
"original" `definitions`. When the timed-out case restored its spies,
the two spies called each other, and the stack overflowed. Its late
`finally` also removed the next case's TLS trust, so that case stayed in
`connecting` with no wire frames until its own 60s timeout. It also
closed the next case's warm sessions with
`closeIdleWarmNativeSessionsForRestart()`. The retry could not help,
because the issue identifier came only from the case name.

**Expected behavior**
A case that times out fails alone. Later cases and the PR retry run
normally.

**Steps to reproduce**
1. Make the `codex cached` case sleep 62s once, after its first run.
2. From `server/`, run `npx vitest run
src/services/native-runtime/plugin-runner-release-resume.integration.test.ts
-t "codex runs through a retained (cached|adopted|adopted restart)
process"`.
3. Before this change, 3 of 3 cases fail, with `Maximum call stack size
exceeded`. With `--retry=1`, 3 of 3 still fail.
4. After this change, only the slow case fails. With `--retry=1`, 3 of 3
pass.

**Paperclip version or commit**
origin/master at the base of this branch (after #21199)

**Deployment mode**
CI and local test only (embedded Postgres, real `paperclip-runnerd`)

**Agent adapter(s) involved**
paperclip_runner (codex and claude ACPX harnesses through the plugin
task driver)

## What Changed

- Create one fixture certificate in `beforeAll`. Set the TLS trust and
`https.globalAgent` once for the file, and restore them in `afterAll`.
- Read the original `PaperclipRunnerToolAuthority.prototype.definitions`
once at module load, before any case spies on it.
- Move the per-case teardown into an `afterEach` that is registered
before the database fixture. The teardown closes idle warm sessions, the
host control, and the TLS proxy, and it restores both spies. Each case
binds its own teardown holder before its first await.
- Add a random suffix to the issue identifier of each attempt, so the PR
retry can run the case again.
- Increase the idle timeout of the adopted cases from 5s to 10s. The
product limits a release to `min(idleTimeoutMs, 30s)`, and #21199
measured a loaded release that took more than 4.8s.
- Probe the Runner port on `0.0.0.0`, the address that the Runner binds.

No product code changed.

## Verification

- Injection test (one case sleeps 62s): see the steps above. Before: 3
of 3 failed, also with `--retry=1`. After: 1 of 3 failed without retry,
and 3 of 3 passed with `--retry=1`.
- Before/after loops of the full file (14 cases), run in parallel on the
same loaded macOS host: see the loop result comment on this PR.
- `tsc --noEmit -p server/tsconfig.json` reports no errors in the
changed file.

## Risks

Low risk. Only the test changed. The adopted (non-restart) cases take
about 5s more each. The file creates one certificate instead of 14. A
case that hangs inside Core execution can still leave a busy warm
session in module state.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with
tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:19:06 -06:00
DottaandClaude Fable 5.2 161ef8121e perf(recovery): prepare the blocker-stall waiter read once instead of writing its SQL for every stalled task (#21201)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a task stalls on a recovery action that needs the board,
recovery wakes the agents of the tasks that wait on it: its parent and
the tasks it blocks (`wakeWaitersOfStalledIssues` in
`server/src/services/recovery/blocker-stall-wakeups.ts`).
> - Each pass reads the waiters of every stalled task with one statement
for each stall. The statement has 18 parameters and two correlated
subqueries. Its SQL text is 0.8 KB, and it is the same for every stall
of the same shape.
> - The server wrote that text with the query builder for every stall on
every pass. On a hosted instance the server main thread was near one
full core and garbage collection used a quarter of it. A statistics
sample showed this statement at about 7.8 executions per second.
> - I measured it on a local PostgreSQL through the real client. The
query builder allocates 103 KB to write and run this statement. The same
statement from a prepared query allocates 15 KB.
> - This pull request writes the SQL text of the waiter read one time
for each client and shape. Each execution then only binds values.
> - The benefit is 76% less allocation and 5 times fewer garbage
collections for a pass over 20 stalled tasks. The statement, its
parameters and its rows do not change.

## Linked Issues or Issue Description

Related: #21195 (the same change for the native chat progress page read;
it added the `preparedStatements` helper that this pull request uses),
#21131 and #18965. No issue exists for this read.

**What existing behavior does this improve?**

The cost, in the server process, of the waiter read that the
blocker-stall wake pass runs for each stalled task.

**Subsystem affected**

Recovery: blocker-stall wakes
(`server/src/services/recovery/blocker-stall-wakeups.ts`).

**Current behavior**

`wakeWaitersOfStalledIssues` builds the waiter query with the Drizzle
query builder inside its loop, and Drizzle then writes the SQL text from
it, one time for each stall and page. Measured on the merge base, Node
26, local PostgreSQL 18, median of 5 passes of 300 calls. The fixture
has 20 stalled tasks, each with a parent and one dependent task that are
already woken, so a pass writes nothing:

| Call | Statements | Waiter SQL texts written | Allocated | Process CPU
| Collections per 1,000 calls (pause time) |
| --- | --- | --- | --- | --- | --- |
| Company pass over 20 stalls | 21 | 20 | 2,298.3 KB | 3,855 µs | 100
(163.7 ms) |
| Pass for one stalled task | 4 | 1 | 355.0 KB | 547 µs | 13.3 (14.3 ms)
|

At the sampled rate of 7.8 executions per second the read allocates
about 0.8 MB per second. This rate is one sample of one instance, taken
soon after a long outage, when recovery had a backlog. It is not a
steady-state figure.

**Proposed behavior**

The query moves into one builder function, `stallWaiterPageQuery`. The
module prepares it one time for each client and shape, with placeholders
for the values. The shapes are: the stalled task has a parent or not;
all waiters or one waiter; first page or later page. The loop executes
the prepared statement.

**Reason and benefit**

Same benchmark, this branch:

| Call | Statements | Waiter SQL texts written | Allocated | Process CPU
| Collections per 1,000 calls (pause time) |
| --- | --- | --- | --- | --- | --- |
| Company pass over 20 stalls | 21 | 0 | 547.3 KB | 1,474 µs | 20 (24.8
ms) |
| Pass for one stalled task | 4 | 0 | 266.9 KB | 496 µs | 10 (9.9 ms) |

One waiter read costs about 88 KB less. At the sampled rate this removes
about 0.7 MB per second of allocation, which is about 0.9% of the 77.7
MB per second that the instance allocated. It is a small part of the
total. The CPU numbers come from a machine with other load and are less
stable than the allocation numbers.

**Breaking changes**

None.

## What Changed

- `server/src/services/recovery/blocker-stall-wakeups.ts`: the waiter
query moves out of the loop into `stallWaiterPageQuery(db, binding)`.
`stallWaiterPageStatement` returns the prepared statement for the shape
and the values to bind. The loop executes it. The lines inside the two
SQL fragments keep their indentation, so the SQL text is byte-identical.
The pull request adds no comment. The one existing comment of this read
stays at the call site, where the code still makes that choice.
- `server/src/__tests__/issue-blocker-stall-waiter-statement.test.ts`
(new): statement equivalence, row equivalence on PostgreSQL, and the
red/green mechanism test.
- `server/src/__tests__/helpers/blocker-stall-fixture.ts` (new): the
fixture that the test and the benchmark share.
- `server/src/__tests__/prepared-statements-driver.test.ts` (new):
proves at the driver that a prepared read and the query builder ask
postgres.js for the same thing, with the driver's prepared-statement
setting unset, false and true. It covers this read and the native chat
progress page read of #21195.
- `server/scripts/benchmark-blocker-stall-waiters.ts` (new, no
comments): the benchmark above. It loads the module at `--baseline-ref`
next to the current module in one process. `sweep` is one company pass
over `STALLS` stalled tasks whose waiters are already woken: one stall
read, then one waiter read for each stall. `task` is the pass for one
stalled task. Before it measures, the script also runs a company with a
paused agent and 101 waiters on a stall with a parent and on a stall
without one, and one pass for a single waiter of each. This covers six
shapes: the first page, the later page and one waiter, each for a stall
with a parent and for a stall without one. The script fails unless both
versions send the same statements, byte for byte, with the same
parameters and return the same result. Passes alternate between the
versions with a full collection before each pass. CPU is the whole
process. Allocated bytes are V8's `total_allocated_bytes`. Optional
environment variables: `STALLS` (default 20), `ITERATIONS` (default 300
calls for each pass), `PASSES` (default 5 for each version).

Not changed: which statements run, their order, the other reads of the
pass (they are small and vary by waiter), the wake itself, the driver's
prepared-statement setting, and the schema. The read takes no lock and
runs on the root client outside a transaction. Its two lists (waiter
statuses and wake statuses) are module constants, so the statement has
no variable-length list parameter.

## Verification

- Statement equivalence (`sends the SQL text and the parameters that the
query builder writes, for every shape of the read`): for all eight
shapes the prepared SQL text equals the text that the builder writes
with the values, and the bound parameters are equal. The test also
proves that a stall without a parent binds `null` for the parent.
- Against the merge base at the driver: the benchmark compares every
statement that the merge base module and this branch send for six of the
eight waiter-read shapes. The texts are byte-identical and the
parameters are equal. The two shapes that it does not reach (one waiter
on a later page) cannot occur: a one-waiter read returns at most one
row, so it has no later page.
- Row equivalence on PostgreSQL (`returns the rows of the query builder:
same values, types and order, and no other company's rows`): two
companies. The fixture has a stall with a parent and six dependents, and
a stall without a parent. One dependent is already woken, one is done,
one is hidden and one has no assignee. One has a failed wake only, and
one has a wake row with the same key in the other company; both stay
waiters. The prepared statement and the builder return strictly equal
rows for all waiters, one waiter, a later page, and a stall read under
the other company's id (no rows). Null columns stay null.
- Mechanism, red then green (`writes the SQL text of the waiter read one
time and then only binds values`): a pass over three stalls writes no
waiter SQL text and sends one identical text three times. On the merge
base module the same test fails: the pass writes three texts.
- Driver behavior (`asks the driver for the same thing as the query
builder with DATABASE_PREPARED_STATEMENTS=...`, three cases): Drizzle's
`.prepare(name)` on postgres.js does not send the name to the database.
`PostgresJsSession.prepareQuery` drops it
(`drizzle-orm/postgres-js/session.js`, lines 87 to 100), and
`PostgresJsPreparedQuery.execute` calls `client.unsafe(text, params)`
(lines 33 and 43). An ordinary query reaches the same call through
`_prepare().execute()` (`pg-core/query-builders/select.js`, lines 795 to
799). The test runs both reads through the builder and through the
prepared statement on one connection for each setting. The arguments and
the options of the driver call are strictly equal. With the setting
unset or false, the server has no named statement after either path, so
a transaction-mode pooler sees no change. With the setting true, the
server has one named statement. Its name is the one that postgres.js
generates for the connection, and both paths use that same statement.
- Existing tests: `issue-blocker-stall-wakeups.test.ts` and
`legacy-workspace-rehome.test.ts` passed (30 tests) with the new file (3
tests). `prepared-statements-driver.test.ts` passed (3 tests).
- `pnpm --filter @paperclipai/server typecheck` passed. `pnpm
check:module-boundaries` passed.
- `pnpm verify` returns exit 2 (broader verification required, because
the change adds a file under `server/scripts`). I did not run the full
local suite. CI supplies that coverage.

Reproduce:

```sh
pnpm --filter @paperclipai/server exec vitest run src/__tests__/issue-blocker-stall-waiter-statement.test.ts \
  src/__tests__/issue-blocker-stall-wakeups.test.ts src/__tests__/legacy-workspace-rehome.test.ts \
  src/__tests__/prepared-statements-driver.test.ts
TSX_DISABLE_CACHE=1 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-blocker-stall-waiters.ts \
  --baseline-ref=549816e6a0
```

Not verified: the effect on a hosted instance. The rate above is one
sample. The local numbers are from Node 26; the hosted instance runs
Node 24.

## Risks

- Low. The statement text and parameters are equal by test, the read
takes no lock, and it runs on the root client outside a transaction.
- A prepared statement keeps the text of the first build. The lists
inside the statement are module constants, so the text cannot go stale
in a running process.
- A test that mocks `db.select` to observe this read no longer sees it.
I found none for this read.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model `claude-fable-5-2` (Fable 5.2), in Claude
Code with tool use and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 16:32:29 -06:00
DottaandClaude Opus 5.5 6ac7211cb9 fix(heartbeat): stop re-delivering a remote-stop follow-up that a succeeded run already read (#21203)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person's follow-up comment must reach the agent once
> - #21175 made a succeeded run's mid-turn read of a comment (through
`get_task_history`) count as delivery for the queued-comment paths
> - The remote-Stop resume path decides delivery with
`undeliveredLegacyUserCommentIds`, which ignores those reads
> - So a follow-up that a succeeded run already read was sent again, and
the agent posted a second DONE on a finished task
> - This pull request applies the same rule on that path and adds a read
helper for a later change
> - The benefit is one answer per follow-up on every resume path, with
no dropped messages

## Linked Issues or Issue Description

Refs #21175

**What happened?**
A saved follow-up comment was resumed after a remote Stop (wake reason
`issue_commented`, idempotency key `remote-stop-comment:…`). A succeeded
run had already read this comment mid-turn. The resumed wake started a
new turn, and the agent posted a second DONE on a finished task. The
cause: `undeliveredLegacyUserCommentIds`
(explicit-native-continuation.ts) checks only wake input and accepted
first turns. It does not check the `task_history.read` records that
#21175 added.

**Expected behavior**
A follow-up that a succeeded run read at its current revision is
delivered. The remote-Stop resume path skips it. A read by a run that
did not succeed does not count, so that follow-up is still delivered
once.

**Steps to reproduce**
1. A native run reads the task history after a person posts a follow-up.
2. The run succeeds. The follow-up stays saved as a deferred wake.
3. The remote-Stop resume path runs for that wake.
4. Before this change, the follow-up is delivered again.

**Paperclip version or commit**
`master` at 549816e6a0.

## What Changed

- `server/src/services/queued-comment-delivery.ts`:
`historyReadDeliveredCommentIds` exposes the #21175 rule for a list of
comment IDs. `taskHistoryReadsForRun({ companyId, runId })` returns each
comment one run read, with the read time and the comment revision that
the read returned. It returns these whatever the run's outcome. It reads
only that run's events through `heartbeat_run_events_company_run_idx`.
No delivery decision uses it.
- `server/src/services/explicit-native-continuation.ts`:
`undeliveredLegacyUserCommentIds` also removes comments that a succeeded
run read at their current revision. Its two callers are the remote-Stop
resume path and the queued explicit-continuation check, and both now use
the #21175 rule.
- Deliberate asymmetry: a read by a run that did not succeed never
counts. A dropped person's message is worse than a duplicate answer.
- `doc/run-log-events.md`: documents both points.
- New test file
`server/src/services/queued-comment-history-read.test.ts`.

## Verification

- RED on `549816e6a0` (`npx vitest run
src/services/queued-comment-history-read.test.ts`): 2 failed, 1 passed.
  - The follow-up that a succeeded run read was still returned as owed.
  - `taskHistoryReadsForRun` did not exist.
- The case where a failed run read the follow-up (still owed) passed, as
intended.
- GREEN: 3/3 pass.
- Neighbouring suites (`first-turn-queued-input`,
`explicit-native-continuation`, `heartbeat-agent-directory-busy`,
`heartbeat-held-comment-wake`, `execution-control-work`): 517/517 pass.
Server `tsc` is clean.

## Risks

- Low risk. The function and its callers keep their contract. It returns
fewer comment IDs only when a succeeded run of the assignee read that
exact revision.
- Comments read through the REST API by non-native adapters are not
recorded, so their behavior does not change.
- No migration. The added lookup uses the time-bounded, index-backed
query from #21175.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:23:34 -06:00
DottaandClaude Opus 5.5 57aeda190b test(runtime): stop one slow plugin Runner eviction case from failing the cases after it (#21199)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The plugin Runner eviction integration test proves that a retained
Runner continues after the host evicts it
> - The test failed on many unrelated pull requests in the "Verify
Paperclip Runner (vitest native server)" lane
> - One slow case caused other cases to fail, and the PR retry could not
run
> - This pull request isolates each case from a case that timed out, and
gives the retry a fresh issue
> - The benefit is that one slow case no longer fails the cases after it

## Linked Issues or Issue Description

**What happened?**

`server/src/services/native-runtime/plugin-runner-eviction.integration.test.ts`
failed in CI on at least 5 unrelated branches in 48 hours, most recently
on #21149. The errors were `Test timed out in 60000ms` and `Timed out
waiting for automatic release A`. Each retry then failed at once with
`duplicate key value violates unique constraint
"issues_identifier_idx"`. The failed cases logged a Runner in
`connecting` with no wire frames.

Root cause: every case changed process-wide state. It set the default
TLS trust and `https.globalAgent` for its own certificate, and it
restored them in its own `finally` block. Vitest continues to the next
case when a case times out, but the timed-out function continues to run.
Its late `finally` then removed the TLS trust of the next case and
closed the warm sessions of the next case with
`closeIdleWarmNativeSessionsForRestart()`. The next case could not open
its Runner connection, waited for the 60s startup deadline, and also
timed out. That case then did the same to the case after it. In CI run
43266976387, one timeout caused 3 more failures. Locally, one timeout
caused 5 more failures.

Two more problems made this worse:
- The issue identifier came only from the case name. A retry of the same
case therefore failed on the unique identifier, so the PR retry never
ran the test again.
- Detached cases used a 5s idle timeout. The product limits a release to
`min(idleTimeoutMs, 30s)`, so these cases had 5s for the drain and the
park. Under load, one park took more than 4.8s, and the release expired.

**Expected behavior**
A case that times out fails alone. Later cases and the PR retry run
normally.

**Steps to reproduce**
1. Make one case slower than 60s. For example, add a 62s sleep after
release A in `codex cached retired`.
2. Run `npx vitest run
src/services/native-runtime/plugin-runner-eviction.integration.test.ts
-t "codex with a (cached|detached) Core owner"` from `server/`.
3. Before this change, 3 of 4 cases fail: the slow case, and 2 cases
that it disturbs. With `--retry=1`, the slow case also fails on the
duplicate identifier.
4. After this change, only the slow case fails. With `--retry=1`, all 4
cases pass.

**Paperclip version or commit**
master at ef6d347f79, rebased on the current origin/master

**Deployment mode**
CI and local test only (embedded Postgres, real `paperclip-runnerd`)

**Agent adapter(s) involved**
paperclip_runner (codex and claude ACPX harnesses through the plugin
task driver)

## What Changed

- Create one fixture certificate in `beforeAll`. Set the TLS trust and
`https.globalAgent` once for the file, and restore them in `afterAll`. A
case no longer changes process-wide trust.
- Move the per-case teardown (close idle warm sessions, close the host
controls, close the TLS proxy) into an `afterEach`. It is registered
before the database fixture, so it runs before the database clients
close, also when a case times out.
- Add a random suffix to the issue identifier of each attempt, so the PR
retry can run the case again.
- Increase the detached idle timeout from 5s to 10s, and wait 200ms past
it as before. This gives the release the time that it needs under load.
- Probe the Runner port on `0.0.0.0`, the address that the Runner binds.

No product code changed.

## Verification

All loops ran on a shared, loaded macOS host (load average 15 to 20 on
18 cores), plus 2 CPU busy loops. Each iteration ran the 12 cases that
match `with a (after inspection|missing state|reused lease) Core owner`.

- Before: 20 iterations, 2 failed (10%). In iteration 10, `claude reused
lease retired` timed out at 60s, and 5 later cases failed on
`readNativeSessionIdleWork() === "none"` because of the case that timed
out. In iteration 17, `claude after inspection retired` failed with
`Timed out waiting for automatic release A`.
- After: see the loop result comment on this PR.
- Injection test (one case sleeps 62s): before, 3 of 4 cases failed, and
with `--retry=1` the retry failed on the duplicate identifier. After,
only the slow case failed, and with `--retry=1` all 4 passed.
- Full file after the change: 40 of 40 passed in 320s locally (335s
before, because the file now creates one certificate instead of 40).
- `tsc --noEmit -p server/tsconfig.json` reports no errors in the
changed file.

## Risks

Low risk. Only the test changed. The detached cases take about 5s more
each (12 cases, about 60s in all), which is partly offset by creating
one certificate per file. A case that hangs inside Core execution can
still leave a busy warm session in module state. This change does not
prevent that. It removes the causes of the cascade that this PR found.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with
tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:18:01 -06:00
DottaandClaude Opus 5.5 7f1cdb842d fix(evals): admit the owner-routed routine follow-up run and brief only the requested change (#21198)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat eval catalog checks that agents in chat do the right thing
on saved product state.
> - In `routines.manage-other-owner`, the CEO cannot change Morgan's
routine. The valid outcome is to route the change to Morgan in one
linked task.
> - That valid path uses three runs: the CEO's turn, Morgan's run on the
task, and the CEO's automatic completion follow-up on the chat. The case
allowed only two runs, so the harness stopped valid attempts with
"exceeded its declared run bound".
> - In one of those attempts, the CEO also told Morgan to use UTC.
Nobody asked for UTC. The routine was on America/Chicago. The oracle
correctly failed that change.
> - This pull request admits the third run only in its declared role,
and it tells agents to brief only the change the person asked for.
> - The benefit is that the eval accepts the valid delegation path and
still fails extra work and unrequested setting changes.

## Linked Issues or Issue Description

**What happened?**

Runner Full-Stack E2E runs 43298807862 and 43300040122 stopped
`routines.manage-other-owner` with "Scenario exceeded its declared run
bound". Both had three runs: the CEO turn, Morgan's run on the routed
task, and the CEO's `chat_task_completed` follow-up. In 43298807862 the
CEO's brief also told Morgan to switch the routine to UTC, and
`same-owned-definition-and-triggers` failed.

**Expected behavior**

The delegation path with exactly one routed owner task passes with three
runs. The refusal path stays at two runs. An agent that relays a change
keeps every setting the person did not ask to change.

**Steps to reproduce**

Run `chat-features.runner-opencode.local.routines.manage-other-owner`
and let the CEO route the change to Morgan.

**Paperclip version or commit**

master at 7747487ee7.

## What Changed

- `tests/runner-e2e/chat-catalog-domains/integrator-routines.ts`: the
case bound is now 3 runs (minimum 1). A new `declared-run-roles` check
allows at most 2 runs, plus at most one run that is the CEO's completion
follow-up (`chatCompletionFollowUp` from `run-roles.ts`) on the source
chat for the single owner task with a bound Morgan run. The observation
now records the source chat id. Oracle version is now
`manage-other-owner.v7`.
- `tests/runner-e2e/chat-catalog-descriptors.json`: the static
descriptor matches the new bound and oracle version.
- `tests/runner-e2e/chat-catalog-domains/integrator-routines.test.ts`:
calibration tests. The delegation path with 3 runs passes. A 4th run
fails. A second follow-up fails. 3 runs without a delegated task fail. A
follow-up for a different task fails. The 2-run refusal path still
passes.
- `server/src/services/agent-conversations.ts`: one sentence in the chat
directive task guidance: "When you relay or delegate a change, state
only what the person asked to change, and keep every other setting, such
as a time zone, owner or schedule detail, as it is."
- `server/src/__tests__/agent-conversations.test.ts`: red/green test
that the directive and the built chat task prompt carry that sentence.
- No scenario yaml change, so the TestOps manifest does not change.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog
tests/runner-e2e/chat-catalog-domains/integrator`: 283 files, 3850 tests
pass.
- `pnpm test:e2e:runner:typecheck`: pass.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
pass.
- `server`: `agent-conversations.test.ts` and
`board-access-denial.test.ts` pass (109 tests);
`codex-local-execute.test.ts` passes. The new directive test failed
before the directive change and passes after it.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs
792f8fb308 --check`: `changed: false`.
- `pnpm verify` exited 2 (descriptor JSON is a global CI input), so
required CI supplies broad coverage.
- Paid single-case run: Runner Full-Stack E2E 43313272302 on a throwaway
branch at this commit. Result is reported in a comment.

## Risks

- Low risk. The eval change only admits a third run in one declared
role. Extra or unbound runs still fail.
- The directive gains one sentence. It can make an agent more
conservative when it relays a change, which is the intent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:10:47 -06:00
DottaandClaude Fable 5.2 549816e6a0 perf(chat): prepare the native progress page read once instead of writing its SQL on every wake (#21195)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run that came from a chat reports safe progress to that
chat. Each progress event wakes a queue, and each wake reads one page of
candidates (`nativeProgressPages` in
`server/src/services/chat-native-progress.ts`).
> - The page read joins six tables and has 44 parameters. Its SQL text
is 3.6 KB. The server wrote that text with the query builder on every
wake, and the text is the same for every wake of the same filter.
> - On a hosted instance the server main thread was near one full core
and garbage collection used a quarter of it. An allocation sample put
46% of all allocation under the query builder's SQL writing. The sample
does not name statements.
> - I measured single statements on a local PostgreSQL through the real
client. The query builder allocates about 20 KB for the smallest
statement, plus about 2 KB for each selected column and about 3 KB for
each predicate. This page read costs 296 KB and 0.34 ms of CPU to write.
The same statement from a prepared query costs 21 KB.
> - This pull request writes the SQL text of the page read one time for
each client and filter shape. Each execution then only binds values.
> - The benefit is 91% less allocation and 14 times fewer garbage
collections for one page read. The statement, its parameters and its
rows do not change.

## Linked Issues or Issue Description

Related: #21131 (fixed SQL text for the native run event append) and
#18965 (an earlier prepared read, in the GitHub review wait). No issue
exists for this read.

**What existing behavior does this improve?**

The cost, in the server process, of the candidate page read that every
native chat progress wake runs.

**Subsystem affected**

Chat channels: native run progress publication
(`server/src/services/chat-native-progress.ts`).

**Current behavior**

`nativeProgressPages` builds the complete query with the Drizzle query
builder for each page, and Drizzle then writes the SQL text from it.
Measured on the merge base, Node 26, local PostgreSQL 18, median of 5
passes of 2,000 calls:

| Call | SQL texts written | Allocated | Process CPU | Collections per
1,000 calls (pause time) |
| --- | --- | --- | --- | --- |
| One page read | 1 | 297.3 KB | 851 µs | 14.5 (30.7 ms) |
| One wake for a task (page read + one transaction) | 7 | 681.3 KB |
2,238 µs | 34 (76.6 ms) |

A statistics sample of one hosted instance on one morning showed this
statement at about 3.9 executions per second. At that rate the read
allocates about 1.1 MB per second for SQL text. This rate is one sample,
taken soon after a long outage. It is not a steady-state figure.

**Proposed behavior**

The query stays in one builder function, `nativeProgressPageQuery`. The
module prepares it one time for each client and each of the six filter
shapes (all tasks, one company, one task; first page or later page),
with placeholders for the values. Each page read executes the prepared
statement.

**Reason and benefit**

Same benchmark, this branch:

| Call | SQL texts written | Allocated | Process CPU | Collections per
1,000 calls (pause time) |
| --- | --- | --- | --- | --- |
| One page read | 0 | 26.8 KB | 219 µs | 1 (1.4 ms) |
| One wake for a task | 6 | 412.1 KB | 1,699 µs | 21 (55.5 ms) |

At the sampled rate this removes about 1.05 MB per second of allocation,
which is about 1.4% of the 77.7 MB per second that the instance
allocated. It is a small part of the total. The remaining six statements
of a wake run inside a transaction, where a prepared query gains
nothing, and this pull request does not change them.

**Breaking changes**

None.

## What Changed

- `server/src/services/chat-native-progress.ts`: the page query moves
into `nativeProgressPageQuery(db, binding)` without a change to its
text. `nativeProgressPageStatement` returns the prepared statement for
the filter shape and the values to bind. `nativeProgressPages` executes
it.
- `server/src/services/prepared-statements.ts` (new, 17 lines, no
comments):
- `preparedStatements(prepare)` keeps one prepared statement for each
client and shape. A prepared statement runs on the client that prepared
it, so the cache key is the client object. A transaction client gets its
own statement and gains nothing.
- `columnPlaceholder(name, column)` makes a placeholder whose value the
column encodes. A bare `sql.placeholder` skips the column's encoding, so
a `Date` would reach the driver unencoded. With this helper it reaches
the driver as the same ISO string as a value that the query builder
binds.
- `server/src/__tests__/chat-native-progress-statement.test.ts` (new):
statement equivalence, row equivalence on PostgreSQL, and the red/green
mechanism test.
- `server/src/__tests__/prepared-statements.test.ts` (new): the helper.
- `server/src/__tests__/chat-native-progress-work.test.ts` and
`chat-channels.integration.test.ts`: three tests watched or failed the
page read through `db.select`. A prepared statement does not call
`db.select`, so these tests now watch the driver (`db.$client.unsafe`).
The idle assertions are stricter: they now fail on any statement.
- `server/scripts/benchmark-native-progress-page.ts` (new, no comments):
the benchmark above. It loads the module at `--baseline-ref` next to the
current module in one process. `page` is `loadNativeChatProgressScopes`
for one task, which sends the page read and nothing else. `wake` is
`reconcileNativeChatProgress` for one task whose progress is already
published, so it writes nothing and every pass does the same work. The
script fails unless both versions send the same statements with the same
parameters and return the same result. Passes alternate between the
versions with a full collection before each pass. CPU is the whole
process (`process.cpuUsage`). Allocated bytes are V8's
`total_allocated_bytes`. A collection counts when it starts inside a
pass. Optional environment variables: `ITERATIONS` (default 2000 calls
for each pass) and `PASSES` (default 5 for each version).
`TSX_DISABLE_CACHE=1` keeps the tsx cache scan out of the first passes.

Not changed: which statements run, their order, the transaction of a
wake, its `FOR UPDATE SKIP LOCKED` reads, the driver's
prepared-statement setting, and the schema. The read has no
variable-length list parameter.

## Verification

- Statement equivalence (`sends the SQL text and the parameters that the
query builder writes, for every shape of the read`): for all six shapes,
and for an empty task id, the prepared SQL text equals the text the
builder writes with the values, and the bound parameters are equal. The
test also proves that the time filter is bound as the ISO string.
- One-time check against the merge base: I derived the builder of the
merge base from its source and compared 24 inputs (4 scopes, with and
without a cursor, 3 page sizes). The prepared SQL text is
byte-identical, and the parameters are strictly equal.
- Row equivalence on PostgreSQL (`returns the rows of the query builder:
same values, types and order, and no other company's rows`): two
companies, four candidate runs, and three runs that are not candidates.
The fixture has a null payload, a nested JSON payload, a run bound
through its task column only, and microsecond timestamps. The prepared
statement and the builder return strictly equal rows for every filter,
for pages of one row walked with the cursor, and for a task of another
company (no rows).
- Mechanism, red then green (`writes the SQL text of a read one time and
then only binds values`): three reads for different tasks write no SQL
text and send one identical text three times. On the merge base module
the same test fails with "expected sqlToQuery to not be called at all,
but actually been called 3 times".
- The benchmark asserts that both versions send the same statements with
the same parameters and return the same result, for the page read and
for a full wake.
- Existing tests: `chat-native-progress-work.test.ts` (19 passed),
`chat-channels.integration.test.ts` (1,954 of 1,963 passed in the
complete local run, including the 49 tests that match `native
progress|saved progress|safe native`. Two tests fail on this machine
because the worktree is inside another checkout: the bundled Slack skill
resolves to that checkout's skills directory, which has no
`slack/SKILL.md` (`ENOENT` in `connector-runtime.ts`). Seven later tests
failed in the same run from the state that those two left. The seven
pass when I run them by name. CI is the evidence for these nine),
`app-private-hostname-gate.test.ts` (passed).
- `pnpm --filter @paperclipai/server typecheck` passed. `pnpm
check:module-boundaries` passed.
- `pnpm verify` returned exit 2 (broader verification required, because
the change adds a file under `server/scripts`). I did not run the full
local suite. CI supplies that coverage.

Reproduce:

```sh
pnpm --filter @paperclipai/server exec vitest run src/__tests__/chat-native-progress-statement.test.ts \
  src/__tests__/prepared-statements.test.ts src/__tests__/chat-native-progress-work.test.ts
TSX_DISABLE_CACHE=1 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-progress-page.ts \
  --baseline-ref=267983356c
```

Not verified: the effect on a hosted instance. The rate above is one
sample. The local numbers are from Node 26; the hosted instance runs
Node 24.

## Risks

- Low. The statement text and parameters are equal by test, and the read
runs on the root client outside a transaction.
- A prepared statement keeps the text of the first build. The lists
inside the statement (safe event types, conversation states, publication
states) are module constants, so the text cannot go stale in a running
process.
- The helper keys statements by client object. A caller that passes a
transaction client gets a statement for that transaction only. This is
correct, and it costs the same as before.
- Tests that mock `db.select` to observe this read no longer see it. I
found three and changed them to watch the driver.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model `claude-fable-5-2` (Fable 5.2), in Claude
Code with tool use and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 15:56:28 -06:00
DottaandClaude Opus 5.5 7747487ee7 fix(budgets): retry contended accounting writes with jittered backoff (#21185)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Budget policies and usage records are written in serialized
accounting transactions for each company
> - When several writes for one company run at the same time, PostgreSQL
rejects all but one and Paperclip retries the others
> - The retries were few and short (4 attempts, 10 to 30 ms, the same
delay for every writer), so concurrent writers collided again and the
request failed with 503
> - The 503 message also said "this run's budget" for a policy write
that has no run
> - This pull request retries contended accounting writes with jittered
exponential backoff and gives the error a correct message
> - The benefit is that normal concurrent budget changes succeed, and a
real failure tells the user what happened

## Linked Issues or Issue Description

**What happened?**
Several budget policy writes for one company at the same time returned
503 after 4 serialization retries. The same retry exhaustion appeared
often in server logs of the chat feature evals. The message said
"Paperclip could not save this run's budget or usage records … Retry the
run" for a policy write.

**Expected behavior**
Concurrent budget policy writes succeed. If contention continues, the
error describes budget or usage changes, not a run.

**Steps to reproduce**
Send 9 budget policy upserts for one company at the same time. Before
this change, about half of them fail with 503.

**Paperclip version or commit**
master

## What Changed

- `withOperationAccountingTransaction` retries up to
`ACCOUNTING_TRANSACTION_ATTEMPTS` (8) times. The delay before each retry
is jittered exponential backoff (`accountingRetryDelayMs`): 10 ms ×
2^(attempt+1), capped at 400 ms, randomized between half and the full
value. The worst case total wait is about 1.4 s.
- `AccountingContentionError` uses a new generic message: "Paperclip
could not save these budget or usage changes because other updates to
the same company kept conflicting with them. Try again."
- A failed run keeps its run-specific `ACCOUNTING_CONTENTION_MESSAGE`
(run completion still uses it).
- The existing exhausted-retry test uses the new attempt constant.

## Verification

- `cd server && npx vitest run
src/__tests__/budget-policy-write-contention.test.ts`
- Red before the change: 5 of 9 concurrent policy writes failed with
`AccountingContentionError`, and the message mentioned a run.
- Green after: all writes succeed (5 of 5 repeated runs; also 3 of 3
runs with 17 concurrent writers).
- `run-accounting-contention`, `run-completion`,
`budget-control-publication`, `budget-evaluation-work` and
`heartbeat-native-predispatch-retry` tests.
- Paid runner E2E evidence for `goals.align-existing-work` is linked in
a comment.

## Risks

- Low. A write that keeps losing now waits up to about 1.4 s before it
fails, instead of about 60 ms. Only transactions that PostgreSQL rolled
back (40001 or 40P01) are retried, as before.
- Moving policy writes out of the accounting transaction was considered.
It is not done here, because the policy fence keeps budget evaluation
consistent with policy changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:33:36 -06:00
DottaandClaude Opus 5.5 de675d924a fix(heartbeat): stop a comment the closing run already read from reopening its finished task (#21175)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can add a comment to a task while the agent's run on that
task is still working
> - The comment must reach the agent once: in the running turn, or in
one later turn
> - In a restart-chaos wave, a comment that the running turn already
read reopened the finished task, and the agent did the completion turn a
second time 15 minutes later
> - This pull request makes a read by the running turn count as
delivery, and holds the comment behind the running run so the run's own
release settles it
> - The benefit is no reopened done task, no repeated completion turn,
and an unread comment that is delivered when its run finishes, not after
a slot wait and a stale cancellation

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261112T181948-s7101`, task t005 (OpenCode,
native Runner), during a two-server overlap deploy (K15).
- 18:26:56 run A started on the old server. 18:26:55 traffic switched to
the new server.
- 18:27:03.953 a board user posted a follow-up comment. The new server
handled it. Run A was not in that process's `activeRunExecutions`, so
`filterZombieCoalesceTarget` treated it as a zombie. Admission returned
`proceed`, and the comment got its own queued run B (18:27:04.252). The
queued run is not part of the release drain.
- 18:33:32 run A called `get_task_history`, received the comment,
applied it, and finished the task (`done`, 18:34:20).
- Run B waited for an agent slot until 18:41:49 and was cancelled as
`continuation_task_ownership_changed`.
- `deferUndeliveredCommentsOfStaleRun` then saved the comment again. Its
"unread" test is only "a run started before the comment and finished
after it". `deliveredQueuedCommentIds` had no record of run A's read.
The release drain reopened the task (done to todo, 18:41:51), and run C
(18:49:22) posted a second "Task complete" summary. The harness grade
`done_comment_once` failed.

**Expected behavior**
- A comment that the running turn received is consumed. No queued work
outlives that run, and the task stays done.
- A comment that the running turn did not receive is delivered once,
through the normal comment-on-done-task path, when that run finishes.
- A stale continuation is never re-issued for a comment that a completed
run already read.

**Steps to reproduce**
1. Start a native run on a task.
2. Post a person's comment while another server instance executes the
run (or seed a queued comment run behind it).
3. Let the run read `get_task_history`, then finish the task.
4. Before this change, the queued comment run is cancelled as stale, the
task reopens, and the agent runs a second completion turn.

**Paperclip version or commit**
`master` at ef6d347f79.

## What Changed

- `server/src/services/queued-comment-delivery.ts`: new run-log event
`task_history.read` and `recordTaskHistoryRead`.
`deliveredQueuedCommentIds` also counts a comment as delivered when a
succeeded run of the assignee on that task read the comment's current
revision (`historyReadDeliveredSql`). All existing consumers use this
one check: the release drain liveness check, the stale-continuation save
(`saveUndeliveredUserComments`), the ownership reconciler, and
first-turn queued input.
- `paperclip-runner-tool-authority.ts` and
`runner-semantic-authority.ts`: `get_task_history` records the returned
comment IDs and `updatedAt` values. The tool output does not change. A
failed record is logged and leaves the comment owed, so it is delivered
again.
- `server/src/modules/wake-queue/application/use-cases.ts`: a comment
follow-up for the same agent's `running` run is held as a deferred wake
even when this process does not track that run. Before, a run executed
by another server instance was treated as a zombie, and the comment got
its own queued run that bypassed the release drain. Coalescing into an
untracked run is still refused.
- `doc/run-log-events.md`: documents `task_history.read`.
- Regression tests in
`server/src/__tests__/heartbeat-held-comment-wake.test.ts`.

Intended behaviour chosen: in t005 run A received the comment mid-turn
through its own `get_task_history` call (run events at 18:33:32 show the
comment ID in the tool result, and the result says it added the
follow-up line), so (a) applied. The fix makes that read the durable
proof of delivery. A comment that no run read still follows the existing
outbox: it is held behind the run and delivered once at the run's
release, reopening the done task as a person's comment does. Envelope
task history of another message's turn still does not count (unchanged
doctrine and test).

## Verification

- RED on master (`ef6d347f79`), `npx vitest run
src/__tests__/heartbeat-held-comment-wake.test.ts -t "mid-turn|another
server|held message"`: 3 failed, 1 passed.
- "consumes a message the closing run read mid-turn, so its stale queued
run reopens nothing": delivered 1, expected 0.
- "holds a message behind a run another server is executing instead of
queueing a run of its own": a queued run was created.
- "settles a held message as soon as its run closes the task (read
mid-turn: true)": delivered 1, expected 0.
- The unread variant (delivered once, no cancelled stale run) passes on
both, as intended.
- GREEN with this change: all 4 pass.
- Review follow-up (f43e79c3): a history read counts as delivery only
for a succeeded run on the read revision. Tests cover an edit after the
read (the stale queued run delivers the comment once) and reads by an
edited or failed run (the comment stays owed). Full file: 37/37 pass.
- Security-scan follow-up (239f0597): the delivery check probes each run
binding (`native_issue_id`, `context_snapshot->>'issueId'`) through its
own existing index with `offset 0`. It reads only the assignee's
succeeded runs of the task with `finished_at >= ` the comment's current
revision. It does not use `started_at`, because the t005 reader started
before the comment. It stops at the first matching `task_history.read`
event of each run through `heartbeat_run_events_company_run_idx`. No
migration is needed.
- Seeded plan (local PostgreSQL: 200 tasks, 4,000 runs, 1.2M run events,
one comment read by one run), `EXPLAIN (ANALYZE, BUFFERS)`:
- Before: 1.202 ms, 6,097 shared buffers. It scanned every event of all
21 runs of the task.
- After: 0.030 ms, 23 shared buffers. It reads one index-matched run and
one event. The full `deliveredQueuedCommentIds` call takes 1.55 ms.
- New case: a succeeded run's read is not delivery when the comment was
edited after that run finished. Full held-comment file: 38/38 pass.
- `muse-control.test.ts` compared the whole run row after a
`get_task_history` call. It now allows that read's own run-log append
(`nextEventSeq` +1, `updatedAt`), and every other column must be
unchanged. That CI failure (General tests server 2/12) came from this
PR. "e2e shard (5/8)" failed because the self-hosted runner lost
communication; that is not related to this PR.
- Runner and Muse suites (`dot-runner`, `paperclip-control-plane-port`,
`runner-prp-coordinator`, `historical-task-questions`,
`paperclip-runner-tool-authority`, `muse-native-follow-up`,
`muse-control`): 509/509 pass. Server `tsc` is clean.
- Neighbouring suites: `heartbeat-held-comment-wake`,
`src/modules/wake-queue/**`, `execution-continuation`,
`paperclip-runner-tool-authority`, `first-turn-queued-input`,
`heartbeat-comment-wake-batching`, `ownership-reconciliation-report`,
`heartbeat-zombie-guard`: 555/556 passed in one loaded run. The 2
timeouts (one batching test, one reconciler `afterAll` hook) pass when
run alone (1/1, 44/44).
- `server` `tsc --noEmit`: clean.
- The restart-chaos harness already grades this class
(`done_comment_once` in staging-mix). No new cell is necessary.

## Risks

- Product choice: a comment counts as delivered when a succeeded run
read it through `get_task_history`, even if the model ignored it. This
is the same trust that applies to a comment delivered as wake input. The
alternative fails worse: it reopens a done task and repeats a completed
turn. An edit after the read, or a reader run that fails, leaves it
owed.
- Legacy (non-native) adapters that read comments through the REST API
do not write the record. Their behavior does not change.
- A comment follow-up behind a truly dead running run now waits as a
deferred wake until recovery releases that run. Before, its queued run
also could not start until the lock was released.
- Each `get_task_history` call appends one small run-log row. This is
not in the path before `turn.start`.
- Upgrade-safe: continuations queued by older builds still go through
the dispatch stale check, and that path now reads the same delivery
record.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:31:11 -06:00
DottaandClaude Opus 5.5 f92f91f0fb fix(native): finish a parent claim when its child results arrived before the run started (#21189)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task gets a wake when one of its child tasks completes. The
wake carries the child results.
> - When the parent already has a run, the child result wake waits
(`deferred_issue_execution`) behind that run.
> - At finalization, a parent completion claim stays open
(`native_child_completion_pending`) while such a wake is still queued,
claimed or deferred. This guard protects child results that arrive while
the parent works.
> - The guard did not check when the child result arrived. A child
result saved before the parent run started was already visible to that
run, but it still blocked the claim.
> - So the task stayed `in_progress`, the deferred wake started a second
run, and the agent repeated its completion summary.
> - This pull request counts a child result as undelivered only when it
was saved after the finishing run started.
> - The benefit is that a parent that already read its child results
finishes in one turn, and child results that arrive during the run still
get a fresh turn.

## Linked Issues or Issue Description

Refs #15372 (added the pending child-completion guard). Refs #20935
(same family: a self-wake after a settled run).

**What happened?**

Restart-chaos wave `w-20261112T181948-s7101`, task t001 (OpenCode,
native runner, staging-mix parent with two subtasks). Both subtasks were
done before parent run `8006a205` started at 18:45:29. The run read
them, posted its final comment and claimed `done` at 18:46:15.860. A
child-completion wake for the parent had been deferred since about 18:42
behind that run. The finalizer found that wake and committed
`native_child_completion_pending`, so the task stayed `in_progress`. 63
ms later the release promoted the deferred wake into run `9282de34`
(`invocationSource: automation`, `wakeCommentId: null`). Its
`paperclip.native-continuation.v1` turn carried the subtask results that
the first run had already used, plus the agent's own two comments
(resume delta since the base run). That run posted a second "complete"
summary and only then moved the task to `done`.

Root cause: `hasPendingNativeChildCompletion()`
(`server/src/services/native-runtime/native-child-completion-delivery.ts`)
treated every queued, claimed or deferred child-completion wake for the
issue as unread, except a wake bound to the finishing run itself. It did
not compare the wake's save time with the run's start.

**Expected behavior**

A child result that was saved before the finishing run started is not
new input for that run. The completion claim completes the task. The
deferred wake is cancelled at release as a wake on a terminal task. A
child result saved during the run still keeps the task open and gets a
fresh parent turn.

**Steps to reproduce**

1. Create a parent task with a done child. Save a deferred
`issue_children_completed` wake for the parent.
2. Start a native parent run after that wake was saved. Let it claim
`done`.
3. Finalize the run. On `master` the decision is
`native_child_completion_pending` and the task stays `in_progress`; the
deferred wake then starts a second run.

## What Changed

- `native-child-completion-delivery.ts`:
`hasPendingNativeChildCompletion()` counts a child-completion wake only
when its `createdAt` is at or after the finishing run's `startedAt`. A
run without a recorded start keeps the old behavior: every such wake
stays pending. The finalizer, the status-commit recheck and status
reassessment all use this function.
- `heartbeat-process-recovery.test.ts`: a regression test finalizes a
real native done claim with a deferred child-completion wake saved
before (task becomes `done`) or after (task stays `in_progress` as
`native_child_completion_pending`) the run started. It then releases the
issue execution through the wake queue. In the `before` case the
deferred wake is cancelled and there is one run total. In the `after`
case the wake is promoted into a fresh run that carries the child
results.
- `doc/architecture/native-status-arbitration.md`: describes when a
child result counts as undelivered.

## Verification

- RED on `master` (`ef6d347f79`): `heartbeat-process-recovery.test.ts`
"treats a child result saved before the parent run started…" fails with
decision `reasonCode: "native_child_completion_pending"`, `toStatus:
"in_progress"`. This is the same decision as the wave. The "after" case
passes on `master`.
- GREEN: both cases pass with the fix.
- Neighbouring suites pass: `native-status-arbiter-corpus` (includes the
queued / claimed / deferred / consumed child-delivery cases from
#15372), `heartbeat-dependency-scheduling` (revised child completion
delivered after a running parent), `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, and `src/services/native-runtime`.
- `pnpm --filter @paperclipai/server typecheck` passes.
- The regression test runs finalization and release end to end. The
cancel-at-release step itself is existing behavior, also covered by
`modules/wake-queue/application/use-cases.test.ts` ("cancels stale
assignee continuations before claiming promotion on %s tasks", `done`
case).

## Risks

- A child result saved before the run started no longer blocks the
claim. The child status was already `done` when the run started, so the
run could read it. If the clocks of two server instances differ, a child
result saved during the run could look older than the run start; the
window is the clock skew.
- A re-dispatched native status wake gets a new row with a later
`createdAt`. Such a row stays pending, which is the old behavior.
- No migration. Saved wakes from older builds work unchanged, because
the check reads only existing columns.
- Scope: this change does not alter the comment-wake promotion path or
the resume-delta builder.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:19:32 -06:00
DottaandClaude Opus 5.5 8a90aa4731 fix(chat): name document revisions and link runs in replies, and record agent-started routine runs as manual (#21183)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and the chat directive tells
the agent how to write replies
> - #21081 added a rule that keeps raw IDs and internal codes out of
replies. It names only tasks, cases, approvals and people as records to
cite
> - After #21081, agents stopped saying "revision 4" and stopped linking
runs. In a full chat cohort, "revision N" mentions fell from 21 to 11,
and run links fell from 9 to 5
> - A person needs the revision number that Paperclip shows and a link
to the run that failed. The intended effect held: UUIDs in prose fell
from 24 to 7
> - Separately, an agent that started a routine on a person's chat
request sent `source: "api"`, so the run was not recorded as a manual
run
> - This pull request extends the same rule to name document revisions
by number and to link runs, and records every agent-started run on the
run-now route as manual
> - The benefit is replies that point to the records the person can
open, and truthful run sources

## Linked Issues or Issue Description

Refs #21081

**What happened?**

After #21081, a restore reply said "the version before the last edit"
and did not name the new revision. A failed-run reply did not link the
failed run. In a different case, the agent called `POST
/api/routines/{id}/run` with `source: "api"` after two 400 responses.
The run was recorded with source `api`, although a person asked for it
in chat.

**Expected behavior**

The reply names the revision as Paperclip shows it ("revision 4") and
links a run it discusses. A run that an agent starts on this route is a
manual run.

**Steps to reproduce**

Run the chat-features OpenCode cases
`documents-artifacts.restore-document`,
`runs-adapters.inspect-failed-run` and `routines.run-now` on master
9e910ff9ae, and read the replies and the routine run source.

**Paperclip version or commit**

`master` at 9e910ff9ae.

**Agent adapter(s) involved**

Not adapter-specific. Seen with the Paperclip runner and OpenCode
(`openrouter/deepseek/deepseek-v4.1-flash-1017`).

**Deployment mode**

The CI chat-eval runner (local server, embedded database).

## What Changed

- `AGENT_CHAT_DIRECTIVE` (`server/src/services/agent-conversations.ts`):
the "Write replies for the person" paragraph now also says to name a
document revision by its number (such as revision 4), and to link a run
it discusses, such as `[the failed
run](/<prefix>/agents/<agentId>/runs/<runId>)`. The ban on raw IDs in
prose does not change.
- `POST /api/routines/:id/run` (`server/src/routes/routines.ts`): for an
agent actor, the server ignores the `source` value in the body and
records `manual`. Board callers keep their `source` value.
- `routines.ts` service: for an agent manual run with an
`idempotencyKey`, the replay lookup also returns that same agent's
earlier `api` receipt (matched by its `routine.run_triggered` activity).
A retry across the upgrade does not start the work twice. A board
receipt is not matched.
- `docs/api/routines.md`: documents the agent rule for the run-now
source.

## Verification

- Red/green: `server/src/__tests__/agent-conversations.test.ts` asserts
the revision-number and run-link sentences and that the raw-ID ban
stays. It failed before the directive change.
- Red/green: `server/src/__tests__/routines-routes.test.ts` sends
`source: "api"` as an agent and expects `manual`; a board caller keeps
`api`. It failed with `source: "api"` before the route change.
- Red/green: `server/src/__tests__/routines-service.test.ts` checks that
a board `api` receipt with the same key does not satisfy an agent's
manual request, and that the agent's own earlier `api` receipt is
returned.
- `server/src/__tests__/routines-e2e.test.ts` (agent routine-to-routine
provenance) still passes.
- `pnpm --filter @paperclipai/server typecheck` passes.
- Paid OpenCode run on the five affected cases (with the companion eval
PR): see the comment below.

## Risks

- Low risk. The directive grows by two sentences. A model can still omit
a revision number or a run link.
- An agent can no longer record a run-now call as `api`. The documented
contract for this route is a manual run, and provenance uses the caller
run, not the source.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:11:52 -06:00
Nicky LeachandPaperclip 2168f50741 Forward provider credentials from verified image catalogs (#21186)
## Thinking Path

> - Paperclip manages AI agents for work.
> - Application images can ship plugins that provide task environments.
> - Paperclip starts plugin workers with a restricted process
environment.
> - An image author can know which provider credentials an image-bundled
worker needs.
> - The image catalog already identifies selected bundles and verifies
their content.
> - This change adds a way for the image catalog to declare the provider
keys that a verified worker receives.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Image-bundled environment drivers can use the worker credential delivery
already available to built-in providers.

**Current behavior**

The host provides fixed credential forwarding for certain built-in
providers.

**Proposed behavior**

A verified image bundle can receive provider API keys declared on its
own catalog entry.

**Reason and benefit**

Image authors can scope deployment credentials to the verified worker
that needs them.

## What Changed

- Add an optional provider credential declaration to the image-owned
distribution catalog.
- Scope forwarding to the verified worker for the selected bundle.
- Publish distribution catalog support for image builders.

## Verification

- The new loader regression failed before the change because the old
catalog rejected the credential declaration.
- Ran six focused Vitest suites for bundled activation, package
identity, distribution catalog validation, worker environment, and
managed configuration. 166 tests passed and one existing test was
skipped.
- Ran `pnpm --filter @paperclipai/server typecheck`. It passed.
- Reviewed the diff for secret values and personal information. Test
keys are synthetic.
- A downstream image build fixture exercised the catalog reader, bundle
verification, and worker environment together. The supported build
passed only the declared provider key to the verified worker. The build
for an older target omitted the declaration.
- Local verification used embedded PostgreSQL and prebuilt fixtures.

## Risks

- Deploy images with catalog credential declarations only to Core
versions that support the field. Older hosts reject unknown catalog
fields.
- The image author controls which verified bundle receives each provider
API key.

## Model Used

- Implementation: GPT-6.1 Sol (`gpt-6.1-sol`), delegated through Codex.
Independent Codex review and verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs
- [x] I have described the issue in the PR
- [x] I have used public references
- [x] My branch name describes the change
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address reviewer comments before requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 20:53:08 +00:00
DottaandClaude Opus 5.5 267983356c feat(mcp): speak stateless MCP 2026-07-28 to provider servers, keep older servers working (#21140)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through Paperclip's MCP client. The client calls
each provider's remote (Streamable HTTP) or local (stdio) MCP server.
> - The client spoke only protocol 2025-06-18. It always did the
`initialize` handshake and it sent client capabilities only in that
handshake.
> - MCP 2026-07-28 removes the handshake and sessions. Every request
carries its version, capabilities and client identity in `_meta`.
Servers that only speak 2026-07-28 reject the old requests, and they
never see that Paperclip renders MCP Apps views.
> - Public servers already speak it: DeepWiki and Context7 answer
`server/discover` with 2026-07-28 today.
> - This pull request makes the client negotiate the version per server.
It speaks stateless MCP to 2026-07-28 servers and keeps the handshake
for 2025-11-25 and older servers.
> - The benefit is that new and old providers both work, and a tool call
to a stateless server is one HTTP request with no handshake.

## Linked Issues or Issue Description

Refs #21119 (MCP Apps host). This PR sends its MCP Apps capability to
stateless servers too.

**Subsystem affected**
Server: the MCP client in `server/src/services/mcp-http.ts`, catalog
discovery, the tool gateway (`tools/call`, context reads, stdio), and
the remote MCP OAuth flow.

**Problem or motivation**
The client used protocol 2025-06-18 only. MCP 2026-07-28 makes MCP
stateless: no `initialize`, no `Mcp-Session-Id`, per-request `_meta`,
`server/discover`, `resultType` with multi round-trip requests (MRTR),
cacheable lists, required `Mcp-Method`/`Mcp-Name` headers, and RFC 9207
`iss` validation. Servers that speak only 2026-07-28 cannot be used.
Paperclip advertised MCP Apps support only in `initialize`, so stateless
servers never saw it.

**Proposed solution**
Probe each server once with `server/discover`, pick the newest mutually
supported version, and save it on the connection. Send stateless
requests to modern servers and keep the handshake for legacy servers.
Tool calls use the saved version and never probe. The saved version is
checked again only when a request built from it is refused.

**Alternatives considered**
Inline negotiation (send each request in modern shape first and fall
back on `400`) adds a failed request to every call to a legacy session
server. Probing on every tool call adds a round trip before each tool
runs. Saving the negotiated version keeps tool calls at one request.

**Roadmap alignment**
MCP protocol currency for the Apps and connections substrate. It follows
the MCP Apps host work in #21119.

**Additional context**
Spec:
https://modelcontextprotocol.io/specification/2026-07-28/changelog. Out
of scope (separate work): the Tasks extension, Skills over MCP,
`subscriptions/listen`, progress and cancellation, icons,
enterprise-managed authorization.

## What Changed

- `mcp-http.ts`: `negotiateMcpProtocolVersion` probes with
`server/discover`. A `DiscoverResult` or a modern error
(`-32020`/`-32021`/`-32022`) means a modern server. Anything else means
a legacy server, which gets `initialize`. A `401`/`403` keeps the
challenge for OAuth discovery. The era is remembered per connection and
endpoint.
- `mcpHttpRequestInit` builds every request. It always sends
`Mcp-Method` and `Mcp-Name` (Base64 sentinel when unsafe). For modern
versions it adds `_meta` (`protocolVersion`, `clientCapabilities` with
`extensions["io.modelcontextprotocol/ui"]` and URL elicitation,
`clientInfo`) and `Mcp-Param-*` headers from `x-mcp-header`.
- Legacy `initialize` now requests 2025-11-25 and advertises the MCP
Apps extension.
- Catalog refresh saves `config.mcpProtocolVersion`. It drops tools with
invalid `x-mcp-header` annotations. It reuses a fresh `tools/list`
result (`ttlMs`, `cacheScope`). It probes again when the saved version
is refused, and it negotiates again after a credential refresh.
- `tools/call`: a saved modern version is called directly (no probe, no
handshake). A modern rejection of a legacy-shaped call, or `-32022`,
gets one retry with a negotiated version. The server rejected the first
request before it ran, so the retry cannot repeat a side effect. A
remembered legacy server that refuses `initialize` is probed again.
- MRTR: `input_required` with URL requests becomes the existing provider
link flow (`provider_interaction_required`). The next identical call
sends `requestState` and `inputResponses`. Form, sampling and roots
requests stop with `provider_input_required` and a plain message. A
missing `resultType` means `complete`.
- Context reads (`resources/*`, `prompts/*`) use the saved version,
cache fresh results, and stop on `input_required`. The error code is
kept, so resource not found (`-32602` or `-32002`) is visible.
- stdio: probe with `server/discover` (2 s limit). Restart a server that
exits on the probe. Send `_meta` to modern servers.
- OAuth: refuse a callback without `iss` when the server advertises
`authorization_response_iss_parameter_supported`. Check `iss` before
acting on a provider error. Send `application_type: "native"` for
localhost callbacks and `web` otherwise. Issuer-bound clients and the
CIMD-first order already existed.
- Docs: `doc/connections/MCP-PROTOCOL-VERSIONS.md`, linked from
`doc/connections/README.md`.

## Verification

- New suite `server/src/__tests__/mcp-protocol-versions.test.ts` (19
tests) uses a fake 2026-07-28 server and a fake legacy session server.
It covers discovery, `_meta`, headers, version fallback, `-32022`,
`resultType`, MRTR link and retry, form stop, cache TTL, legacy flow,
and recovery when a saved legacy server moves to 2026-07-28. Embedded
Postgres runs the end-to-end cases.
- New OAuth tests in `generic-mcp-connection.test.ts`: `iss` required
when advertised, `iss` checked on provider errors, `native` for
localhost, and a stateless server listed again after a revoked token.
- Red/green: with the master service code, all 21 new protocol and OAuth
tests fail. With this change they pass. The token-refresh regression
test fails without its fix and passes with it.
- Existing suites: 12 touched files, 1088 tests pass. `tsc --noEmit`
passes for `server`. Updated expectations count the one-time
`server/discover` probe, and the Calendly DCR test now expects `native`
for its localhost callback.
- `pnpm verify` exits 2 (a doc is a global CI input), so CI supplies the
broad coverage.
- Live check with the changed client, without credentials:
- DeepWiki (`mcp.deepwiki.com/mcp`): discover 200 → 2026-07-28, no
session, `tools/list` 200 (3 tools).
- Context7 (`mcp.context7.com/mcp`): discover 200 → 2026-07-28, no
session, `tools/list` 200 (2 tools).
- GitMCP (`gitmcp.io/docs`): discover 400 → `initialize` (2025-03-26) →
session → `tools/list` 200 (5 tools).
- Circleback (`circleback.ai/api/mcp`): discover 401 with its
`WWW-Authenticate` challenge kept for sign-in.

## Risks

- A legacy server that is not saved as session-required gets one extra
request (`server/discover`) the first time it is refreshed. Then the
version is saved. Saved session-required connections are not probed.
- DCR from a localhost callback now registers a `native` client. This is
what the spec asks for. Existing registrations are reused.
- An authorization server that advertises `iss` support but does not
send `iss` now fails sign-in with a clear message. The spec requires
this.
- MRTR answers are kept in memory for 30 minutes. After a restart the
server asks again, and the user sees the link again.
- No schema change. `config.mcpProtocolVersion` is a new key in the
existing JSON config.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits, web fetch of the MCP spec), extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 14:34:26 -06:00
DottaandClaude Opus 5.5 82c6810107 fix(runtime): post the agent's blocker reply when a retried finalization blocks the task (#21169)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime finalizes an agent's run, applies the agent's
status decision to the task, and posts the agent's reply as a task
comment
> - When the first workspace copy-back fails, or the run reconnects, a
background retry commits the result instead of the live heartbeat
> - That retry path applies the "blocked" status but never posts the
agent's reply, so the task turns blocked with no visible reason
> - The retry path already shares one recovery step for replies, but
that step accepted only "done", "needs review" and conversation replies
> - This pull request lets the shared recovery step present a blocked
task's reply, so the retry path and the normal path converge on one
presentation step
> - The benefit is that a blocked task always tells the user why,
exactly once

## Linked Issues or Issue Description

**What happened?**

An agent reports that it is blocked. The task moves to "blocked". No
comment from the agent explains why. This happens when the run finishes
through a retried workspace copy-back or a reconnect loop instead of the
live heartbeat. The run succeeds and its result has the agent's blocker
summary, but the run has no presentation decision and the task has no
reply. On a staging company, 58 of the 150 most recent agent-blocked
tasks for one agent had no agent comment.

**Expected behavior**

Every finalization path that applies an agent's status decision also
presents the agent's reply. A blocked task always shows the agent's
explanation once. The status decision does not post a second copy.

**Steps to reproduce**

1. Run a native agent on a task. The agent reports `blocked` with a
summary and a task-wide blocker.
2. Make the first workspace copy-back fail
(`native_workspace_sync_out_failed`), so the heartbeat leaves
finalization to the retry.
3. Let the retry finish the copy-back and commit the result.
4. The task is blocked and the run succeeded, but the task has no agent
comment.

**Paperclip version or commit**

`master` at b8207c4ea9

**Deployment mode**

Hosted (staging cloud). The defect is in server finalization, so it
applies to all deployment modes that use the native runtime.

**Agent adapter(s) involved**

Native runtime (`paperclip_runner`), seen with the Codex-based agent.
The defect is adapter-independent.

## What Changed

- The response presentation recovery step
(`recoverCommittedNativeChatResponse`) now presents a non-conversation
task's reply when the agent reported `blocked` and the task is still
blocked.
- For a blocked reply, recovery uses the agent's final message when one
exists. Otherwise it uses the accepted result summary, the same as the
live completion path.
- A blocked reply on a channel task (an external chat run) uses the same
rule as live run completion. Recovery resolves the chat publication
authorization, prefers the final response, and posts with that
authorization. The worker does not refuse these runs, so it does not
record a false publication failure.
- The presentation work query now finds these runs, so the background
worker also presents them and the bounded publication retry applies to
them.
- A blocker reply is not presented when the task is no longer blocked or
when a later status decision was applied. This keeps old blockers from
posting stale comments.
- Presentation stays idempotent: the step records
`presentationDecision`, reuses a reply that the run already posted, and
never posts twice.
- Existing affected tasks need no separate backfill. The presentation
worker loads all pending candidates at startup, so each task that is
still blocked gets its saved reply once after deploy.

## Verification

- Red/green test:
`server/src/services/native-runtime/native-blocked-reply-recovery.test.ts`.
It completes a blocked native run, fails the first copy-back with
`native_workspace_sync_out_failed`, then commits the retry with
`finalizeNativeRun`.
- Red on `master`: the retry-path tests fail (no comment is posted, and
the worker finds no candidate). The channel task test also fails on the
first PR revision (5eb8cdab95).
- Green with this change: 4/4 pass. The tests check that the blocker
summary is posted once, that the final reply is preferred, that a
repeated retry and a repeated worker check do not double-post, and that
a stale blocker is not posted.
- `cd server && npx vitest run
src/services/native-runtime/native-blocked-reply-recovery.test.ts
src/services/native-runtime/native-response-publication-retry.test.ts
src/services/native-runtime/agent-chat-response-recovery.test.ts
src/__tests__/idle-native-finalization.test.ts
src/services/presentation-ownership-read.test.ts
src/services/chat-monitor-continuation.test.ts` passes (172 tests).
- `cd server && npx tsc --noEmit -p .` passes.

## Risks

- After deploy, tasks that are still blocked from earlier affected runs
get the agent's saved reply once. This is the intended backfill. Tasks
that moved on are skipped.
- The candidate query has one more branch with a `NOT EXISTS` on
`status_decisions`, keyed by the existing `(company_id, issue_id,
decision_version)` unique index. The cost is low.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, browser reads of staging API).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 14:30:32 -06:00
Nicky LeachandPaperclip 7f69ef8b44 Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path

> - Paperclip lets plugins own managed Git workspaces.
> - A portable Runner task can move to a different physical allocation.
> - The checkout must be present on the allocation that launches the
Runner.
> - Workspace preparation and finalization need exact placement when
they retry.
> - This change checks the placement at Runner admission and records it
before finalization.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Workspace preparation and finalization for portable Runner tasks.

**Current behavior**
A task replacement can change the allocation after Git preparation. A
lost finalization reply can leave retry without the original placement.

**Proposed behavior**
Prepare the checkout on the actual allocation before admission. Record
finalization placement so an exact retry can use it.

Portable ownership and controller recovery are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066).

## What Changed

- Reserve the managed workspace before acquiring the portable task
lease.
- Pass the task placement to the workspace provider. Reuse the described
task while placement stays the same and prepare again before admitting a
replacement.
- Record the initial preparation's run, agent, generation, and optional
task placement before dispatch. Inspection and retries on the same
Launcher replay the same request; VM and sandbox preparation use the
same recovery path.
- Keep provider request identity separate from Git credential authority:
reservation reconciliation uses the existing project-consent path in a
host-only scope bound to the plugin and workspace. Foreground
preparation still requires an active run; consent, repository, and
revocation checks are unchanged.
- Acquire the current Launcher before retrying preparation. If placement
changes, use ordinary preparation on the replacement so the provider can
settle the prior writer and recover the retained checkout. No additional
provider discovery call is added.
- Select finalization placement before publishing the operation record,
then dispatch and recover using that saved choice. Retain the Launcher
during bounded finalization retries.
- Preserve interrupted pre-upgrade finalization requests without
placement, including their original operation IDs and provider routing.
No migration or new durable mode flag is required.

## Verification

- Final revision `74ea3e6712`: server typecheck and exact SDK source-pin
verification (538 inputs) passed.
- All 185 tests passed across the full plugin worker manager, project
Git authorization, and plugin workspace integration suites (including 56
workspace tests). Independent verification reran the three lost-reply
cases: inspection, same-Launcher retry, and replacement after deleting
the original Launcher. The tests assert the correct request identity,
durable intent, and clearing that intent after success.
- Before the final retry-accounting fix, all 323 tests passed across
workspace admission, runtime execution, native workspace finalization,
plugin workspace integration, and terminal cleanup retry. Regressions
exercise crashes after operation insertion but before dispatch in both
finalization paths, exact replay after lease release, and retry after
ambiguous placement clears without publishing an incomplete operation.
Pre-upgrade interrupted requests retain their operation IDs despite an
available portable lease.
- Final retry-accounting fix: all 84 native workspace finalizer,
failure, and supersession tests passed. A real missing-plugin regression
first failed on the prior commit, then verified healthy-run progress,
three-attempt exhaustion, terminal cleanup, and absence of a misleading
workspace operation. Independent verification also passed the legacy and
crash-before-dispatch cases.
- Worker-process tests cover both preparation RPC methods: the provider
still receives the original run and agent, maintenance receives
project-scoped credential authority, and foreground or mismatched-scope
requests remain denied. Database tests confirm a stopped run cannot
authorize foreground Git access while valid project consent can
authorize maintenance.
- Existing metadata protection coverage proves callers cannot overwrite
the private preparation intent or read it through workspace responses.
- Two existing Launcher Git tests passed: retry preserves dirty files
and the original base commit without fetching, and retry resumes the
saved checkout intent while rejecting a changed pending branch.
- Implementation was delegated to GPT-6.1 Sol and independently
reviewed. The GitHub stack's automatic rebase preserved the original
workspace-placement patch.

- [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706)
passed on `74ea3e6712`, including server and workspace tests, Runner,
end-to-end tests, typecheck, build, and canary dry run. Ordinary
Greptile and
[Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388)
both scored `74ea3e6712` 5/5, with no unresolved findings.

## Risks

- Live Daytona end-to-end qualification is pending. The checkout record
persists on the mounted volume; replacement preparation relies on the
provider confirming the previous writer is settled before using that
retained checkout. Local tests cover Core request routing and Launcher's
durable checkout recovery separately.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 12:12:41 -08:00
Huanyi Xie 8bff629465 fix(environments): skip blank lease ID queries (#15200)
## Thinking Path

> - Paperclip manages work for AI agents.
> - Runs can use environment leases.
> - A lease lookup sends its ID to a PostgreSQL UUID column.
> - An empty ID makes that query fail on every poll.
> - This pull request returns null before querying for blank IDs.
> - Valid lease IDs keep the existing lookup behavior.

## Linked Issues or Issue Description

Fixes: #15143

## What Changed

- Skip database access for empty or whitespace-only lease IDs.
- Cover empty, space, and newline IDs with a database-query spy.
- Keep an unknown valid UUID returning null. Existing lease tests cover
valid lookups.

## Verification

- The branch was rebased onto current master `252d9b50` to resolve the
test-file conflict.
- All 53 environment service tests passed on Node 24.19.0. The run used
cached dependencies and current database and shared-package source
aliases.
- The blank-ID cases assert that no database query occurs. The unknown
valid UUID still returns null. The existing suite covers stored leases.
- `git diff --check`: passed.
- A clean frozen install is blocked because upstream master adds two
OpenTelemetry packages without updating the lock file.
- Full typecheck and build were attempted on both unmodified master
`252d9b50` and this head in the same environment. Both fail with the
exact same runner `SessionRecord.cursor_prompt_usage` error (TS2339 at
`codex-runtime-adapter.ts:1572`). Full validation is pending CI on the
rebased commit.
- The earlier full test run had a Slack callback ordering failure that
also occurred on unmodified master.
- Greptile reviewed the previous head at 5/5 with no inline findings.

## Risks

Low risk. Only blank IDs return early. Non-blank IDs still use the
database. There is no schema or migration change.

## Model Used

OpenAI Codex, based on GPT-6, assisted with source review,
implementation and local test execution. The exact deployment model ID
and context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted suite; full suite
pending above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
user-facing configuration or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (new head checks are pending)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-12 13:49:09 -06:00
DottaandClaude Opus 5.5 4363a3115a fix(recovery): continue any unfinished run in a fresh run from saved work (#21135)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs are interrupted when a server restarts, a deploy swaps
servers, a controller lease is lost, or a sandbox Runner cannot be
reached again
> - Exact recovery (resume the same session, replace, retry) refuses in
many ways, and each refusal leaves the task blocked or idle for a person
> - On staging, 577 of 1,000 runs in 11 hours failed or were
interrupted, across 25 error codes. About 900 earlier PRs each fixed one
refusal code, and the rate did not converge
> - This pull request adds one universal fallback: when an unfinished
run cannot be resumed exactly, for any reason that is not the person's,
the task continues in a fresh run from its last saved workspace and its
conversation
> - The benefit is that a new failure code next week falls back without
code changes, and tasks stop waiting on a person for infrastructure
faults

## Linked Issues or Issue Description

**What happened?**
After a server restart or deploy swap, many runs end with codes such as
`native_continuation_requires_reconciliation`
(`runner_remote_process_identity_unavailable`), `controller_lease_lost`,
`process_lost`, `server_shutdown_interrupted` and
`native_session_interrupted` (including `runner_harness_state_mismatch:
resumed_not_suspended` / `resumed_unavailable`). The disposition worker
then settles the recovery hold as "blocked, no replay". The task stays
Blocked until a person presses Retry or reconciles the run.

**Expected behavior**
A task waits on a person only for reasons that are the person's: an
approval, a question, a credential to share, a budget stop, a provider
usage limit, or a Stop they asked for. For every other failure, the
system starts a new run for the same task from the last verified saved
workspace, without a person.

**Steps to reproduce**
1. Start a task on a sandbox agent.
2. Restart or swap the server while the run is in progress, so the new
server cannot re-reach the Runner.
3. See that the run fails with
`native_continuation_requires_reconciliation` and the task stays Blocked
with no successor run.

**Paperclip version or commit**
`master` at 9e910ff9ae.

## What Changed

- New module `server/src/services/fresh-continuation-fallback.ts`. One
entry point, `ensureFreshContinuation` / `continueFreshInTransaction`,
takes a run that ended without finishing and guarantees one of:
something already follows it (a newer run, a linked replacement, an
operator decision), exact recovery is still under way (checked again
later), the cause is the person's, or one fresh continuation is
recorded.
- Classification, not an allowlist: `PERSON_OWNED_FAILURE_CODES` plus
task-level waits (pending approval, pending question, paused or
unavailable agent, paused company, review in progress, a Stop that a
person asked for, a Blocked status that another decision owns). Every
other code, including unknown future codes, falls back.
- Idempotent per source run. The task row lock serializes callers on
every server; the recorded `freshContinuation.sourceRunId` makes later
calls a no-op.
- Isolation, never waiting on proof. A local process still running on
this host is settled by its own owner first. If the old local writer is
proven gone, the task workspace is kept. Otherwise the continuation gets
a new isolated workspace (`fresh_isolated_workspace`), so two runs never
write one directory. The decision records the stop proof honestly
(`providerStopProof: isolated`).
- Claim admission no longer waits on the cleanup of a predecessor that a
fresh continuation replaced (`heartbeat/queue.ts`, predecessor loop).
Without this, the successor stayed queued behind the unreachable lease,
which is the staging failure itself.
- Only reconciliation holds that this decision owns are settled.
Workspace restore holds that workspace recovery still owns, and resolved
history, are untouched.
- `continuation_user_authorization_missing` is the system's failure (a
Retry's authorization was not carried forward), not the person's.
- Safety: the old run keeps its fencing. When it still owns its remote
workspace epoch, the epoch is abandoned
(`abandonUnverifiedRemoteWorkspaceEpoch`), so the old writer can never
publish and the next run restores the last committed save into a new
root on a new allocation. The unverified files stay in the abandoned
root, unapplied. The native coordinator records `replacementDenied:
fresh_continuation`, so no exact replay also starts.
- The replaced run loses its write authority. The fallback records
`freshContinuationSupersededAt` on the old run. The agent-JWT middleware
(`loadAgentRunAuthentication` reads one more field in its existing row
read) and `assertAgentRunWriteAllowed` refuse every write of the old
token with `agent_run_superseded`. Before this change, a failed or
interrupted run's token could still write. So an agent that is still
alive in an unreachable sandbox cannot act on the task next to its
continuation.
- The opening context (`recoveryOutcomes` in the execution continuation)
says that the previous attempt was cut off and lists the actions with
unverified outcomes, so the agent checks before it repeats a side
effect. No copy asks a person to inspect or repair a sandbox, workspace
or run. The hold text written at run end no longer says "Inspect the
original failure and reconcile".
- Delivery reuses the existing path: the settled hold gets an
`executionReconciliation` decision and `continuationDelivery: pending`,
and `deliverReconciledExecutions` starts the run (fresh session,
`retryOfRunId` = source).
- Budget per task, not per restart: at most 6 fresh continuations per
task per rolling hour (the server-restart lane's cap). At the cap the
hold stays with the system and is checked again when the window has
room. The failure allowance is never charged.
- Call sites:
- `settleUnrecoverableExecutions` (execution-recovery-resolution.ts):
every hold whose exact recovery is ruled out now goes through the
fallback before it would block the task. Interrupted preparations keep
their own replay lane first.
- `continueHeldExecutions` (same worker): picks up holds saved by older
releases (settled-but-blocked no-replay holds, finished or exhausted
saved-work recoveries) for the task's latest unfinished run.
- Wake admission (heartbeat/queue.ts): an ordinary system continuation
wake (sweep, monitor, timer) that a hold would defer now calls the
fallback in the same transaction and becomes the fresh continuation
itself. This is the path the idle-task sweep uses. A person's Retry,
message or queued send keeps its exact admission; the disposition worker
settles the hold, and the saved input then continues.
- `executionControlSchedule.disposition` wakes the worker for held tasks
and for the next due recheck.

### Human gate versus system uncertainty

One documented rule (`isHumanGate` plus `personOwnedTaskWait`) decides
what the fallback never touches. The rule goes by cause, never by owner
type:
- Human gates: a pending approval, a pending question, a user-owned
hold, an owner-gone attestation, a recorded explicit continuation, an
operator reconciliation, a person's Stop.
- System uncertainty: a no-replay record ("outcome unknown after an
interruption"), a hold that is `board`-owned only by default, an
unverifiable controller process space or host.

For uncertainty the fallback starts a fresh run, never a replay. The
interrupted turn is not re-sent. No tool call or write is re-executed.
Governed actions go through their approval gate again. The earlier
`automaticRecovery` decision is kept on the hold as
`automaticRecoveryHistory`, beside the attributed
`freshContinuationSettlement` and an activity entry.

### Restart recovery refused for an unverifiable controller

The case is `controller_process_space_unverified`, its remote variant,
or `controller_host_unrecorded` / `controller_host_unprovable`. Its
lease has lapsed; a live, renewing lease is still waited for (server
overlap). After `NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS` (2 minutes) the
run ends (failed, `native_restart_recovery_blocked`). That frees the
task's execution lock and the agent's slot, and a system-owned hold lets
the fallback continue the task. Recording the stops is optional. The "a
board operator must record both stops" text is replaced.

Board/operator attention sites in native restart recovery:
- `surfaceNativeControllerNamespaceAttention`: copy replaced; the run
now ends after the bounded wait.
- `requestNativeControllerStopConfirmation`
(native-controller-host-recovery.ts): copy says confirmation is
optional; the same bounded end applies.
- The blocked restart-recovery hold (`nextAction` "Inspect the preserved
checkpoint and reconcile…"): copy replaced; the fallback settles the
hold by cause.
- `owner-gone-attestation.ts` (generic resource records from older
releases): unchanged. It is not a run-recovery path, and it is listed as
a follow-up.

### Every wait and its bound

| Wait | Owner that rechecks | Worst-case total before the fresh
continuation |
| --- | --- | --- |
| `exact_recovery_pending` (execution settling, native finalizer,
controller lease, interrupted-preparation lane, saved-work recovery,
restore hold, retained-source fallback, live local process) |
Disposition worker, woken at each deadline; rechecks at 1, 2, 3 and 4
minutes, then every 4 | About 10 minutes from the first pending check.
Up to 15 while the owner shows progress (a new event, output, lease
renewal or coordinator update since the last check). |
| Saved-work (workspace) recovery that is `scheduled`,
`awaiting_evidence`, or `running` without progress | Held-run scan, now
selected and scheduled like the other holds | About 10 minutes; 15 at
most |
| Saved-work recovery that is `running` and advancing (revision,
attempt, state or hold update since the last check) | Held-run scan |
Its own 30-minute budget |
| Unverifiable lapsed controller in restart recovery | Restart recovery
(`retryAt`) | 2 minutes, then the run ends and the fallback continues it
within one disposition pass |
| Hourly cap (`backoff`) | Disposition worker, woken at the deadline |
Until the rolling hour frees a slot, at most 1 hour. This is the
deliberate restart-lane bound, not a stuck state. |
| Person-owned or dependency wait | The person's own action wakes the
task; rechecked hourly | Not bounded: it is the person's |

Restart-recovery classifications (`claimNativeRestartRecoveries`, which
`recoverNativeRunsAfterRestart` calls):

| Classification | Who rechecks, when | Bound |
| --- | --- | --- |
| Claims: `reattach_existing_runner`, `reattach_remote_runner`,
`resume_dead_runner`, `bootstrap_incomplete` | The claiming server
resumes exactly | Exact resume; no fallback |
| `already_finalized` | Nobody; it is terminal | None needed |
| `blocked` | The run is terminal and its hold goes to the fallback
(disposition worker) | Fallback bounds above |
| `awaiting_evidence` / `current_controller_preparing` | This process's
own executor | Ends with that executor |
| `awaiting_evidence` / `recovery_rows_busy` | Restart dispatch after 1
second | Self-resolving: the row lock clears |
| `awaiting_evidence` / a live lease (`controller_lease_held`,
`controller_lease_grace`, `controller_still_alive`) | Restart dispatch
at the lease-renew interval | Never ended while the old controller
renews its lease |
| `awaiting_evidence` / unverifiable lapsed controller (process space or
host) | Restart dispatch every 2 minutes | Ends 2 minutes after the
lease lapsed |
| `awaiting_evidence` / any other reason (startup cleanup, finalizer or
cleanup owner, recovery failure, routine busy, future reasons) | Restart
dispatch at the owner's `retryAt`, or within 2 minutes when it has none
| Ends 2 minutes after the lease lapsed, or after 10 minutes, whichever
comes first. It never ends while a lease is live. |

Ending a run sets it failed (`native_restart_recovery_blocked`),
releases the task lock and the agent's slot, and records a system-owned
hold. The fallback then continues the task once, isolated.

No wait needs longer. Saved-work recovery and retained-source checks
keep their own attempt budgets. After 10 to 15 minutes the fallback
continues anyway in a new root, without the unverified files, which stay
recorded and unapplied.

### When the fallback preempts saved-work recovery

The unverified files are not destroyed. They stay where recovery found
them: in the abandoned remote epoch's root, or in the old local
workspace that the task is no longer bound to. They are recorded and
never applied. Settling the hold records the decision on it, so the
retained-workspace worker stops scheduling it (its query requires no
`executionReconciliation` unless recovery already succeeded). If an
attempt already in flight finishes later, it cannot publish into the
fresh run's workspace. For a remote workspace, the epoch it targets is
abandoned and the writer is cleared. For a local one, the task now
points at a new workspace. Whenever a wait is preempted, the
continuation is isolated: a fenced remote epoch or a new local
workspace.

### What people see

The task's interruption presentation shows "working" while the primary
fresh-continuation record is pending and has not started (coalesced
holds never count, and they follow the primary's final state), and a
timed wait with no Retry while a check is scheduled. The Retry banner
appears only when the system has given up or the cause is the person's.
Holds settled together with the primary record are `coalesced`; they no
longer show "closed or reassigned". A pending continuation is never a
second run: if a person's Retry or saved input started a run after it
was recorded, delivery links to that run instead. A task that finished
before delivery gets no run.

### Readers of what the fallback writes

- Recheck deadline (`freshContinuationCheck`):
- `unresolvedExecutionDispositions` and
`heldExecutionsAwaitingFreshContinuation` read it to skip work that is
not due.
- `nextFreshContinuationCheckAt` and
`executionControlSchedule.disposition` read it to schedule the worker.
- Tests: the cap deadline is scheduled, and a cancelled task goes quiet.
- Isolation decision:
- The task's workspace binding is cleared. Every path that starts the
next run (system continuation, saved input, Retry, sweep wake) resolves
the workspace from that binding at run start (heartbeat.ts
`issueContext.executionWorkspaceId`).
- `freshContinuationIsolation` in the run context is read by
`readFreshWorkspaceReadiness` for the shared-directory case.
- `freshContinuationSupersededAt` on the old run is read by the auth
middleware, `assertAgentRunWriteAllowed` and the claim gate.

## Verification

- `pnpm vitest run src/services/fresh-continuation-fallback.test.ts`
(server): every test passes (it covers the cases below).
- The continuation starts (leaves the queue) while the unreachable old
lease is `active` or `pending_cleanup`. This was red before the
claim-admission fix.
- Isolation: a writer on another host gets a new isolated workspace; a
process alive on this host waits for its owner; a proven-gone writer
keeps the workspace.
  - Owned holds only: a restore hold and resolved history are untouched.
  - The recheck schedule goes quiet when the task is cancelled.
- `continuation_user_authorization_missing` continues under the task's
responsible user.
- One test per top-five staging code (native and legacy hold shapes):
the task gets exactly one fresh successor run from saved state.
- `runner_harness_state_mismatch` (`resumed_not_suspended`,
`resumed_unavailable`) falls back.
  - An unknown code falls back.
- Two concurrent callers plus the disposition worker create one
successor.
- Old-saved/new-resumed: a settled-but-blocked hold from the previous
release, with its coordinator in `awaiting_runner_reattach`, continues
on the first pass.
- An ordinary system continuation wake is admitted as the fresh
continuation.
- Fencing: after the continuation is recorded, the old run's token is
refused for writes (`agent_run_superseded`) and the continuation's token
is not. The old run's remote workspace epoch is abandoned: the writer is
cleared, the location and provider root are cleared, and the epoch
advances. So the next preparation restores the last save into a new root
(`prepareRemoteWorkspace`: `sameAllocation` is false when the location
is null, server/src/services/remote-workspaces.ts:437-470).
- The opening context names the cut-off run and an unverified tool
action.
- Person-owned causes (pending approval, credential not shared, budget
stop) are not continued.
  - The hourly cap backs off without blocking, then continues.
- Red/green: with the wiring reverted to `master`, 9 of these tests fail
(the task stays Blocked with no successor).
- Affected suites pass locally: `execution-recovery-dead-controller`,
`completed-turn-cleanup-hold-settle`, `execution-control-work`,
`heartbeat-process-recovery`, `issue-recovery-actions`,
`native-safe-replacement`, `native-failed-workspace-recovery`,
`task-retry`, `runner-lost-mid-turn-continuation`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
wake-queue `postgres`, `agent-conversations`, `native-startup-retry`,
`governed-wait-runner-loss`, `native-preclaim-continuation`.
- `pnpm verify --explain` selects the full suite (the wake queue is
shared), so CI supplies full coverage.

Existing expectations changed because they asserted the "wait for a
person" behavior this PR replaces:
- `issue-monitor-scheduler.test.ts`: "explains held monitor dispatch
without replaying uncertain execution" is now "continues a held task in
a fresh run when its monitor is due". A due monitor on a task held by a
`process_lost` hold now starts one fresh run instead of deferring.
- `agent-run-authentication.test.ts`: the auth row also reads
`freshContinuationSupersededAt` (still under 512 bytes), and a
superseded run is refused.
- `assigned-work-invariant.test.ts` (#21139): "with the real wake queue,
a hold that defers the continuation leaves a violation…" is now "…a hold
reached by the sweep's wake becomes one pending fresh continuation". The
sweep's wake now yields one pending fresh continuation, and the next
sweep adds no attempt.
- `heartbeat-native-restart-suspension.test.ts` and
`native-runner-restart-recovery.integration.test.ts`: awaiting-evidence
dispositions now carry a `retryAt`, because every wait is rechecked. A
lapsed unverifiable controller no longer waits forever.
- `native-remote-retry-admission.test.ts`: the blocked-hold text is now
"This run stopped before it finished. Your saved work is kept." The old
text asked people to reconcile.

Holdless legacy runs: in the 1,000 staging runs, 0 tasks end on
`controller_lease_lost` or `process_lost` (every one has a later run).
12 end on `server_shutdown_interrupted` with no retry scheduled (attempt
0). On staging now, 9 of those 12 sit `in_progress` with no run and no
hold, 1 is done and 2 have a live run. Those 9 belong to the idle-task
sweep (#21139).

## Risks

- Speed: nothing is added to the normal path before `turn.start`. The
fallback runs only on the disposition worker after exact recovery is
ruled out, or at wake admission when a hold already defers the wake. The
auth middleware reads one more JSON field in the row read it already
does.

- Behavior shift: tasks that today stay Blocked for a person after an
infrastructure failure now continue automatically. A side effect with an
unknown outcome is not replayed; the new run is told about it and must
check first.
- After deploy, the disposition worker continues tasks already blocked
by these codes, at most 25 per pass and at most 6 per task per hour. No
migration. No manual Retry is needed.
- The old executor is not proven stopped before the successor starts.
Fencing is: its API writes are refused, its workspace epoch is
abandoned, the native `replacementDenied` marker is set, and the
successor has a separate run identity.

Known gaps (follow-ups):
- No quarantine attachment: unverified files stay unapplied in the
abandoned workspace root (recorded on the workspace in
`remoteWorkspaceAbandonedEpochs`). They are not read into a task
attachment, because the sandbox is usually unreachable in these
failures.
- Agent Chat conversations are excluded at wake admission. They keep
their owner-bound continuation.
- Under a `reuse_by_environment` lease policy the new allocation can be
on the same provider sandbox. It always uses a new root and a new epoch,
and the old run cannot publish or write.
- Tasks left `in_progress` with no run and no hold (for example after
`server_shutdown_interrupted` with no retry scheduled) belong to the
idle-task sweep. That sweep's wake reaches this fallback when a hold
exists.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:34:34 -06:00
DottaandClaude Opus 5.5 8ff45c0fa9 feat(connections): render MCP Apps views in the task chat, starting with Circleback (#21119)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents reach external apps through MCP connections. Paperclip
records each call and shows it as a tool row in the task chat.
> - MCP Apps (SEP-1865, extension `io.modelcontextprotocol/ui`, spec
2026-01-26) lets an MCP server attach an interactive `ui://` view to a
tool. Claude, Claude Code and VS Code render these views in chat.
Circleback uses them to show meeting cards.
> - Paperclip dropped the tool's `_meta.ui` at discovery and the
result's `_meta` at call time. Its gateway reported MCP App UI as
"unsupported". Circleback was reachable only through Composio, whose
toolkit has no `Display*` tools.
> - This pull request makes Paperclip an MCP Apps host. It also adds
Circleback as a native catalog connection.
> - The benefit is that a Circleback recap in a task shows the same
meeting cards as in Claude, and any other MCP server with views works
the same way.

## Linked Issues or Issue Description

**Subsystem affected**
Connections and apps (remote MCP gateway) and the task chat.

**Problem or motivation**
An agent that uses Circleback in Paperclip can only return text. The
same tools in Claude Code show interactive meeting cards
(`DisplayMeetings`). Paperclip discards the MCP Apps metadata, so it
cannot show them.

**Proposed solution**
Paperclip implements the host side of MCP Apps end to end. It keeps the
tool's view declaration and records each call that has a view. It
renders the view under the tool row in a sandboxed iframe that uses the
MCP Apps postMessage protocol. Circleback becomes a native catalog
connection, so its `Display*` tools are available.

**Alternatives considered**
Composio's `circleback_mcp` toolkit has 11 read tools and no `Display*`
tools, and its meta-tools return plain JSON, so it cannot carry views.
The spec's double-iframe sandbox proxy on a separate origin needs a
second origin that Paperclip does not have. A single opaque-origin
`srcdoc` frame with an injected CSP gives the same isolation.

**Roadmap alignment**
This extends the existing Apps catalog and MCP gateway. It adds no new
integration substrate (decision D4 in doc/connections/README.md).

**Additional context**
The standard, the design and the testing plan are in
`doc/connections/MCP-APPS.md`.

## What Changed

- **Shared:** `packages/shared/src/mcp-apps.ts` adds MCP Apps types,
`_meta.ui` and resource parsing (current and legacy keys), and a CSP
builder that keeps only well-formed `https` origins.
- **DB:** migration `0471` (`online_compatible`) adds
`tool_catalog_entries.ui_meta` and a new `tool_mcp_app_views` table.
- **Server, catalog:** discovery keeps each tool's `_meta.ui`. It is not
part of the version hash, so a view never quarantines a reviewed tool.
- **Server, gateway:**
- Paperclip advertises the MCP Apps client capability when it
initializes a session.
  - App-only tools (`visibility: ["app"]`) are hidden from agents.
- The result `_meta` is kept for the view and is not sent to the agent.
- Each successful call to a tool with a view records a view row.
Recording is best effort.
- The gateway's `initialize` now reports `paperclip/mcp-app-ui:
host_rendered`.
- The context protocol (`resources/read`) now uses the injected test
transport, like the other remote calls.
- **Server, routes:** `server/src/routes/mcp-apps.ts`
- `GET /api/issues/:issueId/mcp-app-views` lists a task's views. It
needs task read access.
- `GET /api/issues/:issueId/mcp-app-views/:viewId` returns the view
document, read live through the agent's connection.
- `POST /api/issues/:issueId/mcp-app-views/:viewId/tool-calls` runs the
view's `tools/call`. It needs `tools:use` and reaches only tools of the
same connection that allow `app`. It uses the connection test-call path,
so policy, Ask first and audit apply.
- **UI:** `ui/src/components/mcp-app/`
- The host protocol: `ui/initialize` with theme, style variables and
size limits; tool-input; tool-result; size-changed; `tools/call`;
`ui/open-link` (http and https only); teardown.
- An opaque-origin iframe (`allow-scripts`, never `allow-same-origin`)
with the CSP placed before the view's markup.
- The views of a run render with that run's reply, in the live turn and
in settled history. They are keyed by the reply's run, so history needs
no transcript.
- **Circleback:** a native catalog app (`mcp-oauth`, DCR, `user
user:write`) with:
  - the official logo;
- a reviewed risk classifier (Share* tools are writes, Delete* tools are
destructive, Display* tools are reads);
  - warnings in the sign-in step;
  - `doc/connections/CIRCLEBACK.md`.
  
Search now prefers a native app over an aggregator listing named "X
MCP".
- **Storybook:** `Tasks/MCP App views` shows the tool row with its view,
the ready state (Refresh calls a tool), loading, view unavailable, and a
tool call that needs approval.

## Verification

- `pnpm exec vitest run packages/shared/src/mcp-apps.test.ts
ui/src/components/mcp-app/`: 19 passed.
- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts`: 3
passed, on embedded Postgres. The test connects Circleback through the
catalog with DCR OAuth against a fake Circleback MCP server. It checks:
  - `ui_meta` is stored;
  - an app-only tool is hidden from the agent;
- an agent call records one view with arguments, structured content and
`_meta`;
  - a call without a view records nothing;
  - the routes serve the view's HTML and CSP;
  - the view's tool call works and a model-only tool is refused;
  - another task, another company and an agent actor are refused.
- Circleback connector: 13 focused files, 645 tests passed. These are
the app definitions, search, aggregator fallback, connection intents,
tool policy, `AppsConnect` and `Browse`.
- Typecheck passed for `@paperclipai/shared`, `@paperclipai/server` and
`@paperclipai/ui`. `node scripts/storybook-catalog.mjs` and the
migration compatibility check also passed.
- Storybook check in the browser:
- In both themes, the view runs the handshake, renders the meeting cards
from `structuredContent`, picks up the theme variables, and resizes to
its content.
  - Refresh sends `tools/call` and re-renders the cards.
- **Live Circleback end to end** on a local instance, with a real
Circleback account:
- Connected through the catalog entry with Circleback's dynamic client
registration and OAuth.
  - Discovery stored the real view declarations:
    - `DisplayMeetings` uses `ui://circleback/meetings.v2.html`.
    - `DisplayActionItems` and `DisplayCalendarEvents` also have views.
- `ListAssigneeOptions` and `SetCalendarEventRecording` are app-only
helpers with no view. They are hidden from agents.
  - A `claude_local` agent called `DisplayMeetings` on two tasks.
- The real 3.5 MB Circleback view rendered the meeting cards with the
agent's reply.
- Clicking a card opened that meeting in Circleback through
`ui/open-link`.
- The live run found two bugs, both fixed in this PR, each with a
regression test:
    - The 1 MB tool-result limit blocked the view document.
    - Views were hidden inside the folded tool history.

## Risks

- **New migration:** a new table and a nullable column. It is
`online_compatible`.
- **Existing connections:** tools get `ui_meta` on the next catalog
refresh. Until then, no view shows and the tool row is unchanged.
- **Untrusted provider HTML:** the view runs as provider HTML in an
opaque-origin sandbox. It cannot reach Paperclip's session, DOM or API.
Its network access is limited to the origins its server declared. A
view's tool calls use the viewer's permissions and the normal policy and
Ask-first path.
- **Agent tool lists:** app-only tools are removed from them. No current
provider marks tools app-only, so no listed tool disappears.
- **Search:** the change that prefers a native app over an "X MCP"
aggregator listing affects every app.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
subagents (one subagent authored the Circleback catalog connector).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:28:05 -06:00
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00
DottaandClaude Fable 5.2 a4a0668a5c perf(attention): count the sidebar summary without the reads that only a card shows (#21159)
GET /attention/summary returns counts. To get them it ran the feed with a
flag that skipped card bodies, and still read every blocked task through the
full task list, every review two more times (one time with its description),
the blocker and the blocked work of each card, and the spend of every budget
policy.

The summary keeps the collection path of the feed, so a card is counted by
the rule that shows it. It no longer reads what only a card shows:

- blocked work: the rows of the task list, in list order, with only the
  columns blocker attention reads (issueService.listBlockerAttentionRoots),
  classified by the function the full list calls
- reviews: the review rows carry the card's task condition, so the summary
  does not read each review again
- tasks of cards: without project and workspace
- budget alerts: the open incidents (budgetService.openIncidents), not the
  spend of every policy
- a fingerprint only for a card that has a dismissal; no queue membership
  and no "transferred by" read

The review classification reads the 20 columns it uses, not the whole task.

The request makes the same writes as before (decision queue seeds, retention
states) through unchanged code. The feed, the task index and the task list
return what they returned before.

One summary request at 3,000 tasks: 133 -> 78 statements, 2,718 -> 842 KB
fetched, 604 -> 144 KB of SQL text, 100 -> 45 MB allocated.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 12:39:21 -06:00
DottaandClaude Opus 5.5 b8207c4ea9 fix(runner): evict finished API snapshots instead of blocking reads at the company quota (#21133)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs read the Paperclip API through `call_api`. Large
responses are saved as "API snapshot" assets so the agent can read them
in pages.
> - Each company has a 20 GiB snapshot quota. Before a large read, the
server reserves 1 GiB of worst-case space against that quota.
> - Snapshot assets were never removed. Reservations from interrupted
captures (crash, restart, ambiguous upload) stayed at 1 GiB each until
"an operator reconciles" them. Nothing did that.
> - After a morning of high retry volume, the quota filled. Every large
read then failed with `api_response_company_storage_limit`, and agents
blocked their own tasks and asked an operator to free storage.
> - This pull request makes company admission reclaim space by itself:
it releases stale reservations of finished runs and evicts the oldest
snapshots of finished runs, oldest first, only as much as the new read
needs.
> - The benefit is that a read never fails because old snapshots filled
the quota, and no person must clear storage by hand.

## Linked Issues or Issue Description

**What happened?**

On staging, about 97 tasks of one agent had successful runs but ended in
`blocked`. In each run, `GET` calls through `call_api` for large records
(for example heartbeat runs and their events) failed with
`api_response_company_storage_limit`: "Company API snapshot storage
quota is exhausted. Read an existing asset or ask the operator to remove
old snapshots or increase the quota." The agent then reported a
task-wide blocker owned by "the instance storage operator". At least 15
tasks had a final comment such as "Blocked: company API snapshot storage
is exhausted". The error message itself told the agent that an operator
must act.

**Expected behavior**

A read must not fail because old snapshots filled the company quota. The
server must reclaim the space automatically. Only snapshots that active
runs still hold, or that users attached to issues, cases or logos, may
stay.

**Steps to reproduce**

1. Set `PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES` to 2 GiB.
2. Store a 1 GiB `runner-api` snapshot asset from a finished run, or
leave two 1 GiB unattached reservations from finished runs that are
older than 20 minutes.
3. In a running native run, call `call_api` for a response larger than
the inline limit.
4. Before this change, the call fails with
`api_response_company_storage_limit`. After this change, the call
succeeds and the oldest finished snapshot is evicted.

**Paperclip version or commit**

Staging deploy `704b3f4d63`. Fix is based on `origin/master` at
`8bdd53c11f`.

**Deployment mode**

Hosted (staging cloud), native runtime with Daytona sandboxes.

**Agent adapter(s) involved**

Native Paperclip runner (Codex app server, `gpt-6.1-sol`). The defect is
in the server-side `call_api` capture admission, so it affects every
native adapter.

## What Changed

- Added
`server/src/services/native-runtime/runner-api-capture-retention.ts`. It
measures company snapshot usage and reclaims space under the existing
per-company advisory lock.
- Reclaim step 1: release unattached reservations older than twice the
capture deadline (20 minutes) whose run is missing or not `queued`,
`running` or `scheduled_retry`.
- Reclaim step 2: evict snapshot assets oldest first until the new
reservation fits. A snapshot is eligible only when it is older than 1
hour, every run that captured it has finished (succeeded, interrupted,
failed, cancelled or timed out), and no issue attachment, case
attachment or company logo links to it. Each eviction writes an
`asset.evicted` activity-log row with the asset id and the owning run
id.
- A capture upload that finishes after its reservation was reclaimed is
rejected in the publish transaction. The same transaction records the
object for deletion, and the bounded cleanup removes it. The upload has
a 10-minute abort signal.
- Failed deletes are logged per company and counted with the
`runner_api_capture_delete_failed` metric. When cleanup keeps failing,
the read error says so in plain words.
- Eviction credits deletes that are already pending before it chooses
more snapshots. Eviction is durable. In the same transaction that
deletes an asset row, it inserts a reservation row with
`pending_delete_object_key`, so the bytes stay counted. Admission then
deletes the stored objects in pages, under a 30-second abort signal, and
removes each row only after its delete succeeds. A failed or interrupted
delete is retried at the next over-quota admission. No background poller
is added.
- Added the `pending_delete_object_key` column and an index on
`runner_api_response_reservations.asset_id` (migration 0472).
- `#reserveApiResponseCapture` now runs reclaim only when the quota
would be exceeded, so reads below the quota do no extra work. After
marking evictions it finishes the deletes and retries admission once.
- When no space can be reclaimed, the error message now names what holds
the space (runs still working, files attached to tasks, deletes in
progress). It no longer tells the agent to ask an operator.
- Replaced the test that asserted old snapshots block reads with tests
for eviction, multi-page eviction, durable delete retry, a bounded
stalled delete, stale-reservation release, and protection of active and
attached snapshots.

## Verification

- Red: the three new tests in `runner-api.integration.test.ts` fail on
`origin/master` (two get `api_response_company_storage_limit`, one gets
the old "ask the operator" message).
- Green: `cd server && npx vitest run
src/services/native-runtime/runner-api.integration.test.ts` passes 70/70
with the fix.
- The multi-page test failed first with
`api_response_company_storage_limit` because pending deletes ran in one
200-row batch. The paged delete loop fixed it.
- `cd server && npx tsc --noEmit -p .` passes.

## Risks

- An agent that reads a saved snapshot of a finished run after it was
evicted gets a not-found response. It can read the API again. Eviction
happens only under quota pressure, oldest first.
- A released stale reservation can leave an unknown storage object from
an ambiguous upload. These bytes were never linked to an asset, so they
were already unreachable. This is the same storage the old code kept
counted forever.
- Eviction runs inside the company capture lock. The candidate query is
bounded to pages of 200 rows and uses the new `asset_id` index.
- Migration 0472 adds one nullable column (`ADD COLUMN IF NOT EXISTS`)
and builds one index with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It
is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. It is additive and
safe for old and new servers during a rolling deploy.
- During a storage outage, an over-quota read still fails until deletes
succeed. The bytes stay counted, so the quota is never exceeded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser reads of staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:30:38 -06:00
DottaandClaude Opus 5.5 0f3a6a867d fix(recovery): continue open agent tasks left with nothing running, pending or waiting on a person (#21139)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents work on tasks through runs, and recovery code starts the next
run when a run stops
> - On staging, many open agent tasks show "running" or "blocked", but
no run, wake or retry is pending, and no person has to act
> - Staging deploys about every 40 minutes. Each deploy stops some runs,
and some tasks then stay idle until a person presses Retry
> - The stranded sweep has many lanes. Some lanes get "no retry" back
and skip the task with no record, and the sweep never looks at `blocked`
tasks
> - This pull request adds one invariant with one pure predicate, and
one event-driven sweep that repairs a violation with one ordinary
continuation wake
> - The benefit is that no open agent task stays with nothing running,
nothing pending and no one else's move

## Linked Issues or Issue Description

Refs #21107 and #21046 (the pattern for leaving a run of another live
server alone). Related open work: #21135 (fresh continuation from saved
work at wake admission), #21124 (sweep calls the bounded scheduler for
pre-provider legacy failures), #21127 (run-start hand-over).

**What happened?**
On a staging snapshot (2026-11-12, about 14:35Z), 1,030 open agent tasks
had no live run. The new predicate, run read-only against the staging
API, classifies 141 of them as violations: 40 offer a Retry only the
system can act on, 38 are blocked with no recorded reason and no hold,
14 are blocked by an automatic recovery outcome with no hold left, 5
have nothing pending, and 44 are blocked under an active recovery hold
(left to the hold's owner, see below). 25 more are blocked by recovery
for more than 7 days. The other tasks wait on a person or on a pending
intent: 542 chat threads, 122 agent-declared blocks, 102 credential
waits, 51 dependencies, 28 questions.

Nine Retry-started `in_progress` tasks show the main gap. A person
pressed Retry, a deploy stopped the run during setup
(`server_shutdown_interrupted`, no provider work), and the bounded
scheduler then refused with `continuation_user_authorization_missing`
("could not revalidate the original user continuation") on every sweep.
The sweep's escalation did not take effect either, so each task stayed
`in_progress` with a Retry button and nothing pending, for up to 10
hours.

**Expected behavior**
Each open task assigned to an agent has a live run, a durable pending
intent, or a named reason that is someone else's move. If it has none of
these, Paperclip continues the task automatically.

**Steps to reproduce**
1. Start a person's Retry on an `in_progress` task with a legacy agent.
Stop the server during setup, so the run ends `interrupted` with
`server_shutdown_interrupted`.
2. Let the bounded retry scheduler refuse to revalidate the person's
continuation.
3. Run the stranded sweep. Nothing is queued. The task stays
`in_progress` with nothing pending.
4. The same happens for a native predecessor
(`enqueueStrandedIssueRecovery` returns `null` for every native run) and
for every `blocked` task (the sweep reads only `todo`, `in_progress` and
`in_review`).

## What Changed

- `server/src/services/recovery/assigned-work-invariant.ts`: the
invariant as one pure predicate. Facts go in. The verdict is `ok` with
its clause (live run, pending intent, or one reason from a closed list
of waits), `settling` with a re-check time, or `violation` with its
reason. An unknown hold, block reason or person action is a violation.
It also holds the per-task continuation allowance: at most 6 per hour
(the server-restart lane's bound, the same cap as #21135) and, below the
cap, a spacing of 2, 4, 8 and then at most 15 minutes. At the cap the
next try waits until the oldest continuation leaves the hour, so a task
that keeps needing continuation is retried at about 6 per hour, never
left for hours.
- `server/src/services/recovery/assigned-work-sweep.ts`: an indexed
pre-filter, a batched fact loader, the repair and a read-only report.
- A running run counts as live unless the shared owner proof
(`owner-proof.ts`) proves its server gone. A held controller lease is
live, so a run of the other server in a deploy overlap is never touched.
A run whose owner is proven gone goes to heartbeat's existing
lost-process settle path. Native running runs are always treated as
live; the native settle watch judges their owners.
- Native active paths match the stranded sweep's check, including
#21108's exclusion of a native finalization that a person's Retry
replaced.
- A run that ended less than 90 seconds ago, or whose controller lease
another server still holds, is `settling`. The sweep looks again at that
deadline.
- A pending question counts as the person's move, unless the person
pressed Retry (or sent a message) after the question was asked.
- The repair takes the task's row lock, checks again that nothing is
pending, applies the allowance and writes a plain-language activity
entry. This entry is the claim, so two servers make one wake. For a
`blocked` task with no recovery hold, it also puts the task back to the
status the block recorded. Then it enqueues one ordinary
`issue_continuation_needed` wake. The wake has no retry link, so it does
not use the task's failure allowance. With #21135, wake admission turns
that wake into a fresh run from saved work when a hold would defer it.
- A `blocked` task under an active recovery hold is reported as a
violation but not repaired here: the hold's owner (#21135) re-continues
it, so the two never continue one task twice. When #21124 escalates once
on a permanent scheduler refusal, the task is blocked under a hold, so
#21135 continues it; when no hold is left, this sweep continues it.
Neither case leaves the task waiting on a person for a reason that is
not the person's.
- The claim re-runs the full predicate on a fresh read under the task's
row lock, so a question, approval, blocker, monitor, wake or run that
appeared after the first read stops the repair.
- A lifted block is restored to the status the blocking entry recorded
(`_previous.status`, `previousStatus` or `fromStatus`). A block that
interrupted a review goes back to `in_review` with no agent wake; the
review's own recovery wakes its participant. If admission refuses the
wake, the block is put back with its original block time.
- The settling window is keyed on three events only: the latest run
ending (a held lease can extend it, but never past 5 minutes after the
run ended), a never-run task's creation, and a blocker finishing.
Routine touches (comments, reads, labels) never postpone a repair. A run
cut by a deploy's shutdown is repaired about 90 seconds after it ended.
- Each process starts at most 10 repairs per minute, newest-updated
tasks first. The other tasks wait for a deadline.
- Each burst of checks writes one structured log line
(`assigned_work_invariant.pass`) with the counts by clause, the repairs
and the deferrals.
- `server/src/services/recovery/work.ts`,
`issue-recovery-notifications.ts`, `delivery-work-notifications.ts`: a
new `assigned-work-invariant` queue on the shared delivery-work
coordinator. The existing issue-recovery notifications (run state
changes, hold resolutions) and server start mark a company dirty. There
is no interval polling. Deadlines exist only for a real obligation
(settling grace, back-off, rate-limit deferral).
- `server/src/services/recovery/service.ts`, `heartbeat.ts`: wire the
sweep with the existing wake, live-execution and settle dependencies.
- `server/src/routes/agents.ts`, `server/src/routes/openapi.ts`,
`docs/api/agents.md`: `GET
/api/companies/:companyId/assigned-work-invariant` (board only,
company-scoped) and filtered by the caller's issue-privacy condition:
private tasks the caller cannot read are absent from every count and
sample returns the counts by clause, the violation count, the held
subset, stale blocks and up to 20 samples. Use it after a deploy to
check for 0 violations.

Why there are two sweeps: the stranded sweep decides how a known failure
resumes (bounded retries, disposition repair, review recovery,
escalation). The invariant decides only that something is pending, and
repairs with the ordinary continuation. It does not change the stranded
sweep's lanes or the recovery-refusal paths.

## Verification

- `server/src/services/recovery/assigned-work-invariant.test.ts` (45
tests): every clause, every wait reason, the unknown cases, settling,
the allowance and its long back-off.
- `server/src/__tests__/assigned-work-invariant.test.ts` (17 tests,
embedded Postgres): a refused or later-declined continuation stays a
violation and is retried under the cap; the real wake queue deferring
behind a recovery hold; scheduling stops when a task is done, cancelled
or reassigned to a person; running vs stopped or blocked session goals;
the deploy-interrupted Retry shape and the native reconciliation shape
(the stranded sweep does nothing, then exactly one wake); the nine
Retry-started tasks whose scheduler refused revalidation, including one
with a question asked before the Retry; a block after a legacy run's
retries ran out with no hold left (`controller_lease_lost`); a block
under an active hold left to its owner; two concurrent sweeps make one
wake; another live server's run is not touched; an owner-gone run is
settled; approval, question, dependency and declared-unblock tasks are
not touched; an unknown block reason is repaired; the per-task cap and
the per-process budget; a run that ended seconds ago; old-shaped rows (a
board-owned stranded action from an older release, native
`awaiting_runner_reattach` and `retryable_failure`, a running run saved
without its controller process, a block older than 7 days); query cost
at staging size.
- Query cost at staging size (800 open tasks, 14,000 runs over 7 days):
one suspect page 2 ms, one task check 165 ms (including the repair), a
full 800-task report 386 ms. The suspect anti-join uses the run status
index (no sequential scan on `heartbeat_runs`, 0.2 ms).
- Route test in `agent-live-run-routes.test.ts`: a board operator reads
its own company; another company and an agent get 403.
- Existing production-wiring suites pass with the new queue:
`issue-recovery-deadlines`, `issue-recovery-conversations`,
`live-recovery-work`, `issue-recovery-work-lifecycle`,
`recovery-deadline-acceptance`, `server-startup-feedback-export`,
`heartbeat-held-comment-wake`, `delivery-work-coordinator`.
- `node scripts/typecheck-layers.mjs` for the server source layers
passes.

## Risks

- On the first deploy, the sweep continues about 97 staging tasks over
about 10 minutes (10 per minute). Each continuation is an ordinary run.
- The stranded sweep can still move a task to `blocked` after its
bounded retries. The task is then continued by the hold's owner or by
this sweep, within the same cap and back-off. The task gets more
activity entries than before.
- On current master without #21135: when the wake reaches a task whose
last run left a recovery hold, admission defers it (a skipped wake, no
pending row). The activity entry then says the continuation did not
start, the task stays a violation, and the sweep tries again with
spacing (2, 4, 8, then 15 minutes at most; at most 6 per hour). It never
loops faster than the cap and never moves the task to blocked for a
person. A test with the real wake queue covers this. With #21135, the
same wake becomes a fresh run from saved work.
- The activity entry is written as an attempt and is published only
after admission: as queued, or as not started. A wake that the queue
declines later (for example behind the old run's cleanup) leaves the
task a violation, and it is retried under the cap.
- Chat-thread tasks are left out (the conversation clause), because
their next move is a person's message.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:30:14 -06:00