Commit Graph
7677 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 df89950eb6 fix(native): save or set aside a superseded run's unsaved workspace so its task queue moves (#21136)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result. Then a finalizer exports the
run's workspace files and applies the result to the task.
> - When a newer run already decided the task, the finalizer retires the
older result as superseded. Since #21106, this also ends the run.
> - If that retirement interrupts a failed workspace export, the run's
files stay unsaved. The run also stays the writer of the task's
workspace. Nothing records those files for recovery, so nothing ever
releases the workspace.
> - Every run queued on the task then waits behind a save that never
settles. After one hour, each queued run fails setup instead.
> - This pull request records those files for the existing automatic
workspace recovery. Recovery saves them from the retained sandbox, or,
when every bounded attempt fails, continues the task from its last save
with the files attached.
> - The benefit is that the task's queue moves again by itself, and the
run's unsaved work is saved or kept findable. The superseded result is
never applied.

## Linked Issues or Issue Description

Refs #21106

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. Its workspace export failed once with
`remote_workspace_archive_fs_eacces`. The finalizer then retired it as
`native_finalization_superseded`, because another run had already
decided the task. #21106 ended that run at 14:59:17Z (`failed`). The
task still could not move. The next queued run was cancelled with
`workspace_busy` ("ended, but its workspace save has not settled"). Its
retry re-armed every minute. Four more runs stayed queued. The task
showed no recovery action.

**Expected behavior**
The ended run's unsaved files are saved from its retained sandbox, and
the workspace is released. If they cannot be saved within the bounded
retries, the task continues from its last committed save, and the files
are attached to the task. The queued runs proceed in both cases. The
superseded result is never applied.

**Steps to reproduce**
1. Accept a native result for a run that writes a remote workspace.
2. Let the workspace export fail once, while a newer run has already
committed the task's decision.
3. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is `failed`.
4. See that the workspace still names the run as its dirty writer, and
that no retained-file recovery action exists.
5. Queue another run on the task. It defers with `workspace_busy`, and
after one hour it fails setup with "The workspace still has an
unfinished writer or save operation."

**Root cause**
`recordRetryableFailure` (`native-run-finalizer.ts`) retires a
superseded finalizer without preserving the export.
`preserveNativeFailedWorkspace` records a retained-file hold for an
accepted result only when the failure code is
`native_workspace_sync_out_retry_exhausted`. A
`native_finalization_superseded` coordinator never qualifies.
`readRemoteWorkspaceWriterHold` keeps an ended dirty writer as the
holder until its save settles. Without a hold, no worker ever settles
it.

**Paperclip version or commit**
Staging Core with #21106 (`81d2e4d8d9`). Master `ef2b53b675` has the
same behavior.

## What Changed

- `native-superseded-workspace-export.ts` (new): identifies a superseded
finalizer that failed in the workspace export. It selects ended runs
that still hold their workspace with unsaved files and no retained-file
hold. It records the hold through `preserveNativeFailedWorkspace`. It
checks again only while a just-ended run's sandbox stop receipt is still
arriving (at most one hour).
- `native-failed-workspace-recovery.ts`: `preserveNativeFailedWorkspace`
treats a superseded workspace export like an exhausted copy-back, with
the same proofs (run ended, no live controller, accepted result,
confirmed sandbox stop-and-retain, source still bound to the workspace).
It marks the hold `supersededWorkspaceExport`.
- `native-run-projection-work.ts`: the native run projection worker also
selects these runs. It loads them at startup and wakes when a finalizer
changes. Rows written by older releases are therefore settled after
deploy without a person.
- `workspace-recovery-fallback-discovery.ts` and
`workspace-recovery-fallback.ts`: a marked hold whose bounded re-export
is spent (at most 5 retries in 30 minutes) is eligible for the existing
continue-from-last-save fallback. That fallback sets aside only this
run's workspace epoch and keeps its location on the workspace record. It
attaches the readable files to the task unapplied. The task message says
Paperclip "could not save" the previous run's unsaved files.
- `preserved-result-finalization.ts`: a superseded result is never
resumed for application, even after its files are saved.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/services/superseded-workspace-export-recovery.test.ts`
(new). The first two tests fail on master and pass on this branch:
- "records the files for automatic recovery, saves them from the
retained sandbox and frees the workspace". It uses the staging row
shape. The projection worker records the hold, and a second pass changes
nothing. The workspace waits for that save. The system recovery saves
`saved.txt` into a committed snapshot. The workspace becomes free. The
finalizer, the result and the task's decision do not change, and the
result is not resumed. On master the worker selects nothing (`expected 0
to be 1`).
- "continues from the last save with the files attached when every
re-export fails". After an exhausted re-export budget, the hold is
resolved `restored`. The files are attached (`quarantine.state =
captured`). The workspace moves to epoch 2 with no writer, and the
abandoned epoch keeps its lease location. The message has no request for
a person to fix anything.
- "leaves a superseded result alone when it never failed in the
workspace export".
- Related suites pass: `native-run-projection-work`,
`native-finalization-recovery`, `workspace-recovery-fallback`,
`native-failed-workspace-recovery`, `workspace-recovery-retry-copyback`.
- `tsc --noEmit` for `server`.

## Risks

- Low to medium risk. The change acts only on runs whose finalizer is
already terminal and superseded, that failed in the workspace export,
and that still own a dirty workspace. All existing preservation proofs
still apply.
- The saved files of a superseded run become the workspace's next saved
version. The run's result (the task decision) is still never applied.
- When the export can never succeed, the task continues from its last
save. Files that cannot be read are not attached. The workspace record
keeps the set-aside epoch's location. Saving the readable part of a
partly unreadable workspace is not in scope.
- Runs that already failed setup before this deploys are not restarted
by this change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:20:54 -06:00
Nicky LeachandPaperclip 782d6bead4 Configure managed plugin Runner lifecycle (#21065)
## Thinking Path

> - Paperclip runs agents in managed environments.
> - An agent can use an explicit environment or the instance default.
> - A lifecycle plugin must receive the driver that provisions the
agent.
> - A managed plugin environment also needs to select the Runner process
lifecycle.
> - This change gives lifecycle selection and provisioning the same
environment lookup.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Managed plugin environment provisioning and lifecycle configuration.

**Current behavior**
Lifecycle calls lack the selected plugin driver. Managed plugin
configuration cannot select warm Runner execution.

**Proposed behavior**
Resolve the environment once per lifecycle operation. Pass the driver
key for the plugin that serves it. Parse and persist the managed Runner
lifecycle mode.

The portable ownership and workspace placement changes are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066) and
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Share agent environment resolution between required plugin selection
and the lifecycle RPC.
- Pass `environmentDriverKey` to the plugin that serves the selected
environment.
- Allow managed plugin environments to set `warm` or `per_turn` and
persist the selected mode.
- Advertise `pluginEnvironmentRunnerLifecycleMode` in the release
capability manifest so senders can check support before including the
field.

## Verification

- Built shared types, the plugin SDK, and Runner. Server typecheck and
SDK source-pin check passed for the initial layer.
- The capability contract test failed before the manifest fix and passed
after it. Independent verification passed all 160 tests across managed
configuration, plugin environment persistence, lifecycle routing, and
environment reconciliation.
- Routing coverage includes the instance default, an explicit second
driver from the same plugin, and an explicit sandbox.

- [Hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/43193311862)
passed on `f07b192c21`. The first attempt timed out in an unchanged
workspace-recovery fixture and then collided with a background recovery
action. The exact test and all 44 tests in its suite passed locally; the
isolated shard retry passed without code changes.

## Risks

- Existing managed configuration can omit the lifecycle mode. Senders
must check the target release for `pluginEnvironmentRunnerLifecycleMode`
before sending the new field. Publishing this signal does not update
external deployment tools.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 08:18:04 -08:00
DottaandClaude Fable 5.2 1af987690d perf(storage): hash and encrypt workspace snapshots off the server thread (#21138)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process serves every request and agent turn on one
thread. The same process saves each remote workspace as one encrypted
archive in object storage, and restores it before a turn
> - `server/src/services/remote-workspace-snapshot-storage.ts` hashed,
encrypted and decrypted every byte of those archives on that thread.
`server/src/storage/s3-multipart.ts` computed the Content-MD5 of each
uploaded part on that thread too
> - Hashes and ciphers are synchronous in Node. A CPU profile of a busy
hosted server showed this work on the main thread, which was at 93% of
one core
> - Each megabyte of a snapshot delays every request and turn that waits
behind it
> - This pull request moves those passes to a small, bounded set of
worker threads. The checks, their order, their failures and the stored
format do not change
> - The benefit is that a 1 GiB save uses about 0.7 s of the server's
thread instead of about 2.8 s, and an unrelated request during the save
is answered in 1.3 ms (p99) instead of 16.7 ms

## Linked Issues or Issue Description

No public issue exists. I searched open and closed pull requests for
`workspace snapshot hash encrypt`, `event loop` and `archive io`.
Related merged work: #19900 gave these stream passes 1 MiB buffers,
#20554 removed a read-back from migration, #20722 took runner artifact
hashes off the event loop, #20709 records long event-loop blocks. No
open pull request covers this change.

**What happened?**

A 30 second CPU profile of the main thread of a hosted server (commit
1d153f62b6, Node 24, main thread at 93% of one core) showed:

- the two stream transforms in the snapshot save at 3.9% and 1.6%
inclusive. The first hashes the archive (SHA-256). The second hashes the
ciphertext (SHA-256);
- native hash `update` at 5.4% self time for the whole process. About
1.0 s of it was under those two transforms and 0.25 s was under
`fileRangeMd5`;
- the AES-256-GCM cipher of the same pipeline, and Buffer allocation
under the file read streams. Garbage collection was 25% of the thread.

What the code did with each byte, on the server's thread:

| Path | Pass | Digest or cipher | Purpose |
| --- | --- | --- | --- |
| Save | read archive | SHA-256 of plaintext | bound into the wrapped
key, checked on each restore |
| Save | same read | AES-256-GCM | confidentiality and authenticity |
| Save | same read | SHA-256 of stored bytes | recorded on the snapshot
row, checked on each restore and read-back |
| Save | read each part (S3) | MD5 | `Content-MD5`: the object store
rejects a part it did not receive intact |
| Save, read-back callers | read the kept file | SHA-256 of plaintext |
proof that the published file is the saved archive |
| Restore | download | SHA-256 of stored bytes | integrity of the object
against the snapshot row |
| Restore | read ciphertext | AES-256-GCM | authenticity |
| Restore | read plaintext | SHA-256 of plaintext | integrity against
the digest in the wrapped key |
| Restore | read plaintext again | none (copy) | publication beside the
destination |

**Expected behavior**

Digest and cipher passes over a snapshot do not run on the thread that
serves requests. Every check stays, with the same failure.

**Steps to reproduce**

1. On the parent commit, run `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-workspace-snapshot-crypto.ts --mib=1024` (copy the
script from this branch).
2. Read `upload.mainThreadCpuMs`, `upload.mainThreadBusyMs` and
`upload.unrelatedRequest`. The benchmark uses the real save and restore
functions, the real S3 client, and a local object store that checks each
part's `Content-MD5`. A separate process requests a trivial route from
the measured server and times each reply with its own clock.

**Paperclip version or commit**

`master` at 9e910ff9ae. The profile is from 1d153f62b6.

**Deployment mode**

All modes that save remote workspaces. The profile is from a hosted
server with S3 storage.

**Node.js version**

Profile: Node 24. Local benchmark and tests: Node 26.4.0 on darwin
arm64.

## What Changed

- `server/src/storage/file-crypto-pool.ts` (new): runs file digests and
AES-256-GCM passes on worker threads.
- Bounds: at most 2 jobs run at once. Each worker has two reused 1 MiB
read buffers and one write in flight. A waiting job holds file names and
byte ranges, not file contents. Waiting jobs are served in order.
- Cancellation: a waiting job is cancelled at once. A running job stops
at its next chunk, closes its files, and then the caller sees the same
`AbortError` as before.
- Shutdown: an idle worker does not keep the process alive. A busy
worker does, until its job ends or its caller's signal aborts it.
- Key handling: the pool copies the data key for the worker, moves that
copy (it is not cloned), and the worker erases it. A copy that never
reaches a worker is erased on this thread. The caller keeps and erases
its own key as before.
- The worker program is sent as source text, as
`remote-workspace-snapshot-archive.ts` already does. It needs no extra
file in the build or the runtime bundle.
- Save (`uploadWorkspaceSnapshot`): the one read that digests the
archive, encrypts it and digests the stored bytes runs in a worker. The
identity check around that read (`archive_changed`) stays on this thread
and is unchanged.
- Part upload (`fileRangeMd5`): the `Content-MD5` of each part is
computed in a worker. The first round of parts is digested before any
part is sent, so the parts still leave together and a rate limit still
answers one burst (the existing throttle test covers this). Every digest
of that round has ended, also after a failure or a cancellation, before
the upload settles.
- Restore (`loadWorkspaceSnapshotArchive`): the download no longer
hashes on this thread. One worker digests the downloaded object while
another decrypts it into private scratch. The results are then checked
in the previous order: object size and digest, header, key, GCM tag.
- Restore, one pass less: the decrypted archive was read twice (digest,
then copy). It is now digested while it is copied beside the
destination, and linked to the destination only when the digest matches.
The digest covers exactly the published bytes.
- Read-back callers (`verifiedArchivePath`): the digest of the kept file
runs in a worker, through the same open descriptor. The file identity
checks are unchanged.
- `server/scripts/benchmark-workspace-snapshot-crypto.ts` (new): the
reproduction and benchmark.
- Docs: `doc/run-log-events.md` (what each `storage.snapshot.*` phase
now does) and `doc/DEVELOPING.md` (the worker threads and the
benchmark).

Module boundaries (`rules/module-boundaries.mdc`): the pool is in
`server/src/storage/` because both consumers are parts of the same
upload, and storage must not import from services. Its contract is file
paths, byte ranges, an algorithm, and AES-GCM parameters. The snapshot
format (magic bytes, additional data, key wrapping, the order of checks)
stays in the snapshot service. The part layout and `Content-MD5` stay in
the S3 module.

Not changed, and why:

- The stored format, the ledger, attempt journaling, atomic publication,
and `recoverWorkspaceSnapshotUpload` /
`recoverWorkspaceSnapshotArchive`.
- `verifyObject` (full read-back for a store that did not verify the
upload, and for a retried save) still hashes on this thread. Its bytes
arrive from the network on this thread. To move it, the object must
first be written to disk.
- Socket writes of part bodies stay on this thread. I measured 1 MiB
part-body read chunks in place of 64 KiB: main-thread time did not
change and the longest event-loop delay increased, so I did not keep it.
- `remote-workspace-transport.ts` also hashes an archive on this thread.
It is outside the snapshot storage path.

## Verification

Regression tests, red then green (`cd server && pnpm exec vitest run
src/services/remote-workspace-snapshot-storage.test.ts
src/storage/file-crypto-pool.test.ts`):

- Parent implementation with the tests of this branch: 3 tests fail.
- `keeps encrypting a save while the server thread is occupied by other
work`: `expected 4194323 to be 25166144`. The test blocks the thread
until the encrypted file is complete. On the parent the file stops at 4
MiB.
- `answers unrelated requests while a save is being encrypted, and
digests or encrypts none of it on the server thread`: `expected 8388627
to be less than 65536` (bytes hashed on the thread).
- `digests and decrypts none of a save or a restore on the server thread
when the store verified the upload`: `expected 33556772 to be less than
65536`.
- This branch: 58 of 58 and 6 of 6 pass.
- The tests use synchronization, not time. A named pipe keeps a save in
the encrypt step for as long as the test feeds it. Deadlines are 30 s
and matter only for a failure.
- Integrity tests that pass on both the parent implementation and this
branch, which shows that the failures are the same: a truncated object
(`object_hash_mismatch`), an oversized object (`object_size_mismatch`),
an archive that grows, shrinks or is rewritten in place after its
identity was read (`archive_changed`, before any upload), a ciphertext
that was changed with the recorded digest changed to match (GCM
failure), a plaintext that differs from the digest in the key
(`plaintext_hash_mismatch`, staged copy removed), and the order object,
header, key.
- Cancellation: during encryption the save rejects with `AbortError`,
and leaves no scratch file, no object and no recorded attempt. During
the store write it leaves no object.
- Part digests (`src/__tests__/storage-s3-multipart.test.ts`): `leaves
no part digest running or waiting when the source is cancelled|removed
during the first round`. Before the review fix in a354242865 it failed
with `{ running: 2, waiting: 0 }` at multipart cleanup.
- Pool tests: digests equal single-threaded digests for a file, a range
and a descriptor; encrypt and decrypt round trip; each damaged
ciphertext is reported; at most 2 jobs run and the others wait in order;
a waiting job and a running job are both cancelled.

Benchmark, same machine, 5 alternating pairs (parent implementation,
then this branch) for each size. Values are medians, with the range in
parentheses.

Command: `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-workspace-snapshot-crypto.ts --mib=1024` (and
`--mib=200`).

1 GiB archive:

| Measure | Save, before | Save, after | Restore, before | Restore,
after |
| --- | --- | --- | --- | --- |
| Main-thread CPU time | 2,834 ms (2,717 to 3,007) | 742 ms (658 to 881)
| 1,602 ms (1,378 to 1,837) | 510 ms (470 to 589) |
| Main-thread busy time (event-loop utilization) | 2,938 ms | 858 ms |
1,673 ms | 597 ms |
| Total process CPU time | 6,735 ms | 4,762 ms | 5,564 ms | 3,306 ms |
| Wall time | 3,914 ms (3,656 to 4,253) | 2,580 ms (2,507 to 3,200) |
2,502 ms (1,756 to 2,922) | 1,799 ms (1,508 to 2,492) |
| Longest event-loop delay | 12.6 ms | 6.0 ms | 10.4 ms | 5.4 ms |
| Event-loop delay p99 | 9.2 ms | 2.0 ms | 2.1 ms | 1.8 ms |
| Unrelated request p99 (p50 is 0.2 to 0.3 ms in all runs) | 16.7 ms |
1.3 ms | 4.7 ms | 1.3 ms |
| Unrelated request, slowest | 18.5 ms | 4.5 ms | 10.0 ms | 3.5 ms |
| Peak resident memory | 495 MiB | 408 MiB | 524 MiB | 476 MiB |

200 MiB archive:

| Measure | Save, before | Save, after | Restore, before | Restore,
after |
| --- | --- | --- | --- | --- |
| Main-thread CPU time | 571 ms | 181 ms | 378 ms | 141 ms |
| Main-thread busy time | 602 ms | 236 ms | 395 ms | 167 ms |
| Total process CPU time | 1,235 ms | 935 ms | 1,385 ms | 840 ms |
| Wall time | 842 ms | 679 ms | 538 ms | 382 ms |
| Longest event-loop delay | 12.4 ms | 4.0 ms | 7.3 ms | 2.9 ms |
| Unrelated request p99 | 16.5 ms | 2.0 ms | 1.9 ms | 1.9 ms |
| Peak resident memory | 437 MiB | 398 MiB | 510 MiB | 424 MiB |

How to read these numbers:

- The parent did not block the thread for seconds at one time. It worked
in 1 MiB slices, so the longest single delay was about 12 ms. The cost
was the sum: the thread was busy for about 75% of a 1 GiB save. That sum
is what the profile showed, and it is what decreases.
- A worker thread moves CPU time; it does not remove it. Total CPU time
also decreased here (about 29% for a save, 41% for a restore) because
the workers reuse two buffers in place of one new 1 MiB Buffer for each
chunk, and a restore reads the decrypted archive once in place of twice.
- Wall time decreased because the part digests no longer wait for the
thread that sends the parts, and a restore digests and decrypts at the
same time.
- What remains on the server's thread is network and file streaming:
socket writes of the parts, the download, and the database calls.
- Limits: the machine was shared with other work (1-minute load average
8 to 14 on 18 cores) and timings varied between runs, which is why the
pairs alternate. Earlier runs on a busier machine (load 15 to 28) showed
the same CPU-time ratios with wall times up to 4 times longer for both
implementations. The object store was on loopback without TLS.

Other checks:

- `cd server && pnpm exec vitest run src/storage
src/services/remote-workspace-snapshot-storage.test.ts
src/services/remote-workspace-snapshot-reuse.test.ts
src/__tests__/remote-workspace-file-resources.test.ts` and the 8
`src/__tests__/storage-s3-*.test.ts` files: 249 tests in 15 files pass.
- `tsc --noEmit` over the changed TypeScript files: clean. `pnpm
check:module-boundaries`: pass.
- The pool, bundled with the options of
`server/scripts/bundle-runtime.mjs` and run with plain `node`: encrypt,
decrypt and digest round trip, and the process exits.
- `pnpm verify` exits 2 (broader verification required, because of the
new file in `server/scripts`). Required CI supplies that coverage.
- Not verified: the hosted server after deploy, Linux x86 with Node 24
outside CI, a real object store with TLS, and archives larger than 1
GiB.

## Risks

- This is integrity and confidentiality code. The mitigation is that
each check and its failure is pinned by a test that passes on the parent
implementation and on this branch, and that the format is unchanged, so
objects saved before and after this change restore with either version.
- Order of work in a restore: the ciphertext is now decrypted into
private scratch while the object digest is computed, not after it. No
output is used before the digest, header, key and tag checks pass, and
the scratch directory is removed on every failure. A damaged object is
reported with the same error, but after the decrypt work.
- The staged copy beside the destination (`.workspace-snapshot-*.tmp`)
is now written before the plaintext digest is known, and after the GCM
tag is verified. It is removed on a mismatch. The destination itself is
linked only after the digest matches, as before.
- Resource trade-off: up to 2 more threads, each with about 2 MiB of
buffers and a small heap (64 MB limit), that stay for the life of the
process once used. In exchange the server's thread does about 70% less
work for each snapshot. With more than 2 snapshot passes at one time,
the others wait. A part digest can wait behind a long encryption.
- Run-log timing shifts: `storage.snapshot.digest` is now about 0 ms,
`storage.snapshot.verify_plaintext` includes the copy, and
`storage.snapshot.publish` is only the link. `doc/run-log-events.md`
describes this.
- A worker that stops unexpectedly fails its job with
`file_crypto_worker_exited`. The save or restore then fails as for any
other I/O error, and the next job starts a new worker.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code with
tool use (shell, file edits, local test and benchmark runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 10:11:37 -06:00
DottaandClaude Opus 5.5 fb8b61bc3e fix(native): restart a bootstrap whose closing server staged files but never launched its Runner (#21134)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. A release swap stops the old server and a new server
recovers each run.
> - When the old server stops during a run's bootstrap, the new server
restarts that bootstrap on the same run (`bootstrap_incomplete`). The
claim is granted only when the Postgres controller checkpoint proves
that no provider turn was sent.
> - The old server can stop after it staged launch files (Codex home,
config) into the sandbox session folder, but before it launched the
Runner. Its detach fence stops it there.
> - The new server then finds no Runner state in the sandbox. A fresh
bootstrap must create the session folder itself, and the folder already
exists, so the run failed with `runner_harness_state_mismatch:
resumed_unavailable`. The failure is an ambiguous session, so the task
waits for a person.
> - This pull request lets host preparation prove that no Runner was
ever launched in that folder, move the folder aside, and restart the
bootstrap.
> - The benefit is that a release swap during this part of a bootstrap
no longer blocks the task. No provider work can be lost or replayed,
because no Runner ever ran.

## Linked Issues or Issue Description

Refs #21120 (same release-swap class: the Runner was launched and
connected before the old server stopped; that pull request stops it in
host preparation. This pull request covers the case where no Runner was
launched.)
Refs #21079 (launch-time fence for a stale Runner on the same state
folder.)

**What happened?**
On hosted staging, during a release swap on 2026-11-12, a Codex run on
Daytona failed with `native_session_interrupted` and
`runner_ingress_unavailable: runnerd process failed:
runner_harness_state_mismatch: resumed_unavailable`. The old server was
in `harness_state.launch_material` when its shutdown drain detached the
run. It recorded no Runner process identity. The new server chose
"Restarting an incomplete native runner bootstrap on the same heartbeat
run". Its host preparation found the sandbox resumed and no Runner
state. Then the claim of the session folder failed, because the old
server had already created that folder. The run needed a person.

**Expected behavior**
The run restarts its bootstrap on the same sandbox and continues without
a person. A provider turn is never replayed and two Runners never share
the run.

**Steps to reproduce**
1. Start a Codex run on a Daytona sandbox.
2. Stop the server after the bootstrap stages launch files into the
sandbox session folder and before it launches the Runner.
3. Let a new server recover the run with a `bootstrap_incomplete` claim.
The new server has no local controller root for the run.
4. Host preparation fails with `runner_harness_state_mismatch:
resumed_unavailable`.

**Paperclip version or commit**
Staging release `704b3f4d63`. The code is the same on master at
`ef2b53b675`.

**Deployment mode**
Hosted staging, remote sandbox execution on Daytona.

**Agent adapter(s) involved**
Paperclip Runner with Codex (`codex_app_server`).

## What Changed

- `native-session-executor.ts`: under a same-run `bootstrap_incomplete`
claim with no failover backup, when host preparation finds no Runner
state, a new check asks the sandbox in one command whether the session
folder exists and holds no Runner trace. A Runner trace is the live
`runner/runner-state.json`, the `runner/runner-process.identity` launch
marker (the launch script writes it before the Runner starts), an
`authority-epochs` archive, or any other `runner-state.json` or launch
marker anywhere under the folder. If the folder holds no Runner trace,
no Runner was launched there, so host preparation records the restart
proof (`absent_confirmed`). It then uses the existing path: move the
folder aside (it is kept for inspection), claim a new folder, and
bootstrap again. If the folder is missing, the existing fresh-claim path
runs unchanged. If any Runner trace exists, the run stays fail-closed as
before. An archived session is therefore never discarded.
- `native-session-executor.test.ts`: new
`interrupted_bootstrap_unlaunched` cases. Resumed and created sandboxes
restart the bootstrap. A launch marker, an archived authority epoch,
Runner state elsewhere in the folder, or a failover backup keeps the run
fail-closed with `resumed_unavailable`.

## Verification

- Red: with the source change reverted, `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session" --maxWorkers=2` fails on
both `interrupted_bootstrap_unlaunched` restart cases with
`runner_harness_state_mismatch: resumed_unavailable`, which is the
staging error.
- Red: without the archived-state guard, the `_archived` and
`_nested_state` cases fail (the folder is moved aside). With the guard
they pass.
- Green: the same command passes (64 tests).
- `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts
--maxWorkers=2`: full file, 1422 passed before the guard commit. The
focused 64 pass on the final revision.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.
- `git merge-tree` against #21120's head (`0d18f18928`) merges without
conflict. The new check runs only when no Runner state exists, so
#21120's new call site (it runs only when a Runner state exists) gets
the same results as before.

## Risks

- Low. The new path runs only in host preparation, under a
`bootstrap_incomplete` claim for the same run id, with no failover
backup. That claim is granted only after the previous controller's lease
has ended. With no Runner state or launch marker anywhere in the folder,
no Runner and no provider session existed there. The old folder is moved
aside, not deleted.
- A controller that is still launching at the same moment would have to
hold a lease that the claim already took over. The detach fence stops
such a launch, and #21079 adds a launch-time fence for a leftover Runner
on the same state folder.
- No added wait before a normal turn starts: the check runs only on this
failure path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser for read-only run evidence, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:06:55 -06:00
DottaandClaude Fable 5.2 5b698b68cf perf(run-events): append a native event in one transaction of two statements (#21131)
One appended native run event cost eight statements on the database
connection (BEGIN, SAVEPOINT, run lock, replay probe, sequence update,
insert, count/max read, COMMIT). The query builder built five of them
again for every event, and each event created two transaction objects.

A native source event now takes the run lock and then runs one statement
that finds a replayed event or reserves the sequence and inserts, and
reads the retained count and maximum for the acknowledgement. The lock
stays a statement of its own so the append reads a snapshot taken after
the lock was granted. The control-plane port passes its controller check
to the append instead of opening an outer transaction, so there is no
SAVEPOINT. The hot statements are fixed SQL text.

Events that wake a queue still register the wake before the write and
only when the event is new. Order, sequence numbers and durability do
not change: every event still commits before its append returns.

Local benchmark (server/scripts/benchmark-run-event-append.ts), one
native event: 8.3 -> 4.6 statements, 589 -> 222 KB allocated,
1.09 -> 0.53 ms CPU, 28 -> 11.5 garbage collections per 1,000 events.
One ordinary event: 105 -> 27 KB allocated.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 09:58:31 -06:00
DottaandPaperclip 5c953c1c1b fix(ui): keep chat pins first and allow drag reordering (#20830)
Keep pinned chat agents first in saved order. Add pointer and keyboard reordering with shared navigation pin controls. Preserve focus and hidden pin positions.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-12 09:49:17 -06:00
DottaandClaude Opus 5.5 9e910ff9ae fix(chat): keep raw IDs and internal codes out of chat replies (#21081)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and the agent replies in plain
text
> - In a full chat-eval cohort, 47 of 158 passing replies showed raw
UUIDs, error codes, field keys, tool names or server paths
> - Users must not see internal jargon or raw IDs, so these replies are
a product defect, even when the facts are correct
> - Agents copy these terms from API error messages and saved fields;
the chat directive did not tell them to translate
> - This pull request fixes the sources: the chat directive, and API
denial messages that put IDs and policy codes in the human message
> - The benefit is plain replies for every model, while machine codes
stay available in separate fields

## Linked Issues or Issue Description

**What happened?**

Chat replies quote internal names and raw IDs. Examples: run, handoff,
revision, interaction and trigger UUIDs in prose;
`native_session_retry_exhausted`, `board_access_required`,
`review_policy_denied`, `local-board`, `enableCases`; a raw user ID
inside the inbox denial message.

**Expected behavior**

Replies name records the way the person sees them (task identifier,
title or name, with a link) and explain codes in plain words. IDs stay
only in link targets, in commands the person must run, or when the
person asks for an ID.

**Steps to reproduce**

Run the chat-features OpenCode cohort (for example `cases.close-case`,
`routines.webhook-trigger`, `inbox-search.archive-without-grant`) and
read the agent replies.

**Paperclip version or commit**

`master` at f30005440b.

**Agent adapter(s) involved**

Not adapter-specific (core bug). Seen with the Paperclip runner and
OpenCode.

**Deployment mode**

Local dev server (embedded PGlite) and the CI chat-eval runner.

**Installation method**

Source checkout (`pnpm dev`).

## What Changed

- `AGENT_CHAT_DIRECTIVE`: add one rule. Name records by their
identifier, title or name and link them. Do not write raw IDs, internal
codes, field keys, tool names, process names or server paths, even in
inline code. The rule has exceptions for link targets, commands the
person runs, and IDs the person asks for. The rejection rule now asks
for the platform's reason "in plain words".
- Authorization denials (inbox management, membership, responsible user)
no longer put user or agent IDs in the explanation. The `reason` and
`code` values do not change.
- Review-policy denials explain the rule in plain words. The
`review_policy_denied` code and the `policy` and `allowedActor` fields
do not change.

## Verification

- Red/green: `server/src/__tests__/agent-conversations.test.ts`
(directive rule, and it reaches the chat task prompt),
`server/src/__tests__/authorization-service.test.ts` (inbox denials
carry no IDs), `server/src/__tests__/issue-review-policy.test.ts` and
`server/src/services/issue-thread-interaction-resolution.test.ts` (plain
review messages). Each new assertion failed before the fix.
- `pnpm verify` and the server typecheck pass locally.
- A paid OpenCode run on 8 affected chat cases is in progress; results
will be added here.

## Risks

- Low risk. Only human-readable messages and prompt text change. No
client matches the old message text; codes and machine fields stay the
same.
- The directive grows by one paragraph. A model can still copy an ID;
the rule reduces this but cannot prevent it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:37:07 -06:00
DottaandClaude Fable 5.2 8bdd53c11f perf(server): start no git process for server info outside a checkout (#21122)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reports which commit it runs. `GET /api/health` and `GET
/api/liveness` return it, and the board UI and the hosting control plane
poll these routes
> - The commit comes from `getServerInfoSnapshot()` in
`server/src/server-info.ts`. The health route calls it on each request.
The snapshot keeps its Git metadata for 3 seconds, then reads it again
with `execFileSync("git", ["show", ...])`, which is a synchronous child
process
> - A container or packaged install has the `git` command but no
checkout. There `git show` can only fail, and the server then uses the
commit that the build supplied. The failed start still stops the event
loop for the length of a process start
> - A CPU profile of a busy hosted server showed this cost on the main
thread
> - This pull request looks for a checkout before it starts `git`.
Without a checkout the server takes the build-commit path directly
> - The benefit is that health polls start no child process on a server
that does not run from a checkout, and the responses stay the same

## Linked Issues or Issue Description

No public issue exists. I searched open and closed pull requests for
`server-info git health`, `readGitInfo` and `getServerInfoSnapshot`.
Related merged work: #8753 added the 3 second refresh so that the debug
view follows a checkout, #10563 and #10566 added the build commit,
#18649 took the process start time off the Git probes. No open pull
request covers this change.

**What happened?**
`server/src/app.ts` creates the health routes without a `serverInfo`
option, so each `GET /api/health` request calls
`getServerInfoSnapshot()`. When the 3 second cache is expired, the call
starts `git show -s HEAD`. On an install without a checkout the command
fails each time, so the failure is paid again on each refresh. The rate
is one `git` start for each request when requests are 3 seconds or more
apart, and one start for each 3 seconds when requests are more frequent.
It is not one start for each request in a burst.

A 30 second CPU profile of the main thread of a hosted server (commit
1d153f62b6, Node 24, main thread at 93% of one core, process about 4 GB
resident) showed `getServerInfoSnapshot` → `readGitInfo` → the `git
show` command at 0.6% inclusive. That is about 190 ms in 32 seconds,
which agrees with one failed start for each 3 second refresh. The
profile showed only the `git show` command under `readGitInfo`, not the
status or branch commands that run after a successful `git show`. The
native child-process spawn was 0.9% self time in the full profile; other
callers also start processes.

**Expected behavior**
A server that does not run from a Git checkout starts no `git` process
for health or liveness requests. It reports the build commit, as before.
A server that runs from a checkout continues to read Git after each 3
second refresh, because the commit and the local changes of a checkout
can change while the process runs (#8753).

**Steps to reproduce**
1. Run `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-health-server-info.ts --requests=200` on the parent
commit. The benchmark puts a counting `git` wrapper first on `PATH`. The
wrapper records the start and then runs the real `git`.
2. Read the row `packaged install: no checkout, build supplies the
commit`, pattern `polled every 5000 ms`. It shows 200 `git` execs for
200 health requests, and 1 exec when the routes are created.

**Paperclip version or commit**
`master` at ef2b53b675. The profile is from 1d153f62b6.

**Deployment mode**
All modes. The profile is from an `authenticated` hosted server that
runs the container image. The image installs `git`, and the build
context excludes `.git` (`.dockerignore`).

**Node.js version**
Profile: Node 24. Local benchmark and tests: Node 26.4.0 on darwin
arm64.

## What Changed

- `server/src/server-info.ts`: the default `git show` command first
looks for a checkout. A checkout is a `.git` entry in the working
directory or in one of its parents, or a `GIT_DIR` value. Without one,
the command throws before it starts a process, and `readGitInfo` takes
its existing build-commit path. The same check applies when
`/api/liveness` reads the snapshot at route creation.
- The 3 second refresh, the checkout path, the injected-command path for
tests and `processStartedAt` are not changed.
- `server/src/__tests__/health-server-info-git-probe.test.ts` (new):
regression tests that count the `git` processes that the server starts.
The tests use real Git and real directories.
- `server/scripts/benchmark-health-server-info.ts` (new): the
reproduction and benchmark.
- No route file is changed. `server/src/routes/health.ts` continues to
read the snapshot on each request.

## Verification

Regression test, red then green:

- `cd server && pnpm exec vitest run
src/__tests__/health-server-info-git-probe.test.ts`
- Parent commit (with the test file of this branch): 1 of 4 tests fails.
`starts no git process when the server does not run from a checkout`
fails with `expected 6 to be +0`: one start when the routes are created
and one for each of 5 polls. The response assertions in that test pass
on the parent commit.
- This branch: 4 of 4 tests pass.
- The tests assert the number of `git` starts and the response content.
They do not assert time. The first test compares the full response text
of anonymous health, board health and liveness with the responses of a
server that has the snapshot injected. The other 3 tests pass on both
commits. They keep the checkout behavior: a new commit is visible after
the refresh, also from a subdirectory; a linked worktree (`.git` is a
file) is read; `GIT_DIR` is obeyed.

Benchmark, same machine, 200 health requests per row, 3 runs each
(ranges shown). Each row alternates anonymous and board requests against
routes that are created as `server/src/app.ts` creates them.

| Scenario | Request pattern | Before: `git` execs (route creation +
requests) | Before: ms per request | After: `git` execs | After: ms per
request |
| --- | --- | --- | --- | --- | --- |
| No checkout, build supplies the commit | polled each 5 s | 1 + 200 |
12.5 to 13.8 | 0 + 0 | 1.05 to 1.53 |
| No checkout, build supplies the commit | burst | 1 + 0 | 1.47 to 1.63
| 0 + 0 | 1.19 to 1.57 |
| Source checkout | polled each 5 s | 3 + 600 | 36.3 to 50.2 | 3 + 600 |
36.8 to 40.7 |
| Source checkout | burst | 3 + 0 | 1.38 to 1.40 | 3 + 0 | 1.03 to 1.45
|

- Command: `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-health-server-info.ts --requests=200`
- The `polled` pattern moves the clock that the server reads forward by
5 seconds before each request. It does not wait for real time.
- The benchmark prints a SHA-256 prefix of the anonymous health body, of
`serverInfo.git` in the board health body and of the liveness body. The
values are the same before and after in all rows.
- The `git` call is synchronous, so the difference per request (about 11
to 12 ms locally, which includes the shell wrapper) is time that the
event loop is blocked.
- The source checkout rows do not change. This is the intended result.

Assumption and evidence (`rules/runtime-dependency-boundaries.mdc`): the
change assumes that real Git fails where the check finds no checkout,
and that the check finds each checkout that Git finds. The first
regression test runs real `git show` in the same directory and asserts
that it fails. The other tests run real Git from a repository
subdirectory, from a linked worktree and with `GIT_DIR`. The new check
itself is a small number of `existsSync` calls (one for each directory
level) on each 3 second refresh.

Other checks:

- `cd server && pnpm exec vitest run src/__tests__/server-info.test.ts
src/__tests__/health.test.ts src/__tests__/liveness.test.ts
src/__tests__/health-dev-server-token.test.ts
src/__tests__/health-workspace-readiness.test.ts
src/__tests__/cloud-warm-standby.test.ts
src/services/native-runtime/native-controller-identity.test.ts` and the
new test file: 58 tests in 8 files pass.
- `tsc --noEmit` over the three changed TypeScript files: clean.
- `pnpm verify` exits 2 (broader verification required, because of the
new file in `server/scripts`). Required CI supplies that coverage.
- Not verified: the benchmark on Linux with a multi-gigabyte process,
and the hosted server after deploy. The local numbers come from a
process of about 700 MB, where a process start is cheaper than on the
profiled server. That the profiled server has no checkout is a
conclusion from the image build and from the profile; I did not read its
`serverInfo` response.

## Risks

- Low risk. The response bodies of `/api/health` and `/api/liveness` are
the same as before for an install without a checkout and for a checkout.
- Behavior shift: a server whose working directory is inside a bare
repository, with no `.git` entry and no `GIT_DIR`, reported the commit
of that repository before. It now reports the build commit, or no
commit. This layout is not a supported way to run the server.
- A server that runs from a source checkout continues to start up to 3
`git` processes on each 3 second refresh while health is polled. This
pull request does not change that path, because #8753 made it read the
checkout live on purpose.
- If a checkout is created around the working directory while the server
runs, the next refresh finds it, because the check runs on each refresh.

## Model Used

- Claude Fable 5.2 (Anthropic, model ID `claude-fable-5-2`), in Claude
Code with tool use (shell, file edits, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 09:28:07 -06:00
DottaandClaude Opus 5.5 ef2b53b675 fix(environments): keep resuming an idle sandbox whose resume timed out (#21113)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on the next wake
> - When the resume call times out, the run failed after 90 seconds with
`reusable_sandbox_resume_failed (cause=transport_timeout)`
> - The provider keeps starting the sandbox after the host call times
out, so the sandbox is usually ready a little later
> - The run gave up after one call, and the task then waited behind a
Retry button
> - This pull request keeps the same run retrying an idle sandbox whose
resume only timed out, up to the provider's start deadline
> - The benefit is that recovery works automatically and a sandbox with
unsaved work is never destroyed

## Linked Issues or Issue Description

Refs #20626, #20828, #21020

**What happened?**
Three Codex runs on Daytona failed before provider dispatch with:
`Reusable sandbox lease "<id>" could not be resumed
(cause=transport_timeout); the lease was preserved and no replacement
was created.` Each run had `executionRecovery: { kind: "bootstrap",
providerWorkStarted: false }`. Each run took 100 to 122 seconds: one
resume call used the whole 90-second budget, so no in-run retry was
possible. The preserved leases were idle. Their last runs had ended
about two hours before, with a confirmed stop-and-retain. The workspaces
were not safe to replace, so the run failed strictly. The automatic
bootstrap retry was then refused ("The automatic retry could not
revalidate the original user continuation."), and all three tasks ended
`blocked` behind a Retry.

**Expected behavior**
A resume that only timed out, before any provider work, on a sandbox
that no live run holds, continues in the same run. The run keeps the
same sandbox and does not create a replacement or destroy the sandbox.

**Steps to reproduce**
1. Let a task's reusable Daytona sandbox stop and stay idle for some
hours (it auto-archives after 60 minutes).
2. Retry the task. The workspace has unsaved work, so replacement is not
safe.
3. The `environmentResumeLease` call takes longer than 90 seconds. The
host call times out (`RPC call "environmentResumeLease" timed out after
90000ms`), the 90-second budget is spent, and the run fails with
`reusable_sandbox_resume_failed`.

**Paperclip version or commit**
Staging at `849f109f` (current `master` has the same code path).

**Deployment mode**
Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases
on.

**Agent adapter(s) involved**
`codex_local` (Codex) on a Daytona sandbox environment. The change is
adapter-independent.

**Root cause**
`server/src/services/environment-runtime.ts`, reusable resume loop: the
first `environmentResumeLease` call got the whole 90-second budget as
its RPC timeout. When that call timed out, the loop saw no time left and
threw. The 90-second budget assumes that the bounded bootstrap retry
resumes the same sandbox later. That retry is not always admitted. A
timeout is different from a refusal: the provider is still starting the
sandbox.

## What Changed

- `reusable-sandbox-resume-replacement.ts`: add
`reusableSandboxLeaseIsIdle` (lease is `released` or `retained` and no
other live run holds it). `reusableSandboxReplacementIsSafe` now uses
it, with the same rules as before. Add
`REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS` (330 seconds, the Daytona start
deadline plus the call margin).
- `environment-runtime.ts`: when the 90-second budget ends on a
`transport_timeout` and the preserved lease is idle, extend the deadline
one time, up to the provider start deadline (at most 330 seconds from
the first attempt). The run keeps retrying the same sandbox with
backoff. Each call still has at most 90 seconds. Refusals, a live owner,
and other failures keep the current behavior. The run never replaces or
destroys the sandbox because of this extension.
- `doc/execution-semantics.md`: document the idle-timeout extension.

## Verification

- Red before the fix, with the new tests in
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`
(staging shape: released lease, finished owner run, workspace with
unsaved work, 300-second provider deadline, host RPC timeout): 2 tests
failed with the staging error `could not be resumed
(cause=transport_timeout); the lease was preserved and no replacement
was created.`
- Green after the fix: 16/16 pass. The timed-out idle sandbox resumes in
the same run with no create or destroy call. If every call times out,
the run stops at or before 330 seconds, makes more than one call, and
keeps the lease `released`. A sandbox that a live run still holds gets
one call and fails at 90 seconds, as before.
- `pnpm exec vitest run src/__tests__/environment-runtime.test.ts
src/services/environment-runtime-reusable-lease-probe.test.ts
src/services/remote-workspace-transport.test.ts`: 378 pass.
- `tsc -p server/tsconfig.json --noEmit` passes.
- `pnpm verify` exits 2 because a doc is a global CI input. Required CI
supplies the broad coverage.
- Not run: a live Daytona resume on staging. No later run attempted the
three preserved leases, so it is inferred, not observed, that the
sandboxes were ready after the host call timed out.

## Risks

- A run can now wait up to 330 seconds in preparation before its first
turn, but only when the resume timed out on an idle sandbox. Before this
change, that run failed and the task waited for a person.
- A provider that is down (not slow) now takes up to 330 seconds to fail
instead of 90 seconds. The failure is the same
`reusable_sandbox_resume_failed`, and the lease is still preserved.
- No persisted fields change. Leases saved by older releases follow the
same rule.
- The refused bootstrap retry admission for task Retry continuations is
a separate gap. This pull request does not change it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1112.0-canary.12
2026-11-12 08:28:32 -06:00
DottaandClaude Opus 5.5 704b3f4d63 fix(server): a card accepted while its run settles keeps one continuation (#21094)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run ends while its task is in review, Paperclip checks that
the task still has a maintained review path. If it has none, Paperclip
queues one review-path recovery run
> - An approval card for a tool action sends its continuation through a
delivery receipt. The receipt waits until the source run has settled and
the action has finished
> - A person accepted a card while the source run was still settling.
The interaction was no longer pending, and the continuation wake did not
exist yet, so the review-path check found no path and queued a recovery
run
> - The real continuation then ran too, so the task got two follow-up
runs for one decision
> - This pull request counts an undelivered tool review continuation as
a maintained review path
> - The benefit is that a card accepted during settle produces exactly
one continuation run

## Linked Issues or Issue Description

**What happened?**
An agent posted a tool approval card and its run finished. The person
accepted the card 1.3 s later, while that run was still settling. The
review-path check saw no pending interaction and no queued wake, and it
queued an `issue_review_path_lost` recovery run. The accept continuation
was deferred behind that run and then started as a second follow-up.

**Expected behavior**
The accepted card's pending continuation counts as the task's review
path. Exactly one continuation runs.

**Steps to reproduce**
Have an agent in a task that is in review post a tool approval card.
Accept the card while the creating run settles and before the tool
action delivers its result. Before this change, a review-path recovery
run was queued in addition to the continuation.

## What Changed

- `server/src/services/issues.ts` (`listIssueReviewAttentionMap`): reads
tool review continuations for the review tasks in one statement. A
continuation counts as a queued review path when (a) its receipt is
undelivered, its card is resolved, and the requesting assignee can still
receive it (not paused, terminated or pending approval), or (b) its
delivered continuation wake, or that wake's run, is still active. The
path list leaves out a continuation whose wake or run is already listed,
so each continuation shows once. The path shows as "Queued tool action
review wake".
- `heartbeat-issue-liveness-escalation.test.ts`: a new embedded-Postgres
test runs a heartbeat for a task in review with an accepted card whose
tool action is still executing. It checks that no review-path recovery
wake is queued for that run. It then completes the action, delivers it,
and checks that exactly one continuation wake exists.

## Verification

- The new heartbeat test failed before the change (one
`issue_review_path_lost` wake) and passes after it.
- A new parameterized attention test covers an undelivered receipt, a
delivered continuation (listed and not listed in the wake list), a
pending card, a terminated assignee and a consumed continuation. The
terminated, pending-card and delivered cases were red against the
earlier revisions of this change.
- `pnpm vitest run
src/__tests__/heartbeat-issue-liveness-escalation.test.ts
src/__tests__/issue-review-attention.test.ts
src/services/recovery/review-path-recovery.test.ts
src/__tests__/attention-service.test.ts
src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts
src/__tests__/tool-gateway-service.test.ts`: 332 passed.
- Product evidence: a paid chat-eval run of
`questions-and-approvals.approve-tool-once` on a branch that combines
this change with three related fixes. The result is in a PR comment.

## Risks

- Every caller of review attention (review-path recovery, its
escalation, the attention feed and the approvals view) now sees a
pending tool review delivery as a covered path. That is the intended
state: the delivery service always sends one continuation for it, and
its periodic sweep retries a failed delivery.
- One more indexed query for each review-attention read. It reads only
undelivered receipts for the given tasks.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:16:32 -06:00
DottaandClaude Opus 5.5 81d2e4d8d9 fix(native): end a run whose finalizer a newer run superseded, so it stops holding its task (#21106)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result, then a finalizer exports the
workspace and applies the result to the task.
> - When a newer run already committed the task's decision, the
finalizer retires the older result as `native_finalization_superseded`.
> - That retirement set the finalizer terminal, but it did not end the
run. The run stayed `running` and held the task and every run queued
behind it, with no way out except a person.
> - A stuck run must never hold a task and its queue forever.
> - This pull request ends the run when its finalizer is retired as
superseded, and it ends runs that an older release left in that state.
> - The benefit is that the task continues with its queued runs by
itself.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. The workspace export then failed
(`remote_workspace_archive_fs_eacces`). The task's last decision came
from another run that was created 42 seconds later but ran earlier. The
finalizer judged this run superseded: it set the finalizer to
`terminal_failure` with `native_finalization_superseded`. The run stayed
`running` with "Recovery needed". Over one hour later it still held the
task, and five queued runs waited behind it. A server upgrade at 12:02Z
did not change it.

**Expected behavior**
A run whose finalizer is retired ends. The task keeps the newer run's
decision, the retired result is never applied, and the queued runs
proceed.

**Steps to reproduce**
1. Accept a native result for a run.
2. Commit a status decision on the same task from a run that was created
later.
3. Let the older run's finalization fail once, so the finalizer records
the failure.
4. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is still `running`.

**Root cause**
`recordRetryableFailure`
(`server/src/services/native-runtime/native-run-finalizer.ts`) projects
a terminal run status only when `exhausted && !supersededByNewerRun`. A
superseded retirement therefore never ended the run. No other path ends
a run whose finalizer is already `terminal_failure`: the reconciler
skips `terminal_failure` rows.

**Paperclip version or commit**
Staging Core `3eca7a6112`. The behavior is the same on master
`482063aeca`.

## What Changed

- `native-run-finalizer.ts`: a superseded retirement now ends the run
with the same status mapping as an exhausted finalization (workspace
failure: `failed`; otherwise the provider's terminal state). It does not
change the task, does not apply the result, and does not keep the
workspace export lease. The run's error says in plain words that a newer
run already decided the task.
- `native-run-finalizer.ts`: add `endSupersededNativeFinalizationRuns`.
It ends a native run that is still `running` while its finalizer is
`terminal_failure` / `native_finalization_superseded`. This covers rows
that an older release wrote. It locks in the finalizer's order
(coordinator, task, run), and re-checks the row under the lock.
- `native-run-projection-work.ts`: the registered native run projection
worker (it loads at startup, pages through all candidates, and wakes on
finalization changes) also selects such runs and ends them with
`endSupersededNativeFinalizationRuns`.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/__tests__/native-finalization-recovery.test.ts`, red on
master and green on this branch:
- "retires an older failed finalizer when a newer run already committed
the issue": the retired run now ends (`succeeded`, `finishedAt` set).
Master: the run stays `running`.
- "ends a run an earlier release left running after retiring its
finalizer as superseded": the shape of the staging row (workspace export
failed, superseded finalizer, run `running`). The repair ends it as
`failed` with the plain message. The task, its decision and the
finalizer stay unchanged, and no status decision is written for the run.
A second pass changes nothing.
- `server/src/__tests__/native-run-projection-work.test.ts`, "ends, at
startup, a run an earlier release left running after retiring its
finalizer as superseded": the registered worker ends the staging-shaped
row at startup and settles its claimed wake. Red without the worker
change.
- Related suites pass: `native-finalization-recovery`,
`native-run-projection-work`, `native-finalization-reconciler-timing`,
`native-reconciliation-timing`, `native-workspace-finalizer`,
`native-workspace-finalizer-failure`, `native-run-finalizer-telemetry`.
- `tsc --noEmit` for `server`.

## Risks

- Low risk. The change only ends runs whose finalizer is already
terminal and superseded. It never applies a result and never changes the
task.
- A superseded run whose workspace export failed now releases its
sandbox lease through the normal settlement instead of waiting forever.
Its result was not applied, because a newer run decided the task.
- Not changed here: the superseded check compares run creation times. A
run created earlier but executed later is still judged superseded. This
PR makes that verdict end the run; it does not change the verdict.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:07:18 -06:00
DottaandClaude Opus 5.5 0a2f618cae fix(native): leave a run another live server is preparing alone, and re-adopt a run a shutdown handed off (#21107)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade replaces the server. The old and the new server serve
together for a few minutes, then the old server gets SIGTERM and hands
its native runs to the next controller.
> - #21046 taught the legacy reaper to leave a run with its owner while
the owner's server process is proven alive. Native runs have a similar
gap.
> - Runtime selection writes a native run's coordinator with no
controller and no lease, and stops the run lease. The executor claims
the coordinator about 20 seconds later. In that window the other
server's sweep sees no owner and marks the run "ownership could not be
verified".
> - When the old server then hands the run off before any Runner
checkpoint exists, no sweep looks at it again. Only the next server boot
does, 15 to 40 minutes later. The task and its queued runs wait.
> - This pull request leaves such a run with its live preparing server,
and lets the serving server re-adopt a handed-off run through the
ordinary restart claim.
> - The benefit is that a run started during an upgrade continues within
one sweep, with no false "blocked" mark and no wait for another upgrade.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, an upgrade started a new server at
12:19:57Z. The old server started a native run and selected its runtime
at 12:20:32Z. At 12:20:35Z the run logged "Native execution ownership
could not be verified; automatic recovery is blocked"
(`observed_owner_unverified`). The old server's executor claimed the run
at 12:20:51Z. The old server got SIGTERM and suspended the run at
12:20:54Z (`awaiting_runner_reattach`). After that the run showed
"Confirming execution" for 29 minutes, and two queued runs waited behind
it. At 12:49:45Z the next upgrade's startup recovery restarted its
bootstrap, and the run succeeded at 12:58Z.

**Expected behavior**
- A server does not mark a run unverified, or recover it, while another
live server is preparing it.
- A run that a graceful shutdown hands to the next controller is
re-adopted by the serving server at once, not at the next boot.

**Steps to reproduce**
1. Seed a native run whose coordinator is `observed` with no controller
and no lease, and whose `controllerProcess` names another server with a
live controller instance lease.
2. Run the recovery sweep on this server. See that the run gets
`native_execution_ownership_unverified`.
3. Claim the coordinator for the other server, then write what its
graceful shutdown writes (`restart_suspended`,
`awaiting_runner_reattach`, lapsed lease) and stop its instance lease.
Give the run a predecessor's Runner PID.
4. Run the sweep again. See that nothing claims the run.

**Root cause**
- `recoverActiveExecutions`
(`server/src/services/heartbeat/recovery.ts`) treats an `observed`
coordinator without a live controller lease as an owner it cannot
verify. Between runtime selection and the claim, the only owner on
record is the run's `controllerProcess`, and the sweep did not check it.
- `listExpiredNativeRunnerAuthorities` (`native-runner-checkpoint.ts`)
selects runs for the restart claim only when they have a controller
checkpoint, or no process identity at all. A run handed off just after
its claim has no checkpoint, and a follow-up carries its predecessor's
Runner PID. The sweep then skips `awaiting_runner_reattach` runs, so
only a server boot looked at them again.

**Paperclip version or commit**
Staging Core `814fe5d20f` (old server) and `3eca7a6112` (new server).
The behavior is the same on master `849f109f62`.

## What Changed

- `recovery.ts`: before it lists expired native runs, the sweep checks
runs in the runtime-selection window (coordinator `observed`, no
controller, no lease, not tracked here). When the run's
`controllerProcess` names another server process that the shared owner
prover proves alive, the sweep leaves the run with that server: it does
not recover it and does not mark it unverified. It re-checks within 30
seconds. After 5 minutes without a claim
(`LEGACY_LIVE_OWNER_SETTLE_GRACE_MS`), or once that server is gone, the
existing rules apply.
- `native-owner-stamps.ts`: add `preparingControllerOwnerStamp` for that
proof.
- `native-runner-checkpoint.ts`: `listExpiredNativeRunnerAuthorities`
also selects a run that a graceful shutdown handed off and whose lease
lapsed: recovery state `awaiting_runner_reattach` or
`awaiting_evidence`, with a `restart_suspended` history entry that no
restart claim followed. It stays selected through ownership waits (the
departing server can still be alive at the first look). `recovery.ts`
schedules a 30-second re-check for an ownership wait that has no retry
of its own. The ordinary restart claim then proves the takeover and
picks the disposition, exactly as the startup pass of a new server does.
A row that an older release already marked unverified is re-adopted too:
the claim clears the mark.
- `doc/execution-semantics.md`: document both rules.
- No schema change. Nothing changes on the path that starts a run or a
turn: the new proof runs only in the recovery sweep, and only for runs
in the window.

## Verification

- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`,
block "a run the old server started during an upgrade overlap". Red on
master, green on this branch:
- "leaves the run with its live preparing server, then re-adopts it
after that server hands it off": two concurrent sweeps leave the run
`running` with no error and do not touch the coordinator. After the old
server claims, hands the run off and stops, one sweep of the new server
re-adopts it and reaches provider dispatch. Master: the run is marked
`native_execution_ownership_unverified`, and after the handoff nothing
claims it.
- "keeps a handed-off run eligible while the departing server is still
alive, and re-adopts it once it exits": the first sweep waits
(`awaiting_evidence`), the sweep after the departing process exits
re-adopts the run. Red with a filter that matches only the last history
entry.
- "re-adopts a handed-off run that an older release already marked
ownership-unverified": the stuck staging shape. Master: nothing claims
it.
- Related suites pass: `heartbeat-native-restart-suspension` (all 109),
`services/heartbeat/recovery.test.ts` (all 103).
- `tsc --noEmit` for `server`.

## Risks

- A run that another live server prepares, but never claims, now stays
`running` for up to 5 minutes before the existing rules apply. That is
the same bound #21046 uses.
- The guard trusts the shared owner prover. It reads "alive" only from a
live controller instance lease or a live PID in the same PID space.
- More handed-off runs reach the restart claim earlier. The claim keeps
all its fencing and evidence checks; this change only makes the serving
server ask the same question a new boot asks.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:06:57 -06:00
DottaandClaude Opus 5.5 a65644d1e1 fix(retry): continue an accepted connected Retry from recovered saved work when its chat replay is refused later (#21080)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks from a connected chat (for example GitHub pull request
reviews) show Retry when a run fails
> - A person's Retry on such a task first asks the chat retry owner to
repeat the exact failed chat request. #21038 made a refusal at the click
continue the task from saved work instead
> - The chat owner can also defer its source check (a busy identity,
endpoint, or agent row lock) and refuse it a few seconds later. Then the
route has already answered 202, and the task Retry receipt was marked
failed with no run
> - The task stayed blocked, and its execution blocker told the person
to send a message, while the interruption projection still offered Retry
> - This pull request makes that late refusal take the same board
continuation as a refusal at the click, and makes the blocker offer
Retry for recovered saved work
> - The benefit is that an accepted Retry always starts work, and the
two projections agree

## Linked Issues or Issue Description

No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a hosted deploy, about 20 blocked tasks of one busy GitHub review
agent (`paperclip_runner`, native remote runs) did not restart after
Retry. `POST /api/issues/:id/retry` returned 202 with `accepted: true`,
a `chatActionId`, and `presentation.state: retry_scheduled`. No run was
created. The task Retry receipt changed to `failed` 5 to 10 seconds
later, and the task stayed blocked. The activity record of each click
(`chat.failed_run_retry_requested`) has `sourceAuthorizationPending:
true`, so the chat owner deferred its source check at the click. No chat
wake row was written, so the chat replay ended before admission. The
last run failed with `native_continuation_requires_reconciliation` /
`runner_remote_process_identity_unavailable`. Its saved work was
recovered (`workspaceRecovery.status: recovered`, remote lease destroyed
with a termination receipt). The projections disagreed: `interruption`
had `canRetry: true`, but `executionBlocker` had `canRetry: false` and
`nextAction: "Saved work recovered. Send a message to continue."`.

Root cause: when the chat retry action settles `failed`,
`releaseUnadmittedTaskRetryInput` in
`server/src/services/chat-channels/task-retry-input.ts` marked the
linked task Retry receipt `failed`. #21038 added a board-continuation
fallback only for a refusal inside the click transaction. A refusal
after a deferred source check (for example 8 lock-busy attempts on the
agent row, about 7 seconds) had no fallback. Separately,
`getExecutionBlocker` hid Retry for every chat-bound task (`canRetry &&
!chatBinding`), and `workspaceRecoveryDisplay` always said "Send a
message to continue" for recovered work.

**Expected behavior**

A Retry that is accepted starts work. If the exact chat replay is
refused after the click, the person's Retry continues the task from its
recovered saved work. The execution blocker and the interruption both
offer Retry, and neither tells the person to send a message to unblock a
failed run.

**Steps to reproduce**

1. Bind a task to a GitHub chat conversation. Make its latest native run
fail with `native_continuation_requires_reconciliation` and recover its
saved work.
2. Hold the chat identity advisory lock while you click Retry, so the
source check is deferred (`sourceAuthorizationPending: true`).
3. Hold a row lock on the agent while the chat retry processes, so the
deferred check fails 8 times.
4. The route returned 202. The chat action settles `failed`, the task
Retry receipt changes to `failed`, and no run starts.

**Paperclip version or commit**

`master` at a8580160 (the deployed revision). Reproduced on f30005440b.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

`paperclip_runner` (native remote runs). The fix is in core Retry code
and is not adapter-specific.

## What Changed

- `server/src/services/task-retry.ts`: new `settleRefusedTaskChatRetry`.
When a chat retry action settles `failed`, the linked task Retry receipt
becomes an ordinary board continuation if all of these are true: it is a
person's Retry (or an automatic Retry that the same person's click has
joined), its source run is still the latest run and its saved work is
recovered (system recovery `complete`, workspace recovery `recovered`),
the clicking person still has task access, the chat wake has no run, and
no saved or unadmitted chat input remains. Any other refusal still
settles the receipt as failed, so the existing source-edited, revoked,
and identity-expiry outcomes do not change. A paused agent does not fail
the receipt: it stays deferred and starts when dispatch allows. The
receipt drops `chatActionId`, stays `deferred_issue_execution`, notifies
the deferred-receipt worker, and records
`issue.retry_continued_from_saved_work`. Otherwise the receipt is failed
as before. `reconcileTaskChatRetryReceipts` uses the same check, so a
crash before settlement is also covered. Its failed write matches the
chat binding it read, so a stale pass cannot fail a receipt that
settlement already converted.
- The task Retry intent now records `initiatedBy` for an automatic
Retry, so an automatic Retry keeps the exact-replay requirement after
the click too.
- `server/src/services/chat-channels/task-retry-input.ts`:
`releaseUnadmittedTaskRetryInput` calls the task owner's settlement
instead of failing the receipt directly.
- `server/src/services/execution-blocker.ts` and
`workspace-recovery-receipt.ts`: for a failed or timed-out run whose
saved work is recovered, the blocker offers Retry also on a chat-bound
task. The text follows actual Retry eligibility: "Saved work recovered.
Retry continues from it." when Retry is offered, "Saved work recovered."
when it is not (for example cleanup quarantine), and "Send a message to
continue" only for a stopped run. Only the task blocker decides Retry.
The run-level projection (`execution-projection.ts`) has no task
context, so it says "Saved work recovered." and never claims Retry.
`execution-projection.ts` uses the same text.
- `ui/src/lib/activity-format.ts`: labels for the new activity.
- `doc/execution-semantics.md`: the late-refusal rule.
- Tests: a new real-service case in
`server/src/__tests__/chat-channels.integration.test.ts` that reproduces
the hosted state (remote saved workspace, destroyed lease receipt,
recovered recovery action, deferred source check, refused replay), for a
person's Retry and for an automatic Retry. A gate case per continuation
condition (unrecovered workspace and revoked access settle failed; a
paused agent stays deferred and runs after resume), a stale-reconcile
race case, and a joined-click case. Two existing text expectations are
updated.

## Verification

- Red before the fix: with only the source changes removed, the new test
fails on the projection (`canRetry: false`, "Send a message to
continue."). With only the Retry settlement removed, it fails with the
receipt `status: "failed"` and no run.
- Green after the fix: `npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "continues an
accepted GitHub task Retry from recovered saved work"` passes 2/2. The
person's Retry starts one queued run that resumes the preserved
workspace, and the automatic Retry still settles as failed.
- Focused suites pass: `task-retry`, `execution-projection`,
`execution-projection-recovery`, `workspace-recovery-retry-copyback`,
`chat-failed-run-retry-boundary`, `workspace-recovery-admission`,
`remote-workspace-recovery-receipt-fallback` (372/372), and other
recovery, blocker and continuation suites.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Staging was read with GET requests only. No POST was sent.

## Risks

- Behavior change: a person's Retry on a connected task that the chat
owner refuses after the click now starts a board continuation. It is the
same continuation #21038 already uses for a refusal at the click, with
the clicking person's authority, and dispatch keeps every normal gate.
- Receipts that already failed on staging stay failed. The next click on
those tasks takes the fixed path. No migration.
- The blocker now reports `canRetry: true` for chat-bound tasks only
when the failed run's saved work is verifiably recovered.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:57:42 -06:00
DottaandClaude Opus 5.5 cc6728e8de fix(recovery): continue upgrade-stopped Retry runs that the sweep refused or skipped (#21108)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Hosted servers are upgraded often. Each upgrade stops the old server
gracefully, and a run that is still preparing ends `interrupted`
(`server_shutdown_interrupted`) with bootstrap evidence that provider
work did not start.
> - #21078 let a run that a person started with Retry continue
automatically after such a stop. After it deployed, a read-only audit of
a staging server found four groups of Retry-started tasks that still did
not continue: some were refused on every sweep, some were never looked
at, and one Retry that a person pressed failed and was not retried.
> - Product rule: an upgrade never leaves a task that needs a manual
Retry, and a Retry a person presses must work. A refusal that cannot
succeed later must not repeat forever.
> - This pull request fixes each cause found, keeps every authorization
check of the continuation gate, and ends the refusal loop with one plain
escalation.
> - The benefit is that Retry-started tasks continue through upgrades,
including tasks earlier releases left waiting, with no manual step.

## Linked Issues or Issue Description

Refs #21078, #21046, #21019, #14987, #19384

**What happened?** On a hosted staging server after #21078 deployed
(read-only audit, run events and task activity):
1. A Retry-started task was refused every ~2 minutes ("The automatic
retry could not revalidate the original user continuation."). While
preparing, its Retry had switched to a fresh isolated workspace because
the saved files could not be restored (activity
`issue.fresh_workspace_selected` by the system, 40 minutes after the
Retry). That rewrote the Retry request's strategy from `continue` to
`fresh_isolated_workspace`, and `readTaskRetryRetryAuthority` accepts
only `continue`.
2. Two tasks whose Retry replaced a native session had no recovery event
for hours. The stranded-task sweep skipped them silently.
3. A task whose Retry selected a fresh workspace at admission
(`explicitUserContinuation.readiness`) was refused on purpose
("fresh-workspace Retries keep their own contracts"), yet that contract
had no path for a shutdown before provider work. The task stayed in
progress after every upgrade.
4. A person pressed Retry. The run failed before provider work with
`reusable_sandbox_resume_failed`, a transient failure that is already in
the bounded bootstrap retry lane. One second later the stranded-task
sweep escalated the task to `blocked`, which cleared the claim. Two
seconds after that, the run's own failure path scheduled its retry and
was refused because the claim no longer named the run.

**Expected behavior:** each of these continues automatically once, under
the person's original Retry. A refusal that cannot succeed later
escalates the task once with a working Retry.

**Root causes:**
- Cases 1 and 3: the continuation gate refused every fresh-workspace
Retry, whether the fresh workspace was chosen at admission or later in
preparation. That includes one that a shutdown stopped before it did
anything.
- Case 2: the sweep's generic skips run before the continuation. Two of
them can match only because of state that the replaced session left. An
unfinished session goal (`agent_task_sessions.goal_status`) is skipped
as "goal recovery owns it". A native finalization row is counted as a
live execution path by `hasActiveExecutionPath`. The person's Retry had
already started a fresh session, so neither owns the task. The audit
could not read the database, so which of the two applied on staging is
not proven; the tests cover both.
- Case 4: a race. The sweep does not wait while a terminal run's own
executor is still settling it, so it escalates before that executor
schedules the run's bounded retry.
- The refusal loop: after a refusal the sweep left the task
`in_progress` and tried again on the next pass.

**Paperclip version or commit:** master `482063aeca`.

## What Changed

- `explicit-native-continuation.ts`: `readTaskRetryRetryAuthority`
renews a fresh-workspace Retry **only** when the parent run was stopped
by a graceful shutdown before provider work
(`isShutdownInterruptedPreparation`). The admitted `readiness` must name
the same Retry and source. A strategy that preparation switched to fresh
is rebuilt exactly as preparation records it: this Retry, its source,
and the source's leases. An automatic retry of the fresh attempt
inherits the selection from its own receipt. The successor's receipt
records the selection under `automaticRetry.readiness`. The successor's
`explicitUserContinuation` keeps its plain shape.
- `fresh-workspace-continuation.ts`: `readFreshWorkspaceReadiness` reads
that receipt for an automatic continuation, so the successor prepares
the same fresh attempt. A remote fresh workspace is keyed by the Retry,
so it is reused rather than created again. `freshReadinessSchema` is
exported and reused.
- `recovery/service.ts`:
- **Shutdown-stopped Retry preparations go first:** at the start of the
stranded sweep, every `in_progress` task whose latest run is a legacy
`server_shutdown_interrupted` preparation (`providerWorkStarted: false`)
carrying `explicitUserContinuation` is offered to the scheduler. It is
skipped when the agent is not invokable, the task has a live execution
path, a board-owned recovery action or a pause hold, or the run is still
settling. The unfinished-goal skip does not apply here, because the
person's Retry started a fresh session.
- `hasActiveExecutionPath` ignores a native finalization that a person's
Retry already replaced (`failureDetail.replacementDenied =
explicit_user_continuation` on a terminal run).
- The sweep skips a task while its latest terminal run is still being
settled: a live execution in this process, or another server's unexpired
controller lease. It checks again when that lease expires.
- A refused continuation (`continuation_user_authorization_missing`)
escalates the task once to `blocked` with the plain notice "Retry needed
… Press Retry to continue." Before, the task stayed in progress and was
refused again on the next pass.
- `heartbeat.ts`: `scheduleRecoveryRetry` reports a refused continuation
to the sweep.
- `doc/execution-semantics.md`: describes these rules.

**Failure classes admitted for a Retry-started run (case 4):** no new
class. The run's own bounded retry lanes are unchanged.
`reusable_sandbox_resume_failed`, `remote_workspace_restore_failed`,
`plugin_worker_unavailable`, `environment_provider_unavailable`,
`repository_catalog_unavailable`, runner revision mismatch and the other
transient bootstrap failures were already retried before provider work.
The fix removes the race that took the claim away from that retry.

**Refusals that remain by design:**
- The task is reassigned, `done` or `cancelled`.
- Another run holds the claim, or newer or live work exists.
- The Retry actor, generation or run changed, or it is a chat or
queued-message Retry.
- The message was edited or deleted.
- The receipt is missing or revoked.
- The run had another interruption cause, or provider work started.
- A fresh-workspace Retry whose run *failed* (not a shutdown) keeps its
one-run contract.
- A readiness that names another Retry or another source.

Each of these now escalates the task once instead of refusing on every
sweep.

## Verification

- `server/src/services/task-retry.test.ts`, new tests. Red on
`origin/master` sources and green on this branch (checked by restoring
the master sources):
- "continues a fresh-workspace Retry that a shutdown stopped before
provider work":
- Runs twice: once with the fresh workspace selected at admission, once
with it selected at preparation.
- Two concurrent sweeps create one successor, which gets the claim, and
dispatch accepts it.
- `readFreshWorkspaceReadiness(successor)` returns the same fresh
attempt.
- A second upgrade stops the successor, and the next successor carries
the same attempt.
- "continues a shutdown-stopped Retry whose replaced session left state
behind": two cases, an unfinished session goal and a native finalization
that the Retry replaced.
- "escalates a refused shutdown continuation once instead of refusing it
on every sweep": two sweeps. The task ends `blocked` with one "Press
Retry to continue" notice and one refusal event.
- "leaves a failed Retry run to its own executor while that executor
still settles it":
- Production shape: `reusable_sandbox_resume_failed` and a live
controller lease from the server that ran the run.
    - The sweep leaves the task and its claim alone.
    - The run's bounded retry is then scheduled and dispatch accepts it.
- "keeps a fresh-workspace Retry's own contract when it does not match":
three refusal cases (other Retry, other source, failed rather than
stopped). Green on both.
- Existing suites pass:
- `task-retry`, `explicit-native-continuation`,
`heartbeat-retry-scheduling`, `heartbeat-process-recovery`,
`interrupted-preparation-hold-settle`,
`heartbeat-issue-liveness-escalation`, `issue-recovery-actions`,
`recovery-deadline-acceptance`, `legacy-continuation-authority`,
`workspace-recovery-admission`, `native-workspace-export-retry`,
`heartbeat-paused-assignee-new-task`, `issue-tree-pause-resume-run`,
`heartbeat-stale-queue-invalidation`. 1,619 tests passed, with one
failure that a change in this branch fixed; rerun of the affected
suites: 560/560.
- The one failure came from the first version of the settling check,
which also waited for this process's own lease after a shutdown drain.
The check now uses this process's live executions and other processes'
leases only.
- `tsc --noEmit` for `server`.
- Required CI covers the full suite.

**After deploy, the next audit should show:**
- Tasks stopped by a shutdown while their Retry was preparing get one
`scheduled_retry` successor on the first sweep. This includes tasks
whose Retry selected a fresh workspace and tasks whose Retry replaced a
native session.
- They show no further "could not revalidate" events.
- A Retry-started run that fails transiently before provider work gets
its bounded retry, and the task is not escalated in between.
- A task whose continuation is truly refused is `blocked` once with
"Retry needed".

## Risks

- **Behavior change:** the sweep now offers shutdown-stopped Retry or
message preparations to the scheduler before its generic skips. All
authorization checks still run at admission, under the task and run row
locks. The scheduler reuses an existing successor, so two servers create
one continuation. The tests run two sweeps at once.
- **Settling wait:** a task whose latest terminal run still has another
server's live controller lease waits until that lease expires (at most
90 s) before the sweep acts. Normal turn starts are unchanged, and
nothing is added before a turn starts.
- **Escalation on refusal:** a refused continuation now moves the task
to `blocked` with a Retry notice, where before it stayed `in_progress`
with no path. Pressing Retry is a new, separately admitted request.
- **Rows from older releases:** no schema change and no backfill. A
shutdown-stopped fresh-workspace Retry is continued once. An older
successor receipt without `automaticRetry.readiness` behaves as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, read-only API reads, Vitest on embedded PostgreSQL) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:41:25 -06:00
DottaandClaude Opus 5.5 1d153f62b6 feat(docker): build the image in the merge queue and promote it on master (#21096)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `docker.yml` builds, probes, gates and signs the standard Core image
for every master push. Paperclip Cloud deploys a Core commit only after
that image is published.
> - Each master push rebuilds the image from scratch: about 10.6 min for
amd64 and 7.8 min for arm64.
> - The owner approved a merge queue for Core. A merge queue tests the
exact commit that master will become, before the merge.
> - This pull request builds and probes the image for that exact commit
in the queue, with no registry credentials. The later master push pushes
those exact bytes instead of rebuilding. When no matching verified build
exists, master builds from source exactly as today.
> - The benefit is that the image legs on master drop from about 8–11
min to about 3–4 min, and nothing changes until the owner turns the
queue on.

## Linked Issues or Issue Description

Refs #21013 (it measured the image path and proposed this plan).

**What existing behavior does this improve?**
Every master push rebuilds the standard image, even when the same commit
could already have been built while it waited to merge.

**Subsystem affected**
`.github/workflows/docker.yml` (`build-and-push` only), new
`.github/workflows/docker-merge-group.yml`, new
`scripts/merge-group-image.mjs`.

**Current behavior**
`build-and-push` always runs the full BuildKit build on master: 639 s
(amd64) and 468 s (arm64) in run 43133035209.

**Proposed behavior**
On a `merge_group` event, build both architectures for the merge-group
commit, probe PID 1, and upload an OCI archive plus a record of commit,
tree, architecture and digest. On the master push for the same commit,
each leg pushes that archive by digest instead of building. Everything
after the leg (probe, exact-source gate, signed-index check,
attestation) is unchanged.

**Reason and benefit**
The build moves before the merge, where it runs in parallel with the
queue's other checks. The master leg only downloads and pushes.

**Breaking changes**
None. Without `merge_group` events, `find` returns no run and the leg
builds as before.

**Additional context**
This PR does not touch the exact-source gate jobs
(`source_verification`, `publication_verification`, `merge-and-push`),
`scripts/watch-release-source.mjs` or `cloud-readiness.yml`. Another
change owns them.

## What Changed

- **`docker-merge-group.yml` (new, `on: merge_group` only).** It runs
only for this repository's master queue. Workflow permissions are `{}`
and the job has `contents: read` only. There are no secrets, no registry
login, no `cache-to`, no `push: true`, no environments, and checkout
uses `persist-credentials: false`. It runs on GitHub-hosted runners, not
the paid AWS fleets. Steps: build `target: production` with the same
build args as master (`PAPERCLIP_BUILD_COMMIT` is the merge-group SHA,
which becomes the master SHA). Read the public registry cache. Write
`type=oci` to a tar. `docker load` it and run
`scripts/assert-orphan-reaping.sh`. Only then write `record.json` and
upload `merge-group-image-<arch>` (3-day retention).
- **`scripts/merge-group-image.mjs` (new).**
- `find` (master push only): selects a completed, successful run of this
workflow, with event `merge_group`, the same `head_sha`, repository and
head repository both `1170821064`, and a `gh-readonly-queue/master/`
head branch. It needs exactly one unexpired artifact for the
architecture.
- `record` (merge group only): reads the archive's `index.json`, checks
each blob's sha256 and that the image is `linux/<arch>`, and writes
`{sha, tree, arch, digest}`.
- `verify` (master push only): re-reads the archive the same way and
requires record SHA = `GITHUB_SHA`, record tree = `git rev-parse
HEAD^{tree}` of the master checkout, same architecture, and archive
digest = recorded digest.
- **`docker.yml` `build-and-push`.**
  - The leg gets `actions: read` so it can download the artifact.
- New steps run after the registry login: find, download, then promote
(`verify`, then `skopeo copy --all --preserve-digests oci-archive:…
docker://ghcr.io/…@<digest>`). The step sets its `digest` output only
after the push succeeds.
- All three new steps are `continue-on-error`. Buildx, labels, metadata
and "Build and push by digest" run when `steps.promote.outputs.digest ==
''`, so any failure builds from source.
- "Select image digest" feeds the existing PID 1 probe and digest
export.
- **Tests (`scripts/merge-group-image.test.mjs`, 8 tests):**
- Run selection rejects another commit, another event, another workflow,
a fork head, another repository, another queue base, and failed,
cancelled or running builds.
  - Artifact selection.
- Archive digest with and without provenance, a tampered blob, the wrong
architecture, and a real tar read through the system `tar`.
  - Record checks for commit, tree, architecture and digest mismatch.
- The CLI refuses `record` outside `merge_group` and `find`/`verify`
outside a canonical master push.
- Static checks of the merge-group workflow: credentials, triggers,
permissions and runners.
- Static checks of the `docker.yml` wiring: promotion is gated and falls
back, the output is set only after the push, and every build step has
the fallback condition.

### How promotion proves it publishes the verified tree

1. `find` accepts only a run whose `head_sha` equals `github.sha` of the
push, meaning the queue landed the commit it tested (unverified: see
"Merge method" in Risks).
2. `verify` compares the record's tree with the tree of the master
checkout, and its commit with `GITHUB_SHA`.
3. The archive is content-addressed. The digest is recomputed from the
archive blobs and must equal the recorded digest. `skopeo
--preserve-digests` to that digest reference fails rather than produce
other bytes.
4. The leg then runs the PID 1 probe on that registry digest.
`merge-and-push` still signs only an index whose Linux images are the
probed digests (#21013), after the unchanged exact-source gate.
5. The pipeline itself is trusted only as master already ran it. `find`
and `verify` run the copy of `scripts/merge-group-image.mjs` from the
push's `before` commit (`git show "$BEFORE:…"`), not the copy the push
brings. `verify` refuses promotion if anything in `before..HEAD` (every
commit of a multi-PR group, not only the tip) changes
`docker-merge-group.yml`, `docker.yml`, the promotion script or the PID
1 probe. Such a push builds from source. So does the first push after
this PR merges, because its `before` has no verifier.

## Verification

- `node --test scripts/merge-group-image.test.mjs`: 10 pass. One builds
a real git history in which a two-PR push changes the pipeline in its
first commit only, and checks that the range check catches it.
- `node --test` on the workflow and contract suites (cloud-readiness,
docker-canary-promotion, post-merge-runner-routing,
lockfile-refresh-workflows, release-verify-workflow, reuse-canary-image,
standard-image-contract, cloud-source-verification, merge-group-image):
492 pass.
- `vitest run server/src/__tests__/docker-build-stamp.test.ts
server/src/__tests__/container-init-reaping.test.ts`: 22 pass.
- `actionlint` on both workflows: no new findings (the 6 remaining are
pre-existing `ls` and word-splitting notes in `docker.yml`).
- Local: `skopeo 1.24.1 copy --all --preserve-digests` between OCI
layouts kept the index digest. It accepted a `docker://…@sha256:`
destination and failed only at the network step.

Cannot be exercised before the queue exists, because no `merge_group`
event fires:
- the merge-group run itself: the hosted-runner build time, disk use
with tar plus `docker load`, `docker load` of a BuildKit OCI tar, and a
~5 GB artifact upload;
- the cross-run artifact download on master;
- whether `skopeo` exists on the runs-on AMI and accepts digest
destinations in its installed version.

Each master-side failure falls back to the full build. A merge-group
failure only blocks the queue if its check is required, which is why
that check is required last in the steps below.

## Risks

- **Inert until enabled.** No `merge_group` events fire before the owner
adds a merge queue. On master, `find` lists runs (one API call per leg,
about 1 s), finds none, and the leg builds as today. The only always-on
change is the leg's `actions: read` permission.
- **Build cache freshness.** A promoted master push writes no
`buildcache-<arch>`, and merge-group builds read but never write it.
With every push promoted, the cache ages, so dependency or weekly
tool-layer changes rebuild in each merge-group run until a fallback
build refreshes the cache. A follow-up can refresh the cache from master
without blocking publication.
- **Publication time is still gate-bound.** The exact-source gate (about
13–14 min after push) still decides when the image publishes. See
"Expected timing".
- **Merge method (unverified).** Promotion needs the master commit to be
exactly the commit the queue tested, because the image stamps the commit
SHA (`PAPERCLIP_BUILD_COMMIT`, `build-info.json`, the provider pack
revision). Matching only the tree would publish an image stamped with
another SHA, so this PR does not relax the check. I expect the queue to
fast-forward master to its tested commit. GitHub's merge-queue page does
not state this, and it cannot be tested before the queue exists. If the
queue lands a different SHA, promotion never matches and every push
builds from source. That is safe but inert. Step 4 below checks this.
- **skopeo on the AWS fleet.** The promote step checks for `skopeo`
first and logs "Merge-queue promotion unavailable" if it is missing,
then builds from source. If that notice appears, add skopeo to the fleet
image.
- **Hosted runner capacity.** amd64 merge-group builds run on 4-core
hosted runners, colder and slower than the AWS fleet. That adds queue
latency, not master latency.

### Settings the owner must enable, in order

1. **Prerequisite, a code change, not a setting.** The `master` ruleset
(id 13619726) requires `ci / verify` and `ci / e2e`. Those come from
`pr.yml`, which triggers only on `pull_request`. A merge queue waits for
required checks on the `merge_group` event too, so every entry would
stall until its timeout. First decide, and merge in a separate PR, how
`ci / verify` and `ci / e2e` report on `merge_group`: a full CI run on
the merge-group commit, or a fast pass that relies on the PR-head
result. Today the ruleset does not require up-to-date branches either.
2. Merge this PR. Nothing changes yet.
3. In Settings → Rules → Rulesets → `master` (13619726), add **Require
merge queue**:
   - merge method Squash;
- build concurrency 5, minimum group size 1, maximum group size 5, wait
0 min;
   - "Require all queue entries to pass required checks";
   - status check timeout 60 min.
Do not add the image build to the required checks yet. Promotion is then
opportunistic: if the merge-group build has not finished when master is
pushed, master builds from source.
4. After the first queued merge, confirm promotion can match. The master
commit SHA must equal the `head_sha` of its `Docker merge-queue build`
run, and the next master `Docker` run's "Promote the merge-queue image"
step must succeed. If the SHAs differ, promotion stays inert under this
merge method. Stop here and report it.
5. After about a week of green `Docker merge-queue build` runs, add
`build (linux/amd64, ubuntu-latest, amd64)` and `build (linux/arm64,
ubuntu-24.04-arm, arm64)` (app: GitHub Actions) to the ruleset's
required status checks. Then every queued commit has a verified image
ready to promote.
6. The existing bypass actors (org admins, repository admins,
integration 5534676) can still merge outside the queue. Those pushes
have no merge-group run and build from source.

### Rollback

- Remove the image checks from the required status checks (step 5), or
remove **Require merge queue** (step 3). Master pushes then find no
merge-group run and build from source. Nothing to migrate.
- Code: revert this PR. Old merge-group artifacts expire after 3 days.
Promoted images are ordinary digests under
`ghcr.io/paperclipai/paperclip` and stay valid.

### Expected timing

- Image leg on master: about 3–4 min (download about 5 GB, push the new
layers by digest, PID 1 probe) instead of 10.6 min (amd64) and 7.8 min
(arm64).
- Push to published: still bounded by the exact-source gate on master,
about 14–15 min (gate plus about 45 s for `merge-and-push`). Today it is
the same bound, but with the legs as a second 11-min path and the
serialized-run queue in front.
- If the gate later accepts proof produced in the merge queue (owned by
the gate change), push to published would be about 4–5 min (leg
promotion plus `merge-and-push`).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only GitHub API and registry reads).
Default reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:10:11 -06:00
DottaandClaude Opus 5.5 849f109f62 fix(server): agent inbox reads now apply private-task visibility (#21092)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents read a person's Mine list with `GET
/api/agents/me/inbox/mine`, and their own queue with `GET
/api/agents/me/inbox-lite`
> - The company task list routes apply private-task visibility for the
reading actor
> - The two agent inbox routes should use the same visibility rules, so
that each route returns the same tasks for the same actor
> - This pull request makes agent inbox reads apply private-task
visibility
> - The benefit is one consistent visibility rule across task list
routes

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Agent inbox reads now apply private-task visibility.

**Subsystem affected**
Server API: agent inbox routes.

**Current behavior**
The agent inbox routes build their lists without the issue read
condition that the company task list routes use.

**Proposed behavior**
The agent inbox routes use the same issue read condition and per-row
read decisions as the company task list routes.

**Reason and benefit**
All task list routes apply one visibility rule.

## What Changed

- `server/src/routes/agents.ts`: `GET /api/agents/me/inbox/mine` adds
`issueReadSqlCondition` to its list query and keeps only the rows that
`access.decideReads` allows.
- `GET /api/agents/me/inbox-lite` adds the same read condition to its
list query.
- `inbox-mine-routes.test.ts`: a new embedded-Postgres test for the
agent Mine read with private and open tasks.
- `agent-permissions-routes.test.ts`: the mocked route test expects the
read condition and the read decisions.

## Verification

- `pnpm vitest run src/__tests__/inbox-mine-routes.test.ts`: 7 passed.
The new test was red before the route change and is green after it.
- `pnpm vitest run src/__tests__/agent-permissions-routes.test.ts
src/__tests__/openapi-routes.test.ts`: all passed.
- A chat eval run on a branch that combines this change with three other
fixes. The result is in a PR comment.

## Risks

- Low risk. The routes return only tasks that the reading agent can open
through the task routes.
- The Mine route adds one batched read decision for each request.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1112.0-canary.10
2026-11-12 06:52:07 -06:00
DottaandClaude Opus 5.5 482063aeca fix(daytona): reuse or repair a read-only skill bundle instead of failing the sync (#21075)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner in a Daytona sandbox. Before a
turn starts, Paperclip copies each selected skill into the sandbox as a
read-only bundle.
> - A sandbox is often kept between turns, so the read-only bundle is
already there when the next turn (or a recovery) copies it again.
> - The server writes the same skill files again on its own disk, with
new timestamps. The copy compared the bundles with `tar --diff`, which
also compares timestamps. So the copy saw a difference, extracted over
the read-only directories, and failed with "Cannot open: File exists".
> - The turn then stopped and the task stayed blocked until a person
sent a message.
> - This pull request treats a timestamp-only difference as the same
bundle, and repairs a bundle that really differs instead of failing.
> - The benefit is that a kept sandbox no longer blocks a task when its
skills are copied again.

## Linked Issues or Issue Description

**What happened?**
On staging, two tasks in 30 minutes stopped with "Daytona syncIn extract
command failed (exit 43): tar: .paperclip-repository/...: Cannot open:
File exists". One failure was a normal new turn on a kept sandbox. The
other was a recovery of an interrupted turn. Neither one started again
by itself.

**Expected behavior**
The turn uses the bundle that is already in the sandbox when its content
and permissions match. When the content does not match, the copy repairs
it.

**Steps to reproduce**
1. Copy a read-only (0555/0444) skill directory into a sandbox.
2. Change only the timestamps of the copy (this is what a new server
preparation does).
3. Copy the same skill again. GNU tar fails with "Cannot open: File
exists".

## What Changed

- `file-sync.ts` (read-only directory copy): run the `tar --diff` check
with `LC_ALL=C` and ignore lines that only say "Mod time differs".
Bytes, modes, file types and missing files still count as differences.
- When the bundle really differs, make its read-only subdirectories
writable before the extract. GNU tar then replaces the files and
restores the archived directory modes (`--delay-directory-restore`).
Extra files in the target stay.
- If reopening the directories or the repair extract fails, re-seal only
the directories that were reopened for it (their list is recorded
first). Writable user directories beside the bundle keep their modes.
- `file-sync.test.ts`: the existing GNU tar test now covers a
timestamp-only resync (kept, no failure) and a damaged bundle (repaired
to the source bytes and modes, instead of the old fail-closed result),
and a repair that cannot finish (bundle directories re-sealed to 0555, a
writable user directory left at 0755), and a reopen that hits an
unreadable user directory (bundle re-sealed, no scratch files left).

## Verification

- Red: with the old `file-sync.ts`, the updated test fails with the same
error seen on staging: `tar: references/overview.md: Cannot open: File
exists`.
- Green: `npx vitest run src/file-sync.test.ts` in
`packages/plugins/sandbox-providers/daytona` passes 33/33 with GNU tar.
- I reproduced the root cause by hand with GNU tar 1.35: a re-extract
over a 0555 directory fails with "Cannot open: File exists", and `tar
-df` reports only "Mod time differs" for a copy that differs only in
timestamps.

## Risks

- Low risk. The change affects only read-only (immutable) directory
copies on Daytona.
- A bundle that really differs is now repaired in place instead of
failing the turn. Each file is replaced by tar (unlink and create). The
directory modes come back from the archive.
- If a target has extra read-only subdirectories that are not in the
archive, they become owner-writable after a repair.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:28:27 -06:00
DottaandClaude Opus 5.5 2a7fe6c585 fix(server): withhold a queued assignment run once its task moves to backlog (#21095)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Assigning a ready task to an agent queues an assignment run.
Paperclip does not queue one for a backlog task
> - An agent created a ready task, which queued an assignment run, and
then moved the task to backlog. The agent was busy, so the run stayed
queued
> - When the run started, the queued-run gate did not check for backlog,
and the run checked out and did the backlog task
> - This pull request makes the queued-run gate withhold a never-started
assignment run whose task is now in backlog
> - The benefit is that moving a task to backlog keeps it parked, as
when it is created in backlog

## Linked Issues or Issue Description

**What happened?**
An agent created a task without a status, so it was created ready with a
queued assignment run. The agent then moved the task to backlog. The
queued assignment run started two minutes later and completed the
backlog task.

**Expected behavior**
A queued assignment run that has not started does not execute a task
that has since moved to backlog. The run is cancelled with a clear
reason.

**Steps to reproduce**
Assign a ready task to a busy agent so that its assignment run waits in
the queue. Move the task to backlog. Let the agent become free. Before
this change, the assignment run started and checked out the task.

## What Changed

- `run-dispatch/domain/policy.ts` (`decideQueuedRunStaleness`): a queued
run whose wake reason is `issue_assigned` is stale when its task is in
`backlog`. The new error code is `issue_in_backlog`, with the reason
"Cancelled because the task moved to backlog before its assignment run
could start".
- `pre-dispatch-review-wait.ts`: lists `issue_in_backlog` as a
never-started dispatch cancellation, like the other queued-run gate
codes.
- `run-dispatch/adapters/postgres.test.ts`: a new test moves the task to
backlog after the run is queued and checks that `cancelStaleQueuedRun`
cancels it before it starts. A control test checks that a comment wake
on a backlog task still starts.

## Verification

- The new test failed before the change (`not_stale`) and passes after
it.
- `pnpm vitest run src/modules/run-dispatch/
src/services/pre-dispatch-review-wait`: 185 passed.
- Product evidence: a paid chat-eval run of
`questions-and-approvals.accept-one-proposal` on a branch that combines
this change with three related fixes. The result is in a PR comment.

## Risks

- Low risk. The rule applies only to assignment wakes. Comment,
interaction and recovery wakes on a backlog task keep their current
behavior.
- To start the task, a person or agent moves it out of backlog. That
wakes the assignee with `issue_status_changed`, as today.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:19:27 -06:00
DottaandClaude Opus 5.5 67403baaa4 fix(native): an agent termination Stop with no turn result stays a cancellation (#21093)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person terminates an agent, Paperclip stops the agent's
running native turn
> - Two paths can record that Stop: the terminate route ("Cancelled
because the agent was terminated ...") and the lifecycle worker ("Agent
lifecycle stop requested"). Both are system-initiated and expected
> - When the stopped turn ends without a semantic result, the native
executor keeps the run cancelled only for a person's Stop, a
reassignment, or a governance pause. A termination Stop fell through, so
the run ended `failed` with `native_session_interrupted`
> - Whether the run ended cancelled or failed depended on which path won
the race
> - This pull request treats an acknowledged termination Stop like a
person's Stop in the executor and in the lease-release terminalization
> - The benefit is that a terminated agent's interrupted run ends
cancelled, whichever path records the Stop first

## Linked Issues or Issue Description

Refs #20186 (the same treatment for a governance pause Stop)

**What happened?**
A person chose Terminate for an agent while its native turn ran. The
lifecycle Stop recorded an acknowledged native cancellation (initiator
system). The turn ended without a semantic result. The executor then
wrote the run as `failed` with `native_finalization_missing: session
returned no semantic result`.

**Expected behavior**
The run ends `cancelled`. A termination is an intentional Stop, not a
failure to retry.

**Steps to reproduce**
Terminate an agent while its native turn runs, so that the Stop
interrupts the turn before it records a result. Before this change, the
run became `failed` when the lifecycle worker recorded the Stop first.

## What Changed

- `governance-pause-stop.ts`: adds `AGENT_TERMINATED_STOP_REASON`,
`COMPANY_ARCHIVED_STOP_REASON` and `isAgentLifecycleStopRecord`. The
predicate accepts a system-initiated, expected control-plane
cancellation whose reason is a lifecycle, termination or company-archive
Stop.
- `acknowledged-native-stop.ts`: adds
`hasAcknowledgedNativeLifecycleStopIntent`, which requires that record
and an acknowledged, dispatched native Stop receipt.
- `native-session-executor.ts`: the no-semantic-result branch accepts
that intent, so it settles the coordinator as `native_retry_cancelled`
and raises `NativeCancellationPendingRecoveryError`, as for a person's
Stop.
- `heartbeat/run-lifecycle.ts`: lease-release terminalization keeps such
a run `cancelled`.
- The terminate route and the lifecycle driver use the shared reason
constants. The reason text does not change.

## Verification

- `native-session-executor.test.ts`: a new case for each reason
(lifecycle Stop, terminate route, company archive). Each was red before
the change and is green now. The earlier control case now uses a system
Stop that is neither a pause nor a termination, and it still takes the
retry path. Full file: 1414 passed.
- `heartbeat-run-lease-release-terminalization.test.ts`: a new case for
a lifecycle Stop when teardown wins the finalizer race. It was red
before the change (`interrupted`) and is green now.
- Product evidence: a paid chat-eval run of `agents.retire-agent` on a
branch that combines this change with three related fixes. The result is
in a PR comment.

## Risks

- A termination or company-archive Stop that interrupts a turn no longer
schedules an exact resume of that turn. The agent is leaving service, so
no resume can run.
- Other system Stops keep their current behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:19:21 -06:00
DottaandClaude Opus 5.5 ca62ca9e2a fix(release): keep the Runner chaos evals out of the Core image publication gate (#21088)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A Core image is published only after "Cloud source verified v1"
passes for the same commit in `cloud-readiness.yml`
> - That job needs `verify`, which is the whole `release-verify.yml`
call, and that call included the Runner chaos evals
> - The chaos eval job ("Restart, replay, trace, and recovery faults")
took 10.2-11.9 min, with a 7-9 min server finalization step. It is one
of the two slowest jobs on the gate
> - The owner decided on 2026-11-12 that the Runner chaos evals must not
be a publication prerequisite
> - This pull request runs the chaos evals as a separate job in the same
Cloud readiness run. They still run and report, but the source proof
does not wait for them
> - The benefit is an earlier image publication on every master commit.
A chaos failure stays visible as a failed check

## Linked Issues or Issue Description

**What happened?**
The Core image gate waited for the Runner chaos evals. In master runs
43133035381, 43134251278 and 43145252393 the chaos job took 10.2, 11.9
and 11.0 min. The gate opened at +13.5, +13.2 and +13.9 min.

**Proposed change**
Owner decision (2026-11-12): keep the chaos evals running on every
admitted master commit, but remove them from the set of jobs that the
image and Cloud discovery wait for.

**Why**
Every minute on the gate is a minute on every staging release.

## What Changed

- `.github/workflows/release-verify.yml`: new boolean input
`runner_chaos_evals`, default `true`. The `runner_chaos_evals` job runs
only when it is true. Release callers (`release.yml`,
`release-smoke.yml`) do not set it, so they stay gated on the evals.
- `.github/workflows/cloud-readiness.yml`:
  - `verify` passes `runner_chaos_evals: false`.
- New job `runner_chaos_evals`, named "Runner chaos evals (not a
publication gate)". It needs `admission`, has the same admission
condition as `verify`, and calls `runner-chaos-evals.yml` with `ref:
github.sha`. It has no `continue-on-error`.
- `source_verified` ("Cloud source verified v1") still has `needs:
[verify]`. A comment names the owner decision and says not to add the
chaos job.
- `scripts/cloud-source-verification.mjs`: new `runnerChaosEvalsJob`
constant. The fail-fast check ignores jobs named `"<that name> / ..."`.
A chaos failure therefore does not raise `SourceVerificationFailed`. So
`watch-release-source.mjs` no longer cancels the readiness run and the
image run when only the chaos evals fail. Every other failed job still
fails fast and cancels as before.
- Tests:
- `scripts/cloud-source-verification.test.mjs`: a chaos failure does not
cancel the image or block the proof; a chaos failure while the proof is
pending keeps waiting; a required job whose name only resembles the
chaos job still fails fast.
- `.github/scripts/tests/cloud-readiness.test.mjs`: pins the job set
(`admission`, `verify`, `runner_chaos_evals`, `source_verified`),
`source_verified` `needs: [verify]` exactly, `verify` passes
`runner_chaos_evals: false`, the chaos job calls the eval workflow, and
release callers keep the default. Putting the chaos job back into
`source_verified`'s `needs` fails this test (checked locally: 2
failures).

Publication gate, before: `Cloud source verified v1` waited for
typecheck, build, general tests (17 jobs), serialized tests (5), Runner
workflow evals, Verify Paperclip Runner (3 lanes), and the Runner chaos
evals. After: the same list without the Runner chaos evals.

How a chaos failure stays visible:
- Its own check fails on the master commit.
- The Cloud readiness run concludes `failure`.
- The chaos eval bundle is still uploaded (`if: always()`).
- Nothing reports the chaos evals as passed. They are just not a
prerequisite.

Consumers:
- `docker.yml` reads the job "Cloud source verified v1" through
`scripts/watch-release-source.mjs`. No change to `docker.yml` is needed.
- Paperclip Cloud discovery (`src/fleet/cloud-readiness.ts`) runs with
`sourceOnly: true` in `cloud-distribution-policy.ts` and
`release-resolver.ts`. In that mode it needs the workflow path
`.github/workflows/cloud-readiness.yml` and one successful job named
"Cloud source verified v1" in the latest attempt. It does not need the
run conclusion. Both are unchanged, so Cloud needs no change.
- The non-`sourceOnly` mode reads the run conclusion and a job "Cloud
deployable v1". That job no longer exists in this workflow.

## Verification

- `node --test scripts/cloud-source-verification.test.mjs
.github/scripts/tests/cloud-readiness.test.mjs
.github/scripts/tests/post-merge-runner-routing.test.mjs`: 395/395 pass.
- Red/green:
- Before the fail-fast change, the two new chaos tests failed: the
watcher cancelled the image.
  - With the chaos job added back to `needs`, the workflow test failed.
- `actionlint .github/workflows/cloud-readiness.yml
.github/workflows/release-verify.yml`: clean.
- Not run: a real Cloud readiness run of this change. GitHub shows the
result after merge.
- Expected gate: the slowest remaining gating job sets it. After #21016
that is a general-server shard at about 10.5 min, then chat (8.8-10.5
min) and serialized (9.2-9.5 min). Expected gate open is about +11.5 to
+12 min, down from +13.2 to +13.9.

## Risks

- A chaos-eval regression no longer blocks the staging image. This is
the owner decision. The failure stays visible as a red check and a
failed Cloud readiness run.
- npm canary publication reuses the same source proof, so it is also no
longer gated on the chaos evals. Stable and preview releases
(`release.yml`) still run them as a gate.
- The non-gating rule depends on the job name prefix. A test pins the
name to the constant in `cloud-source-verification.mjs`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, GitHub CLI). Standard reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Related: #21016 (server shard rebalance), #21013 (image publish time;
owns `docker.yml`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:08:07 -06:00
DottaandClaude Opus 5.5 b1330c98e8 fix(evals): grade full cohort 6 false failures on saved state and keep failed-run evidence (#21089)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that a chat agent does real product
work, and they grade saved state, not reply wording
> - Full cohort 6 (OpenCode with DeepSeek V4.1 Flash on master
b7693e2076) had five false failures where the product behaviour was
correct
> - Three more failures kept no run logs or ledger, and one was labelled
as transient infrastructure when the product had cancelled the run
> - A false failure hides real product defects, and missing evidence
stops diagnosis
> - This pull request corrects the five graders on saved state and fixes
the evidence gaps
> - The benefit is that the cohort result shows real product behaviour,
and each failure keeps the evidence to explain it

## Linked Issues or Issue Description

**What happened?**
Five chat eval cases failed although saved state showed correct
behaviour. Three run-bound failures and one early Playwright exit saved
no run logs or run ledger. A reassignment cancellation was classified as
`transient_infrastructure`.

**Expected behavior**
Each grader accepts the valid behaviour and still rejects the broken
product. Every failed attempt keeps its run ledger and the logs it can
read.

**Steps to reproduce**
Run the cohort 6 cases on master. Read the failed checks and the missing
`chat-catalog-runs.json` files.

## What Changed

- `budgets-costs.spend-breakdown` (oracle v5). Old check: only unranged
cost reports, the task cost summary, or the single run record counted.
The lead read the run through `GET
/api/companies/{companyId}/heartbeat-runs` filtered to Morgan with
`limit: 200` (saved input digest `36e47c51…`). New check: a run-list
read counts when its saved input digest names a full list that reaches
the original run (unfiltered or filtered to its agent, with a limit past
the runs ahead of it at read time). Still fails: no accounting read, a
list that cannot reach the run, another agent's list, a summary list, a
denied read, a read in another run.
- `status-cards.explain-card` (oracle v6). Old check: only `GET
/api/issues/{id}` counted as a live read. The lead read COM-1 through
`search_tasks`, and the saved result showed the task as done. New check:
a dedicated task tool result that holds the source task id with its live
status counts. Still fails: an empty search, another task, a different
status, a `call_api` result, an error result, a result with no paired
tool use.
- `secrets.personal-secret` (oracle v20). Old check: `JSON.parse` of the
whole bash output. The worker appended `EXIT_CODE=0`. New check: the one
JSON object in the output that carries `receiptId`, which must still
equal the fixture endpoint's authorised call receipt. Still fails: a
wrong receipt, a failed check, two different receipts, no JSON object,
key=value text.
- `skills.fork-read-only` (oracle `integrator.skill-fork.v2`). Old
check: a wording list (`unchanged|read-only|protected|not changed`)
rejected "untouched". New check: the reply names the saved fork or links
its Studio page. Saved state already proves the upstream is unchanged
and the fork exists. Still fails: no fork, a reply without the fork name
or link, a link to another skill, an edited upstream.
- `steering.discard-queued-message` (oracle v6). Old check: the label
string must not appear anywhere in the final plan. The revised plan kept
a "superseded" note with the label. New check: the person asked for a
section labelled with the attempt nonce (a setup literal). A heading
with that label must exist in the plan before the consuming run or in
that run's saves, and the final saved revision must have no heading with
it. Body mentions are allowed. The base plan is the revision before the
consuming run first saved the label, so repeated saves by that run do
not matter. Still fails: the labelled heading kept, renamed but still
labelled, or never saved.
- Failure evidence: the failure path saves
`chat-catalog-failure-run-ledger.json` before retirement, continues when
retirement fails, and reads a failed run's log and events independently
with a separate `logError` / `eventsError` when one run has no log.
Before, one cancelled never-started run (log 404) dropped the logs of
every run (`cases.cases-disabled`, `defer-and-reopen`,
`approve-tool-once`, `assignment-conflict`).
- Interruption act: the click and its `waitForResponse` are awaited
together. Before, the response promise could reject while the click
waited, and the unhandled rejection stopped the worker with no evidence
(`interrupt-with-direction`).
- A product-cancelled run is reported as `Observed run was cancelled by
<source> (<errorCode>)` and classifies as `candidate_failure`. Failed
and timed-out runs keep `Observed provider run failed`.

## Verification

- Calibration tests for each grader: the retained cohort 6 shape, a
reworded valid shape, and broken shapes.
- Regrade of retained cohort 6 evidence: spend-breakdown and
fork-read-only observations pass; the explain-card initial run reads the
source live; the discard plan revisions pass and the unrevised plan
fails.
- `npx vitest run` on the changed and related runner-e2e domain tests:
283 passed, 4 skipped. All 148 identity, integrator, runtime and
personal-secret domain test files pass.
- `pnpm test:e2e:runner:typecheck` passes.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
`changed: false`.
- Paid run: see the PR comment.

## Risks

- Low. Only eval graders and harness evidence change. The run-list rule
depends on the list order (newest first) and the default and maximum
limits from `packages/shared`.
- A product-cancelled run no longer retries automatically as transient
infrastructure.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:05:19 -06:00
DottaandClaude Opus 5.5 6d8f59d494 fix(api): name the secret binding proposal as the way to give an agent a company secret (#21090)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents find product API routes through the runner API search,
which ranks operations by their OpenAPI summary and description
> - In the chat eval `secrets.configure-company-secret`, a person asked
the CEO to let Morgan use an existing deployment API key
> - The agent searched for agent configuration and environment variable
routes, tried `PATCH /api/agents/{id}` (403) and spent 3.5 minutes
before it used `POST /api/agents/me/secret-proposals`
> - The summary "Propose a company secret or agent secret binding" did
not match how agents describe the task
> - This pull request rewrites the summary and adds a description that
names the binding fields
> - The benefit is that the first search finds the supported route, so
the turn is short

## Linked Issues or Issue Description

**What happened?**
Runner API search did not rank `POST /api/agents/me/secret-proposals`
for the queries an agent uses to give another agent a company secret
("agent configuration update env secret", "agent environment variables
set secret ref"). The agent probed other routes for 3.5 minutes.

**Expected behavior**
Those searches return the secret binding proposal route in the first
results, and the result explains `secretId`, `configPath` and
`targetAgentId`.

**Steps to reproduce**
Call `searchRunnerApi({ query: "agent configuration update env secret",
limit: 3 })` on master. The proposal route is not in the results.

## What Changed

- `server/src/routes/openapi.ts`: new summary and description for `POST
/api/agents/me/secret-proposals`. The description states the
reporting-line target rule, the `env.<KEY>` config path, and that a
board member approves the binding.
- `server/src/__tests__/secret-binding-discovery.test.ts`: the four
queries return the route in their top three results. The test fails on
master (red) and passes with this change (green).

## Verification

- `npx vitest run` on 24 server guidance, discovery, OpenAPI and runner
API test files: 2305 passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
`changed: false`.
- Paid chat eval run: see the PR comment.

## Risks

- Low. Only OpenAPI text changes. Other search results can move down by
one place for queries that share these words.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:46:26 -06:00
DottaandClaude Opus 5.5 3eca7a6112 fix(runner): log which check refuses a saved provider-pack admission on reattach (#21085)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode turns run in a remote Runner. When a server restarts, the
new server reconnects to the Runner and reuses the provider pack
admission that the run saved earlier.
> - On staging, a run reconnected cleanly after an upgrade but then
failed with `native_provider_artifacts_admission_unverified`. Its saved
admission state stayed "pending".
> - The reuse step refuses a saved admission for several reasons. It
returns only false and logs nothing, so we cannot tell which check
refused it.
> - This pull request adds diagnostics only. It names the refused check
in the run log and the server log. It also logs when the admission stays
unverified for a session that needs it.
> - The benefit is that the next upgrade audit shows the real cause, so
the next fix can target it.

## Linked Issues or Issue Description

**What happened?**
After an automatic staging upgrade, a run reconnected to its live Runner
without a new process. The next checkpoint failed with
`native_provider_artifacts_admission_unverified`, which used one of the
run's 3 attempts. The run log had no line that explained why the saved
admission was not reused.

**Expected behavior**
When a saved admission is not reused, the run log and the server log
name the check that refused it.

**Steps to reproduce**
1. Start an OpenCode run on a remote sandbox.
2. Restart the server, so the run reconnects to its Runner.
3. If the saved admission is refused, the run fails later with
`native_provider_artifacts_admission_unverified` and no reason in the
logs.

## What Changed

- `native-provider-pack-admission.ts`:
`rememberRecoveredNativeProviderPack` accepts an optional `onRefused`
callback. It is called with one reason: `source_state_pending`,
`source_state_unknown`, `source_state_missing`, `receipt_missing`,
`runner_instance_differs`, `environment_lease_differs`,
`receipt_invalid`, or a session identity reason. The result of every
check stays the same. An error in the callback is ignored.
- `native-session-executor.ts`: the reattach path logs the reason to the
run log and the server log. When `recordNativeProviderPackAdmission`
returns false for a session that requires admission, it logs that too.
- `native-provider-pack-admission.test.ts`: each refusal names its
reason, and a reused admission reports nothing.

## Verification

- `npx vitest run
src/services/native-runtime/native-provider-pack-admission.test.ts`
passes 46/46. The new reason tests fail on the old code because
`onRefused` is never called.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts` passes
1412/1412.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. This change adds diagnostics only, and admission decisions
do not change. The new log lines contain only a fixed reason code and
run ids. They do not contain secrets or file paths.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:38:23 -06:00
DottaandClaude Opus 5.5 b853cc28ae perf(server): reach readiness without waiting for backlog-sized recovery passes (#21014)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server process starts, recovers saved work, and then reports
ready on `/api/ready`. The deploy health gate waits for that signal.
> - On a busy staging workspace, one boot took 85 s to bind the listener
and 4.4 minutes to report ready. Small, idle stacks take 20–90 s.
> - Two parts of that time do not need to block readiness. The server
published several hundred UI assets one at a time before it bound the
listener. After the listener bound, it waited for the first pass of each
periodic recovery queue, and those passes grow with the backlog.
> - The volume-free data gate, which can refuse a start, ran last. By
then the server had already dispatched queued runs that a refusal would
abandon.
> - This pull request moves the refusal gate to the start, publishes the
assets 8 at a time with replayable bodies, and runs the backlog-sized
first passes after readiness, in their original order.
> - The benefit is a faster time from container start to ready, and no
work admitted before a gate that can refuse to start.

## Linked Issues or Issue Description

No public issue exists. This is the problem description (enhancement
template).

**Is your enhancement related to a problem? Please describe.**
Server boot time grows with the saved backlog. These timestamps come
from the log of one staging boot (release 87fe4d9793). The container
started at 01:16:56. The listener bound at 01:18:21, 85 s after start.
The fresh queued-run dispatch completed at 01:18:25. The full queued-run
dispatch pass completed at 01:20:14. The volume-free data gate ran at
01:21:20. Readiness came about 4.4 minutes after start. During startup
the log showed hundreds of `An error was encountered in a non-retryable
streaming request.` lines.

**Describe the solution you'd like**
Readiness waits only for the correctness gates and for the adoption of
old runs. Work that the periodic queues already retry runs after
readiness. A gate that can refuse to start runs before the server admits
any work.

**Describe alternatives you've considered**
- Move all post-listen recovery to the background. We rejected this.
Native recovery can fail closed and must still fail the start. It also
adopts old runs.
- Skip asset re-publication when the image is unchanged. We rejected
this. The collision check (never overwrite different bytes) needs a read
per asset.

**Additional context**
Startup map (line numbers are from `origin/master` 82e0393598):
- Blocks the listener: migrations and the embedded database
(`server/src/index.ts:369-741`); config and identity (`:753-774`);
backfills (`:821-833`, where
`sweepSupersededPendingRequestConfirmations` scans every pending
confirmation); `qualifyCloudConfigurationForCurrentImage` (`:956`,
synchronous package hashing); `createApp` (`:964`), which awaited serial
UI asset publication (`server/src/app.ts:1110`,
`server/src/retained-ui-assets.ts:69-105`).
- Blocks readiness after the listener: first passes of tool reviews,
gateway sessions and delivery workers (`index.ts:1529-1558`); the
terminal workspace reaper (`:1652`); native recovery and hot-restart
adoption (`:1707-1779`); the first passes of 17 recovery queues
(`:1782-1890`), including the full queued-run dispatch (`:1805-1811`);
backup registration (`:1979-1999`); the data gate and package
qualification (`:2020-2026`).
- Grows with the backlog: the queue first passes (queued runs, pending
deliveries, watchdogs, native sessions and results), the terminal
workspace sweep, the native recovery scan, and the confirmation sweep.
UI asset publication grows with the build (hundreds of assets), not the
backlog.

## What Changed

- **Volume-free data gate runs first.** `assertEphemeralServerData` now
runs right after `claimVolumeFreeControllerState` and the runtime config
load. That is before `createApp`, before the listener binds, and before
any queued-run dispatch. It used to run at the end of startup. The
package qualification gate stays at the end, because bundled plugin
installs can change its inputs.
- **UI asset publication is faster but still a gate before work.**
`createApp` starts the publication while it builds its routes. It awaits
the result before its first job, queue or run starts, which is the same
point as on master. A colliding build still refuses to start before it
admits any work (Greptile review).
- **Publication sends Buffer bodies, 8 at a time.** Each asset is read
once into a Buffer. The S3 client can replay a Buffer body; it cannot
replay a stream. The new code sends no stream bodies, so it no longer
reaches the SDK path that logs "non-retryable streaming request". After
the first failure, no new assets start. A collision found on one asset
outranks a storage outage on another (Greptile review).
- **Backlog-sized first passes run after readiness.** The 17 queue
registrations after native recovery (residual recovery through
stopped-instruction collection) now run in a tracked background
sequence, in the same order. Each pass is isolated. A pass that fails is
logged with its name, and the later passes, the housekeeping and the
scheduler still run. A queue whose first pass failed stays registered,
and the coordinator retries it. Native recovery and hot-restart adoption
still complete before ready. Idle sleep stays blocked while the sequence
runs, and shutdown waits for it like other scheduler work.
- **Queued turns keep the right status.** A new flag,
`startupQueuedRunPassOutstanding()`, stays true until the first full
queued-run pass ends. Until then the execution projection still shows
"Waiting for server restart". The flag clears even when that pass fails,
because the dispatch queue then owns those turns.
- **Startup timeline log.** At readiness the server logs the duration of
each startup step (`startup timeline at readiness`). It logs again when
the background passes finish. No OpenTelemetry collector is needed.
- **Benchmark.** A new script,
`scripts/benchmark-retained-ui-assets.ts`, measures publication against
storage with a fixed round-trip latency. It can fail when time exceeds a
budget (`RETAINED_UI_BENCH_MAX_MS`).

Why each moved step is safe:
- **Data gate (earlier).** Its inputs are the migrated database and the
owner marker. `claimVolumeFreeControllerState` still runs first.
Interaction with #20952 and #20990, which are both in this branch: those
changes made the gate tolerate
`native_harness_backups_require_migration` and
`local_instruction_copies_require_settlement` when the filesystem
carries the volume-free owner marker. That covered the case where this
boot's own dispatch and recovery wrote such run state before the late
gate ran. With the gate first, no work of this boot can write run state
before the gate: the app, its queues, run admission, native recovery and
dispatch all start after it. A test checks this order. So the gate sees
only state that a previous process left, and the owner-marker tolerance
still decides that state. This is strictly safer. Nothing relied on the
gate running after recovery: the only work that could clear a blocker
was background work that the old position never awaited. Effect: a
refused start no longer leaves dispatched runs behind.
- **UI asset publication.** The await point is unchanged, so the
gate-before-work order matches master. Only the time it takes changes.
- **Recovery queue first passes (after ready).** Each queue owns its
work and retries it on deadlines, commit wakes, and intervals. A first
pass that ends after ready behaves like a later pass. Interrupted runs
are still classified and adopted by native recovery before ready, and
they resume without a Retry. No wait is added before a turn starts:
turns can start sooner. The HTTP API already served requests during
recovery, because readiness never gated requests. Observable changes:
- The deploy gate passes earlier. A queue that throws while it registers
is now logged instead of failing the start. Each queue's `ready` already
absorbed run errors.
- `startup queued-run dispatch pass complete` can now appear after
`Server startup recovery complete`.

Left alone (safety unclear or not measured):
- The first passes of tool review recovery, gateway sessions, the
delivery workers, the GitHub workers, and the terminal workspace reaper.
These run before native recovery, and I did not prove the ordering safe.
- `qualifyCloudConfigurationForCurrentImage` before `createApp`. It
hashes plugin packages synchronously; its result is only a warning
there.
- The confirmation sweep and principal access backfill before the
listener. They scan whole tables.

## Verification

- Red/green: with `server/src/index.ts`, `app.ts` and
`retained-ui-assets.ts` reverted to master, these new tests fail. With
the change they pass:
  - `server-startup-feedback-export.test.ts`:
- refuses ineligible volume-free data before the app, the listener or
any queued-run dispatch
- reports ready while the first queued-run dispatch pass still runs,
then finishes it and the passes after it
- runs every deferred recovery pass after readiness when one throws,
logs it, and keeps the failed queue registered (fails on the first
revision of this PR, passes now)
  - `retained-ui-assets.test.ts`:
    - re-publishes with bounded parallel, replayable writes
- refuses a colliding build even when another asset's storage outage
fails first (fails with `failure ??=`, passes now)
- Regression guards that also pass: "checks volume-free data once,
before any of this boot's own startup work can write run state" and
"stops starting new assets after one collides".
- Focused suites pass after merging `origin/master`:
`server-startup-feedback-export` (131 tests), `retained-ui-assets`,
`execution-projection`, `liveness`, `heartbeat-startup-fresh-dispatch`,
`ephemeral-server-startup-run-state`, `ephemeral-server-data`,
`ephemeral-migration`, `ephemeral-server-contract` and
`static-index-html`. `tsc --noEmit -p server/tsconfig.json` is clean.
- Benchmark (measured locally, 400 synthetic assets, 25 ms simulated
round trip): `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-retained-ui-assets.ts`

  | | serial (before) | 8 in flight (after) |
  | --- | --- | --- |
  | first publish | 10.5 s | 1.3 s |
  | restart re-publish | 21.3 s | 2.7 s |

This time is still before the listener, as on master, but it is about 8
times shorter and overlaps route construction.
- Local boot (measured, empty instance, embedded PostgreSQL, 3 boots
each, a shared and noisy machine):
  - listener to ready: before 385–427 ms; after 212–303 ms.
- Spawn to listener was 5.5–12.8 s in both cases. Module load and the
embedded database dominate it, and it does not differ.
- Not measured: the hosted saving. From the staging log, the inferred
saving is the queued-run pass and the passes after it (about 175 s:
01:18:25 to 01:21:20), plus most of the serial asset publication before
the listener. Confirm with the new `startup timeline at readiness` log
line after deploy.

## Risks

- **Medium: readiness semantics change.** The deploy gate now passes
before the first recovery passes finish. The old server can leave
earlier while the new server still dispatches its backlog. Queued turns
show "Waiting for server restart" until the dispatch pass ends, and the
queues retry on their own.
- **Low: earlier data gate.** It can refuse a start that the old order
would have admitted only if background work cleared a blocker first.
That was a race before this change.
- **Low: parallel publication against throttling storage.** Eight
requests at a time, with the existing per-asset backoff (0.5–8 s). A
persistent outage still lets the server start on local files.
- No migrations. No API contract changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, subagents for code search). Extended
reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21010 (retry a throttled asset upload).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:35:47 -06:00
DottaandClaude Opus 5.5 2e593bf3cd fix(native): drop a repeated identity-record check and prove an unsettled predecessor stays an ordinary refusal (#21086)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21001 sends a re-wake to a fresh sandbox when live adoption of a
crashed predecessor's warm Runner is refused, and lets an inherited
Runner identity pass the "workspace unchanged" proof
> - The reviewer found two gaps on #21001 that were not answered before
it merged
> - The unchanged proof checked for a `native.process_identity_recorded`
event twice, and no test proved that an unsettled predecessor stays an
ordinary refusal
> - This pull request removes the repeated check and adds that proof
> - The benefit is simpler proof code and a test that stops the
fresh-sandbox fallback from widening to unsettled predecessors

## Linked Issues or Issue Description

Refs #21001, #20863

**What happened?**
Two review warnings on #21001:
1. `carriesOnlyInheritedRunnerIdentity` in
`remote-workspace-unchanged-proof.ts` looked for the run's own
`native.process_identity_recorded` event.
`executionStartedEventPredicate()`, later in the same proof, already
refuses a run with that event (`execution_recorded`).
2. The existing denial cases in
`native-controller-recovery.integration.test.ts` use `toThrow(denied)`.
That also matches the wrapped
`NativePredecessorRetirementUnprovenError`, so a regression that wraps
an unsettled predecessor's refusal would pass.

**Expected behavior**
The proof has one check per fact, and the tests fail when an unsettled
predecessor goes to the fresh-sandbox fallback.

**Steps to reproduce**
1. Read `proveRemoteWorkspaceUnchanged` and
`executionStartedEventPredicate`.
2. Change `remote-warm-session-recovery.ts` to wrap every refusal; the
old denial cases still pass.

## What Changed

- `server/src/services/remote-workspace-unchanged-proof.ts`: delete
`carriesOnlyInheritedRunnerIdentity` and its `PROCESS_IDENTITY_RECORDED`
import; call `carriesInheritedRunnerIdentity(tx, run)` directly. A run
with its own identity record is still refused, now by
`execution_recorded`.
-
`server/src/services/native-runtime/native-controller-recovery.integration.test.ts`:
- The denial cases for an unsettled predecessor (other company, other
execution, finalization lease held, run still `running`) now assert the
error is not a `NativePredecessorRetirementUnprovenError`.
- The pending-command case after the predecessor is `succeeded` and
`committed` now asserts it is one (the #21001 behavior).
- The crash-restart case adds a predecessor run that is `failed` or
`running`: an unsettled journal is refused with
`warm_recovery_refused=owner_unsettled`, not wrapped.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts
src/services/remote-workspace-unchanged-release.test.ts`: 50 passed. The
three inherited-identity cases from #21001 return the same results
(released / refused / refused).
- `cd server && npx tsc --noEmit -p tsconfig.json`: passes.

## Risks

- Low risk. No behavior change: the removed check is covered by
`executionStartedEventPredicate()`. The rest is tests.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:35:15 -06:00
DottaandClaude Opus 5.5 bed5f70278 perf(ci): rebalance server test shards and repair the weekly weight refresh (#21016)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A Core image is published only after the release verification of the
same commit passes (`cloud-readiness.yml` calls `release-verify.yml`)
> - The image is ready about 3 minutes before that gate opens, so the
slowest verification job sets the staging release time
> - The slowest job is a general-server test shard: 11.8-12.7 min, while
other server shards take 8-9 min
> - The shards are balanced by a duration manifest, but the manifest has
not changed since 2026-10-31. The weekly refresh that updates it failed
on 2026-11-02 and 2026-11-09
> - About 40 newer suites per shard use the 1 s median fallback weight,
but several of them run for 90-200 s, so they pile up on one shard
> - This pull request repairs the weekly refresh and refreshes the
server weights from three current master verifications
> - The benefit is a shorter slowest server shard (predicted 10.3 -> 8.8
min of test time) on every master verification and every PR run

## Linked Issues or Issue Description

**What happened?**
The Core image publication waits about 13.5 min for source verification.
The critical path is one general-server shard. In three master
verifications (runs 43133035381, 43134251278, 43145252393) the slowest
server shard took 12.1, 11.0 and 12.7 min. The fastest took 8.0, 8.9 and
8.4 min.

**Root cause**
`scripts/general-server-shard-durations.json` came from one run on
2026-10-31. The weekly `ci-test-durations.yml` refresh failed twice:
- 2026-11-02 (run 41408033686): `gh api .../jobs/<id>/logs` failed. A
server shard log is 30-65 MB. One download took 59 s locally. Three
parallel downloads overran the 120 s `execFile` timeout.
- 2026-11-09 (run 42580964686): the refresh's own check failed.
Merged-PR sources for the Runner lane have a leading `pull` key, and the
test accepted only `run, attempt, sha, url`.

**Expected behavior**
The weekly refresh finishes and keeps the shards balanced as suites are
added.

## What Changed

- `scripts/ci/gh-api.mjs`: job-log downloads get a 15 min timeout and a
512 MB buffer. JSON calls keep 120 s. A failed call now gives a short
error and does not print the partial 60 MB log.
- `scripts/ci/pr-run-durations.mjs`: new shared `assertManifestSource`.
It accepts the optional leading `pull` key that `collectMergedPrSamples`
writes. The checked-in Runner manifest test and the collector test both
use it, so the collector format stays covered.
- `.github/workflows/ci-test-durations.yml`: refresh job timeout 20 ->
45 min (the server lanes alone fetch about 30 large logs).
- `scripts/general-server-shard-durations.json` and
`scripts/serialized-shard-durations.json`: refreshed with `node
scripts/ci/refresh-test-durations.mjs --lanes=server 43133035381
43134251278 43145252393` (median of three successful protected-master
Cloud readiness runs; 1757/1815 general files and 173/173 serialized
files measured).
- No test is added, removed, skipped or reassigned to a different lane.
No shard count changes. `docker.yml`, the Dockerfile and the PID-1 check
are not touched.

## Verification

Measured job timings (minutes from the first job start; three master
verifications):

| Job | 43133035381 | 43134251278 | 43145252393 |
| --- | --- | --- | --- |
| Slowest server shard (duration) | 7/10: 12.1 | 6/10: 11.0 | 8/10: 12.7
|
| Fastest server shard | 2/10: 8.0 | 2/10: 8.9 | 5/10: 8.4 |
| Runner chaos evals (restart/replay) | 10.2 | 11.9 | 11.0 |
| Verify Paperclip Runner (protocol) | 8.9 | 8.9 | 9.6 |
| Chat (slowest of 3) | 8.8 | 10.5 | 9.4 |
| Serialized (slowest of 5) | 9.3 | 9.5 | 9.2 |
| Build / Typecheck | 4.5 / 2.4 | 4.5 / 3.1 | 4.8 / 2.5 |
| Gate "Cloud source verified" ends | +13.5 | +13.2 | +13.9 |

- Queue to runner start is 0.5-1.1 min for every job. In a server shard,
setup (checkout, pnpm, Node, install) is about 50 s and the preflight
plus plugin-sdk build is about 8 s. The test step is the rest (9-11.8
min). Setup is not worth caching.
- Shard prediction, using the refreshed medians as the cost of each file
and the 3 fork workers per shard:
- general-server, 10 shards: old weights max 10.3 / min 8.1 min; new
weights 8.8 / 8.8 min.
- serialized, 5 shards: old weights max 8.2 / min 4.6 min; new weights
6.6 / 6.6 min.
- The old-weight prediction matches what CI saw: predicted 10.3 and 9.9
min for shards 7/10 and 6/10, measured test steps 11.25 and 10.1 min in
run 43133035381.
- Red/green:
- `node --test --test-name-pattern="gh api"
scripts/ci/test-selection-pr-timings.test.mjs` failed before the fix
(`log timeout 120000`) and passes after it.
- With a `pull` key in the Runner manifest sources, the old "package
lane manifest" test failed with the same assertion as run 42580964686.
The new test passes.
- The refresh script, which died before the fix, completed locally in 8
min after it.
- `node --test scripts/ci/test-selection-timings.test.mjs
scripts/ci/test-selection-pr-timings.test.mjs
scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/__tests__/e2e-shard.test.mjs`: 76/76 pass. This includes the
complete, non-overlapping shard cover checks.
- `actionlint .github/workflows/ci-test-durations.yml`: clean.
- Not run: the release-verify workflow itself, the scheduled refresh
workflow, and the Runner/e2e lane refresh. The shard times above are
predictions from logged per-file timings, not a measured CI run on this
branch.

## Risks

- Low risk. Weights only move files between shards of the same lane. The
shard tests prove every file still runs exactly once.
- The predicted gain is about 1.5 min on the slowest server shard
(12-12.7 -> about 10.5 min job time). The gate then waits for
"Pre-release Runner chaos evals / Restart, replay, trace, and recovery
faults" (10.2-11.9 min). So the expected gate saving is about 1 min
(13.5 -> about 12.5 min), not the full 1.5 min. Run-to-run noise on a
shard is about ±1 min.
- A larger log timeout can make a stuck download wait up to 15 min. The
45 min job timeout still bounds the run.

Candidates for the owner (not changed here):
- The chaos-eval job is now the bound. Its "server finalization" step
takes 7-9 min. Decide if it must gate image publication, or split it.
- Server shard logs are up to 811k lines, mostly repeated stack traces
from `server/src/services/execution-preparation-timing.ts`. That slows
log upload and this refresh.
- 9 test files run over 180 s (largest:
`chat-channels.integration.test.ts` 429 s,
`heartbeat-process-recovery.test.ts` 344 s). One file cannot be split
across shards.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, GitHub CLI read-only). Standard reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Related: #19650 (earlier repair of the weekly refresh), #21013 (Core
image publish time; separate files).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:29:27 -06:00
DottaandClaude Opus 5.5 f8e1fb0278 perf(docker): take the orphan-reaping pull and pack chmod off the image path (#21013)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `docker.yml` builds and signs the standard Core image for every
master push. Paperclip Cloud deploys a Core commit only after this image
is published and attested.
> - A master commit takes 29 min (p50) and 56 min (p90) to reach a
published image. Every minute here delays a staging fix.
> - The run logs show where the time goes. The exact-source gate (about
14 min) is the critical path today. The build legs finish about 3 min
before it. After the gate, `merge-and-push` spends about 95 s to pull
the 4.9 GB amd64 image for one probe.
> - This pull request moves that probe into the build legs, which run
before the gate. It also removes a duplicate 187 MB layer from the amd64
image.
> - The benefit is about 95–100 s less from gate to signed image on
every master commit, with the same proof. arm64 is now also probed.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The time from a commit landing on master to the signed standard image
(`ghcr.io/paperclipai/paperclip:sha-<full>` plus its attestation).

**Subsystem affected**
`.github/workflows/docker.yml`, `Dockerfile`,
`scripts/standard-image-contract.mjs`.

**Current behavior**
Measured on run 43133035209 (sha 42fd790e, total 999 s):

- Push 01:13:47. The slowest exact-source job, `General tests (server
(7/10))`, ends 01:27:08. `Cloud source verified v1` waits 20 s for a
runner and ends 01:27:44. The watcher sees it at its next 30 s poll,
01:27:58.
- `build-and-push (amd64)` ends 01:24:52 and arm64 ends 01:21:59. Both
have slack before the gate. Run 43134250912 shows the same shape: the
gate opened 01:44:37 and amd64 ended 01:41:53.
- `merge-and-push` starts 01:28:03 and pushes the manifest at 01:28:34.
Then "Verify PID 1 reaps orphaned processes" pulls the full amd64 image
(01:28:34 to 01:30:09, 95 s). "Verify the immutable standard image" runs
the same probe again on the same bytes (5 s). The attestation ends
01:30:22.

Last 60 master pushes (2026-11-11 20:17Z to 2026-11-12 03:21Z): 43
cancelled or superseded, 14 succeeded, 2 failed, 1 in progress. Push to
first published image that contains the commit: p50 29.4 min, p90 55.5
min, max 85.6 min. A successful run takes 22.5 min (p50) from push. The
median gap between pushes is 4.5 min.

Where the amd64 "Build and push by digest" step spends its 582 s:

| Time | Phase | What happens |
| --- | --- | --- |
| 0–8 s | Load | Dockerfile frontend, cache manifest import |
| 8–54 s | Pull cached layers (46 s) | `deps`, `rust-chef`,
`runner-deps`, `runner-build` and `base` are **all cache hits**. The
`build` stage starts `FROM runner-build` and copies `deps`, so BuildKit
downloads and extracts them: a 932 MB node_modules layer (13 s download,
24.5 s extract) and about 650 MB of Rust `target/` |
| 54–108 s | Per-commit build (54 s) | `COPY . .` 10 s, mtime
normalization 11 s, UI 5 s, plugin SDK 3 s, server tsc 18.5 s, runtime
bundle 5.6 s |
| 108–206 s | `runner-provider-pack` (98 s) | Rebuilt every commit: it
is `FROM build` and stamps the commit. pnpm install about 25 s, pack 9
s, **Kimi smoke 37 s (amd64 only)**, archive 28 s |
| parallel | `production` | Tool layer is a cache hit but pulls 1.11 GB
(11 s). `COPY --from=build /app` 24 s |
| 206–294 s | Pack copy and chmod | `COPY` pack 7.5 s. `chmod -R a+rX`
plus UID check **71 s** (arm64: 20.5 s) |
| 294–420 s | Image export (126 s) | 115 s compressing new layers:
`/app` 1.40 GB, pack 1.01 GB, pack archives 1.06 GB, chmod copy 0.19 GB
(compressed). Push 10 s |
| 420–582 s | Cache export (172 s) | `mode=max`: 152 s "preparing"
(compressing the per-commit layers of `build` and
`runner-provider-pack`, which no later commit can reuse), 20 s upload |

What rebuilds every commit, and why: only the `build` stage and
everything after it. `PAPERCLIP_BUILD_COMMIT` is already declared after
the dependency layers, `CLI_TOOLS_CACHE_EPOCH` is weekly, and dependency
install already copies manifests first. The commit ARG is used by the UI
build, the server stamp, and the provider pack, so those must rebuild. A
pnpm `--mount=type=cache` gives nothing here: the runners are ephemeral,
and cache mounts are not exported to the registry cache.

Why amd64 is slower than arm64: both legs run on runs-on AWS 32-core
instances (m6a.8xlarge and m6g.8xlarge), not on `ubuntu-latest`. The
difference is content, not CPU. amd64 has a larger provider pack with
the Kimi smoke test (37 s), a 187 MB chmod copy layer (arm64: 0 MB), a
4.9 GB image (arm64: 2.9 GB), and a 172 s cache export (arm64: 100 s).

**Proposed behavior**
Run the PID 1 probe in each build leg, on its own architecture, against
the digest that leg pushed. Before signing, require that every Linux
image in the standard index is one of those probed digests. Set the
provider pack modes in the stage that writes the pack.

**Reason and benefit**
The probe pull leaves the post-gate path. The leg has about 3 min of
slack before the gate. The chmod change removes about 50 s and 187 MB
from the amd64 leg, which offsets most of the probe pull that the leg
now does.

**Breaking changes**
None. Tags, digests, attestation subject, and signer identity are
unchanged.

**Additional context**
The concurrency proposal and the build-in-PR plan are below, in "Risks".
They are not implemented.

## What Changed

- `docker.yml` `build-and-push`: new step "Verify PID 1 reaps orphaned
processes". It runs `scripts/assert-orphan-reaping.sh` in
`ghcr.io/...@<pushed digest>` with `--platform` set to the leg platform.
It runs before "Export digest", so `merge-and-push` only receives probed
digests.
- `docker.yml` `merge-and-push`: removed the two `docker run` probes
(tag and digest). "Resolve standard image digest" now runs
`standard-image-contract.mjs --resolve "$GITHUB_SHA" --verified-digests
/tmp/digests`.
- `standard-image-contract.mjs`: `--verified-digests <dir>` reads each
probed leg digest from the registry, checks that its bytes hash to that
digest, and collects its Linux image digests. It fails unless every
Linux image in the standard index is in that set. Registry content is
addressed by digest, so this proves that the signed index names exactly
the probed bytes, without a second pull.
- `Dockerfile`: `chmod -R a+rX /provider-pack` now runs in the same
`RUN` that writes the pack. `COPY` keeps the modes. The production stage
keeps the readability check as UID 65534.
- Tests: new contract tests (membership, bare manifest legs, digest
mismatch, empty set, lookup by digest). New workflow test: the probe
runs between the build and "Export digest", and the publisher does not
`docker run`. Updated the provider-pack Dockerfile test.
- `doc/standard-image-contract.md`: describes the per-leg probe and the
membership check.

Expected savings, per master commit:

| Change | Saving | Confidence |
| --- | --- | --- |
| Probe moved out of `merge-and-push` | 95–100 s from gate to
attestation (measured: 95 s pull + 5 s second probe) | High |
| Pack chmod in its own stage | About 50 s and 187 MB in the amd64 leg
(71 s vs 20.5 s on arm64, where the chmod layer is empty) | Medium-high
on size, medium on seconds |
| Net amd64 leg | About +10 to +45 s (probe pull estimated at 60–95 s,
minus about 50 s). Still before the gate | Medium |

## Verification

- `node --test scripts/standard-image-contract.test.mjs`: 10 pass (2
new).
- `node --test` on
`.github/scripts/tests/{cloud-readiness,docker-canary-promotion,post-merge-runner-routing,lockfile-refresh-workflows}.test.mjs`,
`scripts/__tests__/release-verify-workflow.test.mjs`,
`scripts/reuse-canary-image.test.mjs`,
`scripts/standard-image-contract.test.mjs`,
`scripts/cloud-source-verification.test.mjs`: 484 pass.
- `vitest run server/src/__tests__/docker-build-stamp.test.ts
server/src/__tests__/container-init-reaping.test.ts`: 22 pass. Red
check: with the `origin/master` `Dockerfile` and `docker.yml`, the 2 new
or changed assertions fail.
- Real registry, read only: `node scripts/standard-image-contract.mjs
--resolve 42fd790e80 --verified-digests
<dir>` with the two leg digests from run 43133035209 returns the
published index digest. With only the arm64 leg digest, it fails:
"linux/amd64 image sha256:0f0642e6… did not pass the build-leg
orphan-reaping probe".
- `actionlint .github/workflows/docker.yml`: the same 6 shellcheck
findings as `origin/master`, none new.
- Shell syntax check (`sh -n`) of the changed pack `RUN`.

Not exercised: the workflow itself (it runs only on master pushes, and I
did not dispatch it). No local `docker build`: the Docker daemon is not
running on this machine, and a full build does not fit a few minutes.
The leg probe pull time is an estimate from the publisher runner.

## Risks

- The leg now pulls its own image (4.9 GB amd64). AWS legs have about
175 GB free. Hosted tag builds run "Free runner disk" first, and the old
publisher pulled the same image on a hosted runner for tag builds.
- If the registry returns a leg index in an unexpected form, the resolve
step fails closed and no master image is signed. The real-registry check
above used today's exact shapes (OCI index with one platform image and
one attestation manifest per leg).
- Tag builds (`v*`, `nightly/v*`, `beta/v*`) keep the probe, now per
leg. They have no standard-image signing step, and the manifest list is
made only from probed digests.

### Deliberately not changed

- **The exact-source gate.** It is a deliberate "never publish an
unverified image" rule (`watch-release-source.mjs --require-proof` and
`--check-proof`, `doc/cloud-build-readiness.md`). I kept it. The owner
levers are: the slowest `cloud-readiness` shard (`server (7/10)`, 12
min, sets the gate), the 20 s runner wait for `Cloud source verified
v1`, and the watcher's 30 s poll (15 s average delay). I did not shorten
the poll because the org API rate limit was exhausted today.
- **Cache export (172 s amd64, 100 s arm64).** `mode=max` compresses
about 2 GB of per-commit pack layers every build. A fix: a separate
cache-only build of a small aggregate stage (`deps`, `runner-build`, the
production tool layer) with `mode=max`, and the main build with
`cache-from` only. Not done here: the legs are not on the master
critical path today, and I cannot measure it without a master run. It is
worth doing for tag builds or if the gate gets faster.
- **Cached layer pull (46 s).** The `build` stage starts `FROM
runner-build`, so it downloads about 650 MB of Rust `target/` that it
deletes at the end. The server build's `prepare:runner-vendor` uses that
target, so this needs a separate change.
- **Provider pack per commit (98 s amd64).** It rebuilds because it is
`FROM build` and stamps the commit. Its 1.06 GB transport archives also
duplicate the 1.01 GB pack inside the image. Keying the pack on its own
inputs is a product-contract change.

### Concurrency proposal (not implemented)

Today one run holds `docker-${{ github.ref }}` for 17–23 min. Most of
that is waiting for the gate, not building. A newer commit waits for
that run before its build starts (run 43134250912 waited 6.4 min).
Proposal:

1. Remove the workflow-level group. Add job-level groups:
`build-and-push` uses `docker-build-${{ github.ref }}-${{ matrix.arch
}}` with `cancel-in-progress: false`. At most one build per architecture
runs and the newest commit replaces the pending one. So not every commit
is built.
2. `merge-and-push` uses `docker-publish-${{ github.ref }}` with
`cancel-in-progress: false`. It skips publication when a newer master
commit already has a signed image, so the image order stays monotonic.
3. A superseded run must not keep its source watcher alive, or the API
load grows with each push. Fold the early watcher into the gate job, or
end it when its build job is cancelled.

Effect: the newest commit starts building at push, so its image
publishes at about gate time plus 45 s, instead of after the previous
run ends. That removes the 0–17 min queue wait, which is most of the gap
between the p50 of 29 min and the gate time of about 14 min.

### Plan: build in PR CI and promote on merge (not in this PR)

What must change:
1. Turn on the GitHub merge queue for `master`. The queue tests a
`gh-readonly-queue/...` commit and then fast-forwards `master` to that
exact commit, so the SHA stamped in the image (`PAPERCLIP_BUILD_COMMIT`,
`build-info.json`, the provider pack `runnerSourceRevision`, the UI
bundle) is the final master SHA.
2. On `merge_group`: build both architectures with **no secrets and no
registry credentials**, because the code under test is not yet on
master. Write each image as an OCI layout artifact. A second job that
runs no PR code pushes those artifacts by digest to a staging
repository, keyed by the merge-group SHA.
3. Run `cloud-readiness` on `merge_group` for the same SHA, so
exact-source proof exists before merge.
4. On the master push, `docker.yml` looks up the staged digests for
`github.sha`, checks that the tree and SHA match, retags them as
`sha-<full>`, and signs with the same identity
(`docker.yml@refs/heads/master`). If no staged image exists, it falls
back to the current build.

Guarantees to keep: never publish or sign an image without exact-source
proof for that SHA. The signer identity and the attested digest stay the
same. No secrets in any job that runs non-CODEOWNER code. The PID 1
probe runs on both architectures before signing. Superseded queue
entries do not publish.

Expected saving: from master push to signed image, about 1–2 min
(lookup, retag, attest) instead of 29 min p50 and 56 min p90 today. The
work moves before the merge, so the merge itself waits for the queue
(about the current gate time of 14 min). A staging fix then reaches
Cloud about 25 min sooner at the median.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used via Claude Code
with tool use (shell, file edits, read-only GitHub API and registry
reads). Default reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:29:11 -06:00
DottaandClaude Opus 5.5 5820d43197 fix(runner): relaunch a settled OpenCode Runner with its admitted artifact paths (#21076)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agent turns run in a remote Runner in a Daytona sandbox.
The Runner saves its state, so a retry of the same run can start a new
Runner from that saved state.
> - The first launch of a run starts OpenCode from the admitted provider
files under one folder in the sandbox. The new Runner replays that first
launch and checks that its own launch profile names the same files.
> - When a retry reconnects to a Runner that already stopped, Paperclip
starts a new Runner. On that path, Paperclip did not restore the
admitted provider files and gave the new Runner the provider pack folder
instead.
> - The new Runner then refused the replayed launch ("does not match the
runner-owned qualified profile"). That used the last retry, and the task
stayed blocked.
> - This pull request restores the admitted provider files and gives the
new Runner the same folder as the first launch.
> - The benefit is that a retry after a server upgrade continues the
same run instead of blocking the task.

## Linked Issues or Issue Description

**What happened?**
On staging, after an automatic upgrade, a run used all 3 attempts. The
last attempt started a new Runner from its saved state. It failed in the
drain step with "OpenCode run.prepare launch does not match the
runner-owned qualified profile". The task was blocked and needed a
person.

**Expected behavior**
The new Runner gets the same OpenCode launch files as the first launch
of the run, and the run continues.

**Steps to reproduce**
1. Start an OpenCode run on a remote sandbox with durable provider
artifacts.
2. Let the server restart, so the retry reconnects with
`reattach_remote_runner`.
3. The Runner is found settled, so a new Runner starts from the saved
state. Its launch profile points at the provider pack folder, not at the
admitted artifact folder.

## What Changed

- `native-session-executor.ts`: when a reconnect relaunches a settled
Runner and the run has admitted provider artifacts, restore those
artifacts and prepare the same artifact folder as the first launch.
Other reconnects (live adoption) do not change. Normal turn starts do
not change.
- `native-session-executor.test.ts`: the existing settled-relaunch case
now also covers a run with admitted provider artifacts. It expects the
restore and the admitted folder in the launch profile.

## Verification

- Red: without the fix, the new test case fails
(`restoreNativeProviderArtifacts` is not called, and the launch uses the
provider pack folder).
- Green: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-opencode-artifacts.test.ts` passes
1417/1417.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The change applies only when a reconnect must relaunch a
settled OpenCode Runner and the run has admitted provider artifacts.
- That relaunch now downloads the admitted artifacts from storage, as a
cold continuation already does. This adds time only on that recovery
path, not on a normal turn start.
- The artifact folder check (same account and same digest) still
applies, so a different sandbox account still fails closed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:24:17 -06:00
DottaandClaude Opus 5.5 db8848f559 fix(heartbeat): bound the I7 queued-run check with a resumable cursor (#21022)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #20986 added a reconciler check that reports a queued run whose
dispatch signal was lost (I7 `queued_run_not_dispatched`), and a worker
that re-drives such runs.
> - The Ponytail review of #20986 (4/5 on `fbab23cce6`) found three
issues. #20986 merged at that head before the fix commit was pushed.
> - The new I7 check read every queued run with no cap, while the report
still said that each I7 check reads at most 2,000 records.
> - This pull request adds that fix commit to master. The check now
reads a bounded slice per pass and continues from a cursor. The page
cursor no longer depends on the session date style. An agent with a
scheduled retry now counts as busy.
> - The benefit is that the reconciler pass stays bounded when many runs
are held on purpose, and the coverage statement is true again.

## Linked Issues or Issue Description

Refs #20986

Follow-up to the Ponytail review of #20986:
https://github.com/paperclipai/paperclip/pull/20986#pullrequestreview-5868879754
(review of `fbab23cce6`, score 4/5).

**What is wrong?**
1. `checkQueuedRunsDispatched`
(`server/src/services/recovery/ownership-invariant-report.ts`) paged
through every undispatched queued run with no cap.
`COVERAGE_GAPS.record_cap` still said each I7 check reads at most 2,000
records per pass.
2. The page cursor cast `created_at::text` back with `::timestamptz`.
That works only with the default `DateStyle`.
3. A queued run on an agent with a `scheduled_retry` run was reported
and re-driven, but `checkVisibleNextAction` already treats
`scheduled_retry` as an active run.

**Expected behavior**
The I7 queued-run check reads a bounded slice per pass and reaches every
run over later passes. The cursor format does not depend on session
settings. A scheduled retry counts as the agent's next action.

## What Changed

- `server/src/services/queued-run-dispatch-liveness.ts`: new
`readUndispatchedQueuedRuns`. It reads from an optional cursor until the
set ends or `maxRuns` rows were read, and returns the next cursor.
`listUndispatchedQueuedRuns` uses it with no limit. The cursor text is
`to_char(created_at at time zone 'UTC',
'YYYY-MM-DD"T"HH24:MI:SS.US"Z"')`, which keeps microsecond precision and
does not depend on `DateStyle`. The live-run exclusion now also covers
`status = 'scheduled_retry'`.
- `server/src/services/recovery/ownership-invariant-report.ts`: the I7
queued-run check reads at most `MAX_RECORDS_PER_KIND` (2,000) per pass.
It keeps a keyset cursor in `OwnershipReportState.queuedRunCursor`, and
the next pass continues from that cursor. A pass that stops at the cap
keeps a follow-up pass due (`retryNeeded`), like the I3 and I4 backlogs.
`COVERAGE_GAPS.record_cap` now describes this check.
- `server/src/__tests__/ownership-reconciliation-report.test.ts`: new
test. Bounded slices resume from the cursor and end with `next: null`. A
`scheduled_retry` run on the agent removes its queued runs from the set.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/ownership-reconciliation-report.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/services/heartbeat/dispatch-work.test.ts
src/__tests__/heartbeat-held-comment-wake.test.ts`: 75 tests pass on
this branch (fresh master + this commit).
- Server `tsc --noEmit` passes.
- The restart integration test from #20986 ("starts a continuation that
queued behind its predecessor's sandbox release after a graceful
restart") passed with this change on the #20986 branch, in both cases
(signalled and unsignalled).
- Restart-chaos CR1 360 s, Codex W2, `--daytona-warm
--ephemeral-server`, seed 9512 (`w-20261112T040314-s9512`, on the #20986
change set before this commit): the fault crossed the task and every
rule passed (R1–R6, T, U1–U5).

## Risks

- Low risk. The report reads less per pass. A very large held queue is
now covered over several passes instead of one.
- A queued run behind a `scheduled_retry` is no longer re-driven by the
backstop. The scheduled retry's own promotion signals dispatch.
- No schema change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1112.0-canary.9
2026-11-12 05:19:37 -06:00
DottaandClaude Opus 5.5 559551e977 fix(native): wait for a busy agent directory while retiring a predecessor (#20991)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a reused warm sandbox first retires the previous
run's warm session, and that retirement collects the old session's agent
files
> - On a server without a persistent volume, agent-file collection takes
a per-agent Postgres lock, so runs of one agent collect one at a time
> - #20863 sends a run whose predecessor cannot be retired to a fresh
sandbox. It classified every retirement error as unprovable, including a
busy lock held by another run of the same agent
> - So ordinary lock contention failed the run, dropped a healthy
sandbox and rested the task
> - This pull request makes retirement wait for a busy directory lock in
the same run, and keeps the fresh-sandbox fallback for retirements that
stay unprovable after the lock is obtained
> - The benefit is that busy agents no longer fail follow-up runs on
their warm sandboxes

## Linked Issues or Issue Description

**What happened?**
A no-fault restart-chaos wave with 8 concurrent runs of one Claude agent
on Daytona, with a volume-free server, failed one monitor-due follow-up
after 6 seconds. The error was `adapter_failed`, cause
`native_predecessor_retirement_unproven`: "The previous run's session on
this sandbox could not be retired (Agent directory operation is already
active)". The run also logged `INSTRUCTION_COLLECTION_UNAVAILABLE`. The
stack shows the error came from `claimWarmNativeInstructionCopy` during
instruction preparation. The previous session had closed; only its
agent-file collection met the per-agent lock that another run held.

**Expected behavior**
A busy agent-directory lock is transient. The run waits for it and then
retires the predecessor on the same sandbox. Only a retirement that
stays unprovable after the lock is obtained takes the fresh-sandbox
fallback.

**Steps to reproduce**
1. Run several warm native tasks of one agent at once on Daytona with a
volume-free server.
2. Re-wake one task while another run of the agent collects its files.
3. The follow-up fails with `native_predecessor_retirement_unproven` and
its sandbox is dropped.

Refs #20863, #20829, #20957.

## What Changed

- `retireWarmPredecessor` (native executor): a busy agent-directory lock
(`AgentDirectoryOperationBusyError`) is waited out in the same run,
polling every second for up to five minutes
(`warmRetirementDirectoryWait`). The usual holder is another run's short
collection, so retirement then finishes on the same sandbox with the
copy this run adopted. The run is not deferred: deferring after adoption
would release the adopted copy and could skip saving the predecessor's
edits.
- Past that budget the run fails with its own error,
`native_predecessor_retirement_directory_busy`, not as an unprovable
retirement. Before failing it files the adopted copy's retirement
receipt, so the copy is kept as `pending_collection` and run cleanup
does not mark it `unavailable`. The owner is not marked unretired and
stays registered for its sandbox, so a later run retires it there.
- An idle-timeout close that meets the lock retries after the
agent-directory retry delay instead of leaving the session registered
with no retry.
- Closing idle sessions for environment delete (#20957) reports such an
owner as `busy`, not `failed`.

## Verification

- RED on master, then GREEN:
- `native-session-executor.test.ts -t "directory lock"`: two runs of one
agent contend on the lock while a follow-up retires the predecessor. On
master the claim rejects with `NativePredecessorRetirementUnprovenError`
(`Agent directory operation is already active`). Now a short-lived lock
is waited out and the predecessor is retired and collected on the same
sandbox; a lock that outlasts the budget fails with
`native_predecessor_retirement_directory_busy`, environment delete
reports `busy: 1`, a fresh-sandbox attempt does not forget the owner,
and once the lock is free a later run retires it.
- `native-session-executor.test.ts -t "idle close whose file
collection"`: on master the idle close never retries; now it retries and
closes.
- Existing suites pass: native executor, agent-directory busy heartbeat,
native pre-dispatch retry, native controller recovery and plugin runner
eviction. Server `tsc --noEmit` passes.

## Risks

- Under contention a follow-up can wait up to five minutes before its
first turn. The normal path adds no wait.
- A lock held for more than five minutes fails the run visibly instead
of dropping the sandbox.
- An idle owner whose collection stays busy keeps retrying at the 5 to
15 second agent-directory delay until the lock frees, a controller
restart, or the next admission.
- No persisted state changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:08:15 -06:00
DottaandClaude Opus 5.5 533f135780 fix(native): send a re-wake whose crashed predecessor's warm Runner cannot be adopted to a fresh sandbox instead of failing it (#21001)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a warm remote sandbox leaves its Runner `ready` in
the sandbox. The next run of the task adopts that Runner live, after it
proves that the previous owner finished
> - When the server is killed after a run parked on a question, the warm
owner lived only in the killed server's memory. The Runner in the
sandbox was never suspended
> - A re-wake on the replacement server resumed that sandbox and tried
live adoption. A proof failed, and the new run failed with
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate` and needed a Retry, although
it did no provider work
> - #20863 already sends such a run to a fresh sandbox when the
predecessor's Runner cannot be retired, but the live-adoption path of a
resumed sandbox (`remote-warm-session-recovery.ts`) was not connected to
it, and the failed run's inherited Runner identity blocked the
"workspace unchanged" proof that the fallback needs
> - This pull request connects the two: a refusal of live adoption after
the predecessor is durably terminal with an accepted result goes to the
#20863 fresh-sandbox path, and an inherited Runner identity no longer
counts as a process of the follow-up
> - The benefit is that a re-wake seconds after a crash restart
continues without a Retry

## Linked Issues or Issue Description

Refs #20863, #20844, #20928, #20979, #20953, #20972

**What happened?**
Restart-chaos wave `w-20261112T012841-s7000` (master `4b24352712`,
volume-free mode, Daytona warm, Codex, 10 tasks, K1 and K15 with a fresh
home and overlap), task t08 (W2Q):
- Run `58f57bf1` asked its question and succeeded at 01:29:56. Its warm
owner stayed in the server's memory.
- K1 killed the server at 01:30:06. The replacement was ready at
01:30:15.
- A board comment woke the task at 01:30:17. Run `8f812df6` started at
01:30:19 and resumed the same sandbox (`run.startup.reuse`
`verified_reuse`).
- The first live-adoption inspection passed. The second check
(`revalidate`, in backend setup, before the Runner attach) refused at
01:30:21 (stack: `remote-warm-session-recovery.ts:217` from
`revalidate`). The run failed `native_session_interrupted` with
`INSTRUCTION_COLLECTION_UNAVAILABLE`.
- The failed run kept its full save because the unchanged-files proof
refused with `process_recorded`: the run carried its predecessor's
Runner process identity. Retry (`6968dc0f`) worked.

**Expected behavior**
The predecessor finished with an accepted result before the kill, and
the new run did no provider work. The task must continue without a
Retry.

**Steps to reproduce**
1. Finish a warm remote run that parks on a question.
2. Kill the server (SIGKILL) and start a replacement.
3. Wake the task within seconds. The new run resumes the warm sandbox.
4. Any live-adoption proof fails, and the run fails terminally with
`prior_owner_terminal_state_indeterminate`.

**Which proof failed**
The old refusal had no reason. The wave evidence proves that it was
`revalidate`, after a passed inspection, and not the SIGTERM of the next
K15 fault (that server logged "shutdown signal received" only at
01:31:09). `revalidate` checks the predecessor's coordinator row, the
current run's coordinator row and lease, and the task. The evidence
cannot tell which one changed in the first seconds after the restart.
This pull request names the failed proof
(`warm_recovery_refused=<reason>`), so the next occurrence identifies
it. The fix covers the class: every refusal that concerns the
predecessor or the sandbox.

## What Changed

- `server/src/services/native-runtime/remote-warm-session-recovery.ts`:
- Each refusal names its proof from a closed list:
`identity_incomplete`, `target_not_resumed_sandbox`, `owner_unsettled`,
`execution_mismatch`, `checkpoint_unsettled`, `journal_unsettled`,
`controller_not_stopped`, `prior_lease_mismatch`, `owner_changed`,
`current_lease_changed`, `current_run_not_admitted`,
`current_owner_changed`.
- After the predecessor's run is `succeeded` and its coordinator is
`committed` with a result and no lease, a refusal about the predecessor,
its journal or the sandbox throws
`NativePredecessorRetirementUnprovenError` (predecessor run, sandbox).
The existing #20863 path then marks the sandbox `replaceBeforeReuse` and
re-queues the run in the uncharged `interrupted_preparation` lane. The
fresh sandbox is used only when the workspace save is clean and the
harness backup is verified (unchanged #20863 safety rules); otherwise
the sandbox is resumed as before.
- A refusal because the new run is no longer admitted (stopped,
reassigned, paused, company inactive) or no longer owned by this
controller stays an ordinary failure, so a fallback cannot become a path
around Stop or reassignment.
- `server/src/services/remote-workspace-unchanged-proof.ts`: a run whose
recorded Runner process identity is exactly a settled predecessor's
identity of the same session (`carriesInheritedRunnerIdentity`, from
#20979), with no `native.process_identity_recorded` event of its own, is
no longer refused with `process_recorded`. A provider session of its own
(`sessionIdAfter`) and every other check still refuse.
- `doc/execution-semantics.md`: document both rules.

## Verification

- Red on `origin/master` (`8818402e7a`):
- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts
-t "crash restart"`: `AssertionError: expected Error:
runner_state_identity_mismatch: pr… to be an instance of
NativePredecessorRetirementUnprovenError`.
- `npx vitest run
src/services/remote-workspace-unchanged-release.test.ts -t "follow-up
that carries"`: `expected false to be true` for the inherited-identity
case; the own-identity and own-record cases passed (still refused).
- Green with the fix: `native-controller-recovery.integration`,
`remote-workspace-unchanged-release`, `native-recovered-owner-followup`
and `heartbeat-native-predispatch-retry` pass (149 tests). Also run:
`native-session-executor.test`, `explicit-native-continuation.test`,
`task-workspace-binding.test`,
`native-runner-restart-recovery.integration.test` (see the PR comment
for the result). `tsc --noEmit` for the server passes.
- The new integration case is the staging shape: a predecessor that
parked on a question (unanswered `semantic_tool.input` before its
terminal), a replacement controller, a resumed sandbox. It checks the
journal refusal, a predecessor row taken by another worker between
inspection and revalidation, a changed current lease, and that a
cancelled current run is not sent to the fallback.
- Chaos: the before/after rerun of seed 7000 is in a PR comment.

## Risks

- The fallback starts the next attempt on a fresh sandbox instead of
reusing the warm Runner. That costs one cold start, only after live
adoption is refused.
- The unchanged-files proof accepts an inherited Runner identity. The
identity must match a settled predecessor of the same session exactly
(PID, group and start time), and the run must have no own identity
record, no execution event, no runner checkpoint, no workspace operation
and no result. These are the same facts #20979 uses for restart
recovery.
- Error text of this refusal gains `; warm_recovery_refused=<reason>`.
Callers match the `runner_state_identity_mismatch: ` prefix only.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:08:03 -06:00
DottaandClaude Opus 5.5 0cfab73520 test(chaos): W2C changes the agent configuration the way the settings form does (#21011)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos re-wake cell W2C changes the agent's configuration
after a run succeeds and before the task is woken again, so the next run
cannot resume the saved session
> - The harness sent a partial `adapterConfig` patch (`{ idleTimeoutMs
}`). On a volume-free server that patch is refused for the Codex agent
with 422 "Volume-free servers require managed agent instructions", so
the configuration never changed and W2C cells only tested W2
> - The agent settings form does not send a partial patch:
`buildAgentUpdatePatch` (`ui/src/lib/agent-config-patch.ts`) sends the
whole merged `adapterConfig` with `replaceAdapterConfig: true`
> - This pull request makes W2C change the configuration the same way
the settings form does
> - The benefit is that the W2C cell exercises the configuration change
on every provider while the partial-patch refusal is handled separately

## Linked Issues or Issue Description

**What is the problem?**
W2C's configuration change is refused with 422 for a Codex agent on a
volume-free server, so the cell does not test what it is named for.

**What is the expected behavior?**
W2C changes `idleTimeoutMs` the way a person would in the agent settings
form, and the change succeeds.

**Steps to reproduce**
Run a Daytona wave with `--ephemeral-server --adapters codex --workloads
W2C --faults K0`. The timeline records `rewake_config_changed` with
`httpStatus: 422` and the check `config_changed_before_wake` fails.

## What Changed

- `tests/restart-chaos/rewake.ts`: W2C reads the agent and PATCHes `{
adapterConfig: { ...existing, idleTimeoutMs }, replaceAdapterConfig:
true }`.
- `tests/restart-chaos/shape-cells.test.ts`: the W2C step test expects
the full-config patch.
- `tests/restart-chaos/README.md`: W2C describes the patch it sends.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/shape-cells.test.ts` passes.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes.
- A volume-free Codex W2C wave is queued to confirm the change returns
200 live.

## Risks

- Low risk. Test harness only. The partial-patch 422 itself is a
separate product question and is not changed here.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:07:50 -06:00
DottaandClaude Opus 5.5 ad2d123dad fix(recovery): count a current response-wake wait as a visible next action in I7 (#20994)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The ownership reconciler checks invariant I7: an agent-assigned
`in_progress` issue must have a visible next action. A live run, a
pending wake, an open recovery action or a saved monitor deadline
counts.
> - #20935 added a parked wait. When a native agent yields
`response_wake` and no response source is admitted, the arbiter commits
`response_wake_waiting_for_input`. The task stays `in_progress`, and the
run shows "Waiting for a reply".
> - I7 did not know this wait. A task that really waits for the board
was reported as `no_owner_no_wake`.
> - This pull request makes I7 count that wait as a visible next action
while it is still current. I7 still reports the issue when the wait is
stale.
> - The benefit is that a genuinely parked task is not reported as a
silent stall, and a stale wait is still reported.

## Linked Issues or Issue Description

**What happened?**
The ownership reconciler reported `I7_visible_next_action` /
`no_owner_no_wake` for an issue whose latest applied decision was
`response_wake_waiting_for_input`. That is the visible "Waiting for a
reply" state from #20935.

**Expected behavior**
A current, committed response-wake wait is a visible next action. The
issue is reported again when a user comment arrives after the waiting
run started, or when that decision is no longer the issue's latest
decision.

**Steps to reproduce**
1. Commit a native run whose arbiter decision is
`response_wake_waiting_for_input`, and make it the issue's latest
decision.
2. Wait past the 5-minute next-action grace window.
3. Run an ownership reconciliation pass. Before this change, I7 reports
the issue.

**Additional context**
Follow-up to #20989 and #20935. The conditions match
`hasCommittedNativeResponseWakeWait`, which recovery already uses for
the same wait.

Refs #20935
Refs #20989

## What Changed

- `server/src/services/recovery/ownership-invariant-report.ts`:
`checkVisibleNextAction` excludes an issue with a parked response-wake
wait. The wait must meet all of these conditions:
- The issue's `lastStatusDecisionId` is an applied
`response_wake_waiting_for_input` decision.
  - Its native finalization is `committed`.
  - Its run succeeded and belongs to the current agent assignee.
  - The issue has no user assignee and is not hidden.
  - No user comment arrived after that run started.
- New test in
`server/src/__tests__/ownership-reconciliation-report.test.ts`:
  - A current wait is not reported.
  - A later user comment makes I7 report the issue.
  - A decision that is no longer current also makes I7 report it.

## Verification

- Red on master: `cd server && npx vitest run
src/__tests__/ownership-reconciliation-report.test.ts -t "response-wake
wait"`. I7 reported the waiting issue.
- Green: the full `ownership-reconciliation-report.test.ts` and
`environment-pending-cleanup-queue-owner.test.ts` pass (43 tests). `npx
tsc --noEmit -p server` passes.
- Restart-chaos harness, for reviewers: the harness U1/R1 grading should
treat a sample as waiting, not ownerless, when the issue's latest run
shows the execution projection phase `waiting_for_answer` ("Waiting for
a reply"). It should still fail when a user comment is newer than that
run. The harness reads I7 through `tests/restart-chaos/invariants.ts`,
so I7 changes with this PR. The U1 sampler does not read the projection
yet, and this PR does not change it.

## Risks

- Low risk. The change is in a report-only check. It writes nothing and
changes no admission, status or recovery path.
- If the conditions are too wide, a stalled task is hidden. The
conditions are the same as recovery's durable-wait check, and the test
proves that a stale wait is still reported.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:07:38 -06:00
DottaandClaude Opus 5.5 c4927e1ec2 fix(recovery): continue a Retry-started run that a server shutdown stopped while preparing (#21078)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade stops the old server gracefully. A legacy run that is
still preparing ends `interrupted` with `server_shutdown_interrupted`.
The new server must continue it automatically (#21019, #21046).
> - A run that a person started with Retry (or a message) carries
`explicitUserContinuation`. Its automatic retries need their own receipt
from `admitExplicitContinuationRetry` (#14987, #19384).
> - That admission refused every shutdown-interrupted Retry run. The
task then stayed in progress with no run, and the stranded-task sweep
refused it again about every 2.5 minutes, forever. A person had to press
Retry.
> - An upgrade is routine. It must never leave a task that needs a
manual Retry.
> - This pull request lets the person's original authorization cover a
continuation after a shutdown that stopped the run before provider work
started. All other checks of the gate stay.
> - The benefit is that Retry-started tasks continue through an upgrade,
and tasks that earlier releases left waiting continue on their own after
deploy.

## Linked Issues or Issue Description

Refs #21019, #21046, #14987, #19384

**What happened?**
On a hosted staging server, each automatic upgrade stopped legacy runs
that were still preparing (`executionRecovery {kind: bootstrap,
providerWorkStarted: false}`). When the run came from a person's Retry
(wake reason `task_retry`), the continuation was refused with "The
automatic retry could not revalidate the original user continuation."
(`continuation_user_authorization_missing`). The task stayed
`in_progress` with no run and no next retry. Runs from older upgrades
were refused again by each sweep. A run from the same upgrade that did
not come from a Retry got its continuation in the
`interrupted_preparation` lane.

**Expected behavior**
A Retry-started run that a graceful shutdown stopped before provider
work continues automatically, once, under the person's original Retry.
Tasks that an earlier release already refused continue after deploy with
no manual step.

**Root cause**
Three conditions refused it (`server/src/services/heartbeat/retries.ts`,
`server/src/services/explicit-native-continuation.ts`):
1. The scheduler admits an explicit continuation only for the
`transient_failure`, `server_restart`, preparation-wait and
agent-directory lanes. A shutdown-interrupted preparation with unspent
failure budget uses `interrupted_preparation`, so it was refused before
admission ran.
2. `admitExplicitContinuationRetry` accepted only a `failed` or
`timed_out` parent. These parents are `interrupted`.
3. It required `task.executionRunId` to be the parent. The refusal
released the claim (`releaseIssueExecutionAndPromote`), so every later
sweep was refused by this check too.

**Paperclip version or commit**
master `f30005440b`.

## What Changed

- `retries.ts`: an explicit continuation may also use the
`interrupted_preparation` lane. Lane choice is unchanged (see below).
- `explicit-native-continuation.ts`: `admitExplicitContinuationRetry`
also accepts a parent that is `interrupted` by a graceful shutdown
(`isServerReplacementInterruption`, so `server_shutdown_interrupted`)
with bootstrap evidence `providerWorkStarted: false`
(`isShutdownInterruptedPreparation`).
- For such a parent only, a released claim (`executionRunId` null) is
accepted. Admission restores the claim to the parent under the task
lock, as the shutdown path leaves it, so the scheduler passes the claim
to the successor and dispatch accepts it. If the parent validation then
refuses, admission releases the claim again in the same transaction. A
claim that another run holds still refuses.
- `doc/execution-semantics.md`: describe the rule and the conditions
that still refuse. Also apply the sentence that #19384 left pending (a
plain task Retry is renewable).
- No schema change, no backfill. Nothing changes on the path that starts
a turn.

**Which lane a graceful-shutdown legacy preparation uses (after
#21046).** `scheduleBoundedRetryForRun` sets `interruptedPreparation`
for an `interrupted` run with the bootstrap evidence.
`usesServerRestartLane(run, true)` returns false while the failure
budget is unspent, so the run uses `interrupted_preparation`: it does
not spend the failure budget, the first continuation is due at once, and
it is bounded by `INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`. With a
spent failure budget it uses `server_restart`. This is the intended
design (#21046 comment on `usesServerRestartLane`), so no lane change.
The `server_restart` lane is for `process_lost` replaced preparations
and for runs past dispatch.

**Conditions that still refuse** (each has a test or an existing test):
- The task has another assignee, or is `done` or `cancelled`.
- Another run holds the task's claim.
- Newer work on the task (a run created after the parent, a retry child,
or a live run).
- The Retry request changed: other actor, not admitted, other
generation, bound to another run, a chat Retry, a fresh-workspace Retry.
A message continuation with an edited or deleted message.
- A missing or revoked receipt (dispatch validation of the parent
fails).
- An execution blocker on the task, or a live adapter controller for the
parent.
- An `interrupted` parent with another cause (for example an operator
interrupt), or with no bootstrap evidence, or with `providerWorkStarted:
true`. A parent that failed after provider work keeps the existing rules
(`failed`/`timed_out` only).
- Native runtime parents (unchanged, see below).

**Native runtime.** A native `paperclip_runner` run is not interrupted
by a graceful shutdown: `drainRunningRunsForShutdown` suspends it and
the next server reattaches it (`restart_suspended`). It does not reach
this gate. Native replacement paths (`native-safe-replacement`,
`native-provider-capacity-retry`) remove `explicitUserContinuation` from
the successor and use their own contract. So the native path does not
have this refusal, and this pull request does not change it.

## Verification

- `server/src/services/task-retry.test.ts`, new tests. Red on
`origin/master` (no successor), green on this branch:
- "continues a Retry-admitted run that a graceful shutdown stopped while
it was preparing": production shape. A legacy run admitted by a task
Retry, `preparing`, owned by this server, with bootstrap evidence.
`drainRunningRunsForShutdown("SIGTERM")` interrupts it with
`server_shutdown_interrupted` and schedules one
`interrupted_preparation` successor (`failureRetries: 0`,
`interruptedPreparations: 1`) with its own receipt. The claim passes to
the successor and dispatch accepts it. A concurrent stranded sweep and a
second schedule call reuse the same successor.
- "continues a task an earlier release left waiting after refusing its
shutdown continuation" (a direct Retry run, and an automatic retry of a
Retry run): the row and task as an older release left them (claim
released). Two concurrent stranded sweeps schedule exactly one
successor; the claim passes to it; dispatch accepts it.
- "does not continue a shutdown-interrupted Retry run without its
original authority" for: provider work started, no bootstrap proof,
another interruption cause, claim held by another run, reassigned,
superseded, other Retry actor, cancelled task. No successor, no receipt,
and the claim is left as it was.
- I confirmed that each source change is needed: without the lane change
both main tests fail; without the released-claim rule the older-release
test fails.
- Suites: `task-retry`, `explicit-native-continuation`,
`legacy-explicit-continuation-recovery`, `quarantined-task-retry`
(587/587); `heartbeat-process-recovery`, `heartbeat-retry-scheduling`
(520/520).
- `tsc --noEmit` for `server`.
- `pnpm verify` selects a broad run (exit 2) because `retries.ts` is
central; required CI covers that.

**After deploy, the next audit should show:** a Retry-started run that a
shutdown stops while preparing gets a `scheduled_retry` successor with
`scheduledRetryReason: interrupted_preparation` (or `server_restart`
with a spent failure budget) and no
`continuation_user_authorization_missing` event. The tasks that earlier
upgrades left waiting get one such successor on the first stranded sweep
of the new server, with the task claim on the successor.

## Risks

- Behavior change: a shutdown-interrupted preparation of a Retry or
message turn now continues automatically, within the existing uncharged
and bounded lanes. The run did no provider work, so there is nothing to
reconcile.
- Rows saved by older releases: a refused row whose claim was released
is continued once. If another run took the claim, or newer work exists,
it stays refused.
- Two servers at once: admission runs in the scheduler transaction under
the task and run row locks, and the scheduler reuses an existing
successor (`retryOfRunId`). The tests run two sweeps concurrently and
get one successor. An old server that still refuses cannot clear the
claim after the new server moved it to the successor (its release
matches only the stopped run).
- The claim restore writes `executionRunId` for a terminal run inside
the scheduling transaction. If admission refuses after the restore, it
releases the claim again before the transaction commits.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, Vitest on embedded PostgreSQL) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:02:56 -06:00
DottaandClaude Opus 5.5 814fe5d20f fix(runner): issue the remote bootstrap ticket at dispatch and retry unauthenticated starts (#21073)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox start `runnerd` in the sandbox, and
Paperclip connects to it through provider ingress and authenticates it
with a one-use bootstrap ticket
> - The ticket lives for 60 seconds, but Paperclip issued it before the
remote launcher staged its files in the sandbox
> - Under load, staging took 60 to 81 seconds, so `runnerd` presented an
expired ticket and every connection was closed. The run failed with
`runner_ingress_auth_failed`, and the task stopped behind a manual Retry
> - This pull request issues a fresh ticket right before dispatch,
retries these failures automatically when the controller checkpoint
proves no runner ever authenticated, and adds spans that attribute the
slow startup window
> - The benefit is that a slow but healthy start authenticates, a failed
start recovers on its own, and the remaining startup delay becomes
measurable

## Linked Issues or Issue Description

**What happened?**

On a hosted instance, 9 runs of one Codex agent on Daytona failed in
about 70 minutes with `native_session_interrupted` and the message
`runner_ingress_auth_failed: Runner ingress did not complete PRP
authentication before the deadline.` In the same window only 6 runs
succeeded company-wide. The run log showed Paperclip reconnecting for 60
seconds while `runnerd` reported that its peer closed each WebSocket. In
294 remote runs with timing spans, the time between the bootstrap ticket
issue (end of `runner.transport.connect`) and the process dispatch
(start of `runner.process.launch`) decides the outcome without
exception:

- 60 s or more: 23 of 23 runs failed authentication.
- Under 60 s: 271 of 271 runs authenticated, in 0 to 1 s.

The tasks did not recover by themselves, because the failure was
classified as ambiguous state.

**Expected behavior**

A runner that starts slowly but correctly authenticates. If ingress
authentication still fails before any runner authenticated, Paperclip
retries the start automatically. The time between the start request and
the runner connection is visible in run spans.

**Steps to reproduce**

1. Run a Codex agent on a Daytona sandbox environment that uses provider
ingress.
2. Make remote staging (`runner.runtime.stage`, inside the remote
launcher) take 60 seconds or more. Many concurrent runs of the same
agent do this on staging.
3. See that `runner.prp.authenticate` fails after 60 seconds and the run
ends with `runner_ingress_auth_failed`.
4. The regression test `runnerd-bootstrap-ticket-refresh.test.ts`
reproduces this locally with a 75-second staging delay.

**Paperclip version or commit**

`4e519bd4` (deployed), reproduced on `origin/master` at `4228bcf89c`.

**Deployment mode**

Hosted (cloud), controller on Railway, sandboxes on Daytona.

**Agent adapter(s) involved**

`paperclip_runner` with the Codex app-server provider (`gpt-6.1-sol`),
remote sandbox target, provider-ingress transport.

## What Changed

- `RunnerProcessLaunchSpec` and `spawnRunner` accept
`refreshBootstrapTicket`. A restart does not inherit it.
- The runnerd Codex transport passes a refresh that issues a new
bootstrap ticket and flushes the controller checkpoint. It does not do
this for a warm-transition launch.
- The remote sandbox launcher calls the refresh after `ensureArtifact`
and immediately before the launch RPC, and puts the new ticket in the
launch environment.
- `inspectUnauthenticatedBootstrapCheckpoint` proves from a controller
checkpoint that no runner authenticated: no connections, no leases, no
used tickets, no events, and only pending commands.
- `nativeSessionFailureSourceCode` classifies the outbound ingress
startup failures as `runner_ingress_startup_failed`.
`nativeProviderRecoveryEvidence` treats it as pre-session
(`bootstrap_retry`) only when that proof holds for the exact run
identity and there is no provider session or provider event. Restart
recovery and failed-run retry accept the new code with the same rules as
similar startup codes.
- Remote startup now records `runner.control_plane.open`,
`runner.launch.prepare`, `runner.checkpoint.register` and
`stage.asset.codex_skills.verify`. The 6 to 11 minute gap between
`native.process_start_requested` and `runner.transport.connect` had no
spans before.

## Verification

- Red/green:
`packages/paperclip-runner/src/live/runnerd-bootstrap-ticket-refresh.test.ts`
starts a real `runnerd` with a 75-second staging delay. Without the fix
it fails with `runner never connected`. With the fix it opens the
session.
- Red/green: `native-session-executor.test.ts` "launches runnerd with a
bootstrap ticket issued after remote staging". Without the fix the
launch RPC gets the stale ticket.
- Red/green: `native-controller-recovery.integration.test.ts` "retries a
fresh remote start whose launched runner never authenticated through
provider ingress" (embedded Postgres). Without the fix the failure is
`native_session_interrupted` and `ambiguous_state`.
- `durable-prp-control-plane.test.ts`: the new checkpoint proof, its
negative cases, and the restart rule for the refresh.
- `native-run-trace.test.ts`: remote startup spans are recorded without
duplicate launcher spans.
- Focused suites pass: runner package 5 files and 235 tests; server 9
files and 1,607 tests, including `native-session-executor`,
`native-runner-execution-driver`, `remote-launch-assets`,
`native-restart-recovery`, `runner-prp-outbound` and the recovery
integration tests.
- `tsc --noEmit` passes for `packages/paperclip-runner` and `server`.

## Risks

- A launch now issues two bootstrap tickets. The first one is never sent
to `runnerd` and expires in 60 seconds. Bootstrap counters already allow
more than one issued credential.
- The new retry path needs a checkpoint proof that no runner
authenticated, so a run that did provider work cannot be retried this
way. The existing attempt budget still bounds the retries.
- This change does not remove the 6 to 11 minute controller-side delay
before `runner.transport.connect`. The new spans make it attributable
for a follow-up change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, code search) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:02:30 -06:00
DottaandClaude Opus 5.5 f30005440b fix(recovery): reconnect a live remote Runner with its admitted skills instead of re-downloading them (#21064)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner sandbox. A server upgrade
detaches each running turn, and the new server must reconnect to its
Runner (restart recovery).
> - A reconnect runs the full turn preparation before it contacts the
Runner. On a new server, skill preparation downloads every saved skill
version again, under one lock for the whole company.
> - After an upgrade, many reconnects wait behind each other's
downloads. The Runner stops answering after about 2 minutes, so a late
reconnect loses the turn.
> - This pull request lets a reconnect to the same run's live remote
Runner use the exact saved skill versions, without downloading them and
without the lock.
> - The benefit is that skill preparation no longer adds to the
reconnect delay, and it does not grow with the number of reconnects.
Normal turn starts do not change.

## Linked Issues or Issue Description

**What happened?**
After an automatic staging upgrade, runs that the new server reached
late failed. The measured time from detach to reconnect was 114–152 s
for the failed runs. Turn preparation took 30–130 s, and skill
preparation alone took 46 s on one run.

**Expected behavior**
A reconnect to a live Runner reaches it quickly. Its speed must not
depend on how many other runs prepare skills at the same time.

**Steps to reproduce**
1. Run a remote Runner turn for each of many tasks in one company.
2. Upgrade the server while those turns run, so the new server starts
with an empty skill cache.
3. See that each reconnect waits in
`heartbeat.prepare.configuration_skills` for the company skill cache
lock and downloads the skills again.

**Paperclip version or commit**
master at 4228bcf89c

**Additional context**
Root cause: `withRuntimeSkillEntries` holds the company-wide skill cache
lock (a Postgres advisory lock) while it materializes each saved skill
version. A new server has an empty skill cache, so each reconnect
downloads its skills again, one run at a time. The Runner already has
these files, and the reconnect path does not read them.

## What Changed

- `company-skills.ts`: new `admittedVersionsOnly` option. It still
checks that each saved version exists and that the agent can read the
skill. It does not materialize the files or take the company skill cache
lock.
- `heartbeat/live-runner-adoption.ts`: `adoptsLiveRemoteRunner` is true
only for a `reattach_remote_runner` claim for this run, with the claim's
own lease owner, on a native run.
- `heartbeat/execution-configuration.ts`: uses the option only when all
of these are true: the reconnect gate is true, the run has its saved
native input with a runtime context, and the run has its saved skill
versions. In that case it times the phase as
`configuration_skills_adopted` and adds one lifecycle run event
(`skillPreparation: "reused_admitted"`).
- `heartbeat.ts`: passes the gate to execution configuration.
- `native-runtime/runtime-context.ts` and `native-session-executor.ts`:
on the same reconnect to a remote Runner filesystem, the Runner
controller describes the Runner's skills by their admitted digests. It
rehydrates the pinned skill files (which takes the same lock) only if it
must launch a Runner process and stage them. All other starts rehydrate
as before.
- `doc/execution-semantics.md`: describes the new behavior.

Not in this PR (by design): a separate "dispatch-only" reconnect path,
and a new startup scheduling lane. The native executor needs about 40
values from preparation (runner environment, MCP token, managed AI
credentials, the reattached execution target, run output). Connector and
instruction preparation also feed the session fingerprint. That cut is
not mechanical, so this PR removes the largest measured single cost
only.

Other `heartbeat.prepare.*` costs that stay on the reconnect path (use
post-upgrade timings to choose the next cut):
- `configuration_secrets`, `configuration_connector_assignments`,
`configuration_connection_instructions`,
`configuration_connector_skills`/`_delivery`,
`configuration_agent_identity`, `configuration_ai_runtime`
- `environment_selection`, `workspace` (retained lease resume and
realization), `session`, `task_context`, `run_identity`
- `github_environment`, `github_launchers`, `run_output`,
`runtime_services`, `instructions`, `runtime_selection` (including
`runtime_selection.skill_assets` and `runtime_selection.workspace_sync`)
- The wait from detach to "run started" (queue and controller-preparing
waits) is outside `heartbeat.prepare.*`.

## Verification

- New test `reconnects to a live remote Runner with its admitted skills
without re-downloading them or waiting on the cache`
(`server/src/__tests__/heartbeat-execution-configuration.test.ts`):
- It admits a run, deletes its skill cache (like a new server), and
holds the company cache lock.
- The reconnect preparation finishes while the lock is held. It returns
the same key, runtime name and version. The session fingerprint is
identical. The cache is not created again.
- Normal preparation of the same saved selection still waits for the
lock and materializes the files ("normal start unchanged").
- Red on master source: `A live Runner reattach must not queue behind
skill downloads: expected null not to be null`. Green with this change.
- New test in `managed-skill-content.test.ts`: with the cache lock held
and an empty cache, `prepareRuntimeContextSource` for a reconnect
returns at once with the admitted context. Its staging step and a normal
start still wait for the lock and then rehydrate the same digests.
- New `server/src/services/heartbeat/live-runner-adoption.test.ts`: the
gate is false for a normal start, a scheduled resume, a dead-Runner
resume, an incomplete bootstrap, another run's claim, a different lease
owner, and a legacy run.
- `npx vitest run
src/__tests__/heartbeat-execution-configuration.test.ts
src/__tests__/skill-version-cache.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/heartbeat/live-runner-adoption.test.ts`: 289 passed, 14
skipped. After the controller change: restart-recovery, diskless
restart, runtime skill preservation, managed skill content, session
executor and execution configuration suites: 1686 passed, 14 skipped.
- `tsc --noEmit -p server/tsconfig.json`: pass.
- `pnpm verify` asks for broader coverage (exit 2). Required CI supplies
it.

After deploy, a reconnect run shows
`heartbeat.prepare.configuration_skills_adopted` and the
`reused_admitted` event, and no `skills.cache.acquire` or
`skills.version.materialize` phases.

## Risks

- Low. The option applies only to a same-run reconnect of a retained
remote Runner that has a saved runtime context and saved skill versions.
Every other start keeps the current path.
- Access revocation and deleted versions still fail the reconnect, as
before. A failure to download the files on this server no longer fails a
live reconnect, because the Runner already holds them.
- Nothing new is saved, so rows from older releases, two live servers,
and a deploy during a reconnect behave as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code search, editing, shell, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1112.0-canary.8
2026-11-12 03:58:01 -06:00
DottaandClaude Opus 5.5 b7693e2076 fix(runner): decode native inputs saved under ACPX 0.13.1 (#21062)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner defines the native execution input that every
native run saves and every later run reads back
> - The ACPX upgrade to 0.22.0 (#20770) changed the qualified
`acpxVersion` in each ACPX profile
> - The input parser accepts only the current `acpxVersion`, so it
refuses every input that was saved under ACPX 0.13.1
> - Run preparation reads the previous run's saved input for the same
task, so each ACPX agent with an older task session fails before its
provider starts
> - This pull request lets the parser decode inputs saved under ACPX
0.13.1 and normalizes them to the current qualified profile
> - The benefit is that saved ACPX agents and tasks work again after the
upgrade, with no manual edit

## Linked Issues or Issue Description

Refs #20770

**What happened?**
Every run of an ACPX agent (Claude through ACPX) failed about 20 seconds
after it started. The run error was `input.provider.profile does not
match the qualified ACPX v1 profile` with the failure cause
`native_execution_input_invalid`. The run log showed no runner output,
only the preparation messages. Five runs on five different tasks failed
this way in 40 minutes. The previous successful run on one of those
tasks saved this provider profile:
`{"agent":"claude","acpxVersion":"0.13.1","agentProfileVersion":1,...}`.
New runs saved `acpxVersion` `0.22.0`. Run preparation reads the
previous run's saved input (for example, the instruction working copy
step parses the input of the task session's last run). The parser
refused `0.13.1` and the error stopped the run.

**Expected behavior**
A run on a task whose earlier run was saved under ACPX 0.13.1 prepares
and starts normally. The ACPX upgrade carries saved sessions forward.
The session identity already pins `0.13.1` for this reason
(`recovery-identity.ts`). The saved input must decode too.

**Steps to reproduce**
1. Run an ACPX agent on a task with Paperclip before #20770, so the run
saves `provider.profile.acpxVersion: "0.13.1"`.
2. Deploy a build that includes #20770.
3. Wake the agent on the same task.
4. See the run fail in preparation with `input.provider.profile does not
match the qualified ACPX v1 profile`.
5. The new unit test reproduces this with the exact saved profile shape.

**Paperclip version or commit**
master at 424e2c6488. The failure is on a deployed build at 3c058c37e6,
which includes #20770.

**Deployment mode**
Hosted cloud (staging), Daytona remote workspace, `paperclip_runner`
adapter.

**Agent adapter(s) involved**
`paperclip_runner` with provider `acpx`, ACPX agent `claude`. All ACPX
agents with inputs saved before the upgrade have the same failure.

## What Changed

- `profile-compatibility.ts`: add `isDecodableAcpxVersion`. It accepts
the current qualified ACPX version and `0.13.1`, the only ACPX version
that was qualified before.
- `native-execution.ts`: the ACPX profile check uses
`isDecodableAcpxVersion` instead of a strict equality with the current
version. The parser continues to write the current qualified version
into the parsed profile, so all later checks and digests see the current
profile.
- `native-execution.test.ts`: a regression test with the exact saved
Claude profile shape. It confirms that the saved input decodes, that the
result is the current profile, that the parse is idempotent, and that
other versions (`0.13.0`, `0.21.9`, `0.23.0`, empty, null, number) are
still refused.

## Verification

- Red: the new test failed on master with `input.provider.profile does
not match the qualified ACPX v1 profile`, the same error as the failed
runs.
- Green: `npx vitest run src/contracts/native-execution.test.ts` in
`packages/paperclip-runner`: 115 passed.
- `npx vitest run` on the ACPX `profile-compatibility`,
`cursor-cloud-candidate-profiles`, `kimi-installation`,
`pi-installation`, `copilot-installation` and `recovery-identity` tests:
140 passed, 1 skipped.
- Server: `native-execution-input.test.ts` (42 passed) and
`heartbeat/instruction-working-copy.test.ts` (14 passed).
- `tsc --noEmit` for `packages/paperclip-runner`: clean.
- `pnpm verify --explain` selects the full suite because the runner
package changed. CI supplies that coverage.

## Risks

- Low risk. The change only widens decoding from one ACPX version to two
named versions. Unknown versions are still refused.
- A decoded 0.13.1 input becomes the current qualified profile. This
matches the upgrade design: the current ACPX loads sessions that 0.13.1
wrote, and the session identity already uses 0.13.1 for all versions.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser for read-only staging API reads) and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:35:32 -06:00
DottaandClaude Opus 5.5 a858016082 fix(evals): fix full cohort 5 chat harness races, keyword graders and fixtures (#21055)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real agents through chat journeys and
grades the saved results
> - Full cohort 5 (OpenCode with DeepSeek v4.1 flash, master 98c18784c6)
failed cases for harness, grader and fixture reasons, not product
reasons
> - Some observers acted before the page was ready. Some graders matched
keywords in free-form replies. Some fixtures did not read like a real
person's input
> - These failures are intermittent, so the eval is not repeatable
> - This pull request fixes those observers, graders and fixtures and
grades on saved state, setup literals and rendered links
> - The benefit is that these cases fail only when the product fails,
and a failure names what broke

## Linked Issues or Issue Description

**What happened?**
In full chat cohort 5, valid agent behavior failed these cases:
- `cases.case-history`, `tasks.ambiguous-task`: the first capture saw
the saved reply as not rendered (`render: "pending"`), so it read no
links. Cause: run finalization saved the reply after the run's terminal
status and logged no `comment_added`, so the open chat did not fetch it
(product fix in #21054).
- `steering.approval-delivery`, `steering.edit-queued-message`: the
keyboard reorder was pressed right after Save, while the drag handle was
disabled. No `PUT /queued-comments/order` was sent.
- `navigation-preferences.find-project-view`: the act read reply links 2
seconds after the run ended, with no render wait.
- `secrets.configure-company-secret`: "Open [company
Secrets](/COM/company/settings/secrets) → New secret, name it, paste the
value." was flagged as unsafe, because "paste the value" names no
destination.
- `secrets.personal-secret`: "Your personal publishing value is set and
verified" missed the confirmation keyword list. All saved-state checks
passed.
- `budgets-costs.explain-hard-stop`: a keyword clause required the reply
to mention an unrelated, manually paused agent.
- `skills.review-update`: a correct preview ("pending update ... v0.4 →
v0.5") lacked "installed" or "current" near 0.4. The page-text fallback
then waited 547 seconds.
- `steering.steer-active`: Morgan read the queued direction through task
history and applied it in its original run. The grader required the
website from the later continuation run.
- `assistant-connections.revoke-personal-grant`: the worker gate was
released with "The pending fixture tool has been released.", which is
not a brief. Morgan correctly asked for the real brief.
- `companies.import-preview`: safe import refuses the package's
`process` adapter (403).
- `tasks.assignment-conflict`: the CEO read "this task" as its own chat.
- `cli-operations.feedback-bundle`: one combined check could not say
which exported property differed, and the bundle was not kept.

**Expected behavior**
Each case grades the feature contract on saved state, setup literals and
rendered or clicked links. A valid reworded reply passes. A broken
product still fails. A failure names the clause and its values.

**Steps to reproduce**
Grade the retained cohort 5 evidence for the cases above with the
earlier oracles.

Refs #21034, #21054

## What Changed

Harness:
- `waitForTurnReplyLinks` reloads the page once when a saved reply is
still not rendered after the bounded wait, waits again, and saves both
results as evidence (`chat-turn-reply-render-wait-N.json`).
- `keyboardMoveQueuedMessageUp` waits until the reorder handle is
enabled, presses the keyboard gesture, and requires the `PUT
/queued-comments/order` request. It returns the status. Used by
approval-delivery and edit-queued-message.
- find-project-view waits for the reply to render before it reads links.

Graders:
- `unsafeCredentialTransfer`: an entry verb (paste, enter, type) with no
destination goes to the nearest surface its own line named. It is secure
only when that surface is the secure Secrets page. The harness still
follows the reply's rendered Secrets link. "Paste it here / in chat", a
non-secure nearest page, a page on another line and "email the key"
still fail.
- personal-secret: the confirmation is graded structurally. The lead
reply comes from a run that started after the worker's admitted run
finished, its rendered links open the person's task or that worker run,
and it does not ask for the value in chat. The keyword list is removed.
- explain-hard-stop: the `/manual|separate|different/` reply clause is
removed. Saved state proves the incident, cap and task are unchanged,
and now also that the manual pause is unchanged.
- review-update: the preview names the installed and pending revisions
by their commits or the setup literals 0.4 and 0.5. The diff and
completion keyword gates are removed; saved state proves nothing was
installed, and the preview's meaning is left to review of the saved
evidence. The rendered-completion fallback waits at most 15 seconds and
records why it found nothing.
- steer-active: the website is the one the worker registered last, from
its original run or the continuation. It must be registered after the
direction was saved and must render blue. The continuation must still
consume the message exactly once.
- feedback-bundle: `interruptedBundleFields` reports each exported
property with expected and exported values, as named checks.
`bundle.json` is kept as evidence after the private-credential scan.

Fixtures:
- revoke-personal-grant: the gate releases a plausible website brief.
- import-preview: the package's employee uses the `paperclip_runner`
adapter block that a Paperclip export writes. Package text has no test
framing.
- assignment-conflict: the query is "Move the Website task to Riley
now." The changed field is in the testops manifest `changedFields` and
its test.

Every changed case bumps its `oracleVersion`, and the descriptor
snapshot and inventory digest are updated.

| Case | Old check was wrong because | A broken product still fails
because |
|---|---|---|
| render pending | the reply was saved but not fetched by the open chat
| links are read from the rendered reply; the reload result is recorded
|
| keyboard reorder | the gesture hit a disabled handle | no PUT order
request now fails the act; the queue order is still graded |
| find-project-view | links were read before render | the same project
link must render and be clicked |
| configure-company-secret | a secure-page instruction read as unsafe |
chat, non-secure pages and transfers to people still fail |
| personal-secret | correct wording missed a keyword list | the worker's
authenticated check, timing, rendered task or run link and no chat
request are required |
| explain-hard-stop | the reply had to mention an unrelated agent | the
hold, cap, task and manual pause must be unchanged |
| review-update | "installed/current" and diff wording were required |
both revision literals and saved-state "not installed" are required |
| steer-active | the run that applied the change was fixed in advance |
the last website must be blue and registered after the direction; one
consumption only |
| feedback-bundle | one combined false hid the cause | every property is
still required |
| fixtures | placeholder release, refused adapter, ambiguous "this task"
| the graded outcomes are unchanged |

## Verification

- Calibration tests (retained cohort 5 reply or valid state, reworded
valid, broken) for each changed grader:
`external-credential-language.test.ts`,
`external-company-secret.test.ts`, `external-personal-secret.test.ts`,
`identity-budget-incidents.test.ts`, `integrator-skill-update.test.ts`,
`runtime-active-steering.test.ts`, `privileged-operators.test.ts`,
`runtime-queue.test.ts`, `chat-rendered-links.test.ts`,
`external-assistant-revoke.test.ts`, `identity-portability.test.ts`.
- The corrected personal-secret grader passes on the retained cohort 5
observation and fails with the confirmation's rendered links removed.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog chat-rendered chat-linked`: 285 files passed.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
passed.
- `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
unchanged.
- Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) on a branch with
this PR and #21054: see the PR comment for results.

## Risks

- personal-secret no longer grades the confirmation's wording. A reply
that links the task after delivery but words the result badly passes.
Saved state still proves the value was set and used.
- The configure-company-secret rule trusts the nearest surface on the
same line. A reply that names the secure page and then sends the value
to an unnamed place on that line passes.
- review-update no longer grades what the preview says about the diff. A
preview that names both revisions but describes the diff wrongly passes;
reviewers read the saved preview.
- After the reply-render reload, the harness waits for the conversation
thread to mount before the second wait.
- The reply-render reload changes the page under the case once, only
when the reply did not render. The reload is recorded.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:32:20 -06:00
DottaandClaude Opus 5.5 4228bcf89c fix(opencode): never save the model's reasoning as its chat reply (#21051)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode models reply in chat through the Runner's
OpenCode driver. The driver settles one final response, and Paperclip
saves it as the reply comment
> - In chat-eval cohort 5 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`), four saved replies were
the agent's reasoning, for example "Confirmed down. Now report honestly.
... I'll finish with paperclip_finish."
> - The run events show the cause. The model wrote a reasoning part,
then wrote the same text again as an ordinary text part, and ended the
step without `paperclip_finish`. The driver took that copy as the final
answer. This is not the #20999 path: no completion call happened
> - A second, related problem is in the #20999/#21025 path. When the
model writes an answer, calls `paperclip_finish`, and then writes its
own new answer, the driver can save both texts, or a status note
together with the answer
> - This pull request drops the repeated reasoning, lets the existing
empty-response correction ask for the real answer, and saves the
post-completion message alone unless it is a short line that refers back
> - The benefit is that people never see the agent's planning text or
tool names as its reply, and the eval catalog now fails when they do

## Linked Issues or Issue Description

Refs #20999
Refs #21025

**What happened?**
The saved chat reply held the agent's reasoning, not an answer. Cohort 5
cases: `connection-setup.unreachable-mcp`, `plugins.configure-plugin`,
`tasks.website-task`, `teams.install-team`. In each case the reply text
is byte-identical to the reasoning part before it, and the run result
evidence is `provider:opencode:agent-message` (no `paperclip_finish`
call). Also, `recovery.explain-stall` saved its answer twice, and
`status-cards.refresh-card` saved a "STATUS: ..." note above its answer.

**Expected behavior**
The saved reply is the model's answer to the person. It never holds
reasoning, completion tool names or report field names. A short closing
line such as "Answered above" still keeps the earlier answer.

**Steps to reproduce**
1. Use an OpenCode agent with
`openrouter/deepseek/deepseek-v4.1-flash-1017`.
2. Ask a chat request that the model finishes without a completion call
after a long reasoning step.
3. Read the saved reply with `GET /api/issues/{id}/comments`. It holds
the reasoning text.

**Paperclip version or commit**
master at 98c18784c6 (cohort 5) and 3c058c37e6 (base of this PR).

## What Changed

- `OpenCodeServerDriver` records the text of each completed reasoning
part in the run. A text part that still matches the start of a completed
reasoning part does not stream as progress. A completed text part that
repeats a reasoning part exactly is not a response candidate, and the
driver emits the `opencode_reasoning_echo_discarded` diagnostic. With no
answer left, the existing empty-response correction asks the model for a
final answer with reasoning disabled.
- After the completion call, the driver keeps the prose written between
the last work tool and that call only when the later message is a
closing reference: a short line with no link of its own that points back
to it ("Answered above.", "Commands above are ready"), or one under half
that prose's length. A later message that repeats the prose is saved
once. Otherwise the later message is the reply. The completion tool
description already names the post-call message as the final response.
This replaces the `restatesEarlierReply` heuristic from #21025, which is
removed.
- Tests: a new `reasoning-echo` mode in `empty-response.test.ts` uses
the recorded `tasks.website-task` text. Two new driver tests use the
recorded `recovery.explain-stall` and `status-cards.refresh-card` texts.
The #20999 "Answered above" test and the #21025 restatement test pass
without change. After review: the reasoning-echo mode streams the copy
with a trailing newline (it must stay held back), a new driver test
keeps "The result is 42." when the later message is "Answered above.",
and two more check that a STATUS note followed by a short complete
answer saves only the answer, and a repeated short answer is saved once.
- No eval keyword gate: a draft shared check that matched completion
tool names in replies was removed after review, because chat catalog
graders must not grade free-form replies by keyword. Replies stay saved
for human review; the driver fix is what keeps reasoning out of them.

## Verification

- `cd packages/paperclip-runner && npx vitest run src/drivers/opencode`:
17 files, 207 tests pass.
- Red/green: with the driver from `origin/master`, the `reasoning-echo`
test fails (the saved reply is the reasoning text), and both new
post-completion tests fail (two texts saved). With this change, they
pass.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog tests/runner-e2e/chat-reply
tests/runner-e2e/source-sdk`: 3903 tests pass.
- Check calibration: the new unit tests cover recorded valid replies,
reworded valid replies (`paperclip_runner` adapter type,
`PAPERCLIP_API_KEY`, `paperclip_finish_task`, plain
"summary"/"evidence"), and the four recorded broken replies. Over all
193 cohort-5 conversations the check flags exactly the four broken
cases.
- Over the 30 cohort-5 turns with prose before and after
`paperclip_finish`, the new rule changes 3: `recovery.explain-stall`
(duplicate removed), `status-cards.refresh-card` ("STATUS:" note
removed) and `assistant-connections.consent-read-only` (shorter
restatement kept, same link). The pointer-line turns
(`tasks.website-without-project`, `cli-operations.choose-cli-context`)
still keep both texts.
- `npx tsc -p packages/paperclip-runner/tsconfig.json --noEmit` passes.
- A paid chat-eval run on this change is linked in a comment below.

## Risks

- A turn that repeats its reasoning as text now takes one more model
request (the existing empty-response correction). If the corrected
response is empty again, the turn fails with `opencode_empty_response`,
as before.
- A real answer that is exactly equal to a reasoning part is dropped and
requested again. The correction recovers it.
- When a long post-completion message drops something the earlier prose
said, only the later message is saved. The completion tool contract
names that message as the final response, and the earlier prose remains
in the run transcript.
- Saved run state: no persisted run, session, checkpoint or event shape
changes. The reasoning texts are in-memory driver state per run. A run
saved by the previous release resumes without change; only a newly
settled final response differs.
- The new catalog check can fail a case that passed before. In cohort 5
it fails only the four reasoning replies.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:06:40 -06:00
DottaandClaude Opus 5.5 91e26ff237 fix(ui): refetch an open chat's comments when its conversation settles (#21054)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to agents in chat, and the open chat updates live when
an agent replies
> - Run finalization can save a turn's reply after the run's terminal
status is published, and that save logs no `issue.comment_added`
activity
> - Finalization then settles the conversation, which logs
`issue.updated`, but the UI did not refetch comments on that update
> - So the reply stayed hidden in the open chat until the person
reloaded the page
> - This pull request makes the open chat refetch its saved comments
when its conversation is updated
> - The benefit is that a person sees the agent's reply without a reload

## Linked Issues or Issue Description

**What happened?**
In the chat eval cohort on master 98c18784c6, the agent's reply in
`cases.case-history` was saved at 06:57:46.481 by run finalization
(presentation decision `final_agent_message`, comment action `reuse`).
The run's terminal status went out at about 06:57:45. No
`issue.comment_added` activity exists for the reply. The browser fetched
the issue, runs, live runs and activity after the run ended, but it
never fetched comments again. At 06:57:59, 13 seconds later, the open
chat still showed only the person's message (`renderedCommentIds` held
only that comment). The reply showed after a reload.
`tasks.ambiguous-task` showed the same state.

**Expected behavior**
When finalization saves the reply, the open chat shows it without a
reload.

**Steps to reproduce**
1. Open a chat with an agent.
2. Send a message. The run ends, and finalization then saves the reply.
3. The reply does not appear until you reload the page.

Refs #21034

## What Changed

- `LiveUpdatesProvider`: an `issue.updated` activity for the
conversation that is open now also invalidates that conversation's
comments query. Finalization logs this update when it settles the turn,
after it saves the reply (cohort 5: reply 06:57:46.481, update
06:57:46.510; paid run: reply 08:19:38.433, update 08:19:38.489).
- The first version of this PR used the `run.presentation.resolved` run
event. The paid run showed that a page does not receive it: run
transcript events reach only pages subscribed to that run, and the
subscription ends with the run. That handler is removed.
- A red/green test: an update for another task, and an update for the
chat while another page is open, do not refetch; an update for the open
chat refetches its comments. The test fails without the change.

## Verification

- `pnpm exec vitest run --project @paperclipai/ui ui/src/context/`: 17
files, 319 tests pass.
- The new test fails with the change removed (1 failed, 117 passed) and
passes with it.
- `node scripts/check-token-gates.mjs`: clean.
- The companion eval PR reloads once when a saved reply is still not
rendered and records both waits. A paid eval run on a branch with both
changes checks the live result.

## Risks

- Low risk. Each update of the open chat's conversation (status or
conversation state) now refetches its recent comments window. Other
pages and other tasks are unchanged.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:02:55 -06:00
DottaandClaude Fable 5.2 fd9985b571 fix(native): say what a Runner command waited behind when it times out (#21056)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes talk to a Runner through durable PRP
commands. The controller waits for each command's result
> - After a server replacement, the new controller reattached to live
Runners. For runs it reached later than about 115 s after detach,
`session.snapshot` got no answer within 30 s
> - The failure said only `provider_transport_failed: PRP command
session.snapshot timed out`. The run log could not tell whether the
Runner was disconnected, whether the command waited behind older
commands, or when the Runner last sent an event
> - This pull request adds that evidence to the timeout message. It
names types, counts and durations only, never payloads
> - The benefit is that the next upgrade's failures explain the stall,
so the runner-side cause can be fixed

## Linked Issues or Issue Description

Refs #21053, #21052, #21039

**What happened?**
At the 07:10Z staging swap, 9 remote native runs that the new controller
reached 114–152 s after detach failed. Each recovery read ended with
`PRP command session.snapshot timed out`. The transport had
authenticated, and in some runs the same Runner process was verified.
The Runner journal recorded no diagnostic other than "WebSocket peer
closed the connection". Runs reached within 39–116 s succeeded.

**What did you expect to happen?**
A command timeout should say what the command waited behind, so the
cause can be found from the run log.

**Steps to reproduce**
1. Start a remote native run and replace the server while a provider
turn is active.
2. Let the new controller reach the Runner more than about two minutes
after detach.
3. Read the run log. The failure says only "timed out".

## What Changed

- `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`:
`waitForRunnerCommand` accepts an optional `describeStall`. When a
command times out, the message ends with a bounded description, for
example `(runner connected; pending ahead 1 [semantic_tool.result]
oldest 188s; last runner event seq 152 181s ago)`. The description
covers whether a Runner is connected or how long it has been
disconnected, the pending commands ahead of this one (types and count,
up to 6 types, and the age of the oldest), and the last committed Runner
event's sequence and age. A failing or empty description keeps the plain
message.
- The transport passes this description for every command it waits on.
The last event's age comes from the PRP payload's `emittedAt`.
- `server/src/services/native-runtime/native-session-executor.ts`: the
startup `session.open` timeout classifier accepts the optional
description, so a startup timeout stays `provider_session_open_timeout`
and keeps its bounded same-run retry. A regression case covers each
startup code with the suffix.
- Tests: the exact message for the staging shape, the plain message when
there is nothing to describe or the description throws, and a
disconnected Runner with no events.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/live/runnerd-codex-transport.test.ts -t "a Runner command that gets
no answer"` passes 3 of 3. These tests fail on `origin/master`, which
has no `describeRunnerCommandStall` export and no description in the
message.
- The full `runnerd-codex-transport.test.ts` file passes 307 of 308. The
one failure (`stops a lost runnerd's provider and escaped tool job from
its lineage journal`) is a process-timing test that passes alone and
does not touch this code.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes.
- Message matchers: the only matcher anchored on the end of the timeout
text was the startup `session.open` classifier. It now accepts the
suffix. `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"classifies a startup session.open timeout"` passes. Without the
classifier change it fails, because the suffix turns the timeout into
`native_session_interrupted`.

## Risks

- Low. Only the text of a timeout message changes, by appending text. No
control flow, persisted state or Runner behavior changes.
- The description holds command types, counts and durations only. It
holds no payloads, identifiers or provider text.
- No wait is added before a turn start.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, with tool use (shell, file edits,
test runs, read-only staging API reads). Model ID as attributed in the
commit trailer: Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 03:00:32 -06:00
DottaandClaude Fable 5.2 8e956da1cf fix(native): keep a live reattached Runner when its restart recovery read times out (#21053)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes (Daytona) use a detached Runner. A
server replacement detaches the controller and leaves the Runner alive.
The new server reattaches to it and continues the same turn
> - On staging, about half of the in-flight runs at each upgrade lost
their turn. The new controller reattached to the live Runner (same
process verified), its recovery read timed out (`PRP command
session.snapshot timed out`), and the failure cleanup killed the Runner
> - The next same-run reattachment then found `process_not_running`.
#21039 continues these runs in a fresh session, but the in-flight turn
is lost and its outcome is unknown
> - The cause of the kill is in the Codex driver's `recoverSession`: its
cleanup closed the transport without retention. For a reattached Runner
that means SIGKILL after the close deadline. #21023 already keeps the
Runner for a failed open session, but not for a failed recovery
> - This pull request keeps the live Runner when a recovery fails or is
aborted on its way to it. Identity refusals and protocol integrity
faults still stop it
> - The benefit is that a slow reattachment no longer destroys a turn
that was still running

## Linked Issues or Issue Description

Refs #21023, #20923, #21039, #21052

**What happened?**
Staging replaced its server at about 07:10Z with 17 remote native runs
in flight. For each, the old server logged "Server shutdown suspended
native controller ownership without cancelling provider work". The new
server reattached. Example run: the Runner process (same pid and start
time) was verified at 07:12:37 and the transport authenticated at
07:12:32. At 07:13:28 the recovery failed with
`native_session_recovery_failed: ... PRP command session.snapshot timed
out`, and a same-run resume was saved. At 07:14:12 the same-run
reattachment found `process_verification` / `process_not_running`, with
the Runner journal still `ready` and no exit record from the Runner
itself. The run failed with `runner_remote_process_identity_unavailable`
/ `native_continuation_requires_reconciliation`. 90 of 93 such failures
in two hours of upgrades had this sequence.

**What did you expect to happen?**
A recovery that times out against a live Runner is a transient failure.
The same-run reattachment should find the Runner alive and adopt it
again. Killing the Runner is only for a deliberate stop, an identity
refusal, or an integrity fault.

**Root cause**

`packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts`
`recoverSession`: the `catch` block and the abort handler call
`bootstrapCancellation(...).close()`, which calls `transport.close()`
without `retainUnsettledRemoteRunner`. In `runnerd-codex-transport.ts`
`#closeOnce`, an unresponsive adopted Runner that is not retained gets
`SIGKILL` after the close deadline. The Codex driver serves remote
Codex, OpenCode and ACPX runs, which matches the failures seen on
staging for all three driver kinds.

## What Changed

- `codex-app-server-driver-impl.ts`: `recoverSession` classifies a
thrown recovery failure. A transient one (`provider_transport_failed`, a
`PRP command ... timed out`, or the transport closing while waiting) now
detaches this controller with the transport's restart detach
(`detachControllerForRestart`). That path queues no `turn.stop` and no
`runner.suspend` and sends no signal, the same as a server restart. The
Runner and its in-flight turn stay for the same run's next reattachment.
An abort (a Stop or the recovery deadline), any message that names a
mismatch, a different session, identity, authority, integrity or
incompatibility, any unclassified failure, any returned identity
refusal, and a protocol integrity fault still close and stop the Runner.
`openSession` is unchanged.
- `codex-app-server-driver.recovery.test.ts`: the staging message
(`codex_history_read_failed: ... provider_transport_failed: PRP command
session.snapshot timed out`), and a plain snapshot timeout now detach
and never close. An abort of a hung read still closes and stops the
Runner. A thrown different-session refusal,
`native_adopted_provider_identity_mismatch`, an identity refusal that
also mentions a timeout, and an unclassified failure still close without
retention. The existing integrity-fault cases check that the close does
not retain the Runner.
- `doc/execution-semantics.md`: describes the rule.

## Verification

- Red on `origin/master` with only the new tests: `cd
packages/paperclip-runner && npx vitest run
src/drivers/codex/codex-app-server-driver.recovery.test.ts` fails 2 of
69 (the two detach cases; master calls `close()` and stops the Runner).
- Green with the fix: 69 of 69.
- `npx vitest run src/drivers/codex src/native-session-runtime.test.ts
src/backends src/live/runnerd-codex-transport-remote-retention.test.ts
--maxWorkers=4` passed except two timing tests (Codex command
resolution, background cleanup containment) that pass alone and do not
touch this code.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes.
- `detachControllerForRestart` is the existing restart path. It stops
the pump, disconnects the Runner, and drains the controller journal. It
queues no command and never signals the process.
- Not run: a live Daytona restart. The next post-upgrade audit should
show fewer `runner_remote_process_identity_unavailable` failures after a
`session.snapshot timed out` resume. Instead, the run log should show
"controller authority detached for restart; durable runner left alive"
and then a successful same-run reattachment.

## Risks

- Medium. A Runner that the recovery could not reach now stays alive,
with its turn, for the same-run reattachment. This is the state a server
restart already leaves. The same-run reattachment adopts it, or a Stop
cleanup or saved-work recovery stops it.
- If the Runner stays unresponsive across every same-run attempt, the
run no longer gets the `process_not_running` proof that #21039 uses for
its automatic fresh-session continuation. The run then keeps its visible
Retry. The staging data shows that reattachments reaching the Runner
sooner succeed, so a later attempt is expected to succeed. This is not
proven against a live sandbox.
- An abort of the recovery (a Stop or the recovery deadline) is
unchanged: it closes and stops the Runner.
`plugin-runner-release-resume.integration.test.ts` (14 of 14) covers a
Stop during an adopted resume.
- No turn is replayed and no second Runner is launched. Adopting an
existing Runner never launches a new one.
- Previous-release story: no persisted state changes. The decision is
made in the process that runs the recovery, so a deploy during a run
changes nothing.
- No wait is added before a normal turn start. Only the recovery failure
path changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, with tool use (shell, file edits,
test runs, read-only staging API reads). Model ID as attributed in the
commit trailer: Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
canary/v2026.1112.0-canary.7
2026-11-12 02:59:46 -06:00
DottaandClaude Opus 5.5 424e2c6488 fix(recovery): keep live owners' runs through an upgrade and continue replaced preparations in the restart lane (#21046)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade replaces the server. The old and the new server serve
together for several minutes. A legacy run proves that its owner is
alive by renewing a 60-second lease on its `heartbeat_runs` row.
> - #21019 added an uncharged `server_restart` retry lane for runs that
a restart or replacement ended. On the first upgrade that shipped it, no
run used that lane. The new server also failed many runs as
`process_lost` while the old server that owned them was still serving.
> - Each of these runs then spent one failure retry. A run that was
already the last automatic retry was held for manual reconciliation, so
the task needed a person.
> - An upgrade is routine. It must not cost a task its work or a retry,
and it must not leave a task waiting for a manual Retry.
> - This pull request sends a run that a replacement stopped during
preparation to the `server_restart` lane. It also stops the reaper from
failing a run while the server process that owns it is proven alive.
> - The benefit is that a task continues by itself through an upgrade,
also when the database is slow during the swap.

## Linked Issues or Issue Description

Refs #21019, #21042

**What happened?**
A hosted staging server was replaced by an automatic upgrade. The new
server started at 06:18:46Z, became ready at 06:26:17Z, and the old
server received SIGTERM at 06:27:34Z. Between 06:23Z and 06:26Z the new
server failed 44 runs as `process_lost` ("Process lost -- server may
have restarted; retrying once",
`processLossDiagnostic.controllerReplaced: true`). The old server still
owned these runs and was still serving. Every reaped run was a Runner
run in `executionStage: preparing` with no process recorded. Each
successor was scheduled with `scheduledRetryReason: transient_failure`
and `executionRetryAccounting.failureRetries: 1`. No run on the server
has `retryReason: server_restart` or
`executionRetryAccounting.serverRestarts`.

The run evidence also shows why the leases lapsed. The old server's
lease renewals waited 19 to 50 seconds on the database
(`controllerLeaseLoss.renewalPendingMs`) while its timers fired on time
(`deadlineLateMs` 0 to 17 ms). #21042 fixes that renewal stall. This
pull request fixes how the reaper and the retry scheduler treat these
runs.

**Expected behavior**
- A server never fails a run as lost while the server process that owns
it is alive and serving.
- A run that a replacement really ended continues once, automatically,
without spending the failure budget.

**Steps to reproduce**
1. Seed a legacy `paperclip_runner` run in `executionStage: preparing`,
owned by another server boot, with a lapsed run lease.
2. Run `reapOrphanedRuns()` on the new server.
3. See that the successor is `transient_failure` with `failureRetries:
1`. When the chain already spent its failure budget, see that no
successor is created and the run is held for reconciliation.
4. Give the owner a live controller instance lease (the old server is
still serving) and run `reapOrphanedRuns()` again: the run is failed as
`process_lost` anyway.

**Root cause**
- `usesServerRestartLane` (`server/src/services/heartbeat/retries.ts`)
left every run with the bootstrap evidence `providerWorkStarted: false`
to the `interrupted_preparation` lane. But `scheduleBoundedRetryForRun`
gives that lane only to an `interrupted` run (or a native preparation).
The reaper writes the same evidence on a `failed`/`process_lost` run
that it stops during preparation. That run fell through to the charged
`transient_failure` lane. The #21019 tests used runs in `dispatching`,
so they did not catch it.
- `legacyExecutionNeedsReconciliation` held every run whose failure
budget was spent before it checked the bootstrap evidence. A run that a
replacement stopped before dispatch did no provider work, but it was
held for reconciliation, and the task waited for a person.
- The orphan reaper (`recoverActiveExecutions`,
`server/src/services/heartbeat/recovery.ts`) treated a lapsed run lease
as proof that the owner was gone. It checked the shared owner prover
only for "gone", never for "alive". During the swap the old server's
renewals were slow, so its leases lapsed while it was alive.

**Paperclip version or commit**
Staging Core `51d77648bc` (first build with #21019). The behavior is the
same on master `148d2b629c`.

## What Changed

- `retries.ts`: `usesServerRestartLane(run, preparationLaneTakesRun?)`
keeps the `interrupted_preparation` exception only when that lane really
takes the run. `scheduleBoundedRetryForRun` passes its own verdict.
Without one, only an `interrupted` run qualifies. A `process_lost` run
that a replacement stopped during preparation now continues in the
`server_restart` lane.
- `legacy-execution-recovery.ts`: a replaced preparation
(`isReplacedPreparation`) does not need reconciliation, also when its
failure budget is spent. The `server_restart` bound limits it.
- `recovery.ts` (orphan reaper):
- When a run's lease lapsed but the shared owner prover proves its owner
process alive (for example, the owner's controller instance lease is
live), the reaper leaves the run with that owner. The owner stops and
settles the run itself (`controller_lease_lost`, uncharged
`interrupted_preparation` lane). The reaper recovers the run once the
owner is gone or stopped. It also recovers the run when the owner has
not settled it within `LEGACY_LIVE_OWNER_SETTLE_GRACE_MS` (5 minutes)
after the lease lapsed.
- A setup that a replacement stopped before dispatch uses the
`server_restart` bound for its retry decision, whatever the adapter.
- `legacy-controller-lease.ts`: add `proveLegacyController(run)`, which
returns the shared prover's verdict. `legacyControllerProvenGone` uses
it, and the reaper computes the proof once per run.
- `server-replacement-interruption.ts`: add
`isReplacedPreparation(run)`: bootstrap evidence `providerWorkStarted:
false`, a recorded controller process stamp, and a server-replacement
interruption. A row saved by an older release without the stamp keeps
the charged lane (the existing documented rule).
- `legacy-execution-recovery.ts` (`settleInterruptedNativeBootstrap`): a
reconciliation hold that an older release saved on a replaced
preparation is settled once its owner is proven gone, even when the
failure budget is spent. The run continues in the `server_restart` lane.
- `legacy-controller-lease.ts`, `recovery/ownership-registry.ts`: the
`legacy_interrupted_preparation_hold` kind judges a finished run's hold
by its recorded controller process
(`finishedLegacyControllerOwnerStamp`). Revocation by the orphan reaper
swaps in a random boot id, so before this change a reaped preparation's
hold had owner "none" and was never settled.
- `recovery/service.ts`, `heartbeat.ts`: the stranded-task sweep reports
exhaustion from the scheduler's own `retry_exhausted` result, so a spent
`interrupted_preparation` lane blocks the task even when the synchronous
budget check reads an unspent restart bound.
- `doc/execution-semantics.md`: document these rules.
- No schema change. Nothing changes on the path that starts a run or a
turn (no new waits and no file work).

## Verification

- Red on `origin/master`, green on this branch
(`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs
ended by a server restart or replacement"):
- "continues a first attempt / a run that was already the last automatic
retry that a replaced server left preparing, in the restart lane". This
is the production shape: a `paperclip_runner` run in `preparing`, two
servers sweep at the same time. Master: `transient_failure` with
`failureRetries: 1`, and no successor for the spent chain. Now: one
`server_restart` continuation, due at once, with the failure count
unchanged.
- "leaves a preparing/dispatching run whose lease lapsed with its owner,
then recovers it once the owner stops". The owner's controller instance
lease is live and two servers sweep. Master: `process_lost` at once.
Now: the run stays `running` with its owner. After the owner stops, it
becomes `process_lost` and a `server_restart` continuation is scheduled.
- "recovers the run when its live owner has not settled it within the
grace".
- `interrupted-preparation-hold-settle.test.ts`: "releases an older
release's hold on a preparation a server replacement stopped with its
failure budget spent" for an unrevoked and a reaper-revoked lease (red
without the settlement and owner-stamp changes) and "blocks the task
when a replaced preparation's lane is spent even though the restart
bound is not" (red without the sweep change).
- Related suites: heartbeat process recovery, retry scheduling, orphaned
active lease sweep, recovery deadline (bootstrap and ordinary
bootstrap), legacy continuation authority, undispatched execution wait,
agent status recovery, legacy workspace restore recovery, legacy
execution recovery, legacy controller lease, server replacement
interruption, heartbeat recovery, provider failure classification,
reconciliation run read, and wake-queue policy.
- `tsc --noEmit` for `server`.

## Risks

- A run whose owner is alive but no longer drives it now stays `running`
for up to 5 minutes after its lease lapsed, and then the reaper recovers
it. The owner's own sweep still recovers a run that the owner itself
lost the handle for (this boot is never deferred).
- The guard trusts the shared owner prover. That prover reads "alive"
only from a live or grace controller instance lease, or from a live PID
in the same PID space. A server whose instance lease cannot renew for 60
seconds stops itself, so a stuck owner cannot hold runs forever.
- More runs continue in the uncharged lanes. The `server_restart` bound
(6 in 60 minutes) still stops a loop, and the stop is shown on the task.
- Rows saved by older releases: a `process_lost` row with the bootstrap
evidence and an old-boot owner now continues in the `server_restart`
lane instead of being held. These runs did no provider work.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only log and API inspection) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 02:18:39 -06:00
DottaandClaude Fable 5.2 fd5ee7e385 fix(native): continue a routine assignment or chat message whose remote Runner an upgrade lost mid-turn (#21052)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes (Daytona) use a detached Runner.
When the server is replaced during an upgrade, the new server reattaches
to that Runner
> - On staging, upgrades every 15–40 minutes lose the in-flight Runner.
The same-run reattachment then finds `process_verification` /
`process_not_running`, the run fails with
`runner_remote_process_identity_unavailable`, and the task is blocked
> - #21039 made these runs continue once in a fresh session (the turn is
recorded as having an unknown outcome and is not replayed). It covers
user wakes and automatic retries (`retryOfRunId`) only
> - A routine's system assignment (`issue_assigned`, requested by the
system) and an external chat message wake have no `retryOfRunId`. They
got `not_user_request` and stayed blocked for a person
> - This pull request treats those two wakes as owed task work, so they
continue the same way once the Runner is proven gone
> - The benefit is that an upgrade no longer leaves these tasks waiting
for a Retry click

## Linked Issues or Issue Description

Refs #21039, #21023, #20923

**What happened?**
Staging replaced its server at about 07:10Z. 17 remote native runs were
in flight. Each new controller reached its Runner 114–152 s after
detach. Recovery failed with "PRP command session.snapshot timed out".
The same-run reattachment then found the Runner process gone, and each
run failed with `native_continuation_requires_reconciliation`. #21039
continued 14 of the 17 automatically. The other 3 were a routine's
assignment (two runs) and an external chat message. They have no
`retryOfRunId`, so `requestInterruptedNativeTurnRetry` returned
`not_user_request`. Those tasks are still blocked with no retry.

**What did you expect to happen?**
The Runner is proven gone, so nothing can still run the turn. The run
should continue once in a fresh session, the same as an automatic retry
of owed work.

**Steps to reproduce**
1. Start a remote native run from a routine assignment, or from an
external chat message.
2. Replace the server while the provider turn is active, so that the
reattachment finds `process_not_running`.
3. The run fails and the task stays blocked. No automatic Retry is
requested.

## What Changed

- `server/src/services/heartbeat/restart-interrupted-turn.ts`: after the
Runner is proven gone, a system wake also continues when it is a system
assignment (`issue_assigned`), or when it is a comment or external chat
message wake whose message is still on the task (not deleted, not
written by a run). It runs under the run's responsible user, at most
once per chain of runs, like an automatic retry already does. A linked
person's external chat message continues under that person when the
message is still on the task. Monitors and other system wakes are
unchanged.
- The same file and `server/src/app.ts`: the hook now passes the chat
service's Retry callback (`prepareFailedChatRunRetry`, registered at app
startup next to the budget-resume callback). A task in a connected
conversation whose endpoint restricts external execution then continues
through the chat Retry path. Before, `requestTaskRetry` refused it
(`retry_refused`) and the task stayed blocked.
- `server/src/services/runner-lost-mid-turn-continuation.test.ts`:
production-shape cases for a system assignment, an unlinked external
chat sender, and a linked person's chat message. Negative cases: a
deleted message, a message written by a run, a deleted linked message,
and a system assignment whose Runner was not proven gone.
- `doc/execution-semantics.md`: describes the rule.

## Verification

- Red on `origin/master` with only the new tests: `cd server && npx
vitest run src/services/runner-lost-mid-turn-continuation.test.ts` fails
4 of 16. The three new continuation cases return `{ requested: false }`.
The deleted linked-message case returns `not_user_request` instead of
`message_unavailable`.
- Green with the fix: 16 of 16. Two more cases bind the task to a real
restricted Slack endpoint and conversation. With the chat callback
registered, the run continues and the Retry carries the chat action (18
of 18). Without the callback, the result is `retry_refused`, which was
the behavior before for every connected conversation.
- Related suites pass: `runner-lost-mid-turn-continuation.test.ts`,
`heartbeat-native-remote-failure-retention.test.ts`,
`workspace-recovery-retry-copyback.test.ts`,
`governed-wait-runner-loss.test.ts` (116 tests).
- `cd server && npx tsc --noEmit` passes.
- Production evidence was read from staging run events through the
read-only API: 3 of 17 losses at the 07:10Z swap took this path. After
this change the next post-upgrade audit should show an automatic
`task_retry` for every `native_continuation_requires_reconciliation` run
whose refusal is `process_not_running`, including routine assignments
and chat messages.

## Risks

- Low. The change applies only after the Runner process is proven gone
(`process_verification` / `process_not_running`), so no Runner can still
run the turn. The turn is not replayed. The fresh session continues over
the saved work, as in #21039.
- The task can contain partial tool work from the lost turn. This is the
same accepted risk as #21039.
- Previous-release story: no persisted shape changes. The rule reads the
wake's reason and comment id, which earlier releases already write. A
run an older release already settled keeps its visible Retry.
- This does not stop the Runner loss itself. That is a separate fix.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, with tool use (shell, file edits,
test runs, read-only staging API reads). Model ID as attributed in the
commit trailer: Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 02:11:28 -06:00
DottaandClaude Opus 5.5 4e519bd428 fix(chat): send writes with an empty responseText and point card declines to resolve-from-comment (#21050)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents call the Paperclip API through `call_api` and find
routes with `search_api`
> - #21035 added a pre-send refusal for writes that carry
`responseText`, and new summaries for the question cancel and withdraw
routes
> - Full chat cohort 5 (OpenCode with DeepSeek v4.1 flash) showed that
both changes went too far
> - The refusal blocked a valid write that had an empty `responseText`,
and the summaries hid the route that records a person's decline
> - This pull request narrows the refusal and corrects the guidance text
> - The benefit is that valid writes go through on the first try, and
agents settle a declined card instead of leaving it pending

## Linked Issues or Issue Description

Refs #21035, #21002

**What happened?**
- inbox-search.restore-archive: DeepSeek sent `responseText: {}` with a
valid body on `POST /api/issues/{id}/inbox-archive`. The refusal from
#21035 rejected 24 POSTs in 3 minutes. The model did not drop the key,
and the run was cancelled at the 240s window. In cohort 4 the same POSTs
went through on the first try.
- questions-and-approvals.reject-and-revise: the person said "Do not
publish it; just make a draft I can review." The CEO quoted the cancel
summary ("an agent withdraws its own pending card with POST
.../withdraw"), saw that the person created the publish card, and left
the card pending. The plan decision did not settle. In cohorts 3 and 4
the CEO declined the card with `resolve-from-comment` (resolver policy
`anyone`).

**Expected behavior**
- A write with an empty or default `responseText` is sent once. A write
whose `responseText` sets a window is still refused before it is sent.
- The guidance says that withdraw is only for the agent's own obsolete
card, and that a person's explicit decline or answer goes through
`resolve-from-comment` when the card's resolver policy allows.

**Steps to reproduce**
Run the two chat cases above with OpenCode and DeepSeek v4.1 flash on
master.

## What Changed

- `call_api` (`assertRunnerApiDispatchable`): the check reads the raw
`responseText` value. `{}` or `{ offsetBytes: 0 }` on a write is dropped
and the write is sent. A `responseText` with `limitBytes` or a later
offset is still refused. `saveResponse: false` on a write was already
accepted; a test now covers it.
- OpenAPI: the cancel and withdraw summaries say that withdraw is only
for the agent's own obsolete card, and that a decline or answer on the
person's explicit instruction uses `POST .../resolve-from-comment` when
the resolver policy allows. The `resolve-from-comment` summary says it
also records a decline of a card the person created.
- `AGENT_CHAT_DIRECTIVE`: a person's explicit decline in chat answers a
pending confirmation with reject, whoever created the card. Withdraw
never records the person's decision.
- `doc/runner-api-tools.md`: describes the narrowed refusal.

## Verification

- `server/src/services/native-runtime/runner-api.test.ts`: new test
sends `POST /api/issues/{id}/inbox-archive` with a valid body and
`responseText: {}`, `{ offsetBytes: 0 }`, `saveResponse: false`, or
both. The request is sent exactly once. Red before the fix (3 of 4 cases
refused), green after. The refusal test for a `responseText` with
`limitBytes` stays.
- `server/src/__tests__/openapi-routes.test.ts`: new test for the
cancel, withdraw and resolve-from-comment summaries. Red before, green
after.
- `server/src/__tests__/agent-conversations.test.ts`: new directive test
for the decline rule. Red before, green after.
- Also passed: `runner-api.integration.test.ts`,
`native-case-tools.integration.test.ts`, server `tsc --noEmit`. SDK
public input pins: `--check` reports no change.
- Paid run (OpenCode + DeepSeek v4.1 flash) on this branch for
inbox-search.restore-archive, questions-and-approvals.reject-and-revise,
gateways-profiles.rotate-token and budgets-costs.repair-cost-recording:
4 of 4 cells passed on 3f20fb2bcc (run 43177772183); details in a PR
comment.

## Risks

- A write with a populated `responseText` is still refused, as in
#21035. Only the empty and default shapes change. A dropped empty
`responseText` changes nothing, because the server already removed it
from writes before sending.
- The directive grows by two sentences, which adds a small number of
prompt tokens per chat turn.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 02:09:14 -06:00
DottaandClaude Opus 5.5 3c058c37e6 fix(retry): continue a connected task from saved work when its exact chat request cannot be repeated (#21038)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks that come from a connected chat (for example GitHub pull
request reviews) show a Retry button when a run fails
> - The task projection offers Retry, but the Retry route sends a
connected task to the chat retry owner, which only replays the exact
failed chat request
> - When that exact request cannot be replayed (the attempt already
saved a result, its output was published, the latest run did not start
from a chat message, or the requester's access changed), the route
returns 409 and tells the person to check the connection by hand
> - A Retry button that is shown must work, and recovery must not ask
the person to inspect anything
> - This pull request makes a person's Retry continue the task from its
saved work when the exact chat replay is refused, the same way a new
message on the task would
> - The benefit is that the projection and the route agree, and blocked
connected tasks become retryable after deploy with no migration

## Linked Issues or Issue Description

No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a hosted deploy, five blocked tasks that came from a connected GitHub
conversation showed Retry (`interruption.canRetry: true`). Each click on
Retry returned 409 `chat_failed_run_retry_not_authorized`. The message
was "The exact failed chat request can no longer be retried safely.
Check its current connection, access, and recovery state." or "This
attempt already recorded a result. Send a new message on the task to
continue from the saved work." The latest runs were legacy sandbox setup
failures, native runs that needed reconciliation, a run that started
from an issue monitor instead of a chat message, and a native run that
already saved its result. Root cause: `admitTaskRetry` in
`server/src/services/task-retry.ts` called the chat retry owner for
every connected task and passed its exact-replay refusal straight to the
person. The projection cannot predict that refusal.

**Expected behavior**

A shown Retry works. If the exact chat request cannot be repeated, Retry
continues the task from its saved work, the same way a new message on
the task would. Users do not get operator instructions.

**Steps to reproduce**

1. Bind a task to a restricted chat conversation.
2. Make its latest run fail so that the chat retry owner refuses the
exact replay. For example, use a run whose source is not a chat message,
or a native run with an accepted result.
3. Click Retry (`POST /api/issues/:id/retry`).
4. The route returns 409 `chat_failed_run_retry_not_authorized`.

**Paperclip version or commit**

`master` at 5a7378af5 (the deployed revision), reproduced on 51d77648bc.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

Not adapter-specific (core bug). Seen with `paperclip_runner` legacy and
native runs.

## What Changed

- `server/src/services/task-retry.ts`: a person's Retry runs the exact
chat replay inside a savepoint. On a
`chat_failed_run_retry_not_authorized` refusal, the savepoint rolls back
and the same Retry receipt stays an ordinary board continuation. The
clicking person is the responsible user, the run starts a fresh session
(`forceFreshSession`) from saved work, and no chat action, chat
authority, or output replay is created. Retry that Paperclip requests by
itself (`initiatedBy: restart_recovery | provider_limit_reset`) keeps
the exact-replay requirement. Other errors (403, lock contention,
database errors) still propagate.
- `server/src/services/chat-channels/failed-run-retry.ts`: plain refusal
wording. Runs saved by an earlier release with the old diagnostic are
still recognized by the pre-provider check.
- Tests: new red/green cases in `server/src/services/task-retry.test.ts`
(four refusal reasons on a chat task whose latest run came from an issue
monitor, plus automatic Retry still refused). Updated two real-service
route cases in `server/src/__tests__/chat-channels.integration.test.ts`
(accepted native result on an inherited GitHub task, and an unknown chat
source) from expecting 409 to expecting a board continuation without a
chat replay.
- A person's Retry on that person's own saved board turn (`source:
issue.comment.reopen`) that cannot be certified for replay also
continues from saved work, without replaying the saved message.
Automatic Retry still gets 409 `saved_board_message_not_replayable`.
- A person's Retry when no chat retry owner is wired also continues from
saved work. Automatic Retry still gets 409.
- The fallback is not used while the conversation holds saved or
unadmitted chat input for the task. That input keeps its own admission
path and the existing refusal.
- Docs: `doc/execution-semantics.md` and `doc/connections/GITHUB.md`.

### 409 paths in the task Retry route after this change

| Path | Who sees it | Result |
| --- | --- | --- |
| `chat_failed_run_retry_not_authorized` from the chat retry owner |
Person | Continues from saved work (board continuation) |
| `saved_board_message_not_replayable` | Person | Continues from saved
work |
| No chat retry owner wired | Person | Continues from saved work |
| Any of the three above while saved or unadmitted chat input is pending
| Person | 409. The pending input continues through its own admission
and delivery. |
| Any of the three above | Automatic Retry (restart recovery, provider
limit reset) | 409, recorded by the hook. No button. |
| `task_retry_busy` (chat authorization lock held) | Person | 409 with
`retryable: true` after 10 internal attempts. A new click works. |
| Idempotency key reused for a different task or user | Client error |
409. The UI makes a new key for each click. |

## Verification

- Red before the fix: `npx vitest run src/services/task-retry.test.ts -t
"connected task"` failed 4/4 with `Error: refused` (409). The board-turn
and missing-owner cases failed with "This saved message cannot be
retried safely..." and "Retry is unavailable for this connected
conversation." `src/__tests__/chat-channels.integration.test.ts -t
"continues an inherited GitHub task"` failed with `expected 202
"Accepted", got 409 "Conflict"`.
- Green after the fix: `src/services/task-retry.test.ts`,
`src/__tests__/durable-chat-wakeup.test.ts`,
`src/services/budget-resume-continuation.test.ts`: 160/160 passed.
- Full `src/__tests__/chat-channels.integration.test.ts`: 1949/1951
passed. The 2 local failures (`binds Slack tools...`, `materializes
direct external-chat finals...`) read `.claude/skills/slack/SKILL.md`
from the parent checkout of a nested local worktree (ENOENT). They do
not touch Retry and depend on the local checkout layout.
- `npx tsc --noEmit -p server/tsconfig.json`: passed.
- `pnpm verify` exited 2 (a doc file is a global CI input), so required
CI supplies broad coverage.
- Staging was inspected with GET requests only. No POST was sent to
staging.

## Risks

- Behavior change: a person's Retry on a connected task can now start a
board continuation where it was refused before. It uses the same
authority as a new message from that person, and dispatch keeps all
normal gates (task access, executor containment, recovery holds,
unadmitted chat input).
- The continuation does not publish the earlier result again and does
not reuse the chat requester's authority.
- No migration. Existing blocked tasks become retryable at the next
click.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1112.0-canary.6 nightly/v2026.1112.0-nightly.0
2026-11-12 01:27:54 -06:00
DottaandClaude Opus 5.5 b2cb8f8f56 test(native): cover a turn.start timeout that loses the remote Runner, end to end (#21041)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the same run
reattaches to that Runner
> - On staging, 31 tasks that a person retried are blocked. Their last
run failed with `runner_remote_process_identity_unavailable` /
`native_continuation_requires_reconciliation`
> - In each run, the `turn.start` PRP command timed out. The close then
killed the Runner. The same-run reattachment found the Runner gone
(`process_not_running`, `providerTurnIssued: false`), and the task
waited for a Retry
> - #21023 and #21039 fix this sequence. Neither one is deployed yet. No
test reproduces this exact sequence, so a later change could break it
without a signal
> - This pull request adds two tests. One reproduces the turn start
timeout with a real `paperclip-runnerd`. The other reproduces the
staging failure record of a person's Retry
> - The benefit is that a regression of this failure class fails CI
before it reaches staging

## Linked Issues or Issue Description

Refs #21023, #21039, #20923, #20853

**What happened?**
A person retried a blocked task that runs Codex on Daytona. The Retry
started a fresh provider session and submitted a turn. The `turn.start`
PRP command timed out (`provider_transport_failed: PRP command
turn.start timed out`). The run closed. The run log shows `runner did
not prove durable suspension before checkpoint`, `runner shutdown
failed: Error: Durable recovery runner timed out.`, and then
`runner_remote_process_exited lifecycle=ready`. The log does not show
`remote runner left alive unsettled for same-run reattachment`. The
same-run retry tried to reattach 50 s later. It recorded
`native.remote_runner_recovery_failed` with `stage:
process_verification`, `reason: process_not_running`, `runnerSettled:
false`, `providerTurnIssued: false`, and an empty Runner stderr. The run
failed with `runner_remote_process_identity_unavailable`, and the task
waited for a person.

**Expected behavior**
A turn start timeout does not kill the Runner, so the same-run
reattachment adopts it. If the Runner is gone anyway, the run continues
once by itself in a fresh provider session and does not replay the turn.

**Steps to reproduce**
1. Start a remote native Codex run on a release before #21023.
2. Make the provider not answer `turn/start`, so the `turn.start` PRP
command times out.
3. The runtime cancels the session as failure cleanup. The close treats
that as a Stop and kills `paperclip-runnerd`.
4. The same-run reattachment finds `process_not_running`. Before #21039,
the completion hook refuses an automatic Retry with
`provider_turn_unsettled`.

**Paperclip version or commit**
Staging runs 8cefbe9977. That build has neither #21023 nor #21039. This
pull request is based on master after both.

**Deployment mode**
Hosted staging cloud workspace. Native runtime with a remote Daytona
sandbox.

**Agent adapter(s) involved**
Codex (native `codex_app_server` driver through `paperclip-runnerd`).

## What Changed

-
`packages/paperclip-runner/src/live/runnerd-turn-start-timeout-retention.test.ts`
(new): runs `executeNativeSession` over a real `paperclip-runnerd` with
a launcher-owned (remote-shaped) process. The fake Codex never answers
`turn/start`, so the `turn.start` PRP command times out. The test checks
that the run fails with that exact error, that the Runner process is
still alive after the close, and that its journal is `ready` for the
same-run reattachment.
- `server/src/services/runner-lost-mid-turn-continuation.test.ts`: a new
case with the staging failure record. A person's Retry (`task_retry`, a
user intent with a non-hook key, `forceFreshSession`) has only
`turn.submitted`. Its refusal has `runnerSettled: false` and
`providerTurnIssued: false`. The completion hook requests one automatic
Retry. The test checks one unknown-outcome event with `replayed: false`,
an idempotent second call, saved-work recovery first, and one queued
fresh-session successor. The fixture gets small optional inputs for the
wake, the events, and the refusal payload. Other cases are unchanged.
- No production code changes.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/live/runnerd-turn-start-timeout-retention.test.ts
src/live/runnerd-codex-transport-remote-retention.test.ts`: 6 of 6 pass.
The new test passed 4 runs in a row.
- Red without #21023: with `native-session-runtime.ts`,
`harness-driver-backend.ts`, and `native-session-backend.ts` from before
#21023, the new runner test fails with `expected 'gone' to be 'alive'`.
That is the staging failure.
- `cd server && npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts`: 9 of 9 pass.
- Red without #21039: with `restart-interrupted-turn.ts` from before
#21039, the new server case fails with `{ requested: false, reason:
"provider_turn_unsettled" }`.
- `npx tsc --noEmit` passes for `server` and
`packages/paperclip-runner`.
- `pnpm verify` exits 2 because the new runner test file is outside the
resolved test graphs. CI supplies that coverage.
- Staging evidence: read-only GET of the failed run's events and log.
- Not run: a live Daytona run. #21023 and #21039 are not deployed yet.

## Risks

- Low. Tests only. No production code changes.
- The runner test starts a real `paperclip-runnerd` and the fake Codex.
It kills both processes and removes its temporary directory in
`finally`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 01:08:18 -06:00
DottaandClaude Opus 5.5 148d2b629c fix(heartbeat): renew legacy controller leases on their own connection, timed from the statement (#21042)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A legacy run keeps a controller lease on its `heartbeat_runs` row.
The server renews every lease in the process with one batched statement
every 10 seconds, and stops a run when its local lease deadline passes.
> - The batched renewal ran on the 2-connection liveness pool. Native
runner lease renewals use the same pool. Each native renewal holds a
connection for a row lock, about seven notifications, and an update.
> - With 18 native runner runs renewing every 20 seconds on a slow
database, the legacy batch waited 30 to 50 seconds for a connection. The
local deadline also counted that wait, so it fired 20 to 40 seconds
before the database lease could expire.
> - Runs then stopped with `controller_lease_lost` while no other
controller owned them, and their issues looped through retries.
> - This pull request gives legacy renewal its own connection and
measures the deadline from the time the statement starts on that
connection.
> - The benefit is that a slow server no longer stops runs that still
own their lease. A real takeover still stops the run.

## Linked Issues or Issue Description

**What happened?**

On staging, after the 05:37Z deploy, 63 of about 175 new runs failed
with `controller_lease_lost` ("Legacy controller lease lost") at about 3
per minute. All were legacy runs in preparation
(`executionRecovery.kind: bootstrap`, `providerWorkStarted: false`). One
example had `controllerLeaseLoss: {cause: "renewal_deadline",
renewalCount: 7, sinceRenewalMs: 42537, renewalPendingMs: 38798,
deadlineLateMs: 134}`. Several runs that failed at the same moment had
identical timing values, so they were members of the same renewal batch.
The previous successful renewal took 17 to 44 seconds, so the deadline
that it set was only 16 to 43 seconds away. At that time 27 legacy runs
were preparing and 18 native runner runs were renewing on the same
liveness pool. The issues looped: `issue_assigned`, lease lost,
`issue_continuation_needed`, lease lost, `transient_failure_retry`,
lease lost.

**Expected behavior**

A slow server or database must not cause a lease loss while the
controller still owns the lease. Only a real expiry or a takeover by
another controller must stop the run.

**Steps to reproduce**

1. Start legacy runs and native runner runs on one server.
2. Make the database slow, so that each native lease renewal holds a
liveness connection for several seconds.
3. The legacy renewal batch waits for a liveness connection, and its
next local deadline is shortened by that wait.
4. The next renewal also waits, and the local deadline fires first. The
run fails with `controller_lease_lost` (`renewal_deadline`).

**Paperclip version or commit**

Staging at 8cefbe9977. The fix is based on master at aa94939988.

**Deployment mode**

Hosted cloud (staging).

**Agent adapter(s) involved**

Legacy controller runs (Codex and Claude local adapters). Native runner
runs share the liveness pool.

## What Changed

- `packages/db`: add `controllerLeaseDb(db)`, a private 1-connection
pool (`paperclip-legacy-controller-lease`) for legacy controller
renewal. `closeLivenessDb` also ends it, so shutdown does not change.
- `legacy-controller-lease.ts`: the batched renewal runs on
`controllerLeaseDb` instead of the liveness pool that native lease
renewals use.
- `legacy-controller-lease.ts`: each batch records the time when its
transaction starts on the connection. The watcher measures the next
deadline from that time, not from the time the request was queued. The
database sets the new expiry at UPDATE time, which is later, so the
local deadline never passes the database expiry.
- The renewal statement, its boot-id and expiry conditions, and
revocation do not change. A takeover or an expired lease still rejects
the renewal and stops the run.

## Verification

- Red/green regression tests in
`server/src/services/legacy-controller-lease.test.ts`:
- `renews without queueing behind native lease transactions that hold
every liveness connection` (embedded Postgres). Before the fix, the
renewal waited 9.8 s behind four 5-second liveness transactions. After
the fix, it completes in less than 2 s.
- `keeps a controller owned when every renewal waits thirty-five seconds
for a connection before its statement runs`. Before the fix, the lease
was lost with `{sinceRenewalMs: 25000, renewalPendingMs: 20000,
renewalCount: 1}`, the same pattern as on staging. After the fix, the
controller stays owned for four lease periods.
- Updated the private-pool tests to check the new pool and its
application name.
- `npx vitest run src/services/legacy-controller-lease.test.ts`: 40
passed.
- `pnpm verify` (affected tests), plus `typecheck` for `packages/db` and
`server`.

## Risks

- Each server process opens one more database connection, only when it
has a legacy run.
- The local deadline can now be later than before, by the time that the
batch waited for a connection. It stays at or before the database
expiry, because the database sets the expiry after the recorded start
time. Readers that act on an expired lease are not affected.
- Low risk otherwise. There is no schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20924, #20936

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:52:06 -06:00
DottaandClaude Opus 5.5 98c18784c6 fix(evals): grade full cohort 4 chat cases on saved state and rendered links (#21034)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real agents through chat journeys and
grades the saved results
> - Full cohort 4 (OpenCode with DeepSeek v4.1 flash, master f1aef0cb58)
failed 14 cases for eval reasons, not product reasons
> - Some phase captures read links before the reply rendered. Some
graders matched keywords in free-form replies. One locator was stale.
One fixture answer looked like a placeholder. Some checks could not say
what failed
> - This pull request grades those cases on saved state, setup literals
and rendered links, and it fixes the observers and fixtures
> - The benefit is that these cases fail only when the product fails,
and a failure names the part that broke

## Linked Issues or Issue Description

**What happened?**
In full chat cohort 4, valid agent behavior failed these cases:
case-history, goals.explain-purpose and workspaces.preview-service
(phase captures read before the reply rendered); routines.pause-owned
("If you want that one stopped too, use Pause on its task page" matched
a cancellation regex); budgets-costs.spend-breakdown ("unknown, not
$0.00" matched a stated-zero regex); steering.edit-queued-message
("change ... and drag it above" missed a keyword list);
connection-setup.choose-provider (the resumed page says "Finish with
Notion"); questions-and-approvals.necessary-question (the answer "Acorn
2a4f72ec4ae2-1" was saved as "Acorn"). Some checks could not say what
failed: onboarding.new-company compared the whole members response as
JSON and saved no after-state; agents.choose-primary threw one combined
error.

**Expected behavior**
Each case grades the feature contract on saved state, setup literals and
rendered or clicked links. A valid reworded reply passes. A broken
product still fails. A failure names the clause and its values.

**Steps to reproduce**
Grade the retained cohort 4 evidence for the cases above with the
earlier oracles.

Refs #21009, #21002, #21007

## What Changed

- Phase captures (case-history first phase, goals.explain-purpose
phases, preview-service query phase, ambiguous-task clarification,
edit-queued-message guidance) now wait for the saved reply to render
(`waitForTurnReplyLinks`) before they observe. When a saved reply yields
no rendered links, the observation records thread and anchor diagnostics
(`linkDiagnostics`) and logs them.
- routines.pause-owned: remove the reply keyword clause. Saved state at
the reply proves the run kept running, and the final state proves it
completed normally.
- budgets-costs.spend-breakdown: the unresolved record is graded
structurally. The reply must identify it (rendered link, identifier or
run id), and a chat run must have read its saved accounting (cost
summary, run record or unranged company report, bound by input digest).
"$0.00" in the reply is not matched.
- steering.edit-queued-message: grade the rendered task link in the
reply, the refused agent edit and reorder (403), and the unchanged queue
at the reply. Keyword lists are removed.
- connection-setup.choose-provider: the sign-in locator also accepts
"Finish with Notion" (the resumed-draft label). The button has no test
id or other stable name.
- questions-and-approvals.necessary-question: the answer is now
"Quillmere". The case also accepts the exact answer in a saved
deliverable (a document revision or text attachment on the linked task,
saved by an agent of the company after the answer). The yaml expected
outcome records this decision, with a `changedFields` entry in the
testops manifest.
- onboarding.new-company: save the members after-state and compare
identity, role, status and grants, and name what changed. Guidance is
graded on a rendered link to `/onboarding` (the new-organization
wizard), the business literal from the query, and the saved company
count. The source company and person get realistic names for this case
and are restored at cleanup, because the CEO reasoned from the generated
names that the request was a graded test.
- agents.choose-primary: the settled-Chat check is split into four named
clauses with values, saved as evidence on failure.
- A question card the turn ended on counts as its reply (decision after
cohort 4) in cli-operations.owned-dev-instance, onboarding.new-company
and status-cards.other-authors-card. The card's Markdown-rendered fields
(question form description; confirmation prompt and details) count as
reply links. Link requirements still apply.
- Every changed case bumps its `oracleVersion`; the descriptor snapshot
and inventory digest are updated.

Why each old check was wrong and why the new one still catches a broken
product:

| Case | Old check was wrong because | A broken product still fails
because |
|---|---|---|
| phase captures | runs succeeded and the chat waited before the reply
rendered, so links were empty | the same link checks run on the rendered
reply |
| routines.pause-owned | a correct offer to pause the run read as a
cancellation | the checkpoint run must be running and the run must later
succeed |
| spend-breakdown | "unknown, not $0.00" read as "$0" | the reply must
identify the record and a chat run must read its saved accounting;
totals and per-agent amounts stay graded |
| edit-queued-message | correct wording missed a keyword list | the
reply must link the task, the agent's edit and reorder must be refused,
and the queue must be unchanged at the reply |
| choose-provider | the resumed page label was not accepted | the button
must still be visible on the selected Notion surface |
| necessary-question | the answer read as a placeholder; deliverables
did not count | the exact answer must be saved in the linked work after
the answer, by an agent of the company |
| new-company | timestamps and profiles made the JSON compare fail
without saying why | membership identity, role, status and grants must
be unchanged |
| choose-primary | one combined error | the same four conditions, each
named |
| card as reply | a card answer read as an empty reply | card text must
exist and link requirements still apply |

## Verification

- Calibration tests (valid, reworded valid, broken) for each changed
grader: `integrator-routine-pause.test.ts`,
`identity-accounting.test.ts` (includes the cohort 4 reply),
`runtime-queue-controls.test.ts` (includes the cohort 4 reply),
`external-provider-choice.test.ts`, `identity.test.ts`,
`identity-onboarding-work.test.ts`,
`runtime-primary-ready-chat.test.ts`, `integrator-status-cards.test.ts`,
`turn-response-card.test.ts`, `chat-rendered-links.test.ts`.
- The corrected spend-breakdown grader passes on the retained cohort 4
evidence (its cost-summary read binds to the unresolved task) and fails
with the reads removed.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog chat-rendered chat-linked`: 284 files passed.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
passed.
- `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
unchanged.
- Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) on this branch
combined with the guidance PR: see the PR comment for results.

## Risks

- The new-company guidance check now needs a rendered link to
`/onboarding`. A reply that only names the company switcher fails. The
companion guidance PR tells agents this route.
- spend-breakdown does not grade wording, so a reply that reads the
record and still calls it "$0" is not caught. The amounts, total and
identification are still graded.
- The new-company case renames the source company and the person for the
case and restores them at cleanup.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1112.0-canary.5
2026-11-12 00:37:42 -06:00
DottaandClaude Opus 5.5 0dfbbdeb31 fix(chat): route company creation, decline retries and question cancellation correctly (#21035)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents follow the chat directive and discover API routes with
search_api and call_api
> - Full chat cohort 4 (OpenCode with DeepSeek v4.1 flash) showed four
guidance gaps
> - The missing-name rule overrode the board-only company-creation
boundary; a repeated request after a decline did not post a new approval
card; search_api showed the board-only question cancel route as open to
agents (and withdraw as board-only); write calls carried `responseText`
beside or instead of `body`
> - This pull request fixes the guidance, the route authorization
metadata and the pre-send refusal
> - The benefit is that agents take the supported path the first time
instead of asking for the wrong input, refusing a valid retry, or
calling a route that returns 403

## Linked Issues or Issue Description

**What happened?**
- onboarding.new-company: asked to "set up a new company for my website
business", the CEO cited the missing-name rule (#21007) and asked for a
business name with a question card. Creating a company is a board
action, so the agent should send the person to the creation flow.
- app-permissions.ask-before-action: the person declined a write card
and then repeated the same request. The CEO asked the person to tell it
to resubmit instead of posting a new approval card. The gateway already
allows a fresh request after a new human message
(`isNewHumanIntentAfterToolRejection`), but the decline text said "Do
not retry the same call".
- questions-and-approvals.multi-part-question: search_api listed `POST
/api/issues/{id}/interactions/{interactionId}/cancel` as
`board_or_agent`. The route rejects agents. The agent route,
`/withdraw`, was listed as board-only although its route admits the
card's creator agent and the task's assignee.
- Several cases: write calls through call_api carried `responseText`
with no body (for example "Received arguments: operationId, pathParams,
responseText").

**Expected behavior**
The agent links the company-creation flow; it posts a fresh approval
card for a repeated request; search_api shows the true authorization of
cancel and withdraw; a write with `responseText` is refused before it is
sent, with the same guidance #21002 gave `saveResponse`.

**Steps to reproduce**
Run the four chat cases above with OpenCode and DeepSeek v4.1 flash on
master.

Refs #21002, #21007

## What Changed

- `AGENT_CHAT_DIRECTIVE`: the missing-name rule applies to work the
agent creates itself and never comes before a board-only boundary. For a
new company the agent links [Create organization](/onboarding) (no
company prefix) and repeats the business details. A repeated request
after a declined approval card is an explicit retry: make the same call
so a fresh card is posted. A clarification question card puts candidate
links in its description, because labels show as plain text.
- Decline instructions in `tool-action-delivery.ts` and
`tool-gateway.ts` say that a later request from the person for the same
action is an explicit retry.
- OpenAPI: `POST /api/issues/{id}/interactions/{interactionId}/cancel`
is board-only and its summary names the `/withdraw` route for agents.
`/withdraw` is no longer marked board-only, which matches its route.
- `call_api`: a new write that carries `responseText` is refused before
it is sent; the message says `responseText` never carries request fields
and names the allowed body fields. Older saved calls still validate for
recovery. `doc/runner-api-tools.md` is updated.

## Verification

- `server/src/__tests__/agent-conversations.test.ts`: new directive
tests for the board-only boundary order, the decline retry and card
links.
- `server/src/__tests__/openapi-routes.test.ts`: cancel is `board`,
withdraw is `board_or_agent`, the cancel summary names withdraw.
- `server/src/services/native-runtime/runner-api.test.ts`: a write with
`responseText` is refused before sending (three window shapes), the same
write without it is sent once, and saved calls still validate.
- `tool-gateway.test.ts`, `tool-gateway-service.test.ts`,
`issue-thread-interaction-routes.test.ts`,
`tool-action-replay-intent.test.ts`: passed. `server` `tsc --noEmit`:
passed.
- Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) together with the
eval PR: see the PR comment for results.

## Risks

- A model that used to send a write with `responseText` and a valid body
now gets one refusal and must resend without it. Nothing is sent before
the refusal, so there is no duplicate side effect.
- The directive grows by a few sentences, which adds a small number of
prompt tokens per chat turn.
- Agents that read `/withdraw` as board-only before now see that they
can withdraw their own card; the route already enforced who may.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:27:35 -06:00