Commit Graph
7796 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 74096c19f4 fix(evals): admit the in-flight chat completion follow-up on a settled source (#21313)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that a chat request reaches the correct
saved outcome
> - The `source-settled-with-active-work` run policy checks that the CEO
chat is settled while delegated work can continue
> - After a delegated task finishes, the product starts its normal
`chat_task_completed` follow-up on the source chat
> - The policy required every source run to be `succeeded` and the
source to be `waiting`, so a re-grade during that follow-up failed
> - This pull request admits that one product follow-up while it is in
flight, and keeps every other source run strict
> - The benefit is that the eval grades the saved private-delegation
outcome, not the timing of the reload

## Linked Issues or Issue Description

**What happened?**
The OpenCode cell of
`chat-features.runner-opencode.local.tasks.private-task` failed with
"Independent oracle survives browser reload: expected true, received
false". The saved state was correct. The CEO answered, Morgan saved the
private draft in the private project, and the unrelated member could not
read it. The grade passed for its full 5-second stability window.
Morgan's run then finished, and 73 ms later the product started the CEO
`chat_task_completed` follow-up on the source chat. The harness reloaded
the page and re-graded while that follow-up was running. The only failed
check was `catalog-settled-source-separate-work`, because it required
all source runs to be `succeeded`.

**Expected behavior**
The case already counts this follow-up in its run bound
(`expectedRunCount: 3`, `minimumRunCount: 2`). A queued or running
product follow-up on an answered source must not fail the case.

**Steps to reproduce**
Run the private-task case. Let the worker finish within about 5 seconds
of the first passing grade. The post-reload re-grade sees the follow-up
as `running` and fails.

## What Changed

- Add `chatCatalogSourceCompletionFollowUp` to
`tests/runner-e2e/chat-catalog-run-policy.ts`. It identifies the lead's
`chat_task_completed` run on the source whose every completion update
names separate work in the run ledger.
- In the `source-settled-with-active-work` check, admit at most one such
follow-up while it is `queued`, `running` or `succeeded`. Accept a
source `conversationState` of `active` only while that follow-up is in
flight.
- Keep the other rules strict. The original answer must exist and must
have succeeded. Any other active, failed or extra source run still
fails. The declared run-count bound still applies.
- Also admit the follow-up before its turn is prepared. The server
claims the run first and saves the completion updates after that, so a
queued run, or a running run in preparation, has only its delivery IDs.
It qualifies only while queued or running, when automation woke it, and
after a separate worker run in the ledger succeeded.
- Add a red/green test that reproduces the observed race and covers the
negative cases.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-run-policy.test.ts`: the new test fails
before the change (1 failed, 13 passed) and passes after it (14 passed).
The queued-snapshot cases were added after review.
- The related runner-e2e tests (`chat-catalog-run-policy`,
`chat-catalog-readiness`, `chat-catalog-failure-retirement`,
`runtime-dependency`, `internal`) pass: 200 tests.
- `pnpm test:e2e:runner:typecheck` passes.
- Paid run
[43374419951](https://github.com/paperclipai/paperclip/actions/runs/43374419951)
on the first commit of this branch:
`chat-features.runner-opencode.local.tasks.private-task` passed.

## Risks

- Low risk. The change is in eval grading only. It does not change
product code.
- The check accepts a follow-up that is still running at grade time. A
later failure of that follow-up is not observed by this case. Completion
delivery is covered by its own completion cases.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:04:35 -06:00
Nicky LeachandClaude Fable 5.2 23c2343015 feat(server): read a retained provider pack from its Runner image (#21308)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner executes agent work in a sandbox. It needs a provider
pack of agent tools, and the server uploads that pack when the sandbox
does not have it
> - A retained session keeps the pack it started with. When that pack is
not the server's own, the server downloads the pack's transport archives
from a registry (#19779)
> - Today the server looks for them in the application image tagged
`sha-<commit that built the pack>`
> - `master` now publishes the Runner one time for each set of inputs,
as `ghcr.io/paperclipai/paperclip-runner`. A pack from that image names
the commit that published the image, and few commits get an application
image, so that tag often does not exist
> - This pull request reads such a pack from its Runner image, and keeps
the old source for every other pack and for a Runner image that cannot
be read
> - The benefit is that a retained session can still get its pack when
the pack came out of the published Runner image

## Linked Issues or Issue Description

Refs #19779 (the retained pack source), #21228 (the pack field
`runnerInputsDigest`), and #21204, #21229, #21230 (the Runner image). No
issue exists for this change.

**Problem or motivation**

`retained-provider-pack-oci.ts` reads
`ghcr.io/paperclipai/paperclip:sha-<runnerSourceRevision>`. It needs
that image to exist, to carry the pack, and to have one layer with a
known build-history line. For a pack from a published Runner image,
`runnerSourceRevision` is the commit that published the image.
`docker.yml` writes `sha-<commit>` only when its `master` run for that
commit completes, and a newer push can replace a waiting run. The
restart-chaos harness has the same assumption in another place: it
requires the pack of a sandbox image to be built by the commit in the
image's `org.opencontainers.image.revision` label.

**Proposed solution**

A pack that names its Runner image in `payload.runnerInputsDigest` is
read from `ghcr.io/paperclipai/paperclip-runner:inputs-<digest>`. A
sandbox image can name the commit that built its Runner in a label of
its own. No application image carries a pack with that field yet, and no
image carries that label, so behaviour today does not change.

**Alternatives considered**

- Keep the application image as the only source and require an
application image for every Runner publication. Publication would then
depend on `docker.yml`, which serves one ref at a time.
- Take a fixed number of layers ("the image has four, take the last
two"). A server in the field cannot be changed when the image gets
another layer. The server now uses the rule that holds for any number of
layers: the full image is the slim image plus transport layers.
- Fall back to the application image for every failure of the Runner
image. A Runner image that exists and does not match the pack is a
fault, and a second source would hide it.

**Roadmap alignment**

This is build and runtime infrastructure. It adds no product feature and
does not overlap an item in `ROADMAP.md`.

This is part 3 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image (this
pull request)
4. #21310: a registry build cache for the Runner image build

This pull request uses no code from parts 1 and 2. It is stacked on them
because all four parts edit `doc/runner-image-contract.md`.

## What Changed

- `server/src/services/native-runtime/retained-provider-pack-oci.ts`,
for a pack whose `payload.runnerInputsDigest` is 64 hexadecimal
characters:
- The server reads the full index `inputs-<digest>` of the Runner image.
The index must carry `io.paperclip.runner.inputs-digest` equal to the
pack's value, `org.opencontainers.image.revision` equal to the pack's
`runnerSourceRevision`, and a valid digest in
`io.paperclip.runner.slim-digest`.
- It reads the slim index by that digest, never by the slim tag, and
checks the bytes it gets against the digest.
- For the pack's platform, the layer digests of the slim image must be
the first layer digests of the full image, in the same order. The layers
after them are the transport layers. There must be at least one and at
most eight, and their total size has the existing limit.
- The server downloads only those layers. It checks each against its
descriptor and refuses every entry that is not a transport archive, the
receipt or the `provider-pack.archives` directory. A file may not be in
two layers.
- When the Runner image cannot be read, the server reads the application
image as before. That covers a registry error status for the token, for
either index or for either manifest: for example 404 for a tag or a slim
index that does not exist. Nothing is written before that point.
- A Runner image that can be read and does not match fails with
`runner_remote_provider_source_invalid`. The server does not try the
application image then.
- The same file, for every other pack: the old path, unchanged in what
it checks. Its code moved into functions that both paths use (`request`,
`pullToken`, `platformImage`, `receiveTransportLayer`).
- The downloaded layer file is removed as soon as it is unpacked.
Before, the importer removed it after its checks, so
`retained-provider-pack-source.ts` loses that one line.
- The cache record gets an optional `runnerImage` field with the inputs
digest and the digests of the transport layers. `transportLayerDigest`
is the last of them and `transportLayerBytes` their total. An older
server ignores the new field.
- `tests/restart-chaos/daytona-image.ts`: when the image has a non-empty
label `io.paperclip.runner.source-revision`, the pack must be built by
that commit. Otherwise the pack must be built by the commit in
`org.opencontainers.image.revision`, as before. The receipt field
`providerPackRunnerSourceRevision` records the pack's commit.
- `doc/runner-image-contract.md` has a new section, "Reading the
transport archives alone". `tests/restart-chaos/README.md` describes the
label.

One choice needs a reviewer's eye. When the full index can be read and
the slim index cannot (a registry error status), the server falls back
to the application image; it does not fail. Without the slim manifest
the server cannot tell which layers are transport layers, so it has not
accepted any byte of the Runner image, and the old source has its own
complete checks. A slim index that is read and has other bytes than its
digest still fails.

## Verification

Linux (`node:24.22.3-bookworm`, GNU tar 1.34, zstd 1.5.4, the worktree
mounted read-only):

- In `server/`: `retained-provider-pack-source.test.ts` (34 tests, 15 of
them new), `provider-pack-archive.test.ts`,
`remote-provider-pack-verification.test.ts`,
`remote-provider-pack.test.ts`: 127 tests pass in 4 files.
- `node node_modules/vitest/vitest.mjs run --config
tests/restart-chaos/vitest.config.ts
tests/restart-chaos/daytona-image.test.ts`: 25 tests pass, 5 of them
new.

macOS (Node 26.12.0):

- `daytona-image.test.ts`: 25 tests pass.
- `retained-provider-pack-source.test.ts`: the 15 new tests pass. Six
older tests fail, with and without this change. They run the real
uploader, which needs GNU tar, and this host has bsdtar.
- `tsc` on the two changed server files and the test, and on the two
restart-chaos files, with the settings of their test projects: no error.

The new cases:

- Read from the Runner image with 6 registry requests (two indexes, two
manifests, two layers), none to the application image and none for the
slim tag.
- Take every layer after the slim image's: one transport layer, and
three.
- Fall back when the tag does not exist, when the token is refused, and
when the slim index cannot be reached.
- Refuse, with no request to the application image: an index for other
inputs, an index for another revision, no slim annotation, a slim
annotation that is a tag, a slim index with other bytes than its digest,
a slim image that is not a prefix of the full image, a slim image equal
to the full image, nine transport layers, and a transport layer that
holds a pack file.
- Four deliberate faults in the reader each make a test fail: the slim
index bytes are not checked, no prefix check, the slim annotation is not
validated, and no bounds on the number of transport layers.

A real run, in the same Linux container (`linux/arm64`), with no
credential:

- The script took `provider-pack/provider-pack.json` of `linux/arm64`
from the published image
`ghcr.io/paperclipai/paperclip-runner:inputs-0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`.
Pack digest
`sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f`,
`runnerSourceRevision` `25e9a9c2e9277f3be8649be8c809cc81bde6f0c5`.
- `resolveProviderPackUploadSource`, with another pack as the server's
own, returned `{ archiveOnly: true }` after 23 seconds. That function
calls `receiveRetainedProviderPack` and then the importer's checks: the
manifest inside the overlay archive is the selected manifest, every
archive matches the receipt, and the content of every file archive
matches the digest of its file.
- The server code made 9 requests: one anonymous token request, the full
index, the `arm64` manifest, the slim index `sha256:f63ea436…4f3c` by
digest, its `arm64` manifest, and two layers, each with one redirect. No
request went to the application image.
- The cache record:

  ```json
  {
    "schema": "paperclip.retained-provider-pack-source.v1",
    "sourceRevision": "25e9a9c2e9277f3be8649be8c809cc81bde6f0c5",
"providerPackDigest":
"sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f",
"imageIndexDigest":
"sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162",
"imageManifestDigest":
"sha256:73dbe864fbab6208616f0aa6f1610a7ece4ef22f68a73d30901cbaea60139589",
"imageConfigDigest":
"sha256:740e48463d5caf0bfead4b923cdbd8b220acb93d058419c3db9d00c7f1831d65",
"transportLayerDigest":
"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf",
    "transportLayerBytes": 426602696,
    "runnerImage": {
"inputsDigest":
"0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca",
      "transportLayerDigests": [

"sha256:6a340088ca2cfa85f10f0385d9c8f5cdccb22c3a2bbb2062620b1d90616ba5b5",

"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf"
      ]
    },
"receiptSha256":
"2afdd26645845369d9e851b1e6e2884eb6c3884ea7c4866a1eeb2efb9fb230d7",
    "verifiedTarPaths": true
  }
  ```

- `uploadVerifiedProviderPack` then installed the pack from that source
through a local stand-in for the sandbox, with no further registry
request, in 4 seconds. The installed tree is 1,249,060,841 bytes. Its
`provider-pack.json` is the selected manifest, all 7 artifacts of the
manifest have the digest the manifest names, and the pack's own Node
prints `v24.22.3`.

Never run:

- The same real run for `linux/amd64`, whose transport layers are about
1 GB.
- A session in a real sandbox. The real run used a local stand-in for
the sandbox.
- The fallback against the real registry. Fixtures cover it.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The old path was moved into shared functions. Its checks are the same,
and its 19 existing tests pass on Linux, but a review must read that
code as changed.
- The layer file of the old path is now removed before the importer's
checks, not after them. The importer does not read it.
- The reader depends on two rules of the image contract: the full index
names its slim index, and the full image starts with the slim image's
layers. A Runner image that breaks either rule is refused, with no
fallback.
- The fallback covers a registry error status only. A network error ends
the import, as it does today.
- The label `io.paperclip.runner.source-revision` is new and no image
carries it yet. The change to the sandbox image that sets it must use
this name.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:02:19 -08:00
DottaandClaude Opus 5.5 0477f622d6 fix(recovery): keep the sandbox that holds abandoned unsaved files out of replacement and cleanup (#21304)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in remote sandboxes, and the workspace files live in
those sandboxes until a save
> - When a run cannot resume exactly, the fresh continuation fallback
(#21135) abandons the old workspace epoch and keeps its unsaved files
"recorded and unapplied" on the old sandbox
> - The abandonment clears the dirty flag, the writer and the location,
so the workspace looks clean. No code reads the abandonment receipt
> - A later failed resume can then replace the sandbox, and cleanup can
destroy it. That sandbox holds the only copy of the unsaved files
> - This pull request adds one shared hold. The task continues on a new
sandbox, and the old sandbox is parked until its files are saved,
discarded, or kept for 7 days
> - The benefit is that the fallback keeps its promise. Abandoned files
are not lost

## Linked Issues or Issue Description

Refs #21135
Refs #21297

**What happened?**
After the fresh continuation fallback abandoned a workspace epoch, the
receipt in `remoteWorkspaceAbandonedEpochs` was write-only.
`reusableSandboxReplacementIsSafe` saw a clean workspace with a
committed save and allowed replacement.
`retireReplacedReusableSandboxLease` then moved the old lease to
`pending_cleanup`, and the cleanup sweep destroyed the sandbox with the
unsaved files. `nativeWorkspaceExportPendingForLease` (the check that
the sweep, the retained orphan settlement and the scoped destroy paths
use) also answered "nothing pending".

**What did you expect?**
A sandbox that holds abandoned, unsaved files stays until the files are
saved, a person discards them, or a stated retention period ends.

**Steps to reproduce**
1. Abandon a remote workspace epoch whose location names sandbox L.
2. Let the next run fail to resume L.
3. The replacement decision says safe, L goes to `pending_cleanup`, and
the sweep destroys it.

## What Changed

- New `server/src/services/remote-workspace-abandoned-files-hold.ts`
holds the single predicate for "this lease holds abandoned files". It is
one SQL fragment (`abandonedWorkspaceFilesHeldSql`) plus a TypeScript
wrapper (`abandonedWorkspaceFilesHeldOnLease`). A lease is held while a
workspace in its company has an abandoned-epoch receipt whose
`location.providerLeaseId` equals the lease's provider lease id, and
that receipt has no `released` object. A receipt written by the current
release (no new fields) counts as unreleased.
- Replace and park. A failed resume of a held sandbox still lets the run
replace it, so the task continues at once on a new sandbox restored from
the last committed save. `retireReplacedReusableSandboxLease` parks the
old lease as `retained` with the `abandonedWorkspaceFilesParked` marker
(failure reason `abandoned_workspace_files_held`) instead of
`pending_cleanup`. `reusableLeaseCanBeResumed` and
`reusableLeaseCanBeCleanedUp` skip a parked lease, so later runs never
select it again. The check is a metadata read with no query, so the run
path does no extra work.
- Every path that can destroy a sandbox now honours the hold:
- `nativeWorkspaceExportPendingForLease` answers pending for a held
lease. This covers the pending cleanup sweep (defer), the orphaned
active lease sweep, the retained orphan settlement `heldBy`,
`destroyClaimedSandboxLease` (terminal workspace reaper
`destroyRemoteWorkspaceLease`, `destroyReusableSandboxLeases` and the
environment delete path, and run lease release with `destroy`), the
`destroyRunLease` dispatcher, and the plugin sandbox teardown retry.
- The run lease settlement (`environment_run_lease_active`) is the one
caller that ignores the hold (`{ destructive: false }`). It never
destroys a sandbox. It stops the sandbox and keeps it, or it moves a
lease without a sandbox to `pending_cleanup`, where the sweep parks a
held lease. If it waited for the hold, a dead run's lease would stay
`active` and block the retry for the whole retention period.
- `destroyReusableSandboxLease` parks the lease and returns without a
provider call. This covers obsolete reusable lease cleanup, the destroy
after an expired or mismatched resume, release with `expired`, and the
sandbox driver's `destroyRunLease`.
- The `retryPendingSandboxTeardown` dispatcher refuses a destructive
teardown of a held lease. Stop-only cleanup still runs, because a stop
keeps the filesystem.
- The retained orphan listing (`alwaysHeldSql`) uses the same SQL
fragment, so held orphans are not polled.
- Release. `releaseAbandonedWorkspaceFiles` marks a receipt `released`
with a reason (`recovered`, `discarded`, `retention_expired`) and an
actor. It moves the parked leases for that sandbox to `pending_cleanup`
when no other unreleased receipt holds them, and it writes a
`workspace.abandoned_files_released` activity entry on the task in plain
language. No export path for abandoned epochs exists today. The
`recovered` reason is the hook for one; this pull request adds no new
export mechanism.
- Retention expiry. New receipts carry `retainUntil` (7 days). An old
receipt uses `abandonedAt` plus 7 days. `nextEnvironmentCleanupAt`
includes the earliest retention end of any unreleased receipt. The
shared cleanup worker therefore wakes at that real deadline, and a
restart rebuilds it from the database. At the deadline the sweep runs
`expireAbandonedWorkspaceFileHolds`. This releases each due receipt as
an attributed system decision (actor `workspace-retention`) and logs the
held receipts and parked leases for each company. Each expiry always
writes a task entry: "Unsaved files from an interrupted run were kept
for N days, and their retention period has ended." The entry adds "They
are scheduled for removal." only when a parked lease was queued for
cleanup. A separate entry, "... kept for N days and have now been
removed.", is written only after the provider confirms the delete. Each
receipt is released in its own transaction. A failed receipt is logged
and skipped, and it is retried after a backoff (5 minutes, doubling, at
most 6 hours). The other receipts are still processed. The entries
reference a run or a task only if it still exists. If the task was
deleted, the decision goes to the server log only.
- Upgrade. A `pending_cleanup` lease (not in flight, not stop-only) that
an unreleased receipt holds goes back to `retained` and is parked by the
sweep. A sandbox that a release before this change already destroyed
cannot be recovered. Its receipt ends at its retention time like any
other receipt.
- Receipt trim (review P1). `abandonUnverifiedRemoteWorkspaceEpoch`
trims only released receipts and keeps the last 20 of them. It never
drops an unreleased receipt. A new abandonment of the same sandbox and
root replaces the unreleased receipt for that root. So a workspace holds
at most one unreleased receipt for each distinct sandbox and root
abandoned in the last 7 days.

## Verification

- `server/src/__tests__/abandoned-workspace-files-hold.test.ts` (19
tests). It follows the lease through the real acquisition path with a
failed resume, the real `abandonUnverifiedRemoteWorkspaceEpoch`, the
real `sweepPendingCleanupLeases`, and `settleRetainedOrphanSandbox`:
- `gives the next run a new sandbox at once and parks the held one,
which cleanup and later runs leave alone`
- `pulls a held lease already queued for cleanup back to retained, and
leaves it on the provider`
- `treats a receipt written before release fields existed as unreleased`
  - `records a retention expiry on the receipt`
- `hands the parked lease to normal cleanup once the files are
recovered` / `... discarded`
- `removes the files only after the retention period, and says removed
only after the provider confirms it`
  - `does not report removal when the provider delete fails`
- `records the end of retention without scheduling removal when another
receipt still holds the sandbox`
- `records the end of retention when it expires before the lease was
ever parked`
- `wakes the cleanup worker at the receipt's retention end, also after a
restart, and then goes quiet`
- `releases a receipt whose run was deleted and records it without the
run reference`
- `releases a receipt whose task was deleted without writing a task
entry`
- `keeps expiring later receipts when one fails, and retries the failed
one after a backoff`
- `keeps a held sandbox when the retained orphan settlement reviews it`
- `lists the sandbox for the retained orphan settlement again once the
receipt is released`
- `keeps holding the first sandbox after many later abandonments on
another sandbox`
- `replaces and cleans the sandbox as before when the receipt names a
different sandbox`
- `replaces and cleans the sandbox as before when no epoch was
abandoned`
- Red on the base revision (62d673cc3f): the park, upgrade, old-receipt,
expiry, many-abandonments and scheduling tests fail. The controls pass.
- Green: 19 of 19. Also
`environment-runtime-reusable-resume-replacement.test.ts` 17/17,
`environment-retained-orphan-sandbox.test.ts` 32/32,
`heartbeat-pending-cleanup-sweep.test.ts` 51/51,
`fresh-continuation-fallback.test.ts` 58/58.
- `tsc --noEmit -p server/tsconfig.json` and a tsc pass over the new
test file: clean. `environment-cleanup-work.test.ts` +
`heartbeat-pending-cleanup-sweep.test.ts`: 80/80.

- No existing test expectation was changed. Three CI failures on bb43f52
came from this change. Two came from the hold blocking the run lease
settlement: `ownership-reconciliation-report.test.ts > releases the
lease and frees the workspace for the retry once a preparation-only
epoch ends` and `remote-workspace-save-concurrency.test.ts > lets a
retry prepare the workspace again after its first writer was killed
during preparation`. The third, `native-session-executor.test.ts >
restores a warm backup into a created lease only when a settled restore
hold retained the stopped source`, came from a mocked database that has
no `execute`; the predicate now runs through `select`. All three pass
locally now. So do `heartbeat-orphaned-active-lease-sweep`,
`environment-run-lease-ownership`, `native-workspace-warm-reuse` and
`remote-workspaces.integration`.

## Risks

- Cost of a parked sandbox. Parking makes no provider call and changes
only the lease row. A sandbox that could not be resumed is not running.
A reusable sandbox that was released is already stopped and kept. In
both cases the cost is stopped storage for up to 7 days. One case is
different: if a destructive path finds a held warm sandbox that is still
running (`retained` after a keep-running release), the sandbox stays
running until the receipt is released. The expiry log line counts parked
leases for each company, so this cost can be seen.
- Run lease release for an `ephemeral` lease calls the provider's
release. Some providers delete the sandbox on that call. A held sandbox
normally has a reusable lease, so this pull request does not gate that
call.
- No schema change. The receipt is JSON in
`execution_workspaces.metadata`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:00:20 -06:00
Nicky LeachandClaude Fable 5.2 e363b0e52d fix(docker): keep local build output out of the Runner image's build context (#21312)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as the image `ghcr.io/paperclipai/paperclip-runner`, one time
for each set of Runner inputs (#21204, #21229, #21230)
> - The name of a published image is a digest of every file the Runner
build receives from the build context
> - `.dockerignore` keeps the Runner's own build output
(`packages/paperclip-runner/dist`) out of the context, but not the eval
kernel's (`packages/paperclip-eval-kernel/dist`), which the Runner's
build also writes
> - A tree where the Runner was built therefore has another inputs
digest than a clean checkout of the same commit
> - The Runner package's Vite builds write five more directories with
the same problem
> - This pull request adds the missing ignore rules, and a test that
finds the next such directory
> - The benefit is that a developer's tree and a clean checkout of the
same commit name the same Runner image

## Linked Issues or Issue Description

Refs #21204, which added the image identity. No issue exists for this
change.

**Problem or motivation**

The inputs digest covers every file the build receives under
`packages/paperclip-eval-kernel`. The Runner's build compiles the eval
kernel and writes `packages/paperclip-eval-kernel/dist`. After a local
Runner build, that directory is in the build context, so the digest of
the tree differs from the digest of a clean checkout. A lookup of the
published Runner image for such a tree finds nothing.

CI checkouts are clean, so no published image is affected.

**Proposed solution**

Add `packages/paperclip-eval-kernel/dist` and
`packages/paperclip-runner/dist-*` to `.dockerignore`, next to the rule
for the Runner's own `dist`. Add a test that derives every build-output
directory inside the image's inputs and requires a rule for each.

**Alternatives considered**

- Exclude the directory in the identity script only. Then the digest
would not describe what the build receives, which is the property the
identity is built on.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This pull request stands alone. It is based on `master` and needs no
other open pull request.

## What Changed

- `.dockerignore`: two rules.
- `packages/paperclip-eval-kernel/dist`: the Runner's own build writes
it.
- `packages/paperclip-runner/dist-*`: the output of the Runner package's
Vite builds (`dist-browser`, `dist-issue-thread`, `dist-scenarios`,
`dist-sdk`, `dist-standalone`). The Runner's `build` script does not
write them, but a tree where one of those builds ran had the same digest
drift. No tracked path matches the rule, and no later `!` rule
re-includes those directories.
- `server/src/__tests__/runner-image-identity.test.ts`: one new test.
For each package among the identity's inputs it collects the `outDir` of
every tsconfig a script compiles with `tsc -p`, the `target` directory
beside every Cargo manifest a script names, and the `outDir` of every
Vite configuration. It requires that the last `.dockerignore` rule
matching each directory is an exclusion. Today that is eight
directories. A new output directory inside the inputs fails the test
until it has a rule.
- No image build reads that directory from the host. The Runner stages,
the `build` stage of the application image and `docker/daytona-runner`
all compile the kernel themselves (`tsc -p`, not incremental), and the
eval kernel is a development dependency of the Runner only.

## Verification

The inputs digest, as the `runner-artifacts-identity` build target
reports it, on a `linux/arm64` Docker host:

| Tree | Without the rule | With the rule |
| --- | --- | --- |
| clean export at `62d673cc3f` | `0e6662ad…` | `0e6662ad…` |
| the same, with an eval-kernel `dist` | `e4cf3095…` | `0e6662ad…` |
| clean export at `bc71c23e0a` | `88fe2929…` | `88fe2929…` |
| the same, with an eval-kernel `dist` | `ac331275…` | `88fe2929…` |
| clean export at `731b52453c` | `88fe2929…` | `88fe2929…` |
| the same, with two files in `dist-browser` | `44fe867d…` | `88fe2929…`
|
| the same, with one file in each of the five Vite directories | not
measured | `88fe2929…` |

- `0e6662ad…` is the digest of the image that `master` published at
`25e9a9c2e9`, so the rule does not change the digest of a clean
checkout.
- A worktree after a real `pnpm --filter
@paperclipai/paperclip-eval-kernel build` also gave the clean digest
with the rule.
- `runner-image-identity.test.ts` (42 tests) and
`tests/runner-e2e/daytona-image.test.ts` (19 tests), which reads
`.dockerignore`, pass on macOS and in `node:24.22.3-bookworm`.
- The new test fails on the `.dockerignore` of `master` and names
`packages/paperclip-eval-kernel/dist`. It fails without the `dist-*`
rule and names `packages/paperclip-runner/dist-browser`. It fails
without either of the two older Runner rules, and with a trailing `!`
rule that re-includes one of the directories.

Never run:

- Any image build with the new rules beyond the identity target,
including `.github/docker-context-checks.Dockerfile`. On a CI checkout
the directory does not exist, so the rules remove nothing there.
- `linux/amd64`.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- **The sandbox image's content id changes one time.** `.dockerignore`
is an input of that id (`tests/runner-e2e/daytona-image-content.ts`). At
`62d673cc3f` the id goes from `be208823…` to `1e13a151…`. The next
end-to-end run builds a new sandbox image, and the restart-chaos harness
needs an image built from this tree. The image's contents do not change.
- Low otherwise: the rules remove directories that no image build reads.
The `Dockerfile`, `docker/daytona-runner/Dockerfile` and
`.github/docker-context-checks.Dockerfile` name none of them and run no
Vite build.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:55:23 -08:00
DottaandClaude Opus 5.5 bbafa78fe2 fix(retry): offer Retry only when admission can take it, and start held saved-work Retries (#21315)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run fails and its saved work is held for recovery, the task
shows a Retry button. Admission then decides if the Retry can start a
new turn.
> - The task view, the task's execution blocker, and admission each
computed their own answer. The task view said `canRetry: true`. The
blocker said `canRetry: false`, and its own `workspaceRecovery` part
said `canRetry: true`.
> - Admission refused these Retries with no reason. The receipt kept the
generic `execution_recovery` wait ("Retry is waiting for execution
checks"), and each later click joined the same held receipt. Nothing
ran.
> - The refusals had real causes. A preparation proof rejected
host-written workspace notices in the run log. A repair proof did not
accept a snapshot that the failed preparation only restored and never
changed.
> - This pull request fixes both proofs, names every admission refusal
of a held Retry, and makes the task view and the blocker read one shared
Retry answer.
> - The benefit is that Retry starts work in these cases, and Retry is
offered only when admission can take it.

## Linked Issues or Issue Description

**What happened?**

On hosted staging, several blocked tasks showed a Retry button. Pressing
Retry was accepted but nothing ran. The task said "Retry is waiting for
execution checks. Your message is saved." The task view had
`interruption.canRetry: true` and `retryWait.reason:
execution_recovery`. The task's `executionBlocker` had `canRetry:
false`.

- Saved-work recovery failed with `source_unavailable`. The legacy
preparation had failed before the provider started
(`executionRecovery.providerWorkStarted: false`), its sandbox epoch was
abandoned, and its sandbox was destroyed. The blocker contradicted
itself: `workspaceRecovery.canRetry: true`, top-level `canRetry: false`.
- Saved work showed "recovered", but `workspaceRepairRequired` stayed
`true`. The failed preparation had restored a snapshot that an earlier
run produced. The repair proof accepted only a snapshot that the run
itself saved.

**Expected behavior**

The task view, the execution blocker, and admission give the same Retry
answer. A Retry that admission can take starts work. When saved work
cannot be recovered, Retry continues in a fresh session from the
recorded work. Every admission refusal of a held Retry has a named
reason. No Retry is offered for a refusal that another click cannot
change.

**Steps to reproduce**

1. Let a legacy `paperclip_runner` preparation fail before the provider
starts, after the host writes its workspace notices ("Execution
workspace reuse freshness action ...", "Skipping saved session resume
...") to the run log.
2. Let saved-work recovery end with `source_unavailable` and
`use_alternative_strategy`.
3. Press Retry.
4. Observe that the Retry receipt is held with `executionWait.reason =
execution_recovery` and no run starts.

Or:

1. Let a legacy preparation restore the task's saved snapshot and fail
with `setup_failed` before the provider starts.
2. Let saved-work recovery complete.
3. Observe that the blocker keeps `workspaceRepairRequired: true`, and
that Retry is held with `execution_recovery`.

**Paperclip version or commit**

Staging on `origin/master` `62d673cc3f` (after #21267 and #21295).

**Deployment mode**

Hosted cloud (staging), remote Daytona sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (legacy runtime runs, native agent).

Refs #21267, #21295.

## What Changed

- `hasVerifiedPreparationLog` (`verified-preparation-log.ts`) now
accepts a log made only of host notices that the host writes before it
hands the turn to an adapter: the mentioned-skills line (first record
only, checked against the saved skill selection) and the workspace
freshness and session-reset notices. Every record must have the same
attempt, sequential `seq`, `stdout`, and the excerpt and watermark must
match. Any other line still keeps the run held.
- `verifiedUnchangedRestoredSnapshot`
(`remote-workspace-snapshot-reuse-receipt.ts`) proves that a failed
preparation's saved state is the snapshot it restored. It needs
`executionStage: preparing`, `providerWorkStarted: false`, exactly one
succeeded restore of that snapshot at that epoch and lease, and no
snapshot save by the run. `verifiedWorkspaceRepairReceipt` uses it next
to the existing reuse proof.
- `savedWorkOffersRetry` (`workspace-recovery-receipt.ts`) is the one
rule for Retry on a saved-work hold. Retry is offered when the saved
work is recovered and verified, or when recovery is exhausted in a state
that only a Retry restarts (a new recovery generation, or the
fresh-session fallback after `source_unavailable`). Retry is not offered
while recovery re-arms by itself (a stop recheck or a newer capture
policy), or while a copyback owner's stop is unconfirmed. The blocker
uses this rule for the top-level `canRetry` and for
`workspaceRecovery.canRetry`, so the two parts cannot contradict.
- `task-retry-eligibility.ts` lists the held-Retry refusals that another
click cannot change. The task view and the blocker both use it.
- Admission (`explicit-native-continuation.ts`) names each refusal of a
held request that was silent before: `previous_execution_unverified`,
`execution_settling`, `cleanup_pending`, and `remote_cleanup`. It names
a refusal only when an execution blocker holds the task, so a request
without a hold still takes the normal path.
- The task view now reads the blocker's answer for a task with a
saved-work hold. It shows no Retry when the blocker offers none, and it
names the recovery wait.

## Verification

- Red/green: `cd server && npx vitest run
src/services/saved-work-retry-eligibility.test.ts`. The tests fail on
`origin/master` sources and pass with this change. They are built from
the staging shapes:
- offers the same Retry in the task view and the blocker when saved work
is no longer available
- offers no Retry in either view while saved-work recovery waits for an
unconfirmed file copy to stop
- continues in a fresh session from the recorded work when saved work is
no longer available
- verifies the repair of a restored snapshot that the failed preparation
never changed
- keeps the repair unverified when the failed preparation may have
changed the restored snapshot
- names a refusal that another click cannot change and offers no Retry
for it in either view
- names the admission refusal of a held Retry and offers Retry only when
admission would take it
- `npx vitest run src/services/verified-preparation-log.test.ts`: new
cases accept the host notices and keep a provider line, a different
attempt, or a late skills line held.
- Related suites: deferred execution wait, execution blocker read,
explicit native continuation, governed wait runner loss, legacy
preparation retry hold, remote workspace snapshot reuse, task
interruption presentation, task retry, and all `workspace-recovery-*`
suites.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Medium-low. The preparation-log proof accepts more host lines. Each
accepted line has a fixed host format, and the run must still pass every
other preparation proof (events, leases, no provider evidence).
- The restored-snapshot proof applies only when the run recorded that
the provider never started and it never saved its own snapshot.
- Admission now records a named wait instead of the generic one for some
held requests. It changes no admission decision.
- The task view calls `getExecutionBlocker` for tasks that offer Retry
and have a saved-work hold. These tasks are few, so the extra reads are
small.
- Not covered here: a copyback owner on a replaced deployment still
needs its stop confirmed before saved work can be copied out of a full
sandbox. The task view now shows that wait instead of a Retry that
admission refuses.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:52:25 -06:00
DottaandClaude Opus 5.5 b555087f0f fix(native): continue again when a deploy interrupts an automatic continuation (#21314)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner (Daytona) that the
server controls; a deploy restarts the server while runs are in flight
> - When the restart's reattachment proves the Runner process gone
mid-turn, #21039 makes the server request one automatic Retry that
continues in a fresh session from the saved work, without replaying the
turn
> - Staging deploys every 20 to 45 minutes, and Codex runs often take
longer, so that automatic continuation is often interrupted by the next
deploy too
> - The hook treated that second loss as "automatic Retry already used"
and left the task blocked for a person
(`native_continuation_requires_reconciliation`)
> - This pull request gives an automatic continuation one more
fresh-session continuation each time a server restart or deploy took its
controller, and a deploy spends no automatic budget
> - The benefit is that a task interrupted by repeated deploys continues
by itself, and no provider turn that may have run is sent again

## Linked Issues or Issue Description

Refs #21039, Refs #21052, Refs #21272, Refs #21023, Refs #21120, Refs
#21161

**What happened?**

On staging, the largest failure class for a Codex agent on Daytona was
`runner_remote_process_identity_unavailable` with error code
`native_continuation_requires_reconciliation`: 18 runs in 3 hours. The
error name is misleading. It is the shared code for every refused remote
reattachment (`REMOTE_RUNNER_RECOVERY_ERROR_CODE`). In every sampled run
the refusal was `process_verification` / `process_not_running`: the
Runner process was gone, its stderr log was empty, and the last turn
event was `turn.submitted` or `turn.accepted` (unsettled). It was not a
missing identity file.

Sampled 12 such runs. In 9 the #21039 hook worked: it recorded
`native.interrupted_turn_outcome_unknown` and requested the
fresh-session Retry. The tasks that stayed blocked had one shape:

1. A run lost its Runner after a deploy. The hook requested the
automatic Retry (key `task-retry:restart-interrupted-turn:<run>`).
2. The Retry run started a fresh session. The next deploy came during
its turn (a graceful `SIGTERM` shutdown transition, then "Recovering the
existing native runner process after server restart").
3. The same-run reattachment found that Runner gone too
(`process_not_running`).
4. The hook returned `automatic_retry_used`, because the run's Retry
intent came from the hook. The task stayed blocked.

Examples: a `task_retry` run created 40 s after its predecessor failed
(the hook's Retry), interrupted by the 05:17 deploy, failed at 05:18,
task blocked. A second chain (assignment, then the hook's Retry) failed
the same way at 05:12.

**Expected behavior**

A deploy is not the task's failure. When a server restart or deploy took
the controller of an automatic continuation and its Runner is proven
gone, the server continues the task again in a fresh session from the
saved work. It never replays the turn. A loss without restart evidence
keeps the existing one-time rule.

**Steps to reproduce**

1. Start a long Codex run in a remote Daytona Runner.
2. Deploy the server during its provider turn so that the reattachment
finds the Runner gone. The server requests the automatic Retry.
3. Deploy again during the Retry run's turn so that its Runner is also
lost.
4. See the Retry run fail with
`runner_remote_process_identity_unavailable` and the task stay blocked
with no new run.

**Paperclip version or commit**

Staging Core `25e9a9c2e9`. Fixed on `master` after `731b52453c`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `restart-interrupted-turn.ts`: `lostControllerToRestart` finds restart
evidence on the run: a `native.recovery.transition` with `restartKind:
"graceful"` (shutdown), or a restart claim (`reattach_remote_runner`,
`reattach_existing_runner`, `resume_dead_runner`,
`bootstrap_incomplete`) without the `same_run_retry` trigger.
- Both `automatic_retry_used` refusals (a person's Retry that the hook
requested, and a system retry of owed work) now continue when the run
has restart evidence. A deploy interruption spends no budget, so a task
that only deploys interrupt never runs out. Each continuation needs a
new deploy during its run, so deploys bound it. A loss without restart
evidence keeps the one-time rule. All other gates are unchanged: proven
Runner exit, unknown-turn record, the issue still assigned and open,
idempotency per run, and the Retry's admission gates.
- `doc/execution-semantics.md` describes the rule.

## Verification

- Red before the fix, green after:
`server/src/services/runner-lost-mid-turn-continuation.test.ts` › "an
automatic continuation that the next deploy interrupted". The test seeds
the staging sequence: a lost run, the hook's Retry run with a
`turn.submitted` turn, the shutdown and restart transitions with staging
payloads, then a same-run reattachment that proves the Runner gone. On
the old code the two continuation tests return `automatic_retry_used`.
With the fix they request one fresh-session Retry, record the unknown
turn outcome once, recover the saved work and queue the successor with
`forceFreshSession: true`.
- A chain of 12 deploy continuations, each with a system retry that
inherits its Retry receipt, still continues (red on the first revision
of this PR, which had a budget of 8). Losses without deploy evidence
stay refused (with and without earlier deploy continuations).
- `npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts` 23/23 pass.
`governed-wait-runner-loss.test.ts` and
`workspace-recovery-retry-copyback.test.ts` 103/103 pass.
- `pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify` exits 2 (a global doc input changed), so required CI
supplies the broad coverage.

## Risks

- Low to medium. A task whose Runner is lost on every deploy now
continues after each deploy instead of once. Each continuation is a full
run, so it costs tokens. The deploy count bounds it, because each
continuation needs a new restart during its run.
- Not fixed here: why the reattached Runner is gone after a deploy. In
the sampled runs the reattachment after the restart hung in `native
session provider recovery timed out after 900000ms`, and the next
attempt found the process gone with empty stderr. That needs its own
investigation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, browser read-only API GETs on staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:41:32 -06:00
Nicky LeachandClaude Fable 5.2 03f9b83e52 feat(ci): report the Runner image on master and build the app image's Runner path on pull requests (#21307)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work, and `master` now
publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`
> - The application image will soon copy that Runner instead of
compiling its own. Part 1 of this stack adds the resolver that picks the
Runner for a build
> - Two gaps make that next change risky. The resolver has never run in
CI on `master`, so its lookup time and how often a commit has a
published Runner are unknown. And `docker.yml` has no pull-request
trigger, so a change to how the application image gets its Runner first
runs after it merged
> - This pull request adds a report-only job that runs the resolver on
every `master` push, and a pull-request check that builds the Runner
path of the application image
> - The benefit is real data from `master` before any build depends on
the resolver, and a failed check on the pull request instead of a failed
image build on `master`

## Linked Issues or Issue Description

Refs #21204, #21229 and #21230 (the Runner image). No issue exists for
this change.

**Problem or motivation**

The resolver decides whether a build copies the published Runner. On
`master`, a commit that changes the Runner's inputs is ahead of its
publication for some minutes, and nobody knows yet how often an
application build will meet that. The application image itself is built
only on `master` and on tags. The change that removes the Runner's
compile stages from that image will edit the `Dockerfile`, and today
nothing builds those stages on a pull request.

**Proposed solution**

Add one job to `docker.yml` that only reports, and one new workflow that
only checks. Neither changes what an image contains, and no existing job
reads either of them.

**Alternatives considered**

- Run the resolver inside the build legs. A failure there could fail the
image build. A job of its own, with `continue-on-error`, cannot.
- Give the lookup the job token. The Runner package is public, so an
anonymous lookup works and the job needs no credential.
- Build `production` in the pull-request check. It adds three `COPY`
instructions and a readability check to the two stages that this check
builds, at the cost of the tool layer and two more copies of several
gigabytes.
- Trigger the check on every Runner source change. `runner-image.yml`
already builds the Runner on those pull requests, and CI builds the
server and the UI.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 2 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path (this pull request)
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build

## What Changed

- `.github/workflows/docker.yml`: new job `runner_image_report`.
  - It runs only for a `master` push in this repository.
- It runs `runner-image-resolve.mjs inputs`, then `resolve`, and the
result goes to the job summary with the line "Report only: no build read
this result."
- It has `continue-on-error: true`, declares no `outputs`, and no job
needs it. Its failure does not fail the run.
- It holds no credential. Its only permission is `contents: read`, the
checkout does not keep its token, and the lookup is anonymous. It does
not log in to a registry and writes to none.
- Time limits: 8 minutes for the job, 5 for the step that builds, 2 for
the lookup.
- It is the first job in the file. Existing tests cut the other jobs out
of the file by name, up to the next job or to the end of the file, so no
other place leaves those tests unchanged.
- `.github/workflows/docker-app-check.yml` (new): on a pull request,
builds the `build` and `runner-provider-pack` targets with
`PAPERCLIP_BUILD_COMMIT` set, for `linux/amd64` only.
- `permissions: {}` at the top, `contents: read` for the job, no
credential, no `pull_request_target`.
- It exports nothing (`--output type=cacheonly`) and pushes nothing. It
reads the public build cache that `docker.yml` writes and never writes
it.
- It refreshes the lockfile in the build context first, as `docker.yml`
does.
  - Time limits: 60 minutes for the job, 45 for the two builds.
  - Trigger paths, and why:

    | Path | Reason |
    | --- | --- |
    | `.github/workflows/docker-app-check.yml` | the check itself |
| `Dockerfile`, `.dockerignore` | the recipe, and the files it receives
|
| `server/scripts/prepare-runner-vendor.mjs` | the step of `build` that
reuses or rebuilds the Runner |
| `packages/paperclip-runner/scripts/build-inputs.mjs` | the build stamp
that step compares with the files the image build received |

- `server/src/__tests__/docker-runner-workflows.test.ts` (new, 90
lines): only security properties. The report job has no credential and
one read permission, nothing can read it, and its two steps have time
limits. The check has no credential, cannot publish, and exports
nothing. In both, every action is pinned by commit and no shell script
holds an expression.
- `doc/runner-image-contract.md` and `doc/DOCKER.md` name the job and
the check.

## Verification

macOS (Node 26.12.0):

- In `server/`: `docker-runner-workflows`, `runner-image-workflow`,
`docker-build-stamp`, `container-init-reaping`,
`cloud-image-bundled-plugins`, `docker-image-layer-reuse` and
`cloud-image-sentry` tests: 52 tests pass in 7 files.
- `tests/runner-e2e` workflow tests (`workflow-security`,
`private-trust-workflow-security`, `workflow-image-only`,
`paid-runner-routing`): 27 tests pass in 4 files.
- `node --test .github/scripts/tests/*.test.mjs
scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: 962 of 962 pass.
- Both workflow files parse as YAML.
- `docker buildx build --target runner-artifacts-context --cache-from
type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-amd64
--output type=cacheonly .` succeeds, so both flags are accepted and the
cache is readable.

Linux (`node:24.22.3-bookworm`, with `zstd` and `jq` added): the same
three sets pass: 52, 27 and 962 of 962.

The two commands the report job runs were run by hand, with no
credential, on clean exports of the tree. For the inputs `0e6662ad…d1ca`
the answer was `published`, with the two index digests that `master`
published. For a commit whose image was not yet published the answer was
`inline`, `absent`. Part 1 has the output.

With the job at the end of `docker.yml`, two existing tests failed. They
are the reason for its place: `docker-canary-promotion.test.mjs` reads
the last job to the end of the file, and
`release-verify-workflow.test.mjs` allows no BuildKit configuration
after `merge-and-push`.

Never run:

- Either workflow. Only GitHub can run them. This pull request starts
the check by its own path; the report job first runs on `master`.
- The two builds of the check (`--target build`, `--target
runner-provider-pack`) on a local host. Their time and disk use on a
hosted runner are not known.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The check builds the whole `build` stage, with the UI and server
builds. Its duration is not measured. It runs only when one of five
paths changes, and its first real run is on this pull request.
- `docker.yml` holds one concurrency slot for each ref until all its
jobs end. The report job normally ends long before the build legs. A
stalled job can hold the slot for at most 8 minutes when the legs end
sooner.
- `scripts/watch-release-source.mjs` warns about every job of the image
run that waited more than 60 seconds for a runner. The report job is one
more job that it can name.
- `scripts/merge-group-image.mjs` refuses to promote a merge-queue image
for a push that changes `docker.yml`. The push that lands this change is
such a push. The merge queue is not enabled.
- The lookup has 2 minutes. The registry client retries a failing
request for longer than that, so a registry outage ends the step by its
time limit, and the job then reports nothing. The run stays green.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:29:33 -08:00
Nicky LeachandClaude Fable 5.2 ff91bfe2dc fix(runner): withhold Claude's self-scheduling tools from Paperclip ACPX sessions (#21276)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip owns scheduling. An agent uses Paperclip tools such as
task monitors and routines to continue work later.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude Code offers its own scheduling tools to the model:
`CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`. With these
tools the provider can wake itself. Paperclip does not see or govern
that work.
> - Today the Runner stops the Claude process when a turn ends, so these
tools have little effect. A later change keeps the Claude process alive
between turns.
> - This pull request removes the four tools from Claude sessions that
Paperclip starts through ACPX.
> - The benefit is that a Claude agent schedules work only through
Paperclip.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #19912 (the design for turn-scoped ACPX MCP authority).

**What existing behavior does this improve?**

The set of tools that Claude Code offers the model in a Paperclip ACPX
session.

**Subsystem affected**

Paperclip Runner, ACPX provider (Claude).

**Current behavior**

A Paperclip ACPX Claude session offers the model 26 built-in tools. Four
of them are `CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`.

**Proposed behavior**

- The Claude settings that Paperclip writes into the session home deny
the four tools. Claude Code then does not offer them.
- Each Claude launch sets `CLAUDE_CODE_DISABLE_CRON=1`. This stops the
cron scheduler of Claude Code and removes the three cron tools.
- All other tools stay. `SendMessage`, `ListAgents`, `Workflow`, `Agent`
and `TaskStop` do not change.

**Reason and benefit**

The provider cannot schedule its own wake-ups. Paperclip tools stay the
only way to schedule work.

## What Changed

- `packages/paperclip-runner/src/drivers/acpx/permission-policy.ts`: new
constant `CLAUDE_SELF_SCHEDULING_TOOLS` with the four tool names.
- `packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.ts`: the
Claude `settings.json` now always has `permissions.deny` with the four
tools, in each permission mode. The `allow` rules do not change.
- `packages/paperclip-runner/src/drivers/acpx/environment.ts`:
`createSanitizedAcpxSpawnInput` sets `CLAUDE_CODE_DISABLE_CRON=1` for
each Claude launch. The host cannot override it.
- New test `claude-scheduling-tools.claude-acp.test.ts`. It prepares the
session home with the production `prepareAcpxRuntimeSandbox`. It then
runs the pinned `claude-agent-acp` and the pinned Claude Code client
against a local fake Anthropic API. It reads the tools that the client
offers the model.
- Unit tests in `environment.test.ts` and `runtime-sandbox.test.ts`. One
assertion in `runtime-host.test.ts` now expects the `deny` list.
- No change to the `claude-agent-acp` patch, the lockfile or a pinned
digest. No change to Codex.

## State saved by the previous release

- A session that the previous release saved continues with no change.
The new restriction is not part of a saved identity or digest.
- Each time Paperclip starts Claude for a session, it writes its own
`settings.json` again with the deny list and sets
`CLAUDE_CODE_DISABLE_CRON`. This includes a resumed session and a start
after an upgrade.
- A resumed session keeps its identity, its home and its transcript. It
loses the four tools from its next start.
- A Claude process that the previous release started keeps its tools
until it stops. Paperclip does not refuse it and does not classify it
differently. Its next start gets the restriction.
- No refusal code is necessary, because no saved state becomes
incompatible.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/acpx/claude-scheduling-tools.claude-acp.test.ts`
- Without the product change: `2 failed` (`expected [ Array(26) ] to not
include 'CronCreate'`).
  - With the product change: `2 passed`.
- `npx vitest run
src/drivers/acpx/claude-scheduling-tools.claude-acp.test.ts
src/drivers/acpx/environment.test.ts
src/drivers/acpx/runtime-sandbox.test.ts
src/drivers/acpx/runtime-host.test.ts`: `4 passed (4)` files, `147
passed` tests.
- `npx tsc --noEmit -p tsconfig.json` in `packages/paperclip-runner`: no
errors.
- Tools offered after the change, isolated home: the same 26 tools
without `CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`.
- `npx vitest run src/drivers/acpx/runtime-host.test.ts -t "previous
release"`: the new test "resumes a Claude session saved by the previous
release and withholds self-scheduling tools on relaunch" fails without
the product change and passes with it. The test saves a session, puts
its home into the shape of the previous release, and resumes it through
`AcpxRuntimeHost.open`.
- `npx tsc --noEmit -p tsconfig.test.json` in
`packages/paperclip-runner`: no errors.
- The fixture in `claude-scheduling-tools.claude-acp.test.ts` now stops
the process group of the provider and waits for it before it removes the
fixture directory.
- Not run: a live run with a real Claude account, and a remote Runner.
Not run: `pnpm verify` and the full suite. CI supplies that coverage.

## Risks

- **A preconfigured home still offers `ScheduleWakeup`.** In that mode
Paperclip does not write the Claude settings of the user, so the deny
list does not apply. The environment variable removes only the three
cron tools. To remove `ScheduleWakeup` there, the Runner must pass
`disallowedTools` to the session. ACPX 0.22 does not forward that
option. This is a follow-up.
- **The new test runs the real pinned Claude Code client.** It needs the
platform binary, as `runner-tool-bridge.claude-schema.test.ts` does
today. It uses no credentials and no network.
- **A prompt or skill that tells Claude to use these tools no longer
works.** No Paperclip instruction refers to them.
- The prepared harness home and remote Runners use the same code path.
They are not tested separately.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff and ran the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated the options
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:17:24 -08:00
DottaandClaude Opus 5.5 bcadbde449 feat(runner): record the OpenRouter host that served each OpenCode model call (#21298)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agents can use models through OpenRouter, and the Runner
proxies those model calls
> - OpenRouter routes one model across many hosts, and these hosts use
different quantizations (fp4, mxfp4, fp8)
> - Chat evaluations with identical prompts behave very differently
between runs, but the run log does not record which host served a call
> - The run usage shows `provider: "deepseek"` and `biller:
"openrouter"`, and `providerRequestId` is always `null`
> - This pull request records the serving host and the OpenRouter
generation ID for each model call in the run log
> - The benefit is that we can compare evaluation cohorts by host, and
the cost event names a real OpenRouter generation

## Linked Issues or Issue Description

**Subsystem affected**
Runner OpenCode provider proxy, runnerd usage normalization, and native
run usage accounting.

**Problem or motivation**
OpenRouter's default routing sends a request to one of many hosts. Each
host can use a different quantization. Chat evaluation results change a
lot between runs with identical prompts. Paperclip does not record which
host served a call, so we cannot compare results by host.
`usageJson.providerRequestId` is always `null` for OpenCode runs.

**Proposed solution**
Every OpenRouter completion response has a top-level `provider` field
and an `id` (the generation ID). Each streamed chunk has them too. The
Runner proxy reads them from the first chunk after it forwards the
bytes. The Runner attaches them to the usage of that call. The values go
to the run log, and the run's first generation ID goes to
`usageJson.providerRequestId`.

**Alternatives considered**
A new PRP event type such as `provider.upstream.served` would need a new
event enum value. An older control plane rejects an unknown event type.
A `harness.diagnostic` event shows in the task chat as a diagnostic. The
usage payload already flows to every consumer, and older consumers
ignore extra fields. Thus this change uses the usage payload.

**Additional context**
Run-log change only (AGENTS.md section 7). The data stays in
`heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch
`packages/shared/src/telemetry/` or OpenTelemetry.

## What Changed

- New `createUpstreamServedTap`
(`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`).
It is a pass-through stream. It pushes each chunk downstream first, then
it reads `provider` and `id` from the first SSE `data:` event (or from a
JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of
a JSON body. It skips encoded bodies and unknown content types. Any
parse problem or recorder error stops the scan and leaves the response
unchanged.
- `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The
OpenCode session keeps the most recent served call. Each
`item.completed` usage payload and `session.usage()` now carry
`upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint
does not include these fields, so the number of usage events does not
change.
- The OpenCode app-server proxy adds the two fields to
`thread/tokenUsage/updated`. Rust `usage.reported` now fills its
existing `providerRequestId` field and adds an optional
`upstreamProvider`. The TypeScript runnerd transport keeps both fields
when it rehydrates the usage notification. `usage.schema.json` documents
`upstreamProvider` (the protocol manifest and schema bundle are
regenerated).
- Server native turn accounting keeps the first generation ID the run
reports. The usage checkpoint and the native adapter result carry it as
`providerRequestId`, so `usageJson.providerRequestId` and the final cost
event name it.
- `doc/run-log-events.md` documents the fields in a new "OpenRouter
serving host" section.

## Verification

- Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's
stream through unchanged and records the host that served it on the
turn's usage". The test sends a streamed fixture with a comment line, a
host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks
that the bytes are identical and that the turn's usage names `Novita`
and the generation ID. The test fails without the proxy wiring.
- `upstream-served.test.ts`: streamed fixture, chunk split inside a
multibyte character, non-streamed JSON, malformed chunks, a truncated
JSON body, an encoded body, the 64 KiB limit, and a recorder that
throws.
- `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts`
(rehydration), Rust
`provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`,
and server `native-usage-accounting.test.ts` (first generation ID stays
stable across turns).
- Local: OpenCode driver, OpenCode proxy, and runnerd transport suites
(31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes
(one timing test failed once under load and passed alone). Runner and
server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks
pass.
- Latency: a local benchmark proxied a 400-chunk SSE stream 200 times
with the tap and 200 times without it. The median time to first byte was
292 µs with the tap and 259 µs without it (+33 µs, which is the extra
stream stage). The total stream time changed from 491.4 ms to 492.5 ms.
The tap spends about 5.5 µs of CPU to inspect the first chunk, after it
has forwarded that chunk. OpenRouter time to first token is hundreds of
milliseconds.
- Paid evidence: one single-case run of
`chat-features.runner-opencode.local.skills.fork-read-only` on this
branch. The result is in a comment on this pull request.

## Risks

- Low risk. All new fields are optional and additive. An older control
plane ignores the extra usage fields. `usage.schema.json` is not bound
by the event schema, and no validator enforces it on `usage.reported`.
- Saved state from the previous release: this change adds no persisted
run, session, checkpoint, or launch state. A saved session resumes
unchanged. A run that is in flight during a deploy reports usage without
the new fields until its Runner is new. `usageJson.providerRequestId`
then stays `null`, as before.
- `providerRequestId` on the run and its cost event is the run's first
generation ID, not each call's ID. The value is stable, so a late usage
receipt still matches the posted charge. Per-call IDs and hosts are in
the run log.
- If OpenCode starts a new model call before it reports the usage of the
previous call, a usage event can name the newer call. OpenCode reports a
message's usage when its step ends, so this is not expected.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code as an agent with tool use (shell, file edits, test runs). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:15:53 -06:00
DottaandClaude Opus 5.5 c730872974 fix(recovery): a chat turn waiting on the person is not stranded work (#21302)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In a chat, the agent's turn ends and the conversation waits for the
person's next message. The stranded-work sweep must leave that wait
alone
> - The sweep recognizes a finished chat turn by
`finalizationReasonCode: "conversation_turn_finished"` on the run record
> - Run completion re-reads the run through the 64 KiB read projection.
When the stored result is larger, that projection drops the finalization
fields, and completion wrote the reduced copy back
> - The chat turn then lost its finalization reason. Its reply was not
presented, and the sweep queued
`issue.productive_terminal_continuation_recovery` 30 s later. That is an
extra run while the agent waits on a person
> - This pull request keeps the full stored result through run
completion, and makes the sweep read the committed decision from its
durable rows
> - The benefit is that a turn waiting on a person is never treated as
stranded work

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `app-permissions.new-quarantined-action`, the
CEO's chat turn yielded with a `response_wake` continuation and waited
for a person to allow an app action. The stored run result was 113,937
bytes. Its saved record had the read projection's `truncated`,
`truncationReason: oversized_result_json` and `originalSizeBytes` fields
and no `finalizationReasonCode` or `decisionId`. The run's presentation
decision was `interaction_owns_conversation_response`, so no reply
comment was written. 30 s later the sweep queued
`issue.productive_terminal_continuation_recovery`, and the case failed
its run limit.

**Steps to reproduce**
1. Finish a native chat turn that yields `response_wake`, so
finalization records `conversation_turn_finished`.
2. Let the stored run result be larger than 64 KiB when run completion
reads it.
3. Run completion writes back the reduced copy without
`finalizationReasonCode`.
4. Run the stranded-work sweep. It queues a productive continuation run.

**Expected behavior**
The run record keeps its finalization fields, and the sweep leaves the
waiting chat turn alone. No new run starts until the person replies.

## What Changed

- `server/src/services/heartbeat/run-completion.ts`: run completion
reads the run with `unsafeFullResultJson` before it merges the native
result. It no longer writes the read projection's reduced copy back.
- `server/src/services/native-runtime/native-response-wake-wait.ts`:
`hasCommittedNativeConversationTurnWait` checks the run's committed
finalization decision (`conversation_turn_finished`) through
`native_run_finalizations` and `status_decisions` for the conversation's
own agent.
- `server/src/services/recovery/service.ts`: the sweep's
finished-chat-turn skip also uses that durable check, so runs already
saved with the reduced copy are recognized.
- Tests: `run-completion.test.ts` keeps the finalization record when the
stored result is larger than the read projection.
`heartbeat-process-recovery.test.ts` finalizes a real chat turn with the
eval's yielded result, saves the reduced run record, and expects two
sweeps to start no run.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts -t "larger than the read
projection"` — fails on master (the saved result is the truncated copy)
and passes here.
- `cd server && npx vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "chat turn that
yielded"` — fails on master (`continuationRequeued: 1`) and passes here.
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts
src/__tests__/heartbeat-process-recovery.test.ts` (614 passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- Run completion now reads the full stored result for one run. This is
one row; the read projection still serves every API read.
- The sweep check adds one indexed query for chat issues whose latest
run succeeded and whose run record lacks the reason.
- Previous-release saved state: no persisted shape changes. Runs already
saved with the reduced copy are covered by the durable decision check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
nightly/v2026.1113.0-nightly.0 canary/v2026.1113.0-canary.3
2026-11-13 00:11:30 -06:00
Nicky LeachandClaude Fable 5.2 f79e2c5d84 feat(ci): add the Runner image resolver and a no-rebuild mode for the Runner vendor step (#21306)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - `master` now publishes the Runner alone as the image
`ghcr.io/paperclipai/paperclip-runner`, one time for each set of Runner
inputs (#21204, #21229, #21230)
> - The application image still compiles its own Runner on every build.
The next steps make it copy the published Runner instead
> - Before a build can do that, it must answer one question the same way
as every other build: "does a published Runner exist for exactly this
tree?" It must also fail clearly when the Runner it was given does not
match the tree
> - This pull request adds the script that answers the question, and a
mode of the Runner vendor step that fails instead of rebuilding
> - The benefit is that the later change to the image build is small,
and that a wrong or unreachable answer can only mean "build the Runner
yourself"

## Linked Issues or Issue Description

Refs #21204, #21229 and #21230 (the Runner image). No issue exists for
this change.

**Problem or motivation**

Three builders make images that contain the Runner: `docker.yml`, the
merge-queue build and the release preview build. Each one would have to
repeat three steps to find the published Runner of its tree: check the
lockfile, compute the inputs digest in the build, and look up
`inputs-<digest>`. Three copies can decide differently. A build that
copies a Runner built elsewhere also has no Rust toolchain. If the
copied Runner does not match the tree, `prepare-runner-vendor.mjs`
starts a rebuild, and the build fails late at `cargo` with a message
that hides the cause.

**Proposed solution**

Add `scripts/runner-image-resolve.mjs`, the one place that decides. Add
`PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` to
`prepare-runner-vendor.mjs`. Nothing calls the resolver and nothing sets
the variable in this pull request, so no workflow job changes.

**Alternatives considered**

- One command that builds and looks up. Two commands keep a registry
credential, when a caller has one, out of the step that builds.
- A wait for the publication of the commit. The decision whether the
application build waits is open, so the resolver has no wait mode. It
reports `absent` and stops.
- A build argument that names the expected inputs. No `Dockerfile`
declares one yet. The change that makes the application image copy the
Runner will add both ends together.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 1 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode (this pull request)
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build

## What Changed

- `scripts/runner-image-resolve.mjs` (new, 185 lines), with two commands
that share a directory:
- `inputs --dir <dir>` exports the `runner-artifacts-context` target,
runs the lockfile check on that export, and builds the
`runner-artifacts-identity` target when the lockfile is current. It
reads no credential. A lockfile that is behind stops there: no image is
published for such a tree.
- `resolve --dir <dir>` looks up `inputs-<digest>` and prints `mode`,
`reason`, `inputs_digest`, `image`, `slim_image`, `build_contexts` and
`labels`. It also writes them to `$GITHUB_OUTPUT` and a short report to
`$GITHUB_STEP_SUMMARY`.
- `mode` is `published` only for a complete publication whose full index
names these inputs and both platforms. Both images are pinned by index
digest. The slim digest is the one the full index names in
`io.paperclip.runner.slim-digest`; the slim tag is never read on its
own.
- Every other answer is `inline`, with the last four values empty:

  | `reason` | Meaning |
  | --- | --- |
  | `lock-behind` | `pnpm-lock.yaml` is behind the Runner's manifests |
  | `absent` | no tag exists |
| `incomplete`, `broken` | a publication stopped, or its two tags do not
belong together |
| `mismatch` | the full index names other inputs, or lacks a platform |
| `unknown` | the registry refused, failed or could not be reached, or
the lookup threw |

- The resolver reuses `runnerLockState`, `readRunnerImageManifest`,
`runnerPublicationState` and `runnerImageTags` from
`scripts/runner-image-publish.mjs`. That file now exports
`environmentCredentials`; it has no other change.
- `server/scripts/prepare-runner-vendor.mjs`: with
`PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1`, the step reuses a current
build as before. When the build is not current, it prints the reason,
starts no rebuild, leaves the stamp in place and exits with status 1. It
also fails when `PAPERCLIP_RUNNER_VENDOR_REBUILD=1` is set at the same
time. Without the variable the step is the same as before.
- `doc/runner-image-contract.md` has a new section, "Choosing the Runner
for an image build". `doc/DEVELOPING.md` describes the new variable.
- The resolver is not an input of the Runner image, so the inputs digest
does not change.

## Verification

macOS (Node 26.12.0), in `server/`:

- `node node_modules/vitest/vitest.mjs run
scripts/prepare-runner-vendor.test.mjs
src/__tests__/runner-image-resolve.test.ts
src/__tests__/runner-image-publish.test.ts
src/__tests__/runner-image-identity.test.ts
src/__tests__/runner-image-workflow.test.ts
src/__tests__/server-package-build-script.test.ts`: 179 tests pass in 6
files. 27 of them are new resolver tests and 3 are new tests of the
vendor step, whose end-to-end test also has a new case.
- `tsc` on the new test file, with the settings of
`server/src/__tests__/tsconfig.json`: no error.

Linux (`node:24.22.3-bookworm`, GNU tar 1.34, the worktree mounted
read-only): the same files, 179 tests pass.

Real runs, on clean exports of the tree (`git archive`), with no
credential, on a `linux/arm64` Docker host:

- On `master` at `62d673cc3f`: `inputs` gave `lock=current` and
`inputs_digest=0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`.
`resolve` gave `mode=published`,
`image=ghcr.io/paperclipai/paperclip-runner@sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162`
and
`slim_image=ghcr.io/paperclipai/paperclip-runner@sha256:f63ea43666ed708f1b0e5d2ade8bdc002885bf04252b9ed6cee21f7920e54f3c`.
These are the two indexes that `master` published for those inputs.
- The same tree with one line added to a Runner source file: another
digest, `mode=inline`, `reason=absent`.
- On `master` at `bc71c23e0a`, which changed the Runner's inputs, before
its image was published: digest
`88fe2929321b4706c77f11a098572ce990fec4449d505544c62ef3a84c28308a`,
`mode=inline`, `reason=absent`.
- Four deliberate faults in the resolver each make a test fail: no index
check, only `absent` is inline, the identity is built for a lockfile
that is behind, and the slim image is taken by tag.

Never run:

- A lookup with a registry credential. The package is public, so no
caller needs one now.
- Any workflow. No workflow calls the resolver in this pull request.
- `linux/amd64`.
- The full test suite, the full typecheck and `pnpm build`. `pnpm verify
--explain` reports exit 2, "broader verification required", so CI must
supply them.

## Risks

- Low for existing behaviour: nothing calls the resolver, and the vendor
step is the same unless the new variable is set.
- The resolver does not verify the attestation of the image. The
workflow that consumes the image must do that from the digest the
resolver reports.
- The resolver reports both build contexts, `runner-artifacts` and
`runner-artifacts-slim`. A build that reads only one stage gets one
context it does not use.
- A `broken` publication is `inline`, not a failure. The Runner image
workflow fails for that state.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:03:15 -08:00
Nicky LeachandPaperclip 731b52453c fix(environments): verify workspaces on run leases and release settled tasks (#21289)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core owns run leases and delegates remote execution to environment
plugins.
> - A retry must verify its old checkout on the placement that will run
the task.
> - A completed task must still release its provider resources.
> - This change uses the existing lease for workspace verification and
always calls the release hook after settlement.
> - Saved settlement proof lets cleanup retry without repeating
execution work.

## Linked Issues or Issue Description

Refs #21237. That PR records stop proof before lease release. This PR
keeps provider cleanup after settlement and changes workspace
verification sequencing. Both touch task completion; preserve both
changes if the merge order requires conflict resolution.

**What happened?**

Workspace verification ran before a portable task had a lease, so a
provider could not verify the checkout on its selected placement. Core
also skipped the provider release hook after successful task completion.
This could leave resources allocated after the run ended.

**Expected behavior**

Verify the old checkout through the selected run lease. A confirmed
refusal can use the authorized fresh-workspace fallback. A transport
failure must remain retryable. Call the provider release hook after
settlement and retry failed cleanup without changing the run result.

**Steps to reproduce**

1. Retry a task whose prior checkout uses plugin-managed storage and
portable task placement.
2. Verify that the plugin receives the same task lease for verification
and preparation.
3. Complete a task, fail the provider release once, and reconstruct the
cleanup controller.
4. Confirm that cleanup calls only the release hook and retains the
completed run outcome.

## What Changed

- Acquire the logical run lease before provider workspace verification
and reuse it for preparation and execution.
- Preserve the existing session and workspace choices. Recompute
workspace-dependent inputs only after a definite refusal; acquire
credentials and managed runtime inputs once.
- Save the exact refusal and lease before cleanup. Use a
workspace-scoped creation key only for the resulting fresh attempt.
- Replay interrupted verification on the same unlaunched lease. Keep
transport errors retryable and reject replay after execution starts.
- Always call the provider release hook after task settlement. Pass
refreshed settlement proof and route failed releases through the
existing cleanup reaper.
- Add database regressions for call order, restart, retained writers,
refusal, transport errors, and settlement-only cleanup replay.

`plugin-workspaces.ts` grows from 997 to 1,001 lines. The four added
lines extend the existing workspace transaction and lease callbacks.
Keeping the checks there preserves the ownership assertion, provider
result validation, and binding update in one operation. Extracting only
the verification branch would split that transaction without removing a
separate responsibility.

## Verification

- Server TypeScript check passed: `tsc -p server/tsconfig.json
--noEmit`. The complete `server/scripts/typecheck-layers.mjs` check also
passed, including all test-tree layers.
- The first CI typecheck caught a readonly array in a test fixture that
the narrower source check did not include. Commit 8d9ec19e7a corrects
that fixture; all layered checks pass afterward.
- Added a public heartbeat refusal regression in 7c598a4da3. All ten
session-preparation tests pass; independent execution of the refusal
case also passes. It checks successful completion, the saved fresh
workspace, a new conversation, and one credential acquisition.
- Nine database/service suites passed 335 tests: plugin warm tasks,
plugin runner tasks, startup cleanup recovery, credential writeback,
plugin workspaces, session preparation, plugin environments, retry
workspace continuity, and heartbeat workspace preparation.
- After the final restart change, both targeted refusal/restart tests
passed. The primary agent independently reran those cases and the final
server typecheck. After integrating current master, the primary agent's
independent rerun passed all 237 tests across the warm-task,
plugin-workspace, session-preparation, and heartbeat-workspace suites.
- `pnpm verify` returned exit 2 and requested broad CI coverage because
the plugin execution contract documentation changed. No full local
Core-suite pass is claimed.
- Previous-revision CI was green on 8d9ec19e7a: 59 checks passed and 3
were skipped. The first run failed the unchanged legacy-continuation
browser assertion and credential-cleanup server shard. Each passed on
one targeted retry with no code changes. The credential-cleanup
integration suite also passed all 21 tests locally on Node 22. The
browser failure concerned an existing continuation metadata marker; the
server failure cascaded after a cleanup error left a test spy installed.
The successful retries do not establish the cause of the first failures.
- Local database fixtures use simulated provider workers. No live
provider or deployed environment was changed.

- Additional verification for 59c7a1dae8: the layered server typecheck
passed. Independent execution passed all 265 tests across durable task
retries, workspace preparation, and public heartbeat preparation. The
new shutdown regression preserves the fresh Retry choice, waits for
parent cleanup, and prevents the successor from inheriting the parent
lease reference. Same-run recovery retains its receipt. Required CI is
green on 59c7a1dae8: 58 checks passed and 3 were skipped. Standard
review scored 5/5 with no unresolved comments. The separate Superagent
Security Scan is neutral/inconclusive because its workflow failed; a
direct rerun request returned HTTP 404. That scan is not claimed as
passed.

## Risks

- Providers must use the supplied task lease for verification and return
a definite `binding_mismatch` only when the checkout is unconfirmed.
Transport errors retain the retry decision.
- A refused checkout waits for confirmed release before a fresh
allocation. Failed deletion remains owned by the existing cleanup
reaper.
- Task settlement and physical resource release stay separate. Retained
handoff leases do not enter this release path.
- No migration or SDK schema change is required.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) implemented the changes and tests.
The primary Codex agent independently reviewed the connected Core and
plugin paths and ran checks. A separate GPT-6.1 Sol agent reviewed
lifecycle and restart boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue using the bug template labels
- [x] I have not referenced internal or private project details
- [x] My branch name describes the change
- [x] I have added relevant tests and documentation
- [x] I have considered and documented risks
- [x] All required CI gates are green
- [x] Greptile is 5/5 with no unresolved comments
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 21:56:02 -08:00
DottaandClaude Opus 5.5 61ab829d5d fix(queue): another agent's comment no longer hides a pending approval (#21303)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person approves a new plan while the agent is busy, the
approval waits in the task's message queue. Its Interrupt control stops
the current attempt and starts a fresh one with the approved plan
> - The task view shows one queue at a time. A saved card response (an
approval) gives way to any later queued message, so it never hides the
person's own later messages
> - Another agent's comment (here, the CEO's direction with `resume:
true`) is also a queued message. It took over the view, and the approval
and its Interrupt control disappeared
> - The person could then not interrupt for the plan they approved
> - This pull request lets only the person's own later messages hide a
pending approval. A queue that holds only agent comments no longer hides
it
> - The benefit is that the Interrupt control for an approved plan stays
available until the person uses it or writes a newer message

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `steering.interrupt-with-direction`, the person
approved a saved simpler plan while Morgan was busy. The approval queued
with `requiresFreshSession`, and its Interrupt control showed. The CEO
then posted its own direction comment on the task at 04:03:21. That
comment queued as a separate deferred wake. `GET /queued-comments` then
returned only the CEO comment's queue. The approval's Interrupt control
(`task-chat-queued-interrupt-<approval id>`) never rendered, the click
timed out, and no interrupt request was sent. The approval was not lost:
its deferred wake was still saved. Only the view and its control were
hidden.

**Steps to reproduce**
1. Approve a plan on a busy task, so the approval waits for a fresh
session.
2. Let another agent comment on the task, so its comment queues after
the approval.
3. Read `GET /api/issues/:id/queued-comments`. The approval is gone from
the queue.

**Expected behavior**
The approval stays the visible queue with its Interrupt control. A newer
message from the person still takes the view, as before.

## What Changed

- `server/src/routes/issues.ts`: `findQueuedCommentWake` returns a later
message queue only when it holds a message from a person. A queue with
only agent comments yields to a pending saved response. With no saved
response it is still returned, as before.
- `server/src/__tests__/issue-queued-comments-routes.test.ts`: the eval
ordering (a fresh-session approval, then another agent's comment) keeps
the approval as the queue, and a later person message in that queue
still takes the view.

## Verification

- `cd server && npx vitest run
src/__tests__/issue-queued-comments-routes.test.ts -t "another agent's
comment queues after it"` — fails on master (the CEO comment's queue is
returned) and passes here.
- `cd server && npx vitest run
src/__tests__/issue-queued-comments-routes.test.ts` (all tests passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- While a saved response waits, queued agent-only comments are not shown
in the composer queue. They stay saved and are delivered by the normal
deferred-wake path, as they were before the person approved anything.
The person's own messages are never hidden.
- One indexed comment lookup per queued message wake during the queue
read.
- No persisted shape or API contract change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:53:54 -06:00
DottaandClaude Opus 5.5 99d9e608f4 fix(heartbeat): hold the new owner's run while the reassigned owner's run settles (#21301)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person reassigns a task, the old owner's native run is
cancelled and the new owner's run is queued
> - The old run's executor keeps closing its provider session for
several seconds after the run is marked cancelled
> - #21031 holds a queued run while the task's previous session close is
pending. That hold reads the runner's cleanup gate, which exists only
after the close begins
> - The stale-lock sweep cleared the task lock as soon as the old run
was terminal. The new owner's run was then claimed before the close
began, met the close as it escalated, and failed with
`native_session_cleanup_quarantined`
> - This pull request keeps the new owner's run queued while the old
owner's stopped run still settles in this process, and wakes it when
that executor finishes
> - The benefit is that a reassignment never fails the new owner's run
and needs no Retry

## Linked Issues or Issue Description

Refs #21031

**What happened?**
In the paid chat eval `tasks.edit-properties`, the old owner's run
`6cf2ee38` was cancelled for reassignment at 04:03:47.611. Its process
stopped only at 04:03:57. The stale-lock sweep cleared the task lock at
04:03:47. The new owner's run was claimed at 04:03:48.626. The runner
admission waited for the old close, which escalated to operator-required
at 04:03:56, and the new run failed with
`native_session_cleanup_quarantined`.

Why #21031 did not apply: its hold reads the runner's in-memory cleanup
gate. At claim time the old executor had not started its session close,
so no gate existed. The existing `predecessor_settling_in_process` hold
covers only the same agent, and a reassignment changes the agent. #21260
does not touch this path: its new checkpoint runs only before a turn in
a recovered session, and both sessions here were new.

**Steps to reproduce**
1. Run a native task for agent A.
2. Reassign it to agent B. A's run is cancelled, but its executor is
still settling.
3. The stale-lock sweep clears the task lock.
4. B's queued run is claimed before A's session close begins.

**Expected behavior**
B's run stays queued until A's executor has finished. It starts by
itself after that, then #21031's gate covers any close that is still
settling.

## What Changed

- `server/src/services/heartbeat/queue.ts`: the
`predecessor_settling_in_process` hold in `claimQueuedRun` also holds
for another agent's native run on the same task when that run is
terminal but its executor is still active in this process. A live run of
another agent (for example a reviewer) does not hold it.
- `server/src/services/heartbeat/queue.ts`:
`notifyOtherAgentsQueuedOnSettledIssue` sends a queued-run dispatch
signal when a native executor settles and another agent has a queued run
on the same task. It does nothing when no run waits.
- `server/src/services/heartbeat.ts`: the executor calls it after its
cleanup, next to the existing same-agent `startNextQueuedRunForAgent`.
- `server/src/services/heartbeat/queue.test.ts`: the exact ordering
(cancelled old-owner run still in the executor set, lock cleared, new
owner's run queued), a control for a live run of another agent, and the
dispatch wake.

## Verification

- `cd server && npx vitest run src/services/heartbeat/queue.test.ts -t
"reassigned owner|another agent's live run|next owner once"` — the hold
and the wake tests fail on master and pass here.
- `cd server && npx vitest run src/services/heartbeat/queue.test.ts
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/__tests__/heartbeat-workspace-busy.test.ts` (all passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- A queued run on a task waits while another agent's terminal native run
on that task is still settling in this process. That wait ends when the
executor finishes, and the new wake re-drives dispatch. The queued-run
liveness redrive still covers a lost signal.
- The wake is event-driven: one indexed query when a native executor
settles, and one notification only when another agent waits on that
task. It adds no polling.
- Previous-release saved state: no persisted shape changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:53:49 -06:00
DottaandClaude Opus 5.5 f82ec65800 fix(runs): a person's Stop wins over lease-release teardown (#21300)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can stop an agent run with Stop (cancel). The run must then
end as cancelled
> - A run that a person stops just as it starts tears down its
environment lease at the same moment
> - The lease-release teardown wrote `interrupted`
(`lease_released_before_terminal`). It recognized only an acknowledged
Stop, and it decided from a run snapshot that it read before the Stop
request landed
> - An interrupted run looks like an infrastructure failure, and it is a
recovery candidate. That is against the person's decision
> - This pull request makes a recorded person Stop win over the teardown
in both orderings
> - The benefit is that a person's Stop always ends the run as cancelled

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `steering.stop-versus-pause`, the fixture
cancelled a resumed run as it reached native running. The Stop request
was recorded at 04:02:11.049. The teardown wrote `interrupted` with
`lease_released_before_terminal` at 04:02:11.083. The native
cancellation intent was still `pending`, so the teardown did not see it
as a Stop. The eval needs `cancelled`.

**Steps to reproduce**
1. Start a native run.
2. Record a person Stop request on the run (the cancel route does this
first).
3. Let the lease-release teardown run before the cancel finalizer writes
`cancelled`.
4. The run ends `interrupted`.

**Expected behavior**
The run ends `cancelled`. This is true when the Stop is recorded before
the teardown reads the run, and when it is recorded between that read
and the write.

## What Changed

- `server/src/services/acknowledged-native-stop.ts`:
`hasRecordedUserStopRequest` and `recordedUserStopRequestSql` recognize
a person Stop that the cancel route recorded (`cancelledByActorType:
"user"`, a user id, and a requested or acknowledged execution
cancellation or a startup cancellation fence).
- `server/src/services/heartbeat/run-lifecycle.ts`:
`terminalizeRunOnLeaseRelease` re-reads the run before it decides. A
recorded person Stop now gives `cancelled`. The `interrupted` write is
fenced on "no person Stop recorded", so a Stop that lands between the
read and the write makes the write fail. The function then reads again
and writes `cancelled`.
-
`server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts`:
two orderings (Stop recorded before the teardown read; Stop recorded
after the read, with the stale snapshot) and a control (an agent-only
request still reports `interrupted`).

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-lease-release-terminalization.test.ts` —
both new ordering tests fail on master (`interrupted`,
`lease_released_before_terminal`) and pass here.
- `cd server && npx vitest run
src/__tests__/heartbeat-run-lease-release-terminalization.test.ts
src/__tests__/heartbeat-run-terminalize-before-release.test.ts
src/services/heartbeat/run-lifecycle.test.ts` (67 passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- Low. Only the lease-release teardown changes, and only when a person
Stop is recorded on the run. The cancel path already wrote `cancelled`
for the same run when it won the race.
- The teardown now makes one more read of the run row. This is one
indexed primary-key read per teardown.
- Previous-release saved state: no persisted shape changes. Runs saved
by the previous release carry the same Stop fields.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:52:13 -06:00
DottaandClaude Fable 5.2 bc71c23e0a fix(test): remove the Pi fault calibration file race and halve its runtime (#21286)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite checks the Pi provider fault helper. The
helper kills the exact Pi child process on Linux with a pidfd.
> - `tests/runner-e2e/pi-provider-fault.test.ts` failed in 3 unrelated
PRs on 2026-11-12. One failure passed on a rerun of the same commit.
> - The failures had two causes. A fixture file race gave a JSON parse
error. A CPU-heavy calibration ran past the 25 s outer timeout on the
slower shared fleet.
> - This pull request removes the file race and removes about half of
the calibration work. It also states the outer timeout margin.
> - The benefit is a deterministic test that runs in about 3.4 s instead
of about 5.9 s on Linux.

## Linked Issues or Issue Description

**What happened?**
The test "calibrates the exact Pi fault helper with metadata negatives
and native Linux pidfd" failed intermittently in the "Runner support
suite" lane. There were two errors:
- Job 131195539083: `test_real_descriptor_launch_then_exact_child_pidfd
(descriptor=7)` failed at `json.loads(ready.read_text())` with
`JSONDecodeError: Expecting value: line 1 column 1 (char 0)`.
- Jobs 131420813114 and 131420614853: `spawnSync /usr/bin/python3
ETIMEDOUT` after 25 s. Both jobs ran on the `paperclip-public-pr-x64`
fleet (m5n.xlarge, 4 cores, load average about 4). In those jobs, every
file in the lane ran about 1.8x slower than on the hosted `ubuntu-24.04`
image.

**Expected behavior**
The test passes on every run of the same commit.

**Steps to reproduce**
1. Error A: add a delay between file creation and write in the fixture
child (for example `openSync`, then `Atomics.wait` for 500 ms, then
`writeSync`). Run `/usr/bin/python3 -B
tests/runner-e2e/pi-provider-fault.test.py` on Linux. All four real
calibrations fail with the same `JSONDecodeError`.
2. Error B: run the suite on a loaded 4-core runner. The Python suite
needs more than 25 s.

**Paperclip version or commit**
`master` at 9b613562b7.

## What Changed

- Root cause A: the fixture child wrote `child.json` and `exit.json`
with `writeFileSync`. That call creates the file and then writes it. The
test polls only `exists()` and then parses. A reader could see the
created but empty file. Now the fixture writes `<name>.tmp` and renames
it into place, so the file appears only with complete content. This also
fixes the same race on `exit.json`.
- Root cause B: a profile of one calibration showed that about 1.0 s of
1.28 s was spent in `inspect()`. Each `inspect()` call reads and SHA-256
hashes the runner executable and the distribution Node executable. Both
were copies of the 123 MB Node binary. Each calibration calls
`inspect()` five times, and the suite runs four calibrations, so the
suite hashed about 5 GB. Polling sleeps were only 0.09 s. The synthetic
runner root now is a copy of `/bin/sh` that runs a short script. It is
still a distinct, pinned ELF, and `/proc/<pid>/exe` still resolves to
it. The script keeps the same launch shape (direct path, or
`/proc/self/fd/3` or `/proc/self/fd/7` with the executable descriptor
opened only for the wrapper child). It keeps the same SIGTERM cascade.
The distribution Node, the wrapper and the Pi child are unchanged.
- The outer `spawnSync` timeout is now a named constant (45 s) with its
margin in a comment. The Vitest test timeout derives from it (+5 s), so
a slow Python run reports its own output instead of a Vitest timeout.
- No assertion is removed, skipped or weakened. The fault helper
(`pi-provider-fault.py`) is unchanged.

## Verification

All Linux runs use Docker `node:24-bookworm` (Python 3.11, kernel 6.12)
on an Apple Silicon host.

- Red (master + 500 ms create-to-write delay): `Ran 21 tests`, `FAILED
(errors=4)`. Each of the four real calibrations failed with
`json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char
0)`.
- Green (this branch + the same 500 ms delay between creating and
writing the temporary file): `Ran 21 tests in 7.788s`, `OK`.
- Timing, `python3 -B pi-provider-fault.test.py`, 3 runs each:
- Unlimited CPU: master 6.77 / 5.88 / 5.75 s; this branch 3.45 / 3.34 /
3.43 s.
- `--cpus=1`: master 6.16 / 6.08 / 5.53 s; this branch 3.37 / 3.34 /
3.04 s.
- macOS: `pnpm exec vitest run --config
tests/runner-e2e/vitest.config.ts pi-provider-fault` passes. macOS skips
the 3 Linux-only calibrations, as before.
- `pnpm verify` exits 2 because a `.py` file is a global CI input.
Required CI supplies the broad coverage.
- Not run: the test on an x64 GitHub-hosted or fleet runner before this
PR's CI. I could not reproduce the fleet's load locally.

## Risks

- Low risk. The change is test-only.
- The runner root fixture now is a shell instead of Node. The fault
helper checks only the root's executable identity, hash, arguments,
process group and working directory. A copied shell satisfies these
checks in the same way.
- If `/bin/sh` is bash instead of dash, the script still works. It uses
only POSIX `trap`, `kill`, `wait` and `$!`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, Docker, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (not
applicable: test-only change)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:19:29 -06:00
Nicky LeachandClaude Opus 5.5 e65a3095a7 fix(runner): quiesce the Core-dialed route once the Runner commits its suspend (#21245)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Core runs agent sessions through the Paperclip Runner. With provider
ingress, Core dials the Runner over a PRP WebSocket.
> - When Core closes a session, it asks the Runner to suspend. The
Runner suspends, reports the result, and exits.
> - Core's ingress connection did not know that this exit was expected.
It treated the drop as a transport loss and redialed the Runner. Each
redial got a 404 until the close released the route.
> - The release (park) path already quiesces the connection before the
Runner goes away. The suspend path did not.
> - This pull request makes the suspend close quiesce the route in the
same way, once the Runner commits its suspend.
> - The benefit is that a clean suspend no longer looks like a lost
Runner. A drop before the suspend is committed still redials and
recovers as before.

## Linked Issues or Issue Description

**What happened?**
After a provider-ingress session closed cleanly, Core redialed the
Runner's ingress endpoint several times and got 404 each time. The
Runner state was `suspended`, `runner.drain` and `runner.suspend` had
both completed, and the Runner had exited with code 0. The redials
followed the outbound reconnect backoff (about 250, 500, 1000 and 2000
ms) and stopped when the registration release closed the outbound.

**Expected behavior**
After Core commits to `runner.suspend` and the Runner reports it
completed, the Runner's exit is expected. Core must not redial or
classify the drop as a transport loss. The release path already works
this way: it calls `outbound.quiesce()` before it releases the task.

**Steps to reproduce**
1. Open a session through a Core-dialed route (plugin provider ingress
or the `listen_ws` driver).
2. Close the session. Core drains, queues `runner.suspend`, and the
Runner suspends and exits.
3. See that the outbound reconnect loop redials the gone Runner until
the registration release closes it.

**Paperclip version or commit**
`master` at b4ca19f3a8.

**Agent adapter(s) involved**
Not adapter-specific (core bug). It affects any Core-dialed Runner
route: plugin provider ingress and the `listen_ws` driver.

**Additional context**
The redials do not fail the session. The plugin outbound uses a 7-minute
recovery grace, and `#failTransport` ignores failures after
`#closeAndSettle` sets `#closed`. The new test confirms this: without
the fix, the close still settles, but the route redials. This PR removes
the misleading redial. It does not change how a run's own failure is
classified.

## What Changed

- `RunnerExecutionDriver.controlPlaneRegistration` results can now
include an optional `quiesce()`. Core calls it when it has committed the
Runner's exit. The route then stops liveness and reconnect, and it keeps
the established receive/ACK path.
- `DurablePrpControlPlane.onRunnerExitCommitted(listener)` is new. The
authority calls its listeners after it saves a completed
`runner.suspend` or `runner.shutdown` result and before it sends the
acknowledgement. The Runner exits when it gets that acknowledgement, so
a route quiesced here never sees the exit as a loss.
- `awaitRunnerSuspensionBarrier` calls a new `onExitCommitted` callback
once, when its suspend command is `completed`. It also calls the
callback when it reuses a completed receipt and the durable state is
still `suspended`. A rejected, failed or unanswered suspend never calls
it.
- `DurablePrpCodexTransport` registers each control-plane route through
one `#registerControlPlane` helper. The helper records each route's
`quiesce` until that route is released. The suspend close subscribes to
`onRunnerExitCommitted` while its barrier runs and quiesces all of these
routes from that notification. The barrier's own `onExitCommitted`
covers a completed receipt that was acknowledged before this close.
Quiescing clears the tracked routes, so it runs once per close. The
helper copies property descriptors, so getters such as the `listen_ws`
driver's lazy `failure` stay live.
- The plugin task registration (`plugin-runner-task.ts`) and the
`listen_ws` driver registration (`native-runner-execution-driver.ts`)
implement `quiesce` with `outbound?.quiesce()`. This is the same call
that the release path makes.

## Verification

- New transport test with a real runnerd:
`packages/paperclip-runner/src/live/runnerd-codex-transport-suspend-route.test.ts`.
The test route acts like a Core-dialed provider ingress: on a socket
drop that it was not told to expect, it records a redial into a 404.
Before the fix the events were `connected, redial: 404, release`. After
the fix they are `connected, quiesce after runner.suspend completed,
release`. The close settles, and the durable Runner state is
`suspended`. A second case delays the durable-state read by 1.5 s, so
the Runner exits while the barrier still reads. That case fails without
the authority hook and passes with it.
- New barrier unit test in `runnerd-codex-transport.test.ts`. It shows
that the exit is committed only after the retried suspend completes,
never on a rejected or unanswered suspend, and on a reused receipt only
when the state is `suspended`.
- New server tests for the plugin task and `listen_ws` registrations.
`quiesce` reaches the active outbound and does not close it. Release
still closes it.
- Existing tests pass. These include the outbound release-path quiesce
test, the half-open and lease-expiry failure tests (a drop before the
commit still fails after its grace), and the plugin release/resume
facade tests.
- `pnpm verify` exited 2 (the change touches files outside the resolved
test graphs), so I ran focused suites. Locally: 8 runner transport files
and the 4 control-plane test files, 667 tests
(`runnerd-codex-transport*.test.ts` for suspension, release/resume,
remote retention, settlement, warm attachment and attachment recovery,
plus the new suspend-route test), and 5 server files, 148 tests (both
plugin Runner integration tests, the plugin task, the execution driver
and the outbound). Required CI supplies the broad coverage.
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Low risk. `quiesce` is optional. Registrations that do not supply it
behave as before.
- The quiesce happens only after the Runner reports a completed suspend.
A socket drop before that point still redials and recovers.
- The quiesced route keeps its receive/ACK path, so the Runner's last
frames before exit are still acknowledged.
- No persisted state or protocol change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:19:24 -08:00
DottaandClaude Opus 5.5 62d673cc3f fix(retry): start a held Retry once a stopped preparation is proven stopped (#21295)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails, a person can press Retry. Admission starts
a new turn from the recorded work only after it proves that the previous
execution stopped.
> - For a legacy runner run that failed while it prepared its sandbox,
the proof reads the run's events and its environment lease.
> - After the automatic retry budget is spent, execution recovery
settles a no-replay hold and appends its own lifecycle event to the run.
The proof accepted only the error and measured events, so this platform
event made the proof fail.
> - The proof also asked for a workspace repair receipt for every run
with a lease, even when the run never recorded a remote workspace and
its sandbox was destroyed. No repair could ever exist, so the proof
could never pass.
> - Admission then refused the Retry without a named reason. The receipt
kept the generic `execution_recovery` wait ("Retry is waiting for
execution checks"), and every later Retry joined the same held receipt.
> - This pull request accepts the recovery hold's own settlement event
when it is bound to a settled hold for the same run, and asks for a
workspace repair only when the run holds prepared files.
> - The benefit is that Retry starts work at once when the previous
preparation is proven stopped, and it continues from the recorded work.

## Linked Issues or Issue Description

**What happened?**

A board user pressed Retry on a blocked task. The Retry was accepted,
but no run started for more than 13 hours. The task said "Retry is
waiting for execution checks. Your message is saved." A second Retry the
next day attached to the same held request and also did nothing. The
execution blocker said `legacy_execution_requires_reconciliation` with
`canRetry: true` and "A person can use Retry on this task ... to
continue once the previous execution has stopped." The previous run was
a legacy `paperclip_runner` run that failed with `setup_failed` while
preparing (`executionRecovery.kind = bootstrap`, `providerWorkStarted =
false`). It had no process identity, no controller lease, and its
Daytona sandbox lease had a `destroyed` termination receipt. Execution
recovery had settled a no-replay hold on it and appended the "Automatic
recovery stopped" lifecycle event.

**Expected behavior**

When the previous execution is proven stopped, a person's Retry starts a
new turn promptly and continues from the recorded work. The task view,
the execution blocker and admission agree that Retry works.

**Steps to reproduce**

1. Let a legacy `paperclip_runner` run fail while it prepares a remote
sandbox (`setup_failed`, bootstrap evidence with `providerWorkStarted =
false`) on its third automatic attempt.
2. Let the sandbox lease be released with a `destroyed` termination
receipt.
3. Let execution recovery settle the
`legacy_execution_requires_reconciliation` hold. It appends the
"Automatic recovery stopped" lifecycle event to the run.
4. Press Retry on the task.
5. Observe that the Retry receipt keeps `executionWait.reason =
execution_recovery` and no run starts. Press Retry again and observe
that it attaches to the same held receipt.

**Paperclip version or commit**

Staging Core `53608bd1d6`. Fixed on top of `origin/master` `c4f639ed1b`.

**Deployment mode**

Hosted cloud (staging), Daytona remote sandbox.

**Agent adapter(s) involved**

`paperclip_runner` (legacy runtime).

Related: #21267 (a Retry held on a resolved decision,
`decision_pending`). This pull request does not change its files.

## What Changed

- `readFailedPreparationSettlement` (`failed-preparation-settlement.ts`)
accepts one trailing host lifecycle event: the no-replay settlement
event that execution recovery appends. The event must have the exact
payload shape and must name a resolved
`legacy_execution_requires_reconciliation` hold whose evidence binds the
same run. Any other event still fails the proof.
- `failedNativePreparation` (`cancelled-native-startup.ts`) asks for a
verified workspace repair only when the run holds prepared files: a
recorded remote workspace, a workspace restore failure, or required
workspace recovery. A run with no recorded remote workspace and a
stopped sandbox has nothing to repair. The lease termination receipt,
the bootstrap evidence and the settled-event proof are still required.
- New regression test `legacy-preparation-retry-hold.test.ts` uses the
exact shape of the staging run.

## Verification

- Red/green: `cd server && npx vitest run
src/services/legacy-preparation-retry-hold.test.ts`.
- "starts the person's Retry once the stopped preparation's sandbox is
destroyed" fails on `origin/master` (the receipt keeps
`executionWait.reason = execution_recovery`, no run) and passes with
this change (one queued successor with `previousRunId` set to the failed
run and the Retry request id).
- The Retry stays held when the proof does not hold: a settlement event
bound to another run's hold, a provider execution event, a lease without
a termination receipt, and a recorded remote workspace without a repair
receipt.
- Related suites pass (299 tests): `task-retry`,
`saved-bootstrap-continuation`, `interrupted-preparation-hold-settle`,
`workspace-recovery-preparation`,
`workspace-recovery-preparation-gates`, `heartbeat-session-preparation`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low. The change only widens two proofs for a run that never started
provider work: the platform's own settlement event is accepted only when
it binds a settled hold for the same run, and the repair receipt is
skipped only when the run recorded no files to repair.
- Retry still starts a new turn. It never replays the stopped run.
- Not in scope: two other tasks show the same generic wait with a
different cause on the saved-work side. Their blocker says `canRetry:
false` while the task view offers Retry. One has saved-work recovery
that failed with `source_unavailable`. The other shows recovered work
without a verified repair receipt. They need a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:07:17 -06:00
DottaandClaude Fable 5.2 db3d7223e0 fix(verify): report stale SDK public input pins locally, before CI (#21283)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite proves the plugin SDK public types against
pinned source. The pins are in
`tests/runner-e2e/source-sdk-public-inputs.json`.
> - The pins cover every file in `packages/shared/src` and
`packages/plugins/sdk/src`, and the SDK build inputs (for example
`pnpm-lock.yaml`).
> - A PR that edits a pinned file and does not refresh the registry
fails three files in the "Runner support suite" lane.
> - Between 2026-11-12 10:27Z and 2026-11-13 01:14Z, this failure
occurred in 14 CI runs on 12 PRs.
> - The local check, `pnpm verify`, returned exit 2 ("let CI cover it")
for these changes. It did not mention the pins. Authors pushed, and CI
failed.
> - This pull request makes `pnpm verify` find stale pins before the
push. It also makes the opaque fixture error name the file and the fix.
> - The benefit is that authors find and fix the problem in seconds,
locally, instead of after a 6-minute CI lane.

## Linked Issues or Issue Description

**What happened?**
Three files in the "Runner support suite" lane failed together on many
unrelated PRs: `source-sdk-evidence.test.ts`,
`source-sdk-public-inputs.test.mjs` and
`source-sdk-public-type.test.ts`. Two of the three failed with the
opaque error `Reviewed SDK reference source changed`.

**Root cause**
The pins on master are not stale. The tests pass on master (58/58). Each
failing PR edited a pinned input and did not refresh the registry. I
compared the PR heads with their merge base:

- All 14 failing heads (12 PRs) changed a pinned input. None of them
changed `source-sdk-public-inputs.json`.
- All 19 heads in the same window that changed a pinned input and passed
this lane also refreshed the registry. This includes later heads of
three PRs that failed first (#21119, #21121, #21156).
- Two of the 12 PRs are Dependabot PRs that change only `pnpm-lock.yaml`
(#19593, #19594). The lockfile refresh bot already resyncs the pins for
lock changes on master.

**Steps to reproduce**
1. Start from `origin/master`.
2. Add one line to `packages/shared/src/types/heartbeat.ts` and commit
it.
3. Run `node node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts source-sdk-evidence.test.ts
source-sdk-public-inputs.test.mjs source-sdk-public-type.test.ts`.
4. The result is the same as CI: 3 files fail, with the same 4 failures.
5. Run `pnpm verify --explain`. It returns exit 2 and does not mention
the pins.

**Expected behavior**
`pnpm verify` reports the stale pins and the refresh command before the
author pushes.

## What Changed

- `tests/runner-e2e/source-sdk-public-inputs.mjs`: add
`staleSdkPublicInputsInWorkingTree(root)`. It applies the existing drift
guard to the working tree (tracked and untracked files that git does not
ignore). Add `isSdkPublicInput(file)` and the shared failure message
`staleSdkPublicInputsMessage(stale)`. The committed-tree guard keeps its
behavior and now uses a shared comparison.
- `scripts/local-verify.mjs`: when the diff touches a pinned input, the
registry or `report-lock.json`, `pnpm verify` checks the pins first. If
they are stale, it exits 1 and prints the files and the refresh command.
The check reads files only. It does not write the registry.
- `tests/runner-e2e/source-sdk-reviewed-fixture.ts`: the error `Reviewed
SDK reference source changed` now names the file and gives the refresh
command.
- `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: new test for the
working-tree guard (uncommitted edit, untracked file, ignored file). The
CI drift test uses the shared message.
- `doc/DEVELOPING.md`: document the new check under "Incremental local
verification".

No assertion is weakened. No pin is changed. No file under `.github/` is
changed.

## Verification

- Master, without this change: the three files pass (58/58).
- Red: on master plus a one-line edit to
`packages/shared/src/types/heartbeat.ts`, the three files fail with the
CI signature. `pnpm verify --explain` returns exit 2 and does not
mention the pins.
- With this change on the same red commit: `pnpm verify` exits 1 in
about 15 s and prints `tests/runner-e2e/source-sdk-public-inputs.json is
stale for 1 pinned input(s): packages/shared/src/types/heartbeat.ts
(changed). Commit your change, run ... --write, and commit the updated
registry.`
- Green: I followed the printed command (`node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`, then
commit). The three files then pass (59/59, with the new test).
- `node --test scripts/__tests__/local-verify.test.mjs`: 14/14 pass.
- `pnpm verify` on this branch returns exit 2 because
`scripts/local-verify.mjs` is a global CI input. Required CI supplies
the broad coverage.

## Risks

- Low risk. The new check only reads files. It runs only when the diff
touches a pinned input.
- `pnpm verify` now fails (exit 1) for a change that also fails in CI.
Before, it returned exit 2.
- The check needs a working `git ls-files` in the repository. `pnpm
verify` already needs git.
- Dependabot lock PRs still fail this lane until the lockfile bot
refreshes the pins. This PR does not change that path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent with tool use (shell, git,
GitHub CLI). Commits are attributed to Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:03:39 -06:00
DottaandClaude Opus 5.5 0c63f0fdb3 test(skills): budget list read bytes for fixture rows, not the bundled catalog (#21285)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The skill library list reconciles the bundled skill catalog,
folders, and local sources on each read
> - #18023 added a test that limits the `company_skills` bytes one warm
`list()` call reads, so the folder pass and the list query keep their
narrow projections
> - The limit counts every `company_skills` read, so it also counts the
bundled catalog rows that every company gets
> - Each new bundled skill adds about 9.6 KB to that total, so pull
requests that only add skills under `skills/` fail a test about
projections
> - This pull request measures a catalog-only company next to the
fixture company and applies the same limit to the difference
> - The benefit is that the test fails only when the fixture rows are
read too widely, not when the catalog grows

## Linked Issues or Issue Description

Refs #18023 (added the test). Refs #20760, #20764, #20767, #20768,
#20769 (failed on this test).

**What happened?**

`companySkillService.list > keeps folder classification and stored
content while narrowing inventory refresh reads` failed on five open
pull requests that only add bundled skills under `skills/`. The error
was `expected 1163219 to be less than 950000`. Other runs had 983607,
1035956, 982321, and 985561. The test passes on master.

**Expected behavior**

The test must fail when `list()` reads the unused metadata or the
Markdown of company skill rows. It must not fail when the bundled
catalog gets larger.

**Steps to reproduce**

1. Check out master.
2. Run `git checkout <head of #20760> -- skills` to add its 34 bundled
skills.
3. Run `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/company-skills-service.test.ts -t "narrowing inventory
refresh"`.
4. The test fails with `expected 1163504 to be less than 950000`.

**Paperclip version or commit**

`9b613562b70b7bea9e7562ddc1087d4c1b1788b1` (master).

## What Changed

- The test creates a second company that has only the bundled catalog.
- The test lists both companies warm and captures the `company_skills`
response-value bytes of each call.
- The test applies the existing 950,000-byte limit to the difference.
The difference is about 663 KB and comes only from the ten fixture rows.
- The test also checks that the fixture company lists the catalog rows
plus the ten fixture rows.
- No service code changes.

## Verification

Where the bytes come from (master, one warm `list()` call):

| Read | Master catalog | Catalog of #20760 |
| --- | --- | --- |
| Main list query (`company-skills.ts:3324`) | 671,003 B, 20 rows |
691,694 B, 52 rows |
| Bundled reconciliation `getByKey` reads (one per bundled skill) |
165,423 B, 13 calls | 471,771 B, 45 calls |
| Total (old assertion) | 836,465 B, pass | 1,163,504 B, fail |
| Fixture company minus catalog-only company (new assertion) | 663,153
B, pass | 663,166 B, pass |

Red and green:

- Old test, master `skills/` plus the `skills/` tree of #20760: fails,
`expected 1163504 to be less than 950000`.
- New test, same enlarged catalog: passes, difference 663,166.
- New test, master: passes, difference 663,153.

The new test still catches the regression it exists for. I widened each
projection on purpose with the enlarged catalog, ran the test, and
reverted:

- Folder pass reads the full `metadata` column instead of `sourceKind`
only: fails, `expected 1323122 to be less than 950000`.
- Main list query also reads `markdown`: fails, `expected 1183162 to be
less than 950000`.

Commands:

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/company-skills-service.test.ts`: 81 passed.
- `pnpm verify`: not completed. It selected 2,615 of 3,435 test files
(conservative selection), and I stopped it to keep load low on a shared
machine. CI ran the full required checks on the final commit: 54 passed,
4 skipped.

## Risks

- Low risk. The change is in one test only.
- The new test runs one more warm `list()` call, about 0.5 s more.
- The bound is unchanged. A widened projection on the fixture rows still
adds at least 540 KB and fails.
- This test no longer limits the bytes the bundled catalog adds to each
list call. Each bundled skill adds about 9.6 KB per warm call, because
`ensureBundledSkills` calls `getByKey` with the full row for each
bundled skill. That cost is real but separate. A fix should narrow that
read, not let this test limit the catalog size.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (not
applicable: test-only change)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (Superagent Security Scan is
neutral: "Scan inconclusive")
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:03:23 -06:00
DottaandClaude Opus 5.5 c6c7a61e2c fix(retry): release a Retry held on a resolved decision (#21267)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails, the task shows a Retry button. Admission
can hold an accepted Retry while a person must answer a pending approval
or question (`decision_pending`).
> - The hold is saved on the deferred Retry receipt. Nothing cleared
that saved reason when the person answered, so the task kept saying
"Retry is waiting for a pending approval or question" when nothing was
left to answer.
> - The deferred-receipt worker drops a refused receipt from its
schedule. No producer hinted it when the decision resolved, so the held
Retry did not start until some unrelated event rescanned the company.
> - A Retry was also held by a confirmation that was already open when
the failed turn was admitted. That turn had already run past the
confirmation, so the Retry added a gate that the original turn did not
have.
> - A person's comment on the task started a continuation, but the held
Retry receipt stayed deferred next to it.
> - This pull request makes a resolved decision release the held Retry
at once, keeps the projection from showing a resolved decision as the
wait, stops holding a Retry on a decision that predates the failed turn,
and settles a held Retry when a comment or answer continues the same
failed run.
> - The benefit is that Retry always works: a person never waits on a
decision that is already resolved.

## Linked Issues or Issue Description

**What happened?**

A board user pressed Retry on blocked tasks. No run started. The task
said "Retry is waiting for a pending approval or question to be
resolved. Your message is saved." In one task, the only interaction was
a `request_confirmation` that the previous successful run created. The
user's later comment started the turn that failed. The confirmation
stayed pending, so every Retry attached to the same held receipt. After
the user rejected the confirmation and sent a new comment, the task
still showed the same wait. No run started until a server restart
rescanned the deferred receipts about three minutes later. The task
projection said `canRetry: true` with the `decision_pending` wait.

**Expected behavior**

An accepted Retry is never held on a decision that is already resolved.
Accepting or rejecting the decision releases a held Retry right away. A
person's comment on the blocked task starts work, and the held Retry is
settled. The task projection never shows a resolved decision as the
Retry wait.

**Steps to reproduce**

1. Let an agent run on a remote runner create a `request_confirmation`
for the task.
2. Let the runner be lost so the run fails with
`native_continuation_requires_reconciliation`, and let saved work
recovery complete.
3. Press Retry. The Retry receipt is saved with `executionWait.reason =
decision_pending`.
4. Reject or accept the confirmation.
5. Observe that the receipt keeps `decision_pending`, the task keeps
showing the wait, and the Retry does not start until an unrelated
rescan.

**Paperclip version or commit**

Staging Core `57aeda190b`. Fixed on top of `origin/master` `53608bd1d6`.

**Deployment mode**

Hosted cloud (staging), remote sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (native runtime).

## What Changed

- `hasPendingContinuationDecision` (`explicit-native-continuation.ts`)
is now the single check for a decision that holds a person's
continuation. Admission and the task projection both use it.
- For a Retry, a pending interaction that was created before the failed
source run was created does not hold the Retry. That turn was already
admitted while the decision was open. Decisions that the failed run
created (for example a plan review) still hold the Retry.
- `releaseDecisionHeldExecution` (`deferred-execution-wait.ts`) removes
the saved `decision_pending` wait from deferred receipts on the task and
hints each receipt to the deferred-receipt worker. It runs in the
resolution transaction of every interaction resolution path (through
`notifyChatConfirmationResolution`) and in question answers.
- The task projection reads the live decision state. A saved
`decision_pending` wait whose decision is resolved is not shown. This
also repairs receipts saved before this change.
- When a comment or an interaction answer continues a failed run,
`retireTaskRetriesContinuedBy` (`task-retry.ts`) settles held Retry
receipts for the same source run as `task_retry_superseded`.

## Verification

- Red/green: `cd server && npx vitest run
src/services/governed-wait-runner-loss.test.ts`. The five new tests fail
on `origin/master` sources and pass with this change (17/17 pass):
  - releases the held Retry as soon as the person chooses accept
  - releases the held Retry as soon as the person chooses reject
- never shows a resolved confirmation as the Retry's wait, and both
projections offer the same Retry
- a person's comment starts work and settles the held Retry once nothing
is left to answer
- starts a Retry at once when the confirmation was already open before
the failed turn started
- The existing tests that keep a Retry or a message on hold while a plan
review, governed tool action, secret proposal, or connection
authorization is pending still pass.
- Related suites: deferred execution wait, explicit native continuation,
interaction resolution and service, task interruption presentation and
query, task retry, deferred receipt work, interaction routes.

## Risks

- Medium-low. A Retry now starts when the only pending interaction is
older than the failed run. That turn already ran with the interaction
open, so the Retry gets no new authority. Governed tool actions still
need their own approval before the tool runs.
- Interaction resolution now also updates deferred receipts on the same
task. The update is limited to receipts with a saved `decision_pending`
wait. A receipt that is still blocked by another pending decision is
held again on its next admission.
- Issue-linked approvals (`approvals` table) do not yet hint the held
receipt when they resolve. The projection no longer shows them as the
wait after they resolve.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:58:48 -06:00
Pavel-GuseynovandClaude Opus 5.5 0b9a9593d3 fix(tool-gateway): advertise only the MCP capabilities the gateway token may list (#19279)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A named MCP gateway gives a client its assigned tools. Each gateway
token has a list of allowed actions
> - A native run token allows only `tools/list` and `tools/call`
> - `initialize` always advertises the `tools`, `resources` and
`prompts` capabilities, whatever the token allows
> - A client that follows the advertised capabilities calls
`resources/list` and `prompts/list`, and the gateway refuses both with
403 `gateway_token_action_denied`
> - This pull request makes `initialize` advertise only the capabilities
whose list action the token allows
> - The benefit is that a client is not told to call methods that its
token cannot use

## Linked Issues or Issue Description

This continues #17688, which made the same change and was closed
unmerged by its author on 2026-11-02 ("we are no longer contributing
upstream"). That pull request reports that Claude Code requests every
advertised capability at once, drops the HTTP connection after the first
403, and then has no gateway tools for the run. I did not reproduce the
Claude Code part. The gateway part is below.

I searched pull requests for "allowedActions tools/list", "gateway token
allowed actions context tools", "paperclip_list_resources",
"gatewayTokenAllowedActions", "gateway_token_action_denied", "MCP
gateway capabilities resources prompts", "initialize capabilities token"
and "named gateway token actions discovery", and issues for
"paperclip_list_resources" and "gateway token allowedActions". Results:

- #17688 (closed, not merged): this change.
- #14427 (open): filters the four `paperclip_*` context tools in
`tools/list` by the token's allowed actions. It does not change
`initialize`, so the two pull requests do not overlap. They touch
different handlers in server/src/routes/tool-gateway.ts.

No issue matched.

**What happened?**

On master, the `initialize` handler in server/src/routes/tool-gateway.ts
returns `capabilities: { tools: {}, resources: {}, prompts: {} }` for
every token. For a token that allows only `tools/list` and `tools/call`,
a `resources/list` call returns 403 with reason code
`gateway_token_action_denied`.

**Expected behavior**

`initialize` advertises `tools`, `resources` or `prompts` only when the
token allows `tools/list`, `resources/list` or `prompts/list`. A session
with no token action list keeps all three.

**Steps to reproduce**

1. Create a named gateway and a token with `allowedActions:
["tools/list", "tools/call"]`.
2. Send `initialize` with the token. On master the result advertises
`resources` and `prompts`.
3. Send `resources/list` with the same token. The gateway returns 403
`gateway_token_action_denied`.

**Paperclip version or commit**

master at 499712c68b.

**Deployment mode**

Any deployment whose MCP clients use named gateway tokens with
restricted actions, including native runs.

## What Changed

- server/src/routes/tool-gateway.ts: `initialize` reads the session that
`initializeNamedGatewayProtocol` already returns, and advertises each
capability only when the token may list it.
- server/src/__tests__/tool-gateway.test.ts: a new test sends
`initialize` with five tokens (tools only, tools and prompts, tools and
resources, no list action, and no action list) and checks the exact
capabilities. It also checks that the tools-only token gets 403
`gateway_token_action_denied` on `resources/list`.

## Verification

At head e40a1f7d6e, run from server:

```sh
pnpm exec vitest run src/__tests__/tool-gateway.test.ts src/__tests__/tool-access-service.test.ts src/__tests__/tool-gateway-discovery-http.test.ts src/__tests__/mcp-http.test.ts src/__tests__/public-mcp.test.ts src/__tests__/tool-grant-scope.test.ts
```

- With server/src/routes/tool-gateway.ts from master and the test file
from this branch: 1 failed, 706 passed. The new test fails at its first
assertion: the tools-only token gets the tools, resources and prompts
capabilities, where the test expects only tools.
- With this branch: 707 passed.
- Typecheck: `pnpm --filter @paperclipai/server typecheck` reports 29
errors, the same 29 as unmodified master ("Cannot find module
@paperclipai/plugin-sdk" in the sandbox provider plugin packages of my
local install). None is in a touched file, and this pull request adds
none.
- The full upstream test suite runs in CI. I ran only the test files
named above.

## Risks

- Low risk. Only the `initialize` response changes, and only for tokens
that cannot list resources or prompts. Those methods were already
refused for such tokens.
- A client that called `resources/read` or `prompts/get` without listing
first no longer sees the capability advertised for a token that allows
only the read or get action.

## Model Used

- Claude, claude-opus-5-5, with tool use and terminal code execution.
The route change follows #17688.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:52:08 -06:00
DottaandClaude Opus 5.5 8a0f95849d perf(native-runtime): start the first turn without re-copying launch files, and never fail a run on a slow copy (#21287)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode in a remote sandbox must have a durable,
verified copy of the provider launch files before their first turn (the
rule from #19622)
> - Today each new run copies these files out of the sandbox, uploads
them and verifies them again, even when the company already has a ready
artifact with the same files
> - The whole copy runs under one five-minute clock. When eight runs
start together on small sandboxes, the copies compete, and slow copies
fail the run before its first turn
> - This makes cold starts slow (p50 273 s from run start to turn
submitted) and makes half of the first runs fail in a burst
> - This pull request reuses a ready artifact with the same digests,
shares one in-flight copy per company and manifest, and bounds each
network step instead of the whole copy
> - The benefit is that first turns start in about one minute on a cold
burst and in about 20 s when the artifact is already ready, and a slow
copy no longer fails the run

## Linked Issues or Issue Description

Refs #19622

**What happened?**
When 8 OpenCode runs started together on 1 vCPU Daytona sandboxes, 4 of
8 first runs failed before their first turn. Each run copied, uploaded
and verified the same provider launch files. The single 300 s clock over
the whole copy expired. Run start to turn submitted was p50 273 s and
p95 314 s.

**Expected behavior**
A run starts its first turn without a new copy when a ready, verified
artifact with the same files exists. A slow copy does not fail the run.

**Steps to reproduce**
1. Use a company with remote OpenCode agents on Daytona sandboxes with
durable storage.
2. Start 8 runs at the same time on new sandboxes.
3. See the time before the first turn and the failed runs.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- `retainReadyNativeProviderArtifacts` (`native-provider-artifacts.ts`):
reuse a ready, verified company artifact with the same paths and SHA-256
digests through a lookup only. No file is copied, uploaded or verified
again. The lookup (`readyNativeProviderArtifactCandidates`) matches
target, paths and digests in Postgres with jsonb containment on the
company's ready rows (existing partial index), newest first, limit 4. It
does not load or hash other retained versions. No migration.
- `shareNativeProviderArtifactCapture`
(`native-provider-artifact-capture.ts`): one in-flight capture per
company and manifest digest per server. Other runs wait for it, then
reuse its result.
- The single 300 s clock over the whole capture is replaced by a bound
for each network step (upload, verifying download): max(120 s, size at
256 KiB/s). A timeout has the typed cause
`native_provider_artifacts_transfer_timeout`. The capture makes up to 3
attempts from the local copy.
- Stop now interrupts the capture and the wait for a shared capture
(`ActiveNativeSession.admissionAbort`).
- `doc/opencode-cold-recovery.md` describes the new behavior.
- Tests: new `native-provider-artifact-gate.test.ts` (7 tests), 5 new
tests in `native-harness-backup-storage.integration.test.ts`, and new
executor variants `opencode-new-durable-slow-capture` and
`opencode-new-durable-ready` in `native-session-executor.test.ts`.

## Verification

Local (this branch, on master 9b613562b7):

- `npx vitest run
src/services/native-runtime/native-provider-artifact-gate.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
3 files, 1511 passed, 7 skipped.
- `npx tsc --noEmit -p server`: pass.
- Lookup scale: the integration test "loads only the ready artifact
whose launch files match, however many other versions the company keeps"
adds 200 ready non-matching versions. It returns only the matching
artifact. Without the SQL filter it fails (4 rows returned).
- Red/green: with master's `native-session-executor.ts` and the new
tests, `opencode-new-durable-slow-capture` fails with `TimeoutError: The
operation was aborted due to timeout`, and `opencode-new-durable-ready`
fails because the ready artifact is not looked up. With this change,
both pass.

Daytona (same commit content, cherry-picked onto the baseline commit
84a1ee3e11 with #21262; image `sha256:9461cc75…`; 1 vCPU sandboxes;
OpenCode):

| Wave | Shape | First runs OK | Run start → turn submitted | Other |
|---|---|---|---|---|
| Baseline `w-20261113T011027-s1702` | 8 runs started together | 4/8 (4
failed) | p50 273 s / p95 314 s | "only a getter" TypeError ×8,
`bootstrap_not_proven_incomplete` ×8 |
| Fix `w-20261113T031045-s1702` | same seed and shape | 8/8 | p50 66.0 s
/ p95 85.7 s | 8/8 tasks hands-off, 0 Retry banners; first agent event
p50 72.9 s / p95 90.0 s; exactly 1 upload, 7 runs waited on it (~65 s
shared capture); all failure codes 0 |
| Fix `w-20261113T034033-s1702` | 16 tasks, cap 8 | 16/16 | p50 19.1 s /
p95 24.3 s | second batch (artifact already ready, cold sandboxes) gate
time 24, 25, 26, 27, 31, 31, 45, 119 ms; first agent event p50 25.9 s |

## Risks

- The rule from #19622 still holds: a durable, verified artifact exists
before the first turn. Reuse only accepts a ready, verified artifact of
the same company with the same paths and digests.
- The first run per company and manifest per server still pays the full
transfer.
- Two servers can still upload duplicate artifacts for the same company
and manifest. Sharing is per server.
- No test drives Stop through the executor during a capture. Signal
handling is unit-tested only.
- A capture that fails all 3 attempts still ends blocked with a working
Retry. A follow-up PR (pre-turn failures continue automatically) removes
that.
- The documented five-minute deadline per capture is replaced by bounds
per network step. A very large artifact on a slow link can take longer
than five minutes in total, but each step still has a bound.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Default context
window; extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:48:55 -06:00
DottaandClaude Fable 5.2 5d298b1b48 test(server): judge the Linux real-process boundary cases only by the processes they start (#21284)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In `local_trusted` mode, the operator's local processes get implicit
board access. The local board boundary refuses that access to processes
of an agent run
> - #21209 added a Linux rule for a connection whose owner the server
cannot inspect. The server keeps board access only if every same-user
process it cannot inspect belongs to the operator
> - The real-process test "keeps board access for an operator process
the server cannot inspect" used that rule on the whole CI host. Other
processes of the same user on the host can be uninspectable and not the
operator's
> - Since #21209 merged, this test failed in about one in four `PR`
runs, on PRs that did not touch it. 22 failed server-test jobs between
2026-11-12 23:05Z and 2026-11-13 03:06Z
> - This pull request makes the real-process cases judge only the
processes that each case starts. It adds a case that pins the host-wide
rule with a known uninspectable agent-run process
> - The benefit is that the test is stable on shared CI hosts, and the
product rule stays tested

## Linked Issues or Issue Description

Refs #21209

**What happened?**
`server/src/__tests__/local-agent-process-boundary.test.ts` failed in
the "General tests (server)" lane of unrelated PRs. Both cases "keeps
board access for an operator process the server cannot inspect"
(127.0.0.1 and ::1) received `{ type: "none", source: "none" }`. They
expected `{ type: "board", source: "local_implicit" }`. The failures
started with the first runs that included #21209 (merged 22:52Z). They
continued until at least 03:06Z. Two runner image versions failed in the
same way.

**Expected behavior**
The test passes on a CI host that runs other processes.

**Actual behavior**
The boundary looks at every same-user process on the host that it cannot
inspect. When the owner of a connection is hidden, one uninspectable
process that is not clearly the operator's makes the boundary refuse.
Examples are a process of another agent run, a process whose parent is
PID 1, or a process that exits before the server reads its parent chain.
The test did not control which processes were on the host.

**Steps to reproduce**
On Linux, as a non-root user, run `python3 -c 'import ctypes,time;
ctypes.CDLL(None).prctl(4,0,0,0,0); time.sleep(60)' &`. Then make the
operator request from the test. The boundary refuses it.

## What Changed

- The Linux real-process cases use an inspector that wraps the real
Linux inspector. For a hidden owner, it keeps only the candidate
processes that descend from the process that the case started. All other
parts are the real Linux inspector: the TCP table reads, the fd scan,
the recheck, and the ancestry reads.
- Each Linux block starts an idle, uninspectable process with
`PAPERCLIP_RUN_ID`. This process stands for another agent run on the
host.
- A new case, "refuses board access to an operator process the server
cannot inspect while another agent run's process it cannot inspect is
running", uses the unwrapped inspector. It pins the host-wide rule in a
real-process test.
- If the test runner can inspect every process (for example, as root or
with `CAP_SYS_PTRACE`), the idle process is not uninspectable. The
host-wide case then skips. The other cases run as before.
- No product code changed.

## Verification

- Red, Linux (Docker `node:24-bookworm`, non-root user): a script runs
the product module
`server/src/middleware/local-agent-process-boundary.ts` from master with
the same operator request as the test. With no other process, the
operator gets board access 10/10. With one idle uninspectable agent-run
process, it gets 0/10. With one uninspectable process whose parent is
PID 1, it gets 0/10. With short-lived uninspectable processes, it gets
8/10. Agent-run requests are refused 5/5 in all cases.
- Red, CI Linux: in this PR's run, the new host-wide case passes. It
shows that the earlier assertion fails when one uninspectable agent-run
process is on the host.
- Green, CI Linux: in the same run, with that process on the host, both
"keeps board access for an operator process the server cannot inspect"
cases pass. Result: 27 tests passed in the shard that ran the file.
- macOS: `pnpm --dir server exec vitest run
src/__tests__/local-agent-process-boundary.test.ts` gives 15 passed, 12
skipped (Linux-only).
- Not verified: the specific process that was on the CI host in each
failed run. One fresh CI run with a temporary diagnostic found no other
uninspectable process. That run passed. The diagnostic is not in the
final commit.

## Risks

- Low risk. The change is test-only.
- Product note for a maintainer, not changed here: on Linux, an operator
process that the server cannot inspect loses implicit board access when
any other uninspectable same-user process on the machine is not clearly
the operator's. A process of another agent run is an example. This is
the rule from #21209, and the new case pins it. If this is too strict
for operators, that needs a product decision.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, Docker on Linux, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (not
applicable: test-only)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:48:46 -06:00
DottaandClaude Opus 5.5 0c04ff76eb fix(native-runtime): a follow-up that arrives as a warm session closes continues the task (#21279)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent session on a remote sandbox stays warm after a run,
and an idle timeout closes it after about 5 minutes
> - When a person's follow-up comment arrived just as that idle close
began, the new run failed with `native_session_supervisor_busy`
> - The idle close of the finished owner could also fail its final
checkpoint with `native_runner_checkpoint_owner_lost`, because the task
already pointed at the queued follow-up run
> - Then the retry was held for workspace repair and the task stayed
blocked; the environment could not be deleted (409) because its warm
runner could not be stopped
> - This pull request makes the follow-up wait for the idle close that
is in flight, and lets the finished owner save its final checkpoint
while its successor still waits in the queue
> - The benefit is that a follow-up that lands at the warm session's
idle close continues the task with no person involved

## Linked Issues or Issue Description

Refs #18011 (the checkpoint ownership fence this change extends).

**What happened?**

A follow-up comment arrived 295–340 s after the previous run ended,
while the warm session's idle close was in progress. The new run failed
with `native_session_supervisor_busy`. In another case, the idle close
failed its final checkpoint with `native_runner_checkpoint_owner_lost`,
the next run failed
`native_instruction_materialization_handoff_unavailable`, and the retry
was cancelled with `execution_reconciliation_required`. The task stayed
blocked. A hosted instance showed the same class as "previous run's
session on this sandbox could not be retired".

**Expected behavior**

The follow-up run waits for the idle close (inside the existing bounded
release wait) and then starts. The finished owner saves its final
checkpoint and closes. The task completes hands-off.

**Steps to reproduce**

1. Run an opencode agent on a Daytona sandbox with a warm session (idle
timeout 300 s).
2. Let a run finish.
3. Post a comment on the task 298–306 s after the run ended.

**Paperclip version or commit**

master `444dc18d98`.

**Deployment mode**

Local server with Daytona remote sandboxes (restart-chaos harness, light
lane). The same class was seen on a hosted staging instance.

## What Changed

- `claimWarmNativeInstructionCopy` now also waits when an idle close is
in flight for the session scope (`closingWarmNativeSessions`). The wait
uses the same deadline as the existing release wait
(`warmSessionReleaseWait`), so it adds no new wait and nothing before
`turn.start` on the normal path.
- `validateNativeRunnerCheckpointOwner` accepts the finished owner's
final checkpoint (accepted result, no new work) when
`issues.executionRunId` points at a successor run that is `queued` or
`scheduled_retry` for the same company and task and has no
`native_run_finalizations` row.
- Tests: a server test for a follow-up that arrives while the idle
timeout closes the warm session; four new scenarios (`queued_successor`,
`scheduled_retry_successor` × `runner.suspend`/`turn.stop`) and four new
negatives (`queued_wrong_issue`, `queued_new_work` × both commands) in
the early-retirement checkpoint matrix.
- Restart-chaos harness:
`RESTART_CHAOS_MIX_FOLLOW_UP_AFTER_END_MS=<min>-<max>` and
`RESTART_CHAOS_MIX_FOLLOW_UP_COUNT` post the staging-mix follow-up a set
delay after the task's last run ended, so a cell can aim the follow-up
at the warm idle close. The restart-chaos README describes both
variables. A failed post is retried with the same `clientRequestId`, the
grade check `after_end_follow_ups_served` needs an agent reply after
each post and the ledger marker, and the task budget includes the
planned waits.
- Harness: the wave database cleanup
(`tests/runner-e2e/controller-database.ts`) no longer waits forever when
a group signal (for example `timeout` or Ctrl-C) already stopped the
embedded Postgres. Before, `embedded-postgres` `stop()` waited for an
exit event that had already fired, and an interrupted wave hung after
the Daytona R6 sweep.

## Verification

- Fix 1, red without the product hunk: `native-session-executor.test.ts
-t "idle timeout is closing"` fails with `promise rejected "Error:
native_session_supervisor_busy"`. Green with the fix: 1 passed.
- Fix 2, red without the product hunk:
`native-runner-restart-recovery.integration.test.ts` fails 4 of 208
(`queued_successor` and `scheduled_retry_successor` ×
`runner.suspend`/`turn.stop`) with
`native_runner_checkpoint_owner_lost`. Green with the fix: 208 passed,
14 skipped. The two new negative cases pass in both states.
- Harness hang fix, red without the fix: the new
`controller-database.test.ts` case ("finishes cleanup when the database
already stopped because a group signal reached it") times out on cleanup
(`expected 'hung' to be 'cleaned'`). Green with the fix: 5 passed, 1
skipped.
- After the rebase onto master `7154030ef6`, both server test files
together: 1639 passed, 21 skipped.
- Restart-chaos harness tests (`staging-mix`, `staging-mix-retry`,
`observer`): 75 passed, including a test for the after-end retry, the
`after_end_follow_ups_served` grade check and the added budget. `tsc
--noEmit` passes for `server`, `tests/runner-e2e` and
`tests/restart-chaos`.
- `pnpm verify` exits 2 because `tests/restart-chaos/observer.ts` is a
global CI input. Required CI supplies the broad coverage.
- Daytona cell on this branch (4 opencode tasks, warm sandboxes, 1 vCPU,
follow-up comment posted 300.8–308.2 s after each task's last run ended,
image built for master `444dc18d98`): 4/4 tasks done hands-off, 8/8 runs
succeeded, 0 Retry banners, 0 `owner_lost` / `supervisor_busy` / "could
not be retired" / `execution_reconciliation_required` lines in the
server log, environment delete clean, warm-shutdown grade pass, 0
sandboxes left. The harness exited by itself after 1076 s.
- Same cell before the fix (master, 8 tasks): 1/8 tasks blocked
(`native_session_supervisor_busy` →
`native_runner_checkpoint_owner_lost` →
`execution_reconciliation_required`). With fix 1 only: 1/8 still blocked
(`owner_lost` on the idle close). With both fixes: 8/8 done and 0 failed
runs (cut by the 20-minute cap before cleanup), then 4/4 above.
- The cell grade `T:completion_once` fails on all 4 tasks by design: the
timed follow-up comment arrives after the task is done, so the agent
posts a second completion comment.

## Risks

- Fix 1 can delay a follow-up by up to the existing warm release wait
(30 s) while an idle close finishes. Before this change, that run
failed. If the close fails or the wait expires, the claim returns null
and the run uses a fresh instruction copy, as before.
- Fix 2 widens the checkpoint ownership fence from #18011. It does not
reopen what #18011 closed:
- The exemption applies only to the finished owner's final save: the
checkpoint must hold an accepted result and no new work, so no provider
turn can be granted through it.
- The successor must still be unclaimed (`queued` or `scheduled_retry`)
and must have no native coordinator (`native_run_finalizations`) row. No
controller has started it, so nothing can have consumed or depended on
the predecessor's checkpoint. Once the successor is claimed, the
existing preparing/claimed rules from #18011 apply unchanged.
  - The successor must belong to the same company and the same task.
- Cases that must still reject, with tests in the same matrix:
- `queued_wrong_issue`: a queued successor for a different task is
rejected with `native_runner_checkpoint_owner_lost` and the stored
checkpoint does not change.
- `queued_new_work`: a `turn.start` (new work) while the successor is
queued is rejected with `native_runner_checkpoint_owner_lost` and the
stored checkpoint does not change.
- All earlier rejections stay: unclaimed native coordinator,
dispatching, expired or foreign preparation, foreign native preparation,
wrong task, new work, unaccepted result, replaced controller.
- Not changed: an owner whose close fails still blocks environment
delete (409). That is a separate product decision.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Earlier commits on
this branch were written by the same model in an earlier session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:42:50 -06:00
DottaandClaude Fable 5.2 c4f639ed1b fix(ci): re-run cancelled runs that lost a runner, and let runner-loss evidence win (#21291)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - PR CI runs about 50 jobs per run on AWS spot instances. AWS can
reclaim a machine mid-job, and the job dies.
> - `ci-runner-loss-rerun.yml` re-runs such a run once, automatically,
when every failed job lost its runner.
> - In a 17-hour sample, 42 runs lost a runner and were still the PR's
current head. The automation rescued only 14.
> - Two gaps caused most of the misses. A spot reclaim on RunsOn v3.2.3
usually ends the job, and the run, as `cancelled`; the workflow and
script only look at `failure`. Also, the Runner support suite always
carries `gateway_indexing_incomplete` error annotations from a passing
test, so a lost runner there reads as a real failure.
> - On-demand fallback for the fleet was turned off on 2026-11-13, so
reclaims will happen more often. About 20% of runs are expected to lose
at least one job.
> - This pull request makes the re-run act on cancelled runs and jobs,
but only with positive runner-loss evidence, and lets that evidence win
over other error annotations.
> - The benefit is fewer manual "Re-run failed jobs" clicks: an
estimated 34 of 42 instead of 14 of 42 in the sample.

## Linked Issues or Issue Description

No public issue exists.

**What happened?**
Run
[43285064435](https://github.com/paperclipai/paperclip/actions/runs/43285064435)
lost the runner of `ci / General tests (server (8/12))`. The job
concluded `cancelled` (its log has `##[error]The runner has received a
shutdown signal`). The run concluded `cancelled`, so the re-run
workflow's `if:` skipped it.

Run
[43325855069](https://github.com/paperclipai/paperclip/actions/runs/43325855069)
lost the runners of `ci / Runner support suite` and `ci / General tests
(server (2/12))`. Both logs have the shutdown signal. The re-run bot
([decision
log](https://github.com/paperclipai/paperclip/actions/runs/43327873358))
said "1 of 1 failed jobs did not lose their runner", with evidence
`gateway_indexing_incomplete`. It ignored the cancelled job. It read the
annotation as a real failure.

**Expected behavior**
Both runs get one automatic re-run of their failed jobs.

**Steps to reproduce**
The new tests replay trimmed snapshots of both runs
(`.github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json`
and `run-43325855069.json`) through the script.

In a 17-hour sample on 2026-11-12, 42 runs lost a runner and were still
the PR's current head. Only 14 were re-run automatically. 16 stayed red
because the run concluded `cancelled`. 4 stayed red because an error
annotation in the lost job was read as a test failure.

**Additional context**
The `gateway_indexing_incomplete` annotation does not come from the
shutdown. `scheduled-gateway-indexing.test.ts` prints `::error` lines
while it passes, 84 seconds before the shutdown signal in this run.
Attempt 2 of the same run passed with the same two failure annotations.
So, before this change, a lost runner in the Runner support suite could
never be re-run.

## What Changed

- `ci-runner-loss-rerun.yml`: the job also runs when the PR run
concluded `cancelled`. The trigger type, `permissions`, the repository,
path, event and attempt-1 conditions do not change.
- `runner-loss-rerun.mjs`, `eligibleRun`: accepts `completed/cancelled`
as well as `completed/failure`.
- `runner-loss-rerun.mjs`, `decide()`: classifies jobs that concluded
`cancelled` as well as `failure`. It still skips the `ci / verify` and
`ci / e2e` aggregates, and a failed or cancelled routing gate still
blocks the re-run.
- `decide()` now runs `staleReason` (open PR, current head, no newer
run) before it reads any annotation or log, and again just before the
re-run, as on master. A run cancelled by a new push has about 40
cancelled jobs; it now costs two API listings and no log reads. The
final check closes the window where a push lands during the log reads: a
re-run of the old run would join the PR's `cancel-in-progress`
concurrency group (`pr-<number>` in `pr-trusted.yml`) and cancel the new
commit's CI.
- Following CONTRIBUTING.md ("Do not submit a PR that includes code
comments"), this PR adds no new code comments. The reasons for each rule
are in this description and in `doc/DEVELOPING.md`.
- `classifyJob`: runner-loss evidence wins. A lost-communication
annotation means lost. A cancellation annotation plus a shutdown signal
in the log means lost, even if the job also has other error annotations.
A cancelled job without this evidence (for example, a person cancelled
the run) counts as a real failure and blocks the re-run.
- Fixtures: two new trimmed snapshots from the example runs (jobs with
`id`, `name`, `status`, `conclusion`; failure annotations; `##[error]`
log lines only). The two older fixtures get the real annotations and log
error lines of their cancelled jobs. Those jobs were spot-reclaimed too
(each log has the shutdown signal), but the old code ignored them.
- `doc/DEVELOPING.md`: the runner-loss paragraph describes the new
rules.

## Verification

- Red first. Against the unchanged script, the new tests fail for the
reasons above: 43285064435 gives `run is completed/cancelled, not
completed/failure`; 43325855069 gives `1 of 1 failed jobs did not lose
their runner` with evidence `gateway_indexing_incomplete`, the same
decision the bot logged.
- Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`:
25 of 25 pass.
- Push during log reads: the PR head moves after the first log read.
With the final freshness check removed, this test fails (the script
re-runs); with it, there is no re-run and the reason is `no open pull
request still has this run's head commit`.
- Guard cases, all with no re-run and no POST:
- Superseded run: 52 jobs cancelled, the PR head moved. The test asserts
zero annotation and log reads.
- Person cancelled the run: cancelled jobs with only "The operation was
canceled." in the annotation and log, and some never-started jobs with
no annotation.
- Mixed: one cancelled lost job and one real test failure (`1 of 2
failed jobs did not lose their runner`).
- Once only: a cancelled run on attempt 2 is refused after one API read.
The workflow condition test still requires `run_attempt == 1`.
  - Cancelled routing gate: left for a person.
- Policy job command: `node --test '.github/scripts/tests/*.test.mjs'`:
806 of 806 pass.
- `pnpm verify`: exit 2 ("Global CI input changed:
.github/scripts/runner-loss-rerun.mjs"). It selects no local Vitest
files and asks for required CI coverage. I did not run `pnpm verify
--full`; required CI on this PR supplies that coverage.
- This workflow runs on `workflow_run` from the default branch. This
PR's own CI cannot exercise the new behavior. The fixture tests are the
evidence. I did not dry-run the script against live runs.

## Risks

- A test that really fails, and then a spot reclaim of the same job a
minute later, now gets one wasted re-run. Attempt 2 then fails for real.
It cannot loop: the workflow `if:` and `eligibleRun` both accept attempt
1 only.
- A run a person cancels on the current head now starts the re-run job.
The job reads the annotations, and the log of each job with a
cancellation annotation, and then does not re-run. This costs some API
calls on a GitHub-hosted runner.
- A concurrency cancel by a label event that does not request CI (the
newer run is `skipped`) passes the head check. Its cancelled jobs have
no runner-loss evidence, so it is not re-run.
- Trust: no change to `permissions`, the trigger, the accepted
repository, event, workflow path, attempt, or the routing-gate rule. A
cancelled run reuses its attempt-1 gate output, the same as a failed run
does today.
- Estimated effect, from a 17-hour sample: 14 of 42 rescued before; 16
cancelled runs and 4 runs with error-annotation noise become eligible
(about 34 of 42). The other 8 stay skipped on purpose (a real test
failure in the same run, a person re-ran first, or the PR closed).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent. Model ID claude-opus-5-5, with
tool use (shell, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:31:50 -06:00
Awssandcommitperclip[bot] af1c685f5f fix(secrets): return 422 for secret proposals missing a required field (#15348)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can propose a company secret or a secret binding through
the API.
> - The proposal route reads the request body and passes it to the
service.
> - The service trimmed required string fields before it checked them.
> - A request that omitted name or justification threw a TypeError.
> - The route then returned 500 instead of a validation error.
> - This pull request checks the required fields first and returns 422.
> - The benefit is that a client can tell a bad request from a server
fault.

## Linked Issues or Issue Description

Closes #15334

## What Changed

- Check `name` and `justification` in `createSecret` before `trim`, and
return 422 when either is missing.
- Check `justification` in `createBinding` the same way.
- Keep the existing empty-value and malformed-value checks and the
optional `key` field unchanged.
- Add a regression test that covers the missing-field cases for both
proposal kinds and keeps the complete-body case at 201.

No documentation change is needed. The OpenAPI document already lists
422 for this route, so the server now matches the documented contract.

## Verification

- Run the route suite. It passes.

`./node_modules/.bin/vitest run
server/src/__tests__/secret-proposals-routes.test.ts` gives 26 passed.

- Revert only the service change and run the new test. It fails with
`expected 500 to be 422`, which proves the test catches the bug. Restore
the change and it passes again.

- Run the server typecheck. It passes.

`pnpm --filter @paperclipai/server run typecheck` exits 0 with no
errors.

- A `POST /api/agents/me/secret-proposals` that omits `justification`
returns 422 after the fix and returned 500 before it. A complete body
still returns 201.

## Risks

- Low risk. The change adds two field checks before an existing `trim`
call. Valid requests still return 201. A missing required field now
returns 422, which the route already documents.

## Model Used

Claude Opus 4.8 (`claude-opus-4-8`), with extended thinking and tool
use. I chose the issue, set the evidence bar, directed each step, and
reviewed the code, the tests, and the results before this PR. The model
traced the route and the service, wrote the fix and the regression test,
and ran the checks under that direction.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
2026-11-12 22:31:42 -06:00
Nicky LeachandClaude Fable 5.2 25e9a9c2e9 feat(ci): publish both variants of the Runner image from master (#21230)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Parts 2 and 3 of this stack build the Runner alone as an image with
two variants, and check it on pull requests. Nothing publishes it
> - Downstream sandbox and VM images can copy the Runner only from a
registry, and they must be able to verify where it came from
> - A published image must be exactly the bytes that were checked, and
its name must never point at something else later
> - This pull request publishes both variants from `master` as
`ghcr.io/paperclipai/paperclip-runner`, one time for each distinct set
of inputs, and attests them
> - Nothing consumes the image in this pull request. The application
image build does not change
> - The benefit is one verified, attested Runner that later changes can
copy by digest

## Linked Issues or Issue Description

Refs #21204 (part 2 of this stack) and #21096 (the merge-queue image
build). No issue exists for this change.

**Problem or motivation**

The Runner image exists only inside a pull-request check. A consumer
cannot pull it. There is no rule for which commit publishes an image, no
protection for its tags, and no way to verify a copied Runner.

**Proposed solution**

Publish from a `master` push in this repository, and from nothing else.
Build only when no image has the digest of the commit's inputs. Let the
build legs hold no write access. Push the bytes that the legs checked by
digest, attest each multi-platform index, and write the tags last. Write
the full tag after the slim tag, so that the full tag means "both
exist".

**Alternatives considered**

- Publish from `docker.yml`. That workflow holds one concurrency slot
for each ref, and the Runner image needs its own signer identity, so
that its attestation cannot stand in for the application image's.
- Resolve the lockfile in the workflow before a `master` build, as
`docker.yml` does. Then the published Runner depends on the state of the
npm registry at run time, and the digest cannot be computed again from a
commit.
- Hand each variant to the publish job as its own OCI archive. The two
archives hold the two shared layers two times: 1.27 GB for each
architecture in place of 0.85 GB.
- Write the full tag first. Then a consumer can see a full image whose
slim image does not exist yet.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 4 of 4. Each part is on top of the one before it:

1. #21228 (merged): an optional field in the provider pack manifest
2. #21204: build and check the Runner image, with no publication
3. #21229: layers by change rate, and the slim variant
4. #21230: publish both variants from `master` (this pull request)

## What Changed

- `.github/workflows/runner-image.yml`: adds the trigger `push` to
`master` and the job `publish`.
- `identity` checks the committed lockfile against the Runner's
manifests, computes the digest in the build context, and asks the
registry whether these inputs are published.
- `build` is the job of part 3. It also records what it built, on every
event. On a canonical `master` push it uploads that. It cannot write to
the registry on any event. The only token in the job can read packages,
and only the lookup step receives it.
- `publish` runs only for a `master` push in this repository. It builds
nothing and runs no dependency code.
- Lockfile rule. A `master` push builds from the committed
`pnpm-lock.yaml` and resolves nothing, so the digest is a function of
the commit. When the committed lockfile is behind the manifests that the
Runner build reads, the run writes a notice, publishes nothing, and
stays green. The commit that merges the lockfile refresh publishes the
image. Pull requests and manual runs resolve the lockfile first, always
build, and never publish.
- Hand-off. Each leg joins its two OCI archives into one OCI layout that
names the images `slim` and `full`. The layout stores the two shared
layers one time. `record` then reads the layout back, each image by its
name, and checks every blob against its digest. The leg's record binds
the layout to both image digests, the architecture, the inputs digest,
and the commit. Those digests are the digests of the archives that part
3 unpacks and checks, so the tree that is checked and the image that is
pushed are the same bytes. The publish job accepts only records for its
own inputs and commit, and only a layout that still has the recorded
bytes.
- `record` runs on pull requests too. Only the upload is for a `master`
push. A pull request therefore runs the hand-off on real build output.
- Publication order. The publish job pushes the four images by digest
(`skopeo copy --preserve-digests`). It makes a candidate index for the
slim variant, then one for the full variant that names the slim index in
the annotation `io.paperclip.runner.slim-digest`. It verifies that each
index names exactly the recorded images, attests both, and then writes
`inputs-<digest>-slim` and, last, `inputs-<digest>`.
- The full tag is the completion marker. Every "is it published" lookup
reads the full tag. `publication-state` reports one of:

  | State | Registry | What a run does |
  | --- | --- | --- |
  | `absent` | Neither tag | Builds and publishes |
| `incomplete` | The slim tag alone | Builds and publishes. It replaces
the slim tag with its own image, then writes the full tag |
| `published` | The full tag, and the slim tag at the digest that the
full index names | Nothing |
| broken | The full tag, and a slim tag that is missing or points at
another image | Fails, with the steps for an operator |
| `unknown` | No clear answer | Builds. The publish job fails before it
writes a final tag |

- Recovery rule. A run that stops between the two tags leaves the state
`incomplete`. It does not block those inputs. The next run never keeps
the earlier slim image: it writes its own slim tag immediately before
its full tag, verifies both against its own index digests, and then
reads the pair back as every later run will see it. The broken state
cannot be repaired by a run, because no run moves a full tag. An
operator must delete the two package versions and re-run the workflow
run of a `master` push with those inputs.
- Pairing rule for consumers, in the contract: resolve the full tag
first, take the slim digest from the annotation of the full index, and
pull the slim image by that digest.
- `verify-index` accepts `linux/arm64` with the variant `v8`, which some
BuildKit versions record, and no other variant.
- `scripts/runner-image-publish.mjs` (533 lines): the commands
`lock-state`, `record`, `verify-legs`, `publication-state` and
`verify-index`. It reads archives with the reader of
`scripts/runner-image-oci.mjs` (607 lines here), which can now read one
named image of a layout. `lock-state` refuses a checkout, because pnpm
can rewrite the lockfile of the tree that it checks. The workflow gives
it an export that no build reads.
- The workflow test pins the build arguments of the image build. Build
arguments are not part of the image identity.
- `doc/runner-image-contract.md` (197 lines): the section "Publication":
the lockfile rule, the order, the pairing rule, the recovery rule, the
create-only tags, the internal `candidate-` tags, the command that
verifies provenance, and the step to make the package public.
`doc/DOCKER.md` says where the image is published.

## Verification

Tests, and the build jobs of the workflow on GitHub. Nothing in this
pull request ran against a registry, and no publish job has run.

- Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and
zstd): `runner-image-publish.test.ts` (75),
`runner-image-workflow.test.ts` (12), the three other Runner image test
files, the provider pack archive tests and the existing Dockerfile
tests: 372 tests pass in 10 files. The same files pass on macOS.
- The tests of the leg record failed on Linux in the first runs of this
pull request: their fixtures named the archive members `./index.json`,
which bsdtar finds and GNU tar does not. The fixtures now name the
members as buildx does, and the publish script uses the one reader of
part 3.
- Leg records: both variants are accepted for this run's inputs and
commit; other inputs, another commit, a changed layout, a record without
a variant, an archive that lacks a blob, a full image that does not
start with the slim image's layers, and a build that is not a
publication build are refused.
- Publication state: all five states, and the half-finished publication
end to end. A slim tag alone lets the next run publish. A full image
that is left with another run's slim tag is reported as broken.
- Index: exactly the recorded images of one variant; the full index must
name its slim index, and the slim index must name none; `v8` is accepted
on arm64 only.
- Registry lookups: a missing tag only when the registry says so; every
refused, failed, or unreachable request is `unknown`.
- Workflow: job permissions; the publish job and the upload are behind
the canonical `master` push; the build steps receive no credential; the
full tag is written last, after a strict lookup and both attestations;
one export and a time limit for each build command; the two build
arguments; actions pinned by commit; no expression inside a shell
script.
- `record` on the two OCI archives of a real `linux/arm64` build from
part 3: the joined layout is 850,149,376 bytes, and both images read
back by name with every blob under its digest. The two archives together
are 1,273,646,592 bytes. The image digests in the record are the digests
that `unpack` reports for the trees that were checked.
- `tests/runner-e2e` workflow tests: 39 pass. `node --test
.github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image
contract scripts: 33 pass. `docker buildx build --check` reports
nothing. The image identity is the same as in part 3
(`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`),
because this pull request changes no input of the image.

GitHub, at this head:

- The `Runner image` workflow passes with these changes: `Build and
check (amd64)` in 18m30s and `Build and check (arm64)` in 7m48s. Each
leg builds both variants, checks them, and writes the leg record.
`Publish and attest` is skipped, as it must be on a pull request.

Never run:

- The publish job. It runs only for a `master` push.
- The publish path: the registry lookups with a workflow token, `skopeo
copy` from `oci-archive:<layout>:<name>` by digest, `docker buildx
imagetools create` with index annotations, and the tag writes.
- The attestations, and the `gh attestation verify` command in the
document.
- `actionlint`. It is not installed on the local host.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- **Manual step.** GitHub creates the `paperclip-runner` package private
on the first publication. A maintainer must make it public one time.
Until then, consumers cannot pull the image without credentials.
- The first `master` runs of the workflow are the first test of the
publish path. A failure before the tags publishes nothing. A failure
between the two tags leaves the `incomplete` state, and the next run
with those inputs repairs it.
- A slim tag can move while its publication is incomplete. A consumer
must resolve the full tag first and take the slim digest from the
annotation of the full index. It must never resolve
`inputs-<digest>-slim` by tag on its own. The contract says so.
- `candidate-` tags are written before the attestations and can name an
index that is not attested. They are internal.
- The broken state needs an operator, and it fails every `master` run
with those inputs until then. It can arise only when someone changes or
deletes a tag outside the workflow.
- **Known residual.** Two `master` runs that both start before a new
digest is published can each build the image. Each build leg and the
publish job look again before they work, and the publish job runs one at
a time for each digest. The cost is runner time, not correctness.
- A commit whose lockfile is behind the Runner's manifests has no
published Runner image until the lockfile refresh merges. A lockfile
that the refresh cannot fix leaves every `master` run green with only a
notice, for as long as it lasts.
- The tags are create-only because the workflow never moves a tag of a
complete publication. The registry does not enforce that. Consumers must
pin the digest and verify the attestation.
- `payload.runnerSourceRevision` in a pack from this image is the commit
that published the image. Code that reads it as the application's commit
must change before it consumes the image.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 20:27:36 -08:00
Nicky LeachandClaude Opus 5.5 717aae060a fix(environments): let same-run recovery create a replacement once its sandbox is proven stopped (#21223)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute on environment leases, and `createOwnedSandbox`
records each sandbox creation so that a lost reply never creates a
second sandbox
> - The record key is company, environment, run and request key. The
plugin request key is the environment fingerprint, so it is the same for
every attempt of one run
> - A same-run recovery can need a new environment after the first lease
is closed with successful cleanup and a termination receipt
> - The old record is still `attached` to the closed lease, so the
recovery fails with "Sandbox creation has already transferred ownership"
before the provider is called
> - This pull request lets the creation fence accept the same stop proof
that recovery already requires. A creation record whose sandbox is
proven stopped becomes resolved, and a resolved record no longer holds
its key
> - The benefit is that same-run recovery gets one replacement sandbox,
while unconfirmed cleanup still blocks a second sandbox

## Linked Issues or Issue Description

Fixes #21214

Refs #20755. That pull request fixes the same failure with a different
rule. It retries under a new key generation (`<key>#2`, `#3`, ...) when
the recorded lease has ended, and it accepts any ended status, including
`failed`, without a termination receipt. A lease whose cleanup is not
confirmed can then produce a second sandbox while the first one can
still run. This pull request keeps the original key and requires the
same stop proof that Core uses elsewhere
(`hasRemoteTerminationReceipt`). It does not add a key generation.

## What Changed

- `server/src/services/sandbox-creation-operations.ts`: when
`createOwnedSandbox` finds the key held by an `attached` operation whose
lease is not active, it checks the same stop proof that same-run
recovery requires before it acquires:
`hasRemoteTerminationReceipt(resolveRemoteTerminationLease(lease))`, for
the sandbox that this operation created (`creationProviderLeaseId`, or
the lease's `providerLeaseId`).
- With that proof, it runs the same admission transaction one more time.
That transaction first marks the old operation `resolved` (`WHERE id AND
state = 'attached' AND lease_id`), and then inserts the new operation.
  - Without that proof, it throws exactly as before.
- The check covers every source of a stop receipt: `releaseLease`, the
Runner `complete` operation (which writes receipts with direct SQL for
reusable leases), stop-and-retain settlement, and the receipt that
`resolveRemoteTerminationLease` derives for an expired reusable lease.
- A task replacement can leave an earlier operation attached to a lease
that now holds another sandbox. That operation is not proven and keeps
its key.
- Concurrent recoveries serialize on the row update. The second one
updates nothing, finds the first one's unresolved operation through the
partial index, and replays or refuses as before. There is one provider
call.
- Migration `0475_sandbox_creation_owner_unresolved.sql` recreates
`sandbox_creation_operations_owner_idx` under the same name as a partial
unique index (`state <> 'resolved'`). The drop and the create are in one
migration transaction. The Drizzle schema matches.
- The conflict lookup in `createOwnedSandbox` ignores resolved rows. An
active attached lease is still replayed. Pending and
`needs_reconciliation` rows still refuse. `releaseLease` does not
change.
- `doc/DATABASE.md` and `doc/idle-environment-cleanup.md` describe the
operation lifecycle.

## Verification

- `pnpm --filter @paperclipai/db generate`
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/server typecheck`
- `vitest run src/migration-snapshot-drift.test.ts` (db)
- `vitest run src/__tests__/sandbox-creation-operations.test.ts`: 48/48.
With the service change reverted, the 5 replacement tests fail.
- The 9 other server test files that read `sandbox_creation_operations`,
plus `environment-runtime.test.ts`: 564/564
- `pnpm check:migrations` passes

New tests in `sandbox-creation-operations.test.ts`:

- A stop receipt from `releaseLease` (sandbox driver, and the plugin
driver through `acquireRunLease` in the #21214 sequence), from the
`complete` operation's direct write on a retained reusable lease, and
derived only from the destroyed successor of an expired reusable lease
each let the next creation resolve the old operation and create exactly
one replacement. A retry replays that replacement. These tests fail
before the fix.
- Failed cleanup, success without a receipt, a receipt for another lease
or another provider task, a retained lease, and pending cleanup each
still throw "transferred ownership" with no provider call.
- An earlier task's operation that a task replacement left attached
keeps its key.
- Two concurrent recoveries make one provider call. This test fails
before the fix.
- Reapplying migration 0418 keeps the partial index.

I did not run `pnpm verify` or the full suite locally. CI runs them.

## Risks

- Migration: the drop and the rebuild of the index take an exclusive
lock on `sandbox_creation_operations`. The table has one small row for
each sandbox creation, so the lock is short.
- Rolling deploy: the migration is marked `online_compatible`. The
previous release uses an untargeted `ON CONFLICT DO NOTHING`, which also
matches a partial unique index. Its conflict lookup has no state filter,
so it can find a resolved row and throw. That fails closed and never
creates a second sandbox.
- A legacy runner-task operation without `creationProviderLeaseId`
matches any lease it is attached to. Runner-task operations record this
field before they attach, so this is only a theoretical gap.
- An operation stays `attached` until the next creation with the same
key checks it. This is the same as before when no recovery comes.
- Stop-and-retain settlement is not tested directly. The shared receipt
check covers it.
- The fence is unchanged for every case without a stop receipt.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents) and extended thinking. One sub-agent
wrote the change. A second session reviewed it before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:24:34 -08:00
DottaandClaude Opus 5.5 4dc5492c2d fix(recovery): retry a failed Retry run's sandbox resume failure when the sweep reaches it first (#21124)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on its next run
> - When the resume fails before provider work
(`reusable_sandbox_resume_failed`), the run gets a bounded automatic
retry
> - On a hosted staging server, 8 of 14 such failures in 10 hours did
not get that retry. Each task ended `blocked` and needed a person to
press Retry
> - In all 8, the stranded-task sweep blocked the task between the
failed write and the run's own retry scheduling. The block cleared the
claim that the retry needed
> - This pull request makes the sweep give such a run to the same
bounded retry scheduler, not escalate it
> - The benefit is that a transient sandbox resume failure no longer
costs the task, also when settlement is slow or the server stops during
an upgrade

## Linked Issues or Issue Description

Refs #21108, #21113, #21078

**What happened?**
A person pressed Retry on a task. The run tried to resume the task's
saved Daytona sandbox before provider work. The resume failed
(`cause=transport_timeout`, or `provider_unavailable status=503`), and
the run was written `failed` with `reusable_sandbox_resume_failed`. Then
the stranded-task sweep moved the task to `blocked` and cleared its
claim. This happened 1 to 43 seconds after the failed write. The run's
executor scheduled its bounded retry 2 to 81 seconds after the failed
write, and the gate refused it ("The automatic retry could not
revalidate the original user continuation."). Read-only audit of 14
failures (all legacy runtime, `executionRecovery: bootstrap,
providerWorkStarted: false`): 6 got their retry, and the 5 most recent
ones recovered. 8 were refused this way and wait for a person. In one
example the run was written failed at 14:23:31, the task was blocked at
14:24:14, and the retry was refused at 14:24:52. The server was being
replaced at that time.

**Expected behavior**
A transient preparation failure before provider work continues with its
bounded retry. The sweep must not take the claim away from that retry.
When the retry budget is spent, or the continuation is refused, the task
escalates once, as before.

**Steps to reproduce**
1. Press Retry on a task whose reusable sandbox resume fails before
provider work.
2. Let the run's executor be slow to settle the failure (more than the
remaining controller lease, at most 60 seconds), or stop its server
after the failed write.
3. Let the stranded-task sweep run on another server. The task becomes
`blocked`, and no retry is scheduled.

**Root cause**
`classifyContinuationFailure` lists `reusable_sandbox_resume_failed` and
the other transient preparation codes in
`NON_RETRYABLE_CONTINUATION_ERROR_CODES`. The sweep assumes that the
executor already scheduled the dedicated bounded retry, so it escalates
at once and never asks the scheduler. #21108 added a skip while the
executor still settles the run. That skip uses a live local execution or
an unexpired controller lease. The lease stops renewing when the run is
written terminal (renewal requires `status = 'running'`). So the skip
covers at most about 60 seconds after the last renewal, and it covers
nothing when the executor's server stops during settlement.

**Paperclip version or commit**
Staging at `a65644d1e1`. Current `master` has the same code path.

**Deployment mode**
Hosted (cloud), two servers alive during each upgrade.

**Agent adapter(s) involved**
`codex_local` on a Daytona sandbox environment. The change is
adapter-independent.

## What Changed

- `server/src/services/heartbeat/retries.ts`: add
`failedPreparationOwnsBoundedRetry(run)`. It is the persisted form of
the run-completion decision: a failed legacy run with no native session,
bootstrap evidence that provider work did not start, and a transient
preparation code. A restore failure counts only when its owner marked it
retryable. Codes that an accepted interaction retries itself are
excluded.
- `server/src/services/recovery/service.ts`: when the latest run is such
a run, the sweep does not escalate it as non-retryable. It goes to
`scheduleRecoveryRetry`, which is the run's own bounded scheduler. The
scheduler reuses a successor that already exists, so the executor and
the sweep create one successor. The task escalates when the scheduler
reports exhaustion or a refused continuation. When the scheduler
declines the retry for another reason (for example a permanent refusal),
the task escalates once as a non-retryable failure, as before. The
#21108 settling skip still runs first.
- `server/src/services/heartbeat.ts`: pass the predicate to the recovery
service.
- `doc/execution-semantics.md`: document the rule.

## Verification

- New tests in `server/src/services/task-retry.test.ts`, in the staging
shape: a Retry-admitted run failed with
`reusable_sandbox_resume_failed`, bootstrap evidence, and an expired
controller lease from another server.
- "retries a failed Retry run's transient preparation failure when the
sweep reaches it first" (`slow_executor`, `executor_gone`). Red before
the fix: the task became `blocked`, as on staging. Green after the fix:
the task stays `in_progress` with one `scheduled_retry` successor
(`transient_failure`, attempt 1), and dispatch accepts it. For
`slow_executor`, the executor's own scheduling then reuses the same
successor, and there is still one successor.
- "still escalates a failed Retry run's transient preparation failure
once its bounded retries are spent": the task becomes `blocked` with no
successor.
- "escalates a failed Retry run's transient preparation failure once
when the scheduler declines it permanently" (review finding): red before
the follow-up commit (the task stayed `in_progress` with no run), green
after (the task is `blocked` once, and a second sweep adds no notice).
- `pnpm exec vitest run src/services/task-retry.test.ts`: 114/114 pass.
- `pnpm exec vitest run src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/heartbeat-process-recovery.test.ts` (with task-retry): 631
of 633 passed. The 2 failures were 15-second timeouts under parallel
load in native Board-wait tests that this change does not touch. They
passed when run alone.
- `tsc -p server/tsconfig.json --noEmit` passes.
- Not run: a live staging reproduction. The staging evidence is
read-only (run events, task activity, run records).

## Risks

- The sweep can now schedule the bounded retry before the executor does.
The scheduler locks the task and the run and reuses an existing
successor, so there is never a second successor. All continuation gate
checks still apply.
- The sweep acts only after the #21108 settling skip lapses, so a
healthy run is not affected. No wait is added before a normal turn
start.
- No schema change. Rows from older releases follow the same rule
because the predicate reads only persisted fields.
- Out of scope: the provider-side causes (Daytona 503 at 14:21Z, resume
calls that time out). #21113 handles the in-run wait for idle transport
timeouts.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, read-only API reads, Vitest on embedded PostgreSQL) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:20:55 -06:00
Nicky LeachandClaude Opus 5.5 afe86e9ded fix(runtime): record a plugin task's stop receipt when its stop is proven (#21237)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin environment runs each Paperclip Runner turn as a provider
task, and a follow-up run can continue the same Claude/ACPX conversation
from the saved checkpoint
> - Core authorizes that continuation only when the predecessor run's
remote execution has stopped (`remoteExecutionHasStopped`), so two
Runners never write one checkpoint
> - The predecessor's own settlement already proves the stop: the
provider reports `executionStopped`, and Core then completes the task
> - On a per-turn (ephemeral) lease, Core records the stop receipt only
when it releases the lease, and that release can be held for a long time
(for example by a pending workspace export)
> - In that window, an ordinary follow-up fails with `adapter_failed:
Plugin Runner retained session is not authorized`, even though the stop
is already proven
> - This pull request records the stop receipt when the stop is proven,
and the stop check accepts it before the release
> - The benefit is that follow-ups continue as soon as the predecessor
is proven stopped, and the release gates keep their meaning

## Linked Issues or Issue Description

Fixes #21211

Refs #21214 and #21223. Those cover the creation fence for same-run
recovery, which still requires a released lease.

Production evidence for this change: in both observed failures, the
predecessor was a per-turn plugin run. Its `task.settle` completed with
all provider calls successful. Core then did not release its lease for
30 minutes or more, and no release call was made in that time. The
follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint`
→ `resolvePluginTaskResume`) within seconds of admission.

## What Changed

- `server/src/services/environment-task-runtime.ts`
- After a recorded provider stop, `complete` writes the bound stop
receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the
current lease for every lease policy. Before, it did this only for
`reuse_by_environment` leases. Reusable tasks still confirm every owner
and still move retained owners to released.
- New work on a lease (`submit`, `copilot_metadata`, first portable
resume) removes the stop receipt in the same write that clears
`runnerTaskExecutionStopped`. A replacement physical task does not copy
the old receipt.
- `server/src/services/remote-execution-termination.ts`
- New `hasBoundRemoteStopReceipt` checks only the receipt binding
(company, run, lease, provider and physical task).
`hasRemoteTerminationReceipt` uses it and keeps its released-only
semantics. Resource cleanup gates do not change.
- `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a
lease that is released with its receipt, or that is still `active` (not
released) with a bound stop receipt. A lease whose release started and
did not settle does not count. After release starts, the release owns
the receipt.
  - New `remoteExecutionHasReleased` gives the release-based answer.
- `server/src/services/environment-run-orchestrator.ts`: same-run plugin
recovery uses `remoteExecutionHasReleased`. The same-run resume check
after it requires released receipts, so its behavior does not change.
- `resolvePluginTaskResume` keeps its checks and their order. Its
refusal now names the first failing check: `Plugin Runner retained
session is not authorized (reason: <code>)`, for example
`predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The
old message stays as the prefix, and no ids go into it. The
stale-availability recheck in `environment-task-runtime.ts` names its
refusal the same way. Production telemetry for #21211 could not tell
which check fired. The next occurrence will show it.
- `doc/execution-semantics.md` and
`doc/testing-native-remote-continuity.md` describe the stop receipt and
when it is recorded.

Caller audit for the changed meaning of `remoteExecutionHasStopped`:

| Caller | Needs | Decision |
| --- | --- | --- |
| `agent-instruction-working-copies.ts` | stop | Accepts. Directory
cleanup still requires a released "destroyed" receipt. |
| `environment-run-orchestrator.ts` | release | Moved to
`remoteExecutionHasReleased`. |
| `plugin-task-resume.ts` | stop | Accepts. This is the fix. |
| `native-runtime/native-runner-ownership.ts` | stop | Accepts.
`hasPendingRunCleanup` still counts the active lease as pending. |
| `native-runtime/preserved-native-result-finalization.ts` | stop |
Accepts. Workspace recovery has its own gate. |
| `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts.
`hasPendingRunCleanup` still holds active leases. |
| `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop
acknowledgement and quarantined ownership retirement. Holds and blockers
still apply on the normal path. |
| `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors
still fail it. |

## Verification

From `server/`, after `node scripts/agent-preflight.mjs`:

- `npx vitest run
src/__tests__/environment-task-authority.integration.test.ts`: 10/10
pass. The 2 new tests fail without the fix.
- The plugin task and environment runtime suites
(`environment-task-authority.integration`,
`plugin-runner-task.integration`, `plugin-warm-tasks.integration`,
`environment-task-runtime`, `plugin-task-source-recovery.integration`,
`environment-runtime`): 6 files, 366 pass.
- The audited caller suites (`agent-instruction-working-copies`, case
handoff legacy and native, `environment-run-orchestrator`,
`heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`,
`native-remote-retry-admission`, `preserved-native-response-wait`): 9
files, 806 pass.
- `pnpm --filter @paperclipai/server typecheck`: pass.

The new integration tests cover:

- `complete` writes a bound stop receipt on a still-active ephemeral
lease. `remoteExecutionHasStopped` is true and
`hasRemoteTerminationReceipt` is still false.
- `resolvePluginTaskResume` (inspect and launch) authorizes a
follow-up's new task, and the real host `read_state` checkpoint read
succeeds, while the predecessor's release is held.
- Refusals: no provider stop, a receipt for another provider task, an
unstopped sibling lease, and a release in progress.
- New work on the lease removes the receipt. Release overwrites the
receipt with its own receipt. Warm owners keep their behavior.

I did not add an executor-level test. `native-session-executor.test.ts`
mocks `resolvePluginTaskResume`, so the integration test runs the real
checkpoint read path instead. I did not run the full suite locally. CI
runs it.

## Risks

- `remoteExecutionHasStopped` now returns true earlier for a run whose
leases are still held. Each caller is audited above. Callers that need
released resources use the release-based predicates.
- The stop receipt can now be on an `active` lease. The SQL readers of
`remoteExecutionTermination` (cost accounting, execution blockers,
idle-work proof, orphan settlement) also require a released status.
`pinFailedObservedWorkspace` treats an active ephemeral lease with a
receipt as not pinnable. That path applies only to the remote-workspace
flow and falls back to `preserveNativeFailedWorkspace`.
- Leases that settled before this change have no early receipt. They
continue to wait for their release.
- Nothing compares the refusal message exactly. Matchers use substrings,
and run failure classification does not use the message.
- A follow-up still fails closed if the predecessor's stop was never
proven. This change does not add a recoverable wait.
- The reason the release was held in production is not addressed here.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents, Honeycomb trace queries) and extended
thinking. One sub-agent wrote the change. A second session reviewed it
before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:20:45 -08:00
DottaandPaperclip b9308f9828 fix(ui): keep long agent names inside task columns (#21155)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - Task views show the assignee and creator in fixed-width columns.
> - Long names can extend past those columns and cover adjacent values.
> - The identity labels already support text truncation but need a width
limit.
> - This pull request limits each identity to its task column.
> - Names stay on one line, avatars keep their size, and adjacent values
stay readable.

## Linked Issues or Issue Description

**What happened?**

Long agent and human names extend outside the Responsible and Kicked off
by columns in task views.

**Expected behavior**

Names fit inside each column. Long names show an ellipsis. Avatars
remain visible. The title attribute retains the full name.

**Steps to reproduce**

1. Open a task view with Responsible and Kicked off by columns visible.
2. Use a long agent name for the assignee and creator.
3. Observe names covering adjacent columns.

## What Changed

- Limit agent and human identity width to the containing task column.
- Add a production-component Storybook example for long names, names
without spaces, and short names.
- Add browser layout checks for both identity columns, avatar size, full
text, adjacent values, and row height in both themes at 640px and
1200px.

## Verification

- The browser regression failed before the fix: an identity reached
x=424 while its column ended at x=144.
- The focused Chromium suite passes all four cases at 640px and 1200px
in light and dark modes.
- `pnpm --dir ui exec vitest run src/components/IssueColumns.test.tsx
src/components/IssuesList.test.tsx`: 83 tests pass.
- `pnpm --filter @paperclipai/ui typecheck`: passes.
- `node scripts/check-token-gates.mjs` and `git diff --check`: pass.
- `pnpm verify` returned exit 2 because the browser test requires
broader CI coverage. The complete PR CI run supplies that coverage.
- All tests, typecheck, build, and e2e gates pass on final commit
`f73568822add60b4659f2cbdad0196fdfe010d3b`:
https://github.com/paperclipai/paperclip/actions/runs/43271777908.
- The separate commitperclip review passed:
https://github.com/paperclipai/paperclip/actions/runs/43271777344.
- Greptile gives this commit 5/5. There are no open findings or review
threads.
- All 61 checks are complete: 57 success, 1 neutral, and 3 skipped. The
Storybook workflow was skipped; the focused browser suite passed
locally.

## Risks

Low risk. The width limit applies only to identity labels in shared task
columns. The full name remains in the DOM and the existing title
attribute. No API or stored-data changes.

## Model Used

OpenAI Codex, based on GPT-6, with code tools and Chromium browser
checks. The runtime does not expose a more specific model ID.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 22:15:04 -06:00
DottaandClaude Opus 5.5 497ccba8f1 fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner that the server
controls over the durable PRP channel
> - When a run is interrupted, a same-run retry reattaches the live
Runner and reads its session (`session.snapshot`) before it continues
> - On staging, the reattached Runner authenticated and restored its
session, and then did not answer the read
> - The recovery kept that Runner, so each later attempt adopted the
same Runner and failed the same way until the attempt budget ended the
run
> - This pull request makes the next attempt replace that Runner with a
fresh Runner over its saved journal, and makes a failed Codex history
read say what failed
> - The benefit is that an interrupted Codex run continues without a
manual Retry, and its provider turn is not sent again

## Linked Issues or Issue Description

Refs #21161, Refs #21053, Refs #20853

**What happened?**

A Codex agent on Daytona ran recovery runs (wake
`issue_recovery_action_restored`). In one hour, 11 of about 35 finished
runs failed. All failed the same way:

- `native_session_recovery_failed: HarnessCapabilityUnavailableError:
read is unavailable: Error: runnerd exited while waiting for
session.snapshot` (8 runs)
- `native_session_recovery_failed: HarnessReconciliationError:
codex_history_read_failed: thread/turns/list requires supported
paginated history; Error: provider_transport_failed: PRP command
session.snapshot timed out (runner connected; pending ahead 0; last
runner event seq 338 76s ago)` (3 runs)

The run log of each run shows one sequence:

1. Attempt 1 fails with `runnerd did not report the provider turn
identity`. The close leaves the Runner alive and unsettled for the
same-run reattachment.
2. Attempt 2 adopts that Runner (same PID). The Runner authenticates.
The log shows `runnerd restored its durable PRP session and provider
thread`. Then the driver's recovery read (`thread/read` or
`thread/turns/list`, which both send `session.snapshot`) gets no answer.
The Runner still answers WebSocket pings, but it sends no command result
and almost no events (`pending ahead 0`, last event 76 s to 391 s
earlier). The recovery treats this as transient and detaches. The Runner
stays alive.
3. Attempt 3 adopts the same Runner again and fails the same way. In
some runs a protocol fault (`thread_binding_mismatch`) closes the
transport during the read, so the message is `runnerd exited while
waiting for session.snapshot`.
4. The provider attempt budget ends the run.

#21161 did not help. Its wait extension applies only while a queued
answer is being committed. Here the Runner sent no answer at all.

The `paginated history` text was also wrong. `codex-history.ts` called
every failed history read unsupported, also a transport timeout.

**Expected behavior**

A Runner that left a recovery read unanswered is not adopted again. The
next attempt starts a fresh Runner from the saved journal, resumes the
provider session from the saved checkpoint, and does not send the
provider turn again. A failed history read reports the real failure. A
Codex build that refuses paginated history falls back to one complete
thread read.

**Steps to reproduce**

1. Start a Codex run in a remote (Daytona) Runner.
2. Let the first attempt fail at turn start, so that the close keeps the
Runner alive.
3. Let the reattached Runner stop answering commands after it
authenticates (for example, stop the Runner process after its first
answer).
4. See `PRP command session.snapshot timed out (runner connected;
pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3,
with the same Runner PID.

**Paperclip version or commit**

Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on
`master` after `53608bd1d6`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `nativeRecoveryCommandUnanswered` (executor) finds a
`native_session_recovery_failed` whose Runner did not answer a recovery
command (`PRP command <type> timed out` or `runnerd exited while waiting
for <type>`). Identity, authority, integrity and incompatibility
refusals do not match.
- When such a failure ends a `reattach_remote_runner` attempt on a
sandbox target, the failure detail records `adoptedRunnerUnanswered:
true`.
- `claimNativeRestartRecoveries` copies it to the next claim as
`replaceUnansweredRunner: true`, in the same way as
`exitedBeforeSessionOpen`.
- `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps
all identity checks. When the old Runner is still alive, it returns no
adopted process, and the attempt launches a fresh Runner from the saved
journal. The existing launch fence stops the old `paperclip-runnerd` on
that state folder and its whole lineage, and fails the launch closed
when it cannot prove them gone. A Runner that already exited mid-turn
keeps the existing fail-closed path, because its provider or tool
processes can survive where the fence cannot find them. The task then
continues in a fresh session from the saved work. The attempt budget is
unchanged.
- `codex-history.ts`: a history read that fails without a provider
answer is now `codex_history_read_failed: <method> read failed;
<error>`. A provider refusal of the method (`-32601` or `-32600`) on the
first page falls back to one `thread/read` with `includeTurns: true`.
That fallback fails closed when the thread differs, or when turns, items
or turn identities are missing. A refusal on a later page still fails
closed.
- `doc/execution-semantics.md` describes both rules. The launch-fence
code comment now names the replacement case.

### Changed existing test

`codex-history.test.ts` "does not turn an interrupted page or
unsupported method into idle history" used a plain `Error("method not
found")` on page 2 and expected the `supported paginated history` text.
A plain error is now a failed read, not an unsupported method. The test
now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2
and still expects a fail-closed rejection. A new test covers the
failed-read message.

## Verification

- Red before the fix, green after:
- `server/src/services/native-runtime/native-session-executor.test.ts` ›
"a same-run reattachment whose adopted Runner never answered its
recovery read": the replacement and staging-message tests failed on the
old code (the Runner was adopted again, the three staging messages were
not recognized). Two further tests ("never resumes beside an exited
unanswered Runner's lineage") failed on the first revision of this PR,
which relaunched beside an exited mid-turn Runner, and pass now. All 15
pass.
- `native-runner-restart-recovery.integration.test.ts` › "asks the
same-run reattachment to replace a Runner that left its recovery read
unanswered" (embedded Postgres): the claim had no
`replaceUnansweredRunner` on the old code. Now passes.
- `codex-history.test.ts`: 7 new tests failed on the old code. All 16
pass.
- Real `paperclip-runnerd` and fake Codex:
`runnerd-codex-transport-remote-retention.test.ts` › "continues the same
provider thread from a fresh Runner over the journal of a retained
Runner that stopped answering, without replaying its turn". It runs the
staging sequence: turn started, close keeps the stopped Runner, a second
controller adopts it, the Runner stops answering and the controller
detaches, the Runner and provider are killed (the launch fence), and a
fresh Runner over the same journal returns the same thread. The fake
Codex call log shows one `thread/start` and one `turn/start`.
- Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus
the remote-retention, adoption and cold-restore transport suites (442
tests). Two suites timed out under parallel load and pass alone. Server:
`native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation` (1714 passed, 21 skipped).
- `pnpm typecheck` in `server` (all layers) and `tsc` for
`paperclip-runner` source and tests pass.
- Not run locally: the container-gated remote Runner suites
(`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the
full suite.

## Risks

- Medium. The replacement relaunches a Runner over a journal that is not
settled, but only while the old Runner is alive. Safety comes from three
facts: the launch fence stops the old Runner and its lineage with proof
or fails the launch; the Runner restores its own durable journal (as
local `resume_dead_runner` does); and the real-runnerd test shows that
the turn is not sent again. An exited Runner is never relaunched beside
its possible survivors.
- If the previous close left a `runner.suspend` pending, a fresh Runner
can execute it and exit. The attempt then fails as it does today. The
usual detach path after an unanswered read sends no suspend.
- The replacement uses the existing attempt budget. It does not add
attempts.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events
and logs), extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:06:24 -06:00
Nicky LeachandClaude Fable 5.2 9b613562b7 feat(docker): split the Runner image into stable layers and add a slim variant (#21229)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Part 2 of this stack builds the Runner alone as the image
`runner-artifacts`, in one layer of about 850 MB
> - Most of those bytes are provider binaries and Node. They change only
with a provider version or the Node pin, but one layer means that every
Runner source change makes all 850 MB new
> - About half of the image is the pack's transport archives. A consumer
that copies the pack as files does not need them
> - This pull request splits the image into layers by how often their
content changes, and adds a slim variant without the transport archives
> - The benefit is that a new image shares about 93% of its bytes with
the one before it, and a consumer that needs no transport archives pulls
half as much

## Linked Issues or Issue Description

Refs #21204 (part 2 of this stack) and #21167 (provider pack layers). No
issue exists for this change.

**Problem or motivation**

The image from part 2 is one layer. A registry and a builder store and
move layers, not files. A change to one Runner source file therefore
makes a new 850 MB layer, although 790 MB of its content is the same as
before. The image also holds the provider pack two times: as files, and
as transport archives of those files. Some consumers need only the
files.

**Proposed solution**

Write the image as four layers: the pack's large dependency files,
everything else, the transport archives of those large files, and the
other transport content. Make the two large layers byte-identical from
build to build. Add the target `runner-artifacts-slim`, which is the
first two layers. The full image is the slim image plus the two
transport layers, as the same layer blobs. Check all of this on the
built images.

**Alternatives considered**

- Two unrelated images. Then the registry stores the shared content two
times unless the layers happen to match, and nothing ties the two
together.
- Split by path only, with all files of 1 MiB or more in the first
layer. Six of those files are the Runner's own bundles under `dist`.
They change with the Runner's source, so the first layer would change
with it too.
- Leave the transport archives as they were written. Their bytes then
depended on timing (see "What Changed"), and the archive layer never
repeated.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 3 of 4. Each part is on top of the one before it:

1. #21228 (merged): an optional field in the provider pack manifest
2. #21204: build and check the Runner image, with no publication
3. #21229: layers by change rate, and the slim variant (this pull
request)
4. #21230: publish both variants from `master`

## What Changed

- `Dockerfile`: two final stages. `runner-artifacts-slim` is two `COPY
--link` layers. `runner-artifacts` starts from it and adds two more. The
build stage sorts its output into the four layers. Every path is where
it was in part 2: consumers copy by path and see no difference.

  | Layer | Content | Changes with |
  | --- | --- | --- |
| 1 | The pack's dependency files of 1 MiB or more: provider binaries
and Node | A provider version or the Node pin |
| 2 | Everything else, with `runner-image.json` | Every Runner source
change |
| 3 (full only) | The transport archive of each file in layer 1 | The
same as layer 1 |
| 4 (full only) | The other transport archives, the overlay archive and
the receipt | The same as layer 2 |

- Stable bytes. The files of layers 1 and 3 get timestamp 0, and the
large dependency files get it before their transport archives are
written. A file under the pack's `dist` is the Runner's own output, so
it and its archive are in layers 2 and 4.
- `scripts/create-provider-pack-archive.mjs`, two separate commits.
**This script also writes the application image's archives.**
- The same file now gives the same archive every time. The script piped
tar into Node's zstd compressor. When the pipe ended while the
compressor still had its last chunk queued, Node finished the zstd
stream two times, and the archive got an empty frame of nine bytes at
its end. That depended on timing. In the pinned Linux base, 37 of 60
runs of the old code had the frame. The compressor is now ended only
after it has taken every chunk.
- Archive N holds file N of the receipt, and the receipt listed the
files in path order. `dist` sorts before `node_modules`, so one more
large file in the Runner's own output gave every dependency file another
archive name, and layer 3 became a new blob with the same contents. The
receipt now lists the files outside `dist` first. The readers
(`provider-pack-layers.ts`, `retained-provider-pack-source.ts`) pair
file N with `N.tar.zst` and do not use the path order, so they need no
change. For the application image, the archive numbers change one time,
and the bytes of each archive do not change.
- `runner-image.json` is the same file in both variants. Its new field
`fullImageOnly` names the two paths that only the full image has.
`paths` did not change.
- `scripts/runner-image-layout.mjs check` takes `--variant slim|full`.
It is required and has no default. A slim tree that holds a transport
path fails. A full tree without them fails.
- `scripts/runner-image-oci.mjs` (new, 585 lines). It is not an input of
the image, because the image build does not run it.
- `unpack <oci archive> --out <tree>` writes the tree of an image from
the layers of its archive, in order. It checks every blob against its
digest. It refuses a whiteout, an absolute path, a path with `..`, a
path that is written two times, an entry behind a link, and every entry
that is not a file, a directory or a link.
- `check-layers --slim <oci archive> --full <oci archive>` checks the
table above on the built images. It reads the receipt to decide which
archive belongs in which layer. It fails when the full image does not
start with the slim image's layer blobs in the same order, when a path
that is not a directory is in two layers, and when a shared layer holds
transport content under any name or directory.
- The script reads the archive and its layers itself. It does not use
the system's `tar`, so every platform reads them the same way. It finds
a member only under the name that buildx gives it.
- `.github/workflows/runner-image.yml`: each leg builds each variant one
time, as an OCI archive and nothing else. It unpacks both archives, runs
`check` on each tree with its variant, runs `check-layers`, and runs the
smoke matrix on the slim tree. The tree that is checked is therefore the
bytes of the image, not a second export of the same build. Every step
that builds has a time limit. The workflow still has no registry access.
- Why one export: in the first runs of this pull request, the slim build
stalled on all four legs. Each build command had two exports, a tree and
an archive. The slim command, with every stage from the cache, wrote its
archive and then made no progress on the tree for 50 minutes. A test now
holds the rule of one export and a time limit for each build command.
- The smoke matrix runs on the slim tree. The daemon is in a layer that
both variants share, so one run covers both.
- `server/src/__tests__/runner-image-identity.test.ts`: the test that
holds the rules for the Runner stages now finds a stage wherever
BuildKit starts one. It fails for `FROM x as name`, a flag on `FROM`, a
lowercase or indented `COPY`, an `ADD` or `ONBUILD`, a `COPY` with
another flag, and a `COPY` that continues on the next line. Nine samples
show each case.
- Two comments are corrected. The comment on
`PAPERCLIP_RUNNER_INPUTS_DIGEST` in `build-provider-pack.mjs` said that
no check reads the field. The `Dockerfile` comment on the publication
argument named a workflow "that builds the image for publication".
- `doc/runner-image-contract.md` (159 lines): the two variants, the
section "Layers", how the checks get their tree, and the new checks.
`doc/DOCKER.md` names the slim target.

Script sizes: `runner-image-layout.mjs` 969 lines,
`runner-image-oci.mjs` 585, `runner-image-identity.mjs` 166,
`create-provider-pack-archive.mjs` 114.

## Verification

Image identity at this commit:
`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`.

Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and
zstd, the worktree mounted read-only):

- `runner-image-identity.test.ts` (41), `runner-image-layout.test.ts`
(100), `runner-image-oci.test.ts` (50), `runner-image-workflow.test.ts`
(8), `provider-pack-archive.test.ts` (47, none skipped),
`retained-provider-pack-source.test.ts` (19),
`remote-provider-pack-verification.test.ts`,
`docker-image-layer-reuse.test.ts`, `docker-build-stamp.test.ts`: 293
tests pass in 9 files.
- The same files pass on macOS. The tests of `check-layers` failed on
Linux in the first runs of this pull request: their fixtures named the
archive members `./index.json`, which bsdtar finds and GNU tar does not.
The fixtures now name the members as buildx does.

`linux/arm64`, on a local Docker host, with the new export path:

- Each variant builds as an OCI archive only: the full target in 156
seconds without a cache, the slim target in 5 seconds from the cache.
- `unpack` writes the full image (10,519 entries that are not
directories) in 4 seconds, and the slim image in 4 seconds.
- `check <tree> --variant <variant> --expect-inputs-digest <digest>
--expect-source-revision <sha> --expect-architecture arm64` passes on
each unpacked tree. `runner-image.json` is the same file in both.
- `check-layers --slim <slim> --full <full>` passes.
- The smoke matrix passes on all five base images, from the unpacked
slim tree.

| Layer | Digest | Bytes | In |
| --- | --- | --- | --- |
| 1 | `sha256:3f21b15b3a99…70ac1c` | 379,609,589 | slim and full |
| 2 | `sha256:d5567acc977f…b4f12a` | 43,906,956 | slim and full |
| 3 | `sha256:6a340088ca2c…6ba5b5` | 410,938,259 | full |
| 4 | `sha256:789e97f5db88…3eab44` | 15,656,249 | full |

- The slim image is 423,516,545 bytes in two layers. The full image is
850,111,053 bytes in four.

Stability, with three builds that share no cache for the build step:

| Build | Source revision | Runner build output | Layer 1 | Layer 2 |
Layer 3 | Layer 4 |
| --- | --- | --- | --- | --- | --- | --- |
| A | `aaaa…` | this commit: 6 files of 1 MiB or more under `dist` |
`3f21b15b…` | `d5567acc…` | `6a340088…` | `789e97f5…` |
| B | `bbbb…` | this commit | `3f21b15b…` | `a2f8a4fe…` | `6a340088…` |
`4e6c492d…` |
| D | `dddd…` | one new source file that adds two large files under
`dist` | `3f21b15b…` | `33b0bdaf…` | `6a340088…` | `0ea322bf…` |

- Layers 1 and 3 are byte-identical in all three: 790,547,848 of
850,111,053 bytes, 93%. Build D has two more large files under `dist`,
eight in place of six, and another image identity. Its dependency files
keep the archive numbers 0 to 20. With the earlier numbering, those
numbers moved by two, and layer 3 was a new blob with the same contents.
- Layer 1 has the same digest as before this round. Layer 3 has a new
digest one time, because the archives have new numbers.

Other checks:

- `tests/runner-e2e` workflow tests: 39 pass. `node --test
.github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image
contract scripts: 33 pass. `check-docker-deps-stage.mjs`,
`check-node-version-policy.mjs` and `docker buildx build --check` report
nothing.

GitHub, at this head:

- The `Runner image` workflow passes on both architectures with the new
export path: `Build and check (amd64)` in 14m41s and `Build and check
(arm64)` in 8m41s. Each leg builds both variants, checks both layouts
and the layer rule, and runs the smoke matrix. The first runs of this
pull request stalled in the slim export before any check.

Never run:

- `native-session-executor.test.ts`. It does not load on this host, with
or without this change, because `@paperclipai/plugin-sdk` is not built
here. It imports the reader of the retained pack and no archive number.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The two changes to `create-provider-pack-archive.mjs` change the
application image's transport archives: about half of them lose nine
bytes, and all of them get new numbers one time. An archive's content
does not change. A reader pairs file N of a receipt with archive N of
the same build.
- The large dependency files in the pack, and in their transport
archives, now have the modification time 0 (1970). Before, they had the
build time. Nothing in this repository reads those times.
- `unpack` is a reader of tar and gzip that this repository owns. It
reads the layers that BuildKit, GNU tar and bsdtar write (tested with
each), and it fails for a header it does not know.
- The check on the built images, not the recipe, is what proves the
layer rule; it runs in each leg of the workflow.
- `check` now needs `--variant`. A caller that does not pass it fails
with a clear message. No caller exists outside this repository yet.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
canary/v2026.1113.0-canary.2
2026-11-12 19:52:02 -08:00
DottaandClaude Opus 5.5 3b4c3265ed fix(recovery): offer Try again on a model-unavailable failure and recheck the model first (#21244)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native run fails, the task thread shows a failure marker, and
some markers offer Try again
> - A `native_provider_model_rejected` failure blocks the task for the
person and shows "Model unavailable", but it has no Try again control
> - Recovery must just work: after the person fixes the model, they need
one obvious control that continues the task, and that control must not
start a run that is sure to fail
> - This pull request shows Try again on the "Model unavailable" marker
and makes the retry recheck the selected model before it starts a run
> - The benefit is that the person fixes the model, presses Try again,
and the task completes, and a too-early press explains which setting to
change instead of failing again

## Linked Issue or Issue Description

Refs #21210

**What happened?**
Paid run 43321773136, `recovery.retry-exhausted` (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`). The run failed with
`native_provider_model_rejected`. The task stayed blocked and the thread
showed "Model unavailable", but no Try again control appeared
(`task-chat-run-failed-try-again` not found in 30 s). Only the "Run
failed" and "Usage limit reached" markers offer Try again.

**Expected behavior**
The "Model unavailable" marker offers Try again. Pressing it checks the
selected model first. If the provider-rejected model is still selected,
no run starts, and the person sees which setting to change with a link
to it. After the person changes the model, Try again starts one retry of
the original run.

**Steps to reproduce**
1. Set an agent's model to an id the provider rejects.
2. Assign it a task. The run fails with
`native_provider_model_rejected`.
3. Open the task. Before this change, the marker has no Try again.

## What Changed

- `server/src/services/native-runtime/model-rejected-retry.ts`:
`rejectedModelRetryRefusal` reads the model the failed run sent
(`runnerProfileJson.nativeExecutionInput.provider.model`) and the model
a new run would use (the task's model override for its assignee, else
the agent's model). If they are the same, it returns a 409 refusal with
code `native_provider_model_still_unavailable`, the setting to change
(`agent_model` or `task_model_override`) and the rejected model.
- `server/src/routes/agents.ts`: the exact failed-run retry (`POST
/agents/:id/wakeup`, `reason: retry_failed_run`) applies this check
before it queues anything.
- `ui/src/components/task-chat/adjacent-errors.ts`,
`TaskChatThreadView.tsx`: one `isRetryableFailureMarker` rule, now
including "Model unavailable".
- `ui/src/lib/model-retry-refusal.ts`, `ui/src/pages/IssueDetail.tsx`: a
refused retry shows "Model still unavailable" with the server's message
and an "Open <agent> model settings" link to `/agents/<agent>/runtime`.
A task override refusal names the override in the message.
- Eval `recovery.retry-exhausted` (oracle `runtime.retry-exhausted.v7`):
the rejected model stays selected after the failure. The person presses
Try again (409, toast names the model, its link opens the agent's model
settings, no run starts in 20 s, task stays blocked), restores the
working model, presses Try again (202), and the one retry of the
original run uses the working model, saves the brief and completes the
task. 3 runs: failed 1, succeeded 2. The budget-hold and two follow-up
chat phases are removed; the case now tests the model recheck.
`recovery.yaml` context and expected outcome updated, recorded in
`chat-full-catalog-testops-manifest.json` `changedFields`, pinned in its
test, descriptors and inventory digest updated.

## Verification

- RED on master: `TaskChatThread.test.tsx` "offers Try again on a
model-unavailable failure and retries that run" fails (no control).
`agent-live-run-routes.test.ts` "refuses a retry while the rejected
model is still selected and starts no run" fails (the route queues the
run).
- GREEN: both pass. `agent-live-run-routes.test.ts` 101/101.
- `server/src/services/native-runtime/model-rejected-retry.test.ts`
(embedded PostgreSQL), 6/6: refuses while the agent still selects the
rejected model (names the agent setting); names the task override;
allows the retry when the agent or the override now selects another
model, or the run failed for another reason; ignores another company's
run.
- `ui/src/lib/model-retry-refusal.test.ts`: names the setting and links
to `/agents/morgan/runtime`; no link for a task override; other errors
fall through to the generic toast.
- Server `tsc --noEmit` clean. UI `tsc` clean apart from the
pre-existing generated storybook catalog. `node
scripts/check-token-gates.mjs`: clean.
- Eval: `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog tests/runner-e2e/chat-catalog-domains
tests/runner-e2e/chat-*.test.ts`: 339 files pass. New calibration
rejects an accepted early retry, a run started while the model is
rejected, a refusal without the model, setting link or blocked task, a
retry that used the old model, and duplicate or missing retries. `pnpm
test:e2e:runner:typecheck`: pass. Manifest test 10/10.
`source-sdk-public-inputs --check`: `changed: false`.
- Paid run: Runner Full-Stack E2E 43338695850 on throwaway branch
`chat-eval/qa-mr-paid`. Result to follow in a comment.

## Risks

- The recheck compares the selected model with the model the provider
rejected. It does not call the provider. A model the provider rejected
stays refused until the person selects another one; a different but also
unsupported model starts a run that fails with the same marker again.
- The refusal message is a toast (15 s). The marker detail still says to
choose a supported model.
- No persisted shape changes; runs saved by older builds carry the same
`runnerProfileJson` field.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:42:52 -06:00
DottaandClaude Opus 5.5 7154030ef6 fix(heartbeat): draining/yielding servers claim no queued run, and closed tasks get no system continuation (#21258)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - During a release swap, the outgoing server drains. It must not start
new runs, because its own exit would cut them off.
> - The drain gate (run admission closed at SIGTERM) and the hand-over
gate (#21127, yield to a replacement that already dispatches) were
checked only by the queued-run dispatch passes.
> - Other start paths (executeRun of a queued run, delivered
continuations, immediate wake starts) went straight to `claimQueuedRun`,
so an outgoing server could still start runs after traffic moved.
> - Also, a system continuation queued while its task was open started
after another run had finished the task, and took an extra turn on a
done task.
> - This pull request makes `claimQueuedRun`, which every start path
uses, honour the drain and hand-over gate, and cancel a system
continuation whose task closed while it was queued.
> - The benefit is fewer runs cut off by a swap, and no extra turns on
finished tasks.

## Linked Issues or Issue Description

Refs #21127, #21135

**What happened?**
1. An outgoing server kept starting runs minutes after traffic moved to
its replacement. Each such run was cut off when the old server exited.
The run start went through `executeRun` of a queued run, which called
`claimQueuedRun` without the drain or hand-over check.
2. In release-swap waves, a continuation (execution reconciliation
delivery, invariant or recovery continuation, blocker-stall wake) was
queued while its task was open. Another run then finished the task. The
continuation still started and took an extra turn on a done task.

**Expected behavior**
A draining or yielding server leaves a queued run queued for the serving
server, on every start path. A system continuation does not start on a
task that was closed while it was queued. A person's message on a done
task still runs.

**Steps to reproduce**
1. Close run admission (drain) or set the hand-over yield on a server.
2. Start a queued run through `executeRun` (for example a delivered
continuation).
3. See the run claimed and started on the draining server.

## What Changed

- `claimQueuedRun` (`server/src/services/heartbeat/queue.ts`) refreshes
and checks the run-admission gate first. When admission is closed
(draining or yielding), it leaves the run queued and returns. The
dispatch passes already did this check; now every start path does.
- `claimQueuedRun` re-reads the task for a system continuation
(`issue_recovery_action_restored`, `issue_continuation_needed`,
`issue_assignment_recovery`, blocker-stall wake). If the task is `done`
or `cancelled`, it cancels the run as `issue_closed_before_start`, skips
its wake, and releases the task. Other wake reasons, including a
person's message, are not affected.
- That cancel never consumes a person's message. Admission can attach
deferred comments to a queued system continuation without changing its
wake reason. The cancel locks the run's wake and the run (the order
admission's coalescing uses), reads the run's current comment ids, and
if any is a person's comment that no run of this agent has received yet
(read-only use of `deliveredQueuedCommentIds`, which includes the
history-read rule), the run is not cancelled: it is claimed as that
person's follow-up, which may reopen the task through the normal path. A
comment attached concurrently is either seen under the lock or, after
the cancel commits, saved behind the cancelled run by admission.
- Admission is checked again inside each claim transaction (ordinary
claim, queued-comment claim, native review claim), right after the write
that moves the run to `running`. If admission closed during the awaited
checks, the transaction rolls back: the run stays queued with no
`startedAt`, its wake and the task's execution lock are untouched, and
nothing is charged.
- Every cancellation `claimQueuedRun` makes from the caller's copy of
the run is now a compare-and-set from `queued`
(`setRunStatusFromLive(..., ["queued"])` through a new
`cancelRunIfStillQueued` helper): the closed-task cancel, the
refused-interaction cancel, the missing-@mention cancel, the daily-cap
cancel and the blocked-dependency cancel. A run another dispatcher
already started or finished keeps its status and result, and no wake or
release follows. The queue module no longer receives the unconditional
`setRunStatus`.
- The closed-task cancel takes its decision under the wake and run
locks, then cancels through `setRunStatusFromLive` fenced on the row
version (`xmin`) the decision read. That saves the same never-started
proof (`executionRecovery: { kind: "bootstrap", providerWorkStarted:
false }`) as every other queued-run cancel, so task release promotes a
person's message saved behind it. A message attached after the decision
changes the run, the cancel matches nothing and the decision is taken
again. `HeartbeatRecoveryDependencies.setRunStatusFromLive`
(`recovery.ts`) now types its existing `cancellationCondition` parameter
as `SQL`.
- The dispatch ranking in `startNextQueuedRunForAgent` is not changed.

### What I checked in `claimQueuedRun` for writes from the caller's copy

- Cancellations through `setRunStatus`: all five now conditional on
`queued` (above).
- The final claims (ordinary, queued-comment, native review) already
lock and re-read the run and update `where status = 'queued'`.
- The case-handoff cancel and the chat-control stop already lock the run
and update `where status = 'queued'`.
- The budget-stamp update already writes `where status = run.status`
(`queued`).
- `cancelRunInternal` (agent removed, agent not invokable, board-owned
budget stop, pause hold, budget cancel) re-reads the run and only
cancels a non-terminal status, so it cannot overwrite a finished result.
It can still stop a run another server has just started; that is a
policy stop (agent removed, budget, hold) that would apply to that run
anyway, and it is unchanged here.
- `cancelStaleQueuedRun` already takes `expectedStatus: "queued"`.

## Verification

Red-to-green tests in `server/src/services/heartbeat/queue.test.ts`:
- "claims no queued run once run admission is closed, whatever path asks
(draining or yielding server)": red before the gate was added to
`claimQueuedRun`, green after.
- "does not start a %s run whose task another run already finished":
`issue_continuation_needed` and `issue_recovery_action_restored` are
cancelled as `issue_closed_before_start`; `issue_commented` (a person's
message, carrying its `wakeCommentId`) is claimed and the stored run is
`running`.
- "answers a person's message attached to a system continuation on a
closed task once": still undelivered, the continuation is claimed and is
the only unfinished run (red when the cancel ignores comment ids, green
after); already received by the closing run, the continuation is
cancelled as `issue_closed_before_start` and no other run exists.
- "keeps the result of a continuation another dispatcher already started
and finished": the stale queued copy is passed after the run succeeded;
the stored run stays `succeeded` with its result, and no wake or release
follows. Red before the compare-and-set (the run was overwritten to
`cancelled`), green after.
- "returns the claim to the queue when run admission closes while the
claim is being checked": admission closes inside an awaited check; the
run stays `queued` with no `startedAt`, the task's execution lock and
the wake are unchanged. Red before the in-transaction re-check, green
after.

Red-to-green test through the real release path, in
`server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts`:
- "delivers a person's message saved behind a system continuation
cancelled at claim because its task closed": a never-started legacy
continuation on a done task, with a person's deferred message behind it.
Red before the fix: the cancelled run had `resultJson: null`, with no
never-started proof. Green after: the proof is present, the wake is
skipped, and the message is promoted and runs once.

Test expectations changed because they asserted the old behaviour:
- `server/src/services/agent-interaction-delivery.test.ts` "cancels
refused queued runs through the public queue claim without dispatch":
now expects the cancel through `setRunStatusFromLive(runId, "cancelled",
["queued"], …)` instead of `setRunStatus`.
- `server/src/services/heartbeat/queue.test.ts` (refused-assignment held
case): asserts `setRunStatusFromLive` is not called, instead of
`setRunStatus`.
- The `issue_commented` case of "does not start a %s run…" now asserts
the run is claimed and `running` (it previously asserted only that the
error code was not `issue_closed_before_start`).

Local command, on this branch based on current master (includes #21234;
the dispatch ranking it changed is untouched here):
- `cd server && pnpm exec vitest run
src/services/heartbeat/queue.test.ts
src/__tests__/recovery-wake-herd.test.ts`: 102 tests passed. After the
proof fix, `queue.test.ts` again passed 97/97, and the new stale-queue
test passed. (The file's `afterAll` database cleanup hook times out on
this loaded machine on master too; it is not related to this change.)

Why the local release-swap wave does not show the yield fix:
- The "yield to the replacement" signal exists only when the run-start
handover is created (`server/src/index.ts`, about line 1320). That
happens only when a direct deployment listen URL exists, that is, with
`PAPERCLIP_EPHEMERAL_SERVER=true` and `PAPERCLIP_DATABASE_LISTEN_URL`
(the volume-free configuration).
- The local short form of the wave leaves `--ephemeral-server` out, so
the handover is disabled and `isRunAdmissionClosed()` becomes true only
at SIGTERM. The wave therefore cannot exercise the yield.
- The evidence for this fix is the unit test above and the reasoning
that every start path now reaches the same gate in `claimQueuedRun`.
Hosted confirmation must come from a volume-free instance.
- The closed-task fix was found in release-swap waves (an extra turn on
a done task in two tasks).

## Risks

- `claimQueuedRun` can now await an admission refresh. The refresh
returns at once when the cached state is fresh, so the normal path adds
no wait before `turn.start`.
- A system continuation for a closed task is cancelled. This matches
admission, which already refuses such wakes for a closed task.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:36:05 -06:00
DottaandClaude Opus 5.5 d169dd5a90 fix(presentation): offer Retry only when no automatic continuation or scheduled retry is pending (#21264, includes #21200)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a run fails with a transient error, Paperclip schedules the
next attempt by itself and shows "retrying automatically" with a
countdown.
> - The task presentation also offered the Retry button next to that
countdown.
> - The system owns the next attempt while the run is `scheduled_retry`,
so the button is misleading: it suggests that a person must act.
> - This pull request stops offering Retry while a scheduled retry is
pending. Retry returns when the scheduled retry is past due with nothing
pending, or when the retries are exhausted.
> - The benefit is one clear message: Paperclip retries by itself, and
Retry appears only when it is the way forward.

## Linked Issues or Issue Description

**What happened?**
A legacy run in status `scheduled_retry` (for example reason
`transient_failure`, which is not a shared-resource wait and has no
system recovery) projected `state: "retry_scheduled"` with `canRetry:
true`. The UI then showed the Retry button beside "retrying
automatically". The cause is in `projectTaskInterruptionState` in
`server/src/services/task-interruption-presentation.ts`: the `retryAt >
now` branch set `canRetry: !preparationPending && !nativeResumeAt`,
which ignores that the run itself is the scheduled attempt.

**Expected behavior**
While a run is `scheduled_retry` and its retry time is in the future,
the task shows the countdown and offers no Retry. When the scheduled
retry is past due with nothing pending, or the retries are exhausted,
Retry is offered.

**Steps to reproduce**
1. Let a task's run fail with a transient failure that schedules an
automatic retry.
2. Open the task before the retry time.
3. The task shows "retrying automatically" and the Retry button.

**What a Retry click does today in this state**
`requestTaskRetry` (`server/src/services/task-retry.ts`) treats a latest
run in `scheduled_retry` as busy. It accepts the click (202), records
the request as `coalesced` onto the scheduled run and returns
`scheduledRetryRunId`. It does not start a second attempt next to the
scheduled one. The button is therefore misleading but not unsafe, and
this pull request changes only what the task shows.

## What Changed

- `server/src/services/task-interruption-presentation.ts`: in the
`retry_scheduled` branch, `canRetry` is now also false when `run.status
=== "scheduled_retry"`.
- `server/src/services/task-interruption-presentation.test.ts`: three
tests for a scheduled transient-failure retry.
- This branch is based on current `master` and includes #21200, which
has not merged yet and changes the same file. The diff of this pull
request against #21200 is one line and the tests.

## Verification

- `cd server && npx vitest run
src/services/task-interruption-presentation.test.ts -t "automatic
scheduled retry"`
- "shows a pending transient-failure retry as scheduled without a
competing Retry": red on `master` (`canRetry: true`), green with this
change.
- "offers Retry once the scheduled retry is past due with nothing else
pending": green before and after.
- "offers Retry once the scheduled retries are exhausted": green before
and after.
- Locally: `task-interruption-presentation.test.ts`,
`task-interruption-query.test.ts` and
`task-interruption-automatic-path.test.ts` pass (107 tests). Server
typecheck: 0 errors. The wider suites that read the presentation run in
CI; they were not repeated locally because the machine was under memory
pressure.

## Risks

- Low risk. The change affects presentation only. The Retry route still
accepts a click and coalesces it onto the scheduled run, as before.
- If a scheduled retry never dispatches, the task offers Retry again
once the retry time passes, as before this change.
- No schema, API or UI change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in
Claude Code with tool use (shell, file edits, test execution). Extended
reasoning was available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:14:41 -06:00
DottaandClaude Opus 5.5 6cb0dd0c6b fix(recovery): continue twice-interrupted tasks and waiting follow-ups after a release swap (#21250)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is cut off by a restart or a release swap, the
fresh-continuation fallback continues the task in a new run.
> - The continuation goes through normal admission, or through a
person's saved follow-up when one is waiting.
> - In release-swap waves, three gaps left tasks stopped with no run:
admission refused an isolated continuation of a task with no project, a
person's saved follow-up behind the replaced run was never promoted, and
the fallback relied on a saved follow-up that the cut-off run had
already read.
> - This pull request closes those three gaps.
> - The benefit is that a twice-interrupted task, and a task with a
waiting follow-up, continue by themselves.

## Linked Issues or Issue Description

Refs #21135, #21222

**What happened?**
1. A task with no project was interrupted twice. The second continuation
used a fresh isolated workspace, which clears the task's workspace
binding. The admission preflight still required that binding for an
isolated task with no project, so it refused the continuation as
`workspace_worktree_requires_project` about every 15 s. Each refusal
re-blocked the task and added a comment (329 to 428 skipped wakes per
wave). The runtime itself supports this case: it prepares a new
workspace (`create_new`).
2. The fallback continued a task with delivery `saved_input` (a person's
deferred follow-up is the next turn). The stranded-queue pass refused to
promote that follow-up, because the replaced native run's finalization
was `terminal_failure` (restart recovery ended it), not `committed`. The
release path would also have re-blocked the task with a reconciliation
incident. The follow-up never ran.
3. The fallback chose `saved_input` because a deferred user wake
existed, but the cut-off run had already taken that message into its
turn. Promotion settled the message as delivered, and no run followed.
The task showed "working".

**Expected behavior**
An interrupted task continues by itself in each case, with no person
action and no repeated comments.

**Steps to reproduce**
1. Run a release-swap wave with projectless tasks in isolated mode and
person follow-ups during runs.
2. Interrupt the same task twice, or interrupt a run that has a deferred
follow-up behind it.
3. See the task stay blocked or "working" with no run.

## What Changed

- Admission (`server/src/services/execution-workspace-policy.ts`,
`server/src/services/heartbeat/queue.ts`,
`server/src/services/fresh-continuation-fallback.ts`):
`isUnrunnableWorktreeCombo` accepts `freshIsolatedContinuation`. The
admission preflight sets it from the new
`isFreshIsolatedContinuationWake`, which returns at once (no query)
unless the wake context carries `freshContinuationIsolation`. This does
not touch the dispatch ranking.
- Follow-up promotion
(`server/src/modules/wake-queue/adapters/postgres.ts`): the
stranded-queue pass and the release path skip the native finalization
gate and the reconciliation incident for a run that the fallback
superseded (`agentRunSupersededByFreshContinuation`).
- Saved-input choice
(`server/src/services/fresh-continuation-fallback.ts`): a deferred user
wake counts as saved input only when it has no comment ids, or still has
a message that the agent has not received
(`undeliveredLegacyUserCommentIds`, called, not changed). Otherwise the
fallback records its own continuation.

- Review round 1: (a) whenever the old writer is not proven stopped
(`fresh_isolated_workspace`), the fallback records its own isolated
continuation first, also when the task had a binding; saved input
follows it on the new workspace. (b) Release skips the manual-admission
hold and the pending-cleanup check for a run the fallback replaced with
an isolated continuation (`fresh_isolated_workspace` or `remote_epoch`,
via `freshContinuationIsolatesRun`); other replacements still wait.
- Review round 2 (the isolated continuation must run before the waiting
follow-up): while the fallback's `fresh_isolated_workspace` continuation
is still pending delivery, release leaves the person's follow-up
deferred. Otherwise the follow-up would become the task's next run
without the isolation marker, and delivery would count it as the
continuation and skip the isolated run. The continuation's own release
promotes the follow-up behind it. `freshContinuationIsolatesRun` now
also reports `continuationPending`.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `server/src/__tests__/execution-workspace-policy.test.ts`: admission
keeps a fresh isolated continuation of a projectless task runnable.
- `server/src/modules/wake-queue/adapters/postgres.test.ts`
"restart-ended native run": saved input behind a run the fallback
replaced is promoted.
- `server/src/services/fresh-continuation-fallback.test.ts` "relies on
saved input only while it still holds an undelivered message" (taken:
true gives `recorded`, taken: false gives `saved_input`).
- `postgres.test.ts` "promotes saved input behind an exhausted restart
hold ... (released, continuation pending)": the follow-up stays deferred
while the isolated continuation is pending, then is promoted when the
continuation's run is released. It was red before the fix (`expected
'promoted' to be 'released'`).
- Local, round 2: `cd server && npx vitest run
src/modules/wake-queue/adapters/postgres.test.ts
src/services/fresh-continuation-fallback.test.ts`: 2 files, 177 tests
passed.

Local command, on this branch based on current master (includes #21222
and #21234):
- `cd server && npx vitest run
src/services/fresh-continuation-fallback.test.ts
src/__tests__/execution-workspace-policy.test.ts
src/modules/wake-queue/adapters/postgres.test.ts`: 3 files, 221 tests
passed (re-run after rebasing on #21234).

Local release-swap cells:
- L1 (6 tasks, 3 swaps): 5 fresh isolated continuations delivered,
including a second one on the same task; 0
`workspace_worktree_requires_project` rows (before: 329 to 428 skipped
wakes per wave).
- R2 (6 tasks, 3 swaps): a task interrupted three times had its second
and third continuations delivered. The same cell found gap 3 (a
follow-up already read by the cut-off run), which the last commit fixes.
- Hour-long release-swap wave (w-20261112T233630-s7201) on the branch
that holds these fixes and the related recovery fixes: 11/11 tasks
finished hands-off (master: 9/11). The lost-continuation and
follow-up-promotion fixes were visibly working in it.

Review-round tests: "moves the task off the old writer's workspace and
runs the isolated continuation before saved input" (changed expectation:
`recorded` instead of `saved_input`; red on the previous head);
"promotes saved input behind an exhausted restart hold once a %s
continuation replaced the run" (isolated: promoted, red before;
`writer_ended`: held). 3 files, 224 tests passed.

Conflict note: the last commit was resolved onto current master. It no
longer imports `taskHistoryReadsForRun`, which belongs to a separate
opening-note change.

## Risks

- An isolated projectless continuation now passes admission. It runs in
a new workspace that the run prepares, so it never shares a workspace
with the cut-off run.
- A superseded run's follow-up is promoted without its native
finalization committed. The superseded run is fenced (its writes are
refused), so there is still one writer.
- No migration. No schema change. The normal admission path adds no
query.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:10:47 -06:00
DottaandClaude Opus 5.5 a298ca7fbc fix(runner): prove ownership right before starting a turn in a recovered session (#21260)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner runs native agent sessions (Codex, Claude, and
others) and keeps them alive across server swaps. A new server
generation recovers the session and continues it.
> - Only the generation that owns the run may submit a turn into the
provider session. The Runner proves ownership with a fenced checkpoint
(controller boot, generation and lease, judged by the database clock).
> - On a recovered session, the startup turn and the restart
continuation took that checkpoint right after recovery, then prepared
the turn's input (attachments, queued input, steered messages), then
submitted.
> - A generation that lost the run during input preparation still
submitted a turn into the shared provider session.
> - This pull request repeats the fenced checkpoint right before
`turn.start` on those two recovered-session paths.
> - The benefit is that a server generation that no longer owns a run
cannot add a turn to its provider session. New sessions are unchanged,
so nothing is added before their first turn.

## Linked Issues or Issue Description

**What happened?**
On staging, a Codex run crossed three server swaps. After its turn
finished without a result, a turn that no generation could account for
appeared in the same provider thread. Every path that submits a turn
into an existing session is preceded by the fenced checkpoint, but the
two startup paths took it before input preparation, not right before the
submit. A generation that lost the run in that window still submitted.

**Expected behavior**
A generation submits a turn into a recovered session only if it still
owns the run at the moment of the submit.

**Steps to reproduce**
1. Recover a native session on a new server generation.
2. While it prepares the startup turn's input, fence the generation
(another generation takes the lease).
3. Before this change, the turn is still submitted. After it, the
checkpoint fails and no turn starts.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
- The restart continuation calls the fenced `checkpoint` right before
`session.startTurn`, after attachments and steered messages are
prepared.
- The startup turn on a recovered session (`recovered === true`) calls
`checkpoint` right before `session.startTurn`. A new session's normal
path is unchanged.
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: new
tests "proves it still owns a recovered session right before starting a
turn in it" (startup turn) and "proves it still owns a recovered session
right before a restart continuation" at startup and when the lost turn's
terminal arrives (`resumeInterruptedTurn`).

Turn starts into an existing session, and how each proves ownership
right before the submit:
- Startup turn on a recovered session: new checkpoint, covered by the
first test.
- Restart continuation (`startRestartContinuation`), at startup and via
`resumeInterruptedTurn`: new checkpoint, covered by the second test
(both cases).
- Continuation disposition (`requestContinuationDisposition`): already
persists the fenced checkpoint immediately before `startTurn`, with only
a snapshot read between them and no input preparation. No change.
- Compaction turn: started right after the fenced recovery checkpoint
with an empty message; there is no input preparation between them (the
only possible await is the disposition-replay read, which applies only
when a disposition turn was submitted, and then no compaction turn
starts). No change.
- Runner-only change. No change to `protocol/`, `generated/`, Rust
shapes or the ACPX patch, so the provider-pack `contractDigest` and
interface revisions are unchanged and older image packs stay compatible.
The Runner build and image content for this source change are produced
by CI (`runner-artifacts`); nothing is deployed by this PR.

## Verification

- Red-to-green: "proves it still owns a recovered session right before
starting a turn in it" fails without the startup-turn checkpoint (the
turn is submitted after the generation is fenced) and passes with it.
Both restart-continuation cases fail with the checkpoint removed from
`startRestartContinuation` and pass with it.
- Local: `cd packages/paperclip-runner && pnpm exec vitest run
src/native-session-runtime.test.ts`: 290 passed.
- Local: `node scripts/typecheck-test-programs.mjs
packages/paperclip-runner/tsconfig.test.json`: passed (the first CI run
failed on the test's attachment prompt returning `undefined`; it now
returns a string).
- **Not run locally:** one test case that needs the Rust
`paperclip-runnerd` binary, and
`src/live/runnerd-restart-continuation.test.ts` (10 cases). The Rust
binary was not built on this machine. CI must cover them.

## Risks

- One extra checkpoint round trip before `turn.start` on a recovered
session and on a restart continuation. These are recovery paths; a new
session's first turn gets no added wait, so boot and time-to-first-token
for new runs are unchanged.
- If the checkpoint fails (the generation lost the run), the turn is not
submitted. That is the intended behaviour; the owning generation
continues the run.
- No migration. No schema change. No pack contract change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:08:19 -06:00
DottaandClaude Opus 5.5 db802d1230 fix(heartbeat): deliver a queued comment again unless the reading run cites it as handled (#21243)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can add a comment to a task while the agent's run on that
task is still working
> - #21175 made a `get_task_history` read of a queued comment's current
revision count as delivery
> - A read is not handling. In `agent-chat.multiple-handoffs`, Morgan's
running turn read the "Launch sooner" comment, set it aside, and saved
"Launch later". No continuation ran, so the instruction was lost.
> - This pull request counts a read as delivery only when the run's
accepted result cites the comment as handled. Otherwise one continuation
still delivers it, and that continuation is told the comment may already
be applied.
> - The benefit is that a person's comment is never silently lost, and
the #21175 protection against repeated completed work stays in place

## Linked Issue or Issue Description

Refs #21175
Refs #21203
Refs #21210

**What happened?**
Paid run 43321773136 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`), case
`agent-chat.multiple-handoffs`. Morgan's running turn called
`get_task_history` and received the CEO's queued "Launch sooner" comment
at its current revision (`task_history.read` at seq 75). The model
judged the comment as an unauthenticated agent instruction and saved
"Launch later" from the brief. Its accepted result did not mention the
comment. `deliveredQueuedCommentIds` counted the read as delivery, the
release drain dropped the held comment, and no continuation ran.

**Expected behavior**
A read counts as delivery only when the run records the comment as
handled. A comment that was read but not recorded as handled is
delivered once in a new turn. The new turn's prompt says that an earlier
turn already read the comment and may have applied it.

**Steps to reproduce**
1. Start a native run on a task.
2. Queue a comment behind the run.
3. Let the run read `get_task_history` and finish without applying the
comment.
4. Before this change, no continuation runs and the comment is lost.

## What Changed

- `server/src/services/queued-comment-delivery.ts`: the history-read
delivery rule now also requires the reading run's `run.result.accepted`
event to cite `comment:<id>` (top-level `evidence[].ref` or a
criterion's `evidenceRefs`, prefix match so an annotated ref counts).
The read revision, succeeded-run and finished-after-revision rules are
unchanged. All consumers (release drain, stale-continuation save,
ownership reconciler, first-turn queued input, remote-Stop resume) use
this one rule.
- New `readByEarlierTurnCommentIds`, `commentsArrivedDuringRun` and
`taskHistoryHandlingGuidance`.
- `get_task_history` (both tool authorities) returns
`arrivedDuringThisTurn` (the `comment:<id>` refs of comments other
authors posted or edited after the run started) and a `handling`
instruction to cite each applied comment in `paperclip_finish` /
`paperclip_block` evidence. The comment list itself is unchanged. A
failure to compute the list returns no guidance, so the comment stays
owed.
- `buildPaperclipWakePayload` marks a wake comment `readByEarlierTurn`
when another finished run of the agent read its current revision.
`renderPaperclipWakePrompt` adds one line after that comment: check the
current task state first, and do not repeat work that is already done.
- `doc/run-log-events.md`: the delivery rule and the new fields.
- Evals: `agent-chat.multiple-handoffs` (oracle
`runtime-multiple-handoffs.v6`) and `steering.edit-queued-message`
(oracle `edit-queued-message.structural.v6`) accept in-turn consumption
only with a current-revision read and a citation. A read without a
citation must get exactly one continuation whose saved wake marks
`readByEarlierTurn`, and the final saved revision must hold the change
("Launch sooner"; blue). `runtime-queue.ts` reads citations from the
run's `run.result.accepted` event. No scenario YAML changed.

## Verification

- RED on master `b1ef1f3022`,
`server/src/__tests__/heartbeat-held-comment-wake.test.ts`: 10 of 41
fail. The key one: "settles a held message as soon as its run closes the
task (read mid-turn: true, cited: false)" delivered 0, expected 1.
- GREEN: 41/41. The three required shapes:
- read, not cited: one continuation, and its wake marks
`readByEarlierTurn` (`...read mid-turn: true, cited: false` and the
stale-queued-run variant).
  - read and cited: no continuation (`...cited: true`).
  - never read: one continuation, as before.
  - Edited after the read, or a failed reader: still owed.
- `server/src/services/queued-comment-history-read.test.ts` (remote-Stop
resume): succeeded+cited not owed; succeeded+uncited owed; failed+cited
owed. 4/4.
- `packages/adapter-utils/src/server-utils.test.ts`: the prompt line
renders only for `readByEarlierTurn`.
- Neighbouring suites: `muse-control`, `dot-runner`,
`paperclip-runner-tool-authority`, `paperclip-control-plane-port`,
`muse-native-follow-up`, `runner-prp-coordinator`,
`native-session-handoff`, `execution-continuation-history-read`,
`heartbeat-comment-wake-batching`, `ownership-reconciliation-report`,
`run-preparation`: all pass. Server and adapter-utils `tsc --noEmit`:
clean.
- Eval graders: `npx vitest run --config
tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog
tests/runner-e2e/chat-catalog-domains`: 283 files pass. New calibration:
the observed lost-instruction case (read, not cited, no continuation)
fails; read-not-cited with one warned continuation passes; an unwarned
continuation, a missing final change, and a cited read plus a
continuation fail.
- `pnpm test:e2e:runner:typecheck`: pass. `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10.
`node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
`changed: false`.
- Paid run: Runner Full-Stack E2E 43338695850 on throwaway branch
`chat-eval/qa-mr-paid` (this change plus the model-retry fix on master
19f8ed1ea2). Result to follow in a comment.

## Risks

- A run that applies a queued comment but does not cite it gets one more
turn on that task. That turn is told the comment may already be applied.
In the restart-chaos t005 shape (#21175) this can reopen a done task
when the agent does not cite the comment. This is the chosen tradeoff: a
lost comment is worse than one extra checked turn.
- Each `get_task_history` call adds one small indexed read of the run's
`started_at`.
- Upgrade-safe: runs saved by older builds have no citation, so a
read-only delivery from before this change stays owed and is delivered
once more. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:07:08 -06:00
DottaandClaude Opus 5.5 04b50e6007 fix(native-runtime): a timed-out step records its real cause instead of leaving the run unrecoverable (#21262)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs start a Runner and a provider session, then send the
first turn
> - When a startup step fails, the native session executor redacts the
error, classifies it and records the failure
> - A timed-out step fails with a `DOMException` (`TimeoutError` or
`AbortError`). Its `message` property has only a getter
> - The redaction assigned to `error.message`. The assignment threw a
`TypeError`, so the classifier did not run and the real cause was lost
> - The run then fell into a restart-recovery chain that ended in
"requires reconciliation" wording for an operator
> - This pull request makes the redaction safe for errors with a
read-only message
> - The benefit is that a timed-out step records its real cause and
settles through the normal failure path

## Linked Issues or Issue Description

**What happened?**

In a local chaos cell with 8 opencode tasks on Daytona sandboxes, four
first runs failed before their first turn. A step before the first turn
ran under a five-minute `AbortSignal.timeout`. The step took longer than
five minutes. The run log shows `TypeError: Cannot set property message
of which has only a getter` at the startup failure handler, then
`native_finalization_missing`. The run never recorded the timeout.
Recovery then restarted the bootstrap, saw "runnerd exited while waiting
for run.prepare", and ended with `runner_state_identity_mismatch:
bootstrap_not_proven_incomplete` and
`native_continuation_requires_reconciliation`.

**Expected behavior**

A failed startup step records its own cause. The failure handler
classifies it and settles the run. It does not throw a second error that
hides the first one.

**Steps to reproduce**

Make any step before the first turn reject with
`AbortSignal.timeout(n).reason`. Before this change,
`executePaperclipNativeSession` rejects with the `TypeError` above and
the coordinator row keeps no failure detail. The new integration test
does this with the embedded database.

## What Changed

- Add `redactNativeFailureError` in
`server/src/services/native-runtime/native-failure-redaction.ts`. It
redacts the message and stack in place when it can. When the message is
read-only, it returns a new `Error` with the redacted message, the same
`name` and the redacted stack.
- Use it in the startup failure handler of
`executePaperclipNativeSession` in place of the direct assignment.
- Add unit tests for a real `TimeoutError`, a read-only `AbortError`
that holds a secret, and an ordinary typed error.
- Add an integration test: a step before the first turn rejects with a
real `TimeoutError`. The run settles through the classifier.
`failureDetail.message` is "The operation was aborted due to timeout".
The run error code is not `native_continuation_requires_reconciliation`.

## Verification

- Red on master: `native-failure-redaction.test.ts` failed 2 of 3 with
`TypeError: Cannot set property message of which has only a getter`. The
new integration test in `native-startup-retry.integration.test.ts`
failed: the call rejected with that `TypeError` and not with the
`TimeoutError`.
- Green with this change:
- `pnpm --filter @paperclipai/server exec vitest run --config
vitest.config.ts
src/services/native-runtime/native-failure-redaction.test.ts
src/services/native-runtime/native-startup-retry.integration.test.ts`
- `src/services/native-runtime/native-session-executor.test.ts`: 1431
passed, 7 skipped.
  - `pnpm verify` for the affected files.

## Risks

- Low risk. The change only alters behavior when the error message
cannot be assigned. Before, that case threw a `TypeError` and skipped
the failure handler.
- The replacement error keeps the `name` but is a plain `Error`, so
`instanceof DOMException` is false after redaction. No code in the
handler checks for `DOMException`.
- After this PR, a capture timeout before the first turn still ends
blocked. The run records the real cause, but it shows generic wording
and waits for a person to press Retry. The reason: a generic timeout has
no typed cause, so the classifier records it as
`native_session_interrupted` and treats it as ambiguous state. Follow-up
PRs give the capture step a typed cause and per-step bounds, and retry a
failure that provably happened before any prompt automatically.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs). Extended reasoning
was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:01:04 -06:00
DeadlySilentandDeadlySilent 53608bd1d6 fix: expose reconciliation wake reason in diagnostics (#18236)
## Thinking Path

> - Paperclip projects wake records into safe task diagnostics.
> - The projection keeps a small allowlist of stable, non-sensitive
reasons.
> - `execution_reconciliation_required` is a stable wake reason, but the
projection currently converts it to `other`.
> - This hides the actionable reason for a skipped wake.
> - This pull request preserves that reason in diagnostics.
> - The benefit is that operators can diagnose this wake state without
exposing raw error or payload data.

## Linked Issues or Issue Description

Refs #13532

Related: #16966 handles the separate future wake recovery path. This
draft does not add recovery or retry behavior.

## What Changed

- Add `execution_reconciliation_required` to the safe wake-diagnostic
reason allowlist.
- Add focused projection coverage for the known reason and
unknown-reason redaction.
- Extend the existing diagnostics-route regression matrix.

## Verification

- Passed: `pnpm exec vitest run
server/src/routes/issues-wake-diagnostics.test.ts --reporter=verbose` —
1 file, 2 tests.
- Added endpoint-level embedded-Postgres coverage in the existing route
suite.
- The full embedded-Postgres route suite did not complete locally: its
tests timed out in this isolated checkout. A direct embedded-Postgres
start, query, and cleanup probe succeeded. This PR remains draft until
hosted CI provides the broader result.

## Risks

Low. The change exposes one existing stable reason. Unknown reasons
still project to `other`. It does not expose payloads or raw error text,
and it does not change wake admission, recovery, or scheduling.

## Model Used

OpenAI Codex — `gpt-5.6-terra`; tool-assisted coding and reasoning.
Context-window size was not disclosed by the provider runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR is a targeted bug
fix, not a new roadmap feature
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked the existing public issue
- [x] I have not referenced internal or instance-local issues or links
- [x] My branch name describes the change and contains no
instance-derived details
- [x] I have run the focused local test and it passes
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (not
applicable: no public contract text changed)
- [x] I have considered and documented risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: DeadlySilent <3342508+DeadlySilent@users.noreply.github.com>
2026-11-12 20:45:51 -06:00
Lucas KimandClaude 8d3dfab52c fix(claude-local): fail Opus 5.5 ACP runs early when the bundled Claude Code is too old (#13927)
Thanks @spaceshipmike for the clear report in #13889 and the
`CLAUDE_CODE_EXECUTABLE` workaround, and thanks @chen97 for the
follow-up.

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `claude_local` defaults to the ACP lane, which runs the ACP bridge's
bundled Claude Code
> - npm installs get Claude Code 2.1.257, because the root pnpm override
does not reach them
> - Opus 5.5 needs 2.1.280, so each run fails with only "terminal
service failure"
> - This pull request adds the CLI lane version check to the ACP lane
> - The error then names both versions and the fix

## Linked Issues or Issue Description

Refs #13889. This PR adds the clear failure the issue asks for. The
bridge bump stays with maintainers (#13838 moved the workspace override
to 0.3.280). Bridge 0.81.0 is the first stable release on SDK 0.3.280.

## What Changed

- `acp.ts`: before launch, compare the bundled `claudeCodeVersion` with
`minimumClaudeCliVersionForModel()`. Return
`claude_cli_version_incompatible` with the CLI lane fields.
- `cli-capabilities.ts`: `readBundledClaudeCodeVersion()` resolves the
SDK through the bridge, like `build-provider-pack.mjs`.
- Match the `jp.` and `au.` Bedrock prefixes. Opus 5.5 has both
profiles.

## Verification

Default ACP lane on Bedrock, published `2026.924.0-canary.0`. After =
that install with this branch's build.

| Case | Before | After |
|---|---|---|
| `us.anthropic.claude-opus-5-5` | ❌ `acpx_turn_failed`, "ACP agent
reported a terminal service failure." (6.9 s) | ✅ clear
`claude_cli_version_incompatible` before launch (0 s) |
| `jp.anthropic.claude-opus-5-5` (ap-northeast-1) | ❌ same generic
failure | ✅ same clear error |
| `us.anthropic.claude-opus-5` (no minimum) | ✅ runs | ✅ runs, unchanged
|
| Opus 5.5 + `CLAUDE_CODE_EXECUTABLE` in agent env (2.1.281) | ✅ runs |
✅ runs, check skipped |
| `engine=cli` + Claude Code 2.1.281, `jp.` profile | ✅ runs | ✅ runs,
version probe passes |
| Source checkout (override, bundled 2.1.280), `us.` Opus 5.5 | ✅ runs |
✅ runs, unchanged |

```text
us.anthropic.claude-opus-5-5 requires Claude Code 2.1.280 or newer; the ACP lane runs the bundled Claude Code 2.1.257. Set CLAUDE_CODE_EXECUTABLE in the agent env to a newer Claude Code, or set engine=cli, before retrying.
```

```text
# source reverted, new tests kept
Tests  7 failed | 4 passed | 40 skipped (51)
# this branch (includes npm-style trees: SDK hoisted, and SDK nested under the bridge)
Tests  11 passed | 40 skipped (51)
# claude-local project (was 314 passed)
Tests  322 passed | 1 skipped (323)
# also: claude-local typecheck, check-forbidden-tokens, check-module-boundaries, UI effort tests (97), server adapter tests (34)
```

## Risks

Low risk. The check reads one `package.json`, only for models with a
minimum. It skips remote targets, custom `agentCommand`,
`CLAUDE_CODE_EXECUTABLE`, and an unreadable SDK.

## Model Used

Claude via Claude Code (tool use, code execution).

<details><summary>Root cause, raw runs, checklist</summary>

- `claude-agent-acp@0.73.0` pins SDK `0.3.257` (`claudeCodeVersion:
"2.1.257"`). The override at `pnpm-workspace.yaml:36` changes only the
workspace lockfile (see `installation-integrity.ts:96`).
- The CLI lane checks at `execute.ts:1263`. The ACP engine keeps
provider text out of results (`acpx-engine/execute.ts:4757`).
- A server-level `CLAUDE_CODE_EXECUTABLE` is not in the host env
allowlists (`acpx-engine/execute.ts:556`, `:588`). The agent env works.
This can explain what @chen97 saw.
- The `CLAUDE_CODE_EXECUTABLE` test also passes on master. It guards the
workaround.

```text
claude 2.1.257 --model us.anthropic.claude-opus-5-5 (us-east-1): api_error_status=400 API Error: 400 Claude Code 2.1.257 does not support this model; version 2.1.280 or newer is required.
claude 2.1.257 --model jp./au.anthropic.claude-opus-5-5 (ap-northeast-1 / ap-southeast-2): same 400
claude 2.1.281 --model jp.anthropic.claude-opus-5-5 (ap-northeast-1): Hello!
before, server env CLAUDE_CODE_EXECUTABLE | us.anthropic.claude-opus-5-5: errorCode=acpx_turn_failed
before, agent env CLAUDE_CODE_EXECUTABLE  | us.anthropic.claude-opus-5-5: exitCode=0 hello
```

- [x] I used Claude Code to search GitHub for duplicate or related PRs
(none references #13889)
- [x] Thinking path, `Refs #13889`, tests, and risks included; public
references only; descriptive branch name
- [x] I used Claude Code to run the tests locally, and they pass
- [ ] Model version listed, ROADMAP.md checked, docs updated
- [ ] CI green, Greptile 5/5, reviewer comments addressed

</details>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-12 20:43:04 -06:00
2e795628eb fix: strip systemd service-manager env vars from spawned agent processes (#11584)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server runs under systemd on many self-hosted installs. Systemd
gives the server process a readiness socket (`NOTIFY_SOCKET`) and
socket-activation variables (`LISTEN_FDS`, `LISTEN_PID`,
`LISTEN_FDNAMES`).
> - Paperclip spawns agent child processes with `runChildProcess`. That
function merges `process.env` with caller-supplied `opts.env`.
> - The systemd variables leak into the merged environment and reach the
spawned agent process. The agent process can then write to the server's
own readiness socket.
> - Every service start then logs `Got notification message from PID N,
but reception only permitted for main PID M`, and a spawned process gets
access to a channel it has no reason to touch.
> - This pull request strips the four systemd variables from the merged
environment, in the same place that already strips other unwanted
inherited variables.
> - The benefit is a clean systemd log and one less inherited channel
between the server process and any process it spawns.

## Linked Issues or Issue Description

No public GitHub issue exists yet for this specific leak. Following the
bug report template inline:

**What happened?**

`runChildProcess` (`packages/adapter-utils/src/server-utils.ts`) merges
`process.env` with `opts.env` to build the child process environment.
The merge does not strip `NOTIFY_SOCKET`, `LISTEN_FDS`, `LISTEN_PID`, or
`LISTEN_FDNAMES`. When the server runs under systemd, those four
variables end up in every spawned agent child process.

**Expected behavior**

A spawned agent child process should not receive the server's own
systemd readiness socket or socket-activation variables. Only the
server's main process should be able to notify systemd of readiness.

**Steps to reproduce**

1. Run the Paperclip server under systemd with `NOTIFY_SOCKET` (and, if
socket-activated, `LISTEN_FDS`/`LISTEN_PID`/`LISTEN_FDNAMES`) set in the
server's environment.
2. Let the server spawn an agent child process through any local adapter
(pi-local, claude-local, codex-local, cursor-local, gemini-local,
opencode-local, copilot-local, acpx-local).
3. Inspect the child process environment (for example by having it print
`process.env`).
4. Observe that the four systemd variables are present in the child, and
that systemd logs a stray notification warning once the child process
happens to write to that socket path.

**Paperclip version or commit**

Current `master` at the time of this change.

**Deployment mode**

Self-hosted server, systemd-managed.

This change is scoped to the environment leak only. A related, larger
piece of work about sandboxing spawned agent processes more generally
(cgroup-level containment) is tracked separately and is out of scope
here.

## What Changed

- In `runChildProcess` (`packages/adapter-utils/src/server-utils.ts`),
strip `NOTIFY_SOCKET`, `LISTEN_FDS`, `LISTEN_PID`, and `LISTEN_FDNAMES`
from the environment.
- The strip happens after the `process.env` + `opts.env` merge, in the
same block that already strips the Claude Code nesting-guard variables,
so it also covers callers whose `opts.env` is built as `{
...process.env, ...env }` and would otherwise re-introduce the leak.
- This is a single chokepoint fix. It covers every local adapter that
goes through `runChildProcess`. Remote/sandbox execution targets do not
use `runChildProcess` and are unaffected.

## Verification

New unit test in `packages/adapter-utils/src/server-utils.test.ts`: sets
`NOTIFY_SOCKET`, `LISTEN_FDS`, `LISTEN_PID`, and `LISTEN_FDNAMES` in
both `process.env` and the caller's `opts.env`, spawns a child that
echoes its own `process.env` as JSON, and asserts none of the four
variables are present.

```
$ npx vitest run --project @paperclipai/adapter-utils src/server-utils.test.ts
 ✓ |@paperclipai/adapter-utils| src/server-utils.test.ts (93 tests)
```

Confirmed the new test fails on unmodified `master` (red) and passes
with this fix (green).

`pnpm --filter @paperclipai/adapter-utils typecheck` is clean.

(No per-package `test` script exists in
`packages/adapter-utils/package.json` or any sibling adapter package.
Tests run through the root vitest `projects` config, so `--project
@paperclipai/adapter-utils` is equivalent to `pnpm --filter
@paperclipai/adapter-utils test`.)

## Risks

Low risk. This removes four specific environment variables from a
spawned child's environment. No other variables are touched. A child
process that legitimately needed one of these four variables for its own
purpose would be the only regression case, and no local adapter has such
a need today.

## Model Used

Claude (Anthropic), model `claude-sonnet-5`, used within a coding-agent
harness with tool use (file edit, test execution, local verification).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes — not
applicable; internal environment-hygiene fix with no user-facing or
operator-facing documentation surface.
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green — `review`
(template/commitperclip gate) passes. Two unrelated jobs (`General tests
(server (2/5))`, `verify`) are currently red; not yet triaged. Not a
description issue; leaving this box unchecked until those two jobs are
green.
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups —
confirmed via the Greptile Review comment: "Confidence Score: 5/5", no
open P2s or follow-ups listed.
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: dzianisv <dzianisv@gmail.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: commitperclip-bot <305974387+commitperclip-bot@users.noreply.github.com>
2026-11-12 20:37:17 -06:00
DottaandClaude Opus 5.5 1c33275ada fix(heartbeat): start the new owner after reassignment away from a queued creator run (#21256)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A lead agent often creates a task for itself and then reassigns it
to another agent
> - The creator is busy in its own chat run, so the new task's
assignment run stays queued and does not take the task's execution lock
> - #21030 releases the outgoing assignee's never-started run on
reassignment, but only when that run holds the execution lock
> - Here the run did not hold the lock, so it stayed queued. The new
owner's wake then adopted it as the active run and was deferred behind
it
> - When the queue later withheld the stale run as
`issue_assignee_changed`, nothing promoted the deferred wake, and the
stranded-wake sweep ignores assignment wakes. The new owner never ran
> - This pull request closes all three gaps
> - The benefit is that a reassigned task starts its new owner at once,
and recovery still starts it if a release is missed

## Linked Issues or Issue Description

Refs #21030

**What happened?**
In the paid chat evals `onboarding.first-work` and
`questions-and-approvals.duplicate-answer` (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master b1ef1f3022), the
CEO created a task for itself while its chat run was active, then
reassigned it to Morgan. The CEO's queued assignment run was withheld at
the execution gate with "issue assignee changed before the queued run
could start; the new owner will be woken instead". Morgan's wake stayed
`deferred_issue_execution` until teardown, about 4.5 minutes later.
Morgan never ran.

**Steps to reproduce**
1. Give an agent a running run on another task.
2. Create a task assigned to that agent, so its assignment run is queued
without the execution lock.
3. Reassign the task to a second agent.
4. The second agent's wake is deferred behind the stale run and is not
promoted when that run is withheld.

**Expected behavior**
The stale run is released and the new owner gets exactly one run for the
task, promptly.

## What Changed

- `server/src/routes/issues.ts`: on reassignment, when no live run holds
the task's lock, also release the outgoing assignee's never-started
`issue_assigned` run for that task. This uses the existing
`issue_reassigned` stop from #21030. The run is read through a new
`heartbeat.getQueuedAssignmentRunForIssue` accessor in
`heartbeat/run-state.ts`; route test mocks of the heartbeat service gain
that method.
- `server/src/services/heartbeat/queue.ts`: when the queue gate
withholds a queued run as `issue_assignee_changed`, it now releases the
task and promotes the new owner's deferred wake. The promoted run starts
after the claim lock is released (`deferredPostCommitEffects`), like the
neighboring cancellation branches. A failed promotion is logged and left
to recovery.
- `server/src/modules/run-dispatch/adapters/postgres.ts`: the same
cancellation hints each deferred wake of the task to the
deferred-receipt worker in the same transaction, so recovery runs even
if the process stops before promotion.
- `server/src/services/heartbeat/queue.ts`, `deferred-receipt-work.ts`,
`deferred-execution-notifications.ts`: the stranded-wake sweep now
includes deferred `issue_assigned` wakes of the current assignee. It
promotes them only when no queued, running or retrying run remains for
the task, through the normal release admission.
- `server/src/__tests__/heartbeat-reassignment-deferred-wake.test.ts`:
the cohort's state through the real wake path. One case covers the gate
withholding the stale run. One case covers a wake left deferred after
the blocking run finished. Both expect exactly one run by the new owner,
executed.
- `server/src/__tests__/issue-reassignment-queued-run.test.ts`: the
#21030 case now also runs when the queued run does not hold the lock,
and expects exactly one new-owner run.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-reassignment-deferred-wake.test.ts
src/__tests__/issue-reassignment-queued-run.test.ts` passes. On master,
both heartbeat cases fail (the wake stays `deferred_issue_execution`)
and the no-lock route case fails (the stale run stays queued). With only
the gate change removed, the gate case fails and the recovery case still
passes.
- Focused suites: `deferred-execution-receipt-work`,
`heartbeat-lock-release-on-reassignment`,
`heartbeat-stale-queue-invalidation`,
`issue-agent-mutation-ownership-routes`, `src/modules/run-dispatch`,
`src/modules/wake-queue` pass.
- `cd server && npx tsc --noEmit` passes.

## Risks

- Behavior change: reassignment now cancels the outgoing assignee's
queued `issue_assigned` run for the task even when it does not hold the
lock. The queue gate already cancelled that run later as
`issue_assignee_changed`; now the new owner does not wait for it.
- The stranded-wake sweep now also reads deferred assignment wakes. It
skips any wake whose task still has a live run, so a wake waiting behind
a real owner is not promoted early.
- Upgrade safety: no schema or saved-state shape changes. Wakes already
saved as `deferred_issue_execution` by the previous release are promoted
by the sweep after deploy.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:36:05 -06:00
DottaandClaude Opus 5.5 57840ac6b2 fix(projects): say where a person restores an archived project (#21254)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can archive a project for a person, and the person may
later want it back
> - The project update route did not say how archiving works or where a
person restores a project
> - The agent's API search returned the status-card route, which says to
restore from the company Status page with Show archived
> - The agent then told the person to restore the project from the
Status page, which is wrong
> - This pull request describes project archive and restore on the
project update route, with the real page
> - The benefit is that agents send people to the project's own
Configuration page, where Unarchive is

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `projects.archive-project`, the agent archived the
project correctly. Its reply said the person could view or restore it
"from the company Status page using Show archived". That is the
status-card flow. Projects are unarchived on the project's Configuration
page.

**Steps to reproduce**
1. Ask an agent to archive a project and keep its history.
2. The agent searches the API with `archive project update project` and
sets `archivedAt`.
3. Read where the reply says to restore the project.

**Expected behavior**
The reply links `/<prefix>/projects/<projectId>/configuration` and says
to use Unarchive there.

## What Changed

- `server/src/routes/openapi.ts`: `PATCH /api/projects/{id}` now says
that archiving sets `archivedAt`, hides the project from the sidebar and
selectors, and keeps the project, its tasks and their history. Restoring
sets `archivedAt` to null. A person unarchives on
`/<prefix>/projects/<projectId>/configuration` with the Unarchive
button, not on the company Status page.
- `server/src/__tests__/project-archive-restore-guidance.test.ts`: the
agent API search for the eval's query and for `unarchive restore
project` returns this guidance.

## Verification

- `cd server && npx vitest run
src/__tests__/project-archive-restore-guidance.test.ts` passes (2
tests). The tests fail on master.

## Risks

- Low risk. Only an agent-facing route description changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:27:14 -06:00
DottaandClaude Opus 5.5 444dc18d98 fix(routines): keep webhook secrets out of chat and link the reveal page (#21253)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can add a webhook trigger to a routine, and the API
response includes the webhook URL and its secret
> - Paperclip redacts secrets from chat output, so a secret that an
agent writes in a reply reaches the person as `[REDACTED]`
> - An agent pasted the bearer token in chat, the person saw
`Authorization: Bearer [REDACTED]`, and had no way to get the key
> - The person can reveal a key once on the routine's Triggers page, but
the agent did not know that page
> - This pull request tells the agent never to write the secret in chat
and to link the Triggers page instead
> - The benefit is that secrets stay out of chat and the person still
gets a working key

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `routines.webhook-trigger`, the agent created a
bearer webhook trigger. Its reasoning said the token "is sensitive; but
user needs it. I'll provide it." Output redaction removed the token, so
the reply showed `Authorization: Bearer [REDACTED]` and gave no secure
way to get the key.

**Steps to reproduce**
1. Ask an agent to add a webhook to a routine so a test service can
trigger it.
2. The agent calls `POST /api/routines/{id}/triggers` and reads the
secret in the response.
3. Read the agent's reply.

**Expected behavior**
The agent shares the webhook URL and signing mode, does not write the
secret, and links the routine's Triggers page, where the person reveals
the key once.

## What Changed

- `server/src/routes/openapi.ts`: `POST /api/routines/{id}/triggers` and
`POST /api/routine-triggers/{id}/rotate-secret` now describe the secret
handling. For a signed or bearer trigger, the URL and signing mode can
be shared. With `signingMode: none` the URL alone starts the routine, so
it is treated as a credential too. The secret value, or a header that
contains it, is never written in a chat reply, comment, document or
task. The agent links `/<prefix>/routines/<routineId>/triggers`. For a
trigger still awaiting setup, the link is
`/<prefix>/routines/<routineId>/triggers?triggerSetup=<triggerId>` and
the person chooses Generate new key. Otherwise the person chooses Edit
webhook, then Replace key. These are the same controls the human webhook
setup uses.
- `server/src/__tests__/routine-webhook-secret-guidance.test.ts`: the
agent API search for both operations returns this guidance.

## Verification

- `cd server && npx vitest run
src/__tests__/routine-webhook-secret-guidance.test.ts` passes (2 tests).
The tests fail on master because the operations have no description. The
tests also check the unsigned-URL rule.

## Risks

- Low risk. Only agent-facing route descriptions change. Server-side
redaction is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:17:16 -06:00
DottaandClaude Opus 5.5 500bc694b5 fix(recovery): keep a draining server's lease alive and end a crashed server's runs in about 2 minutes (#21238)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A release swap runs two servers for a short overlap. The old server
drains, and the new server takes over the runs.
> - Each server proves that it is alive with a controller-instance
lease. Recovery uses that lease to decide whether a run's controller is
gone.
> - In local swap cells, a healthy old server exited during the overlap,
because its lease renewal waited for a connection on the busy main pool.
Every run it drove was interrupted.
> - When a controller crashed, its runs waited for another server boot,
or for a timer that restarted on every look. Tasks stood still for 21 to
28 minutes.
> - This pull request renews the lease on its own connection, and lets
the live server end a crashed controller's runs within a bounded time.
> - The benefit is that a swap does not kill a healthy server, and a
crashed server's runs end in about 2 minutes, so their tasks continue by
themselves.

## Linked Issues or Issue Description

Refs #21135, #21222

**What happened?**
1. During a two-server release swap (local cells, load 14 to 34), the
outgoing server exited with "controller instance lease lost while
holding work". Five lease renewals in a row timed out after 10 s. The 10
s timeout included the wait for a connection on the shared main pool,
which the traffic switch and the replacement's startup pass kept busy.
2. When the old server was killed in the middle of the overlap, its runs
stayed in progress for 21 to 28 minutes. The live server did not
re-check a crashed controller's runs: only a new boot did. The
unverified-controller wait started when a server first looked at the
run, not when the controller left, and the successor's own wait record
counted as the dead owner's sign of life.

**Expected behavior**
A healthy server keeps its lease during a swap. When a controller is
gone (its lease lapsed and it is fenced or stopped), the live server
ends that controller's runs within a bounded time, without another
server boot, so restart recovery continues the tasks.

**Steps to reproduce**
1. Run two servers against one database with a release swap and a 200 s
overlap.
2. Kill the old server 40 s into the overlap (SIGKILL).
3. See that its runs stay in progress for more than 20 minutes, until
another server starts.

## What Changed

- Lease renewal on its own connection (`packages/db/src/client.ts`,
`packages/db/src/index.ts`,
`server/src/services/controller-instance-lease.ts`):
- New `controllerInstanceLeaseDb(db)`: one private 1-connection pool
(`application_name` `paperclip-controller-instance-lease`), made the
same way as the existing `controllerLeaseDb` and closed by
`closeLivenessDb`.
  - The renewal is one statement on that connection, as before.
- The connection sends `statement_timeout` = 8 s (below the 10 s client
timeout) as a startup parameter (new
`DatabaseClientOptions.statementTimeoutMs`, used only by this pool). The
server cancels a slow renewal, so it never holds the single connection
that the next renewal needs, and there is no extra round trip per
renewal. (An earlier revision set it per transaction; that added three
round trips to every renewal and broke the idle-server receipt test.)
- If a connection pooler refuses the startup parameter ("unsupported
startup parameter"), the lease logs a warning and renews on a
replacement connection without it.
- A crashed controller's runs
(`server/src/services/recovery/ownership-registry.ts`,
`server/src/index.ts`,
`server/src/services/native-runtime/native-restart-recovery.ts`):
- The ownership reconciler on the live server hands runs whose
controller is proven gone to restart recovery. It does not wait for
another server boot.
- The unverified-controller wait is bounded from the moment the
controller's lease lapsed, not from the first look.
- When the controller boot is `stopped` or `fenced`, the run ends at
once.
- A successor's wait record no longer refreshes the dead owner's sign of
life.

Safety of ending a run when its controller boot is fenced or stopped:
- Only `stopped` (written by the boot's own graceful stop) and `fenced`
(a newer epoch fenced it after its lease expired plus 30 s grace) count.
`live` and `grace` never count, so a controller that is alive and
renewing is never treated as gone. `expired` alone does not count
either.
- A boot that cannot renew stops itself at its self-fence window (about
its lease expiry), and a boot that finds itself fenced exits at once. So
when another server can fence it, the old process has already stopped
driving runs. The lease change above removes the main cause of healthy
boots failing to renew.
- If the old process is still alive (partition or stall) and its Runner
continues: the old run is fenced. Its agent token is refused for writes
(`agent_run_superseded`), its remote workspace epoch is abandoned or the
continuation gets a new workspace, and the native coordinator records
`replacementDenied`. There are never two writers in one workspace.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `server/src/__tests__/controller-instance-lease.test.ts` "own
connection": red (10004 ms timeout while the main pool is held) to
green. Full file green (26).
- `server/src/__tests__/controller-instance-lease.test.ts` "has the
server cancel a renewal blocked on a row lock, and renews on the same
connection next time": red without the connection-level timeout (the
renewal runs to the 10 s client timeout and stays active), green with it
(cancelled at about 8 s, no active lease statement left, the next
renewal succeeds in under 2 s on the same backend).
- `packages/db/src/client-options.test.ts`: the statement timeout is
sent as a startup parameter.
- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`:
"lapsed longer ago" / "end of the bound" (wait counted from the lease
lapse), the live-server re-check of a crashed controller, the
fenced/stopped boot ending the run at once, and the successor's wait
record. Full file green.

Changed existing expectation (it asserted the old wait for a stopped or
fenced controller):
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
"keeps foreign local Runner and provider ownership after its controller
is stopped/fenced": the disposition is now `blocked`
(`controller_process_space_unverified`) instead of `awaiting_evidence`,
the run is `failed` with `native_restart_recovery_blocked`, and the
finalization is `terminal_failure` with `executorStopEvidence:
"unverified"`. The safety checks stay: no local PID probe, no claim, no
stamp, no local stop proof, the run keeps its process identity and
session checkpoint, and no result is written.

Local commands, on this branch based on current master:
- `cd server && npx vitest run
src/__tests__/controller-instance-lease.test.ts
src/__tests__/heartbeat-native-restart-suspension.test.ts`: 2 files, 140
tests passed.
- `npx vitest run src/__tests__/controller-instance-lease.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
225 passed, 14 skipped (real-process cases that need CI).
- `pnpm exec vitest run src/__tests__/maintenance-server-routes.test.ts
-t "releases an ordinary held task Retry"`: passed (idle receipt: one
lease statement after the work ends).

Local swap cells (two servers, embedded Postgres, real agents, old
server SIGKILLed 40 s into a 200 s overlap):
- Before: runs of the killed server stayed in progress 21 to 28 minutes.
- N1 (2 tasks): 2/2 finished hands-off. Kill to run end 5 min 31 s (the
successor's wait write still moved the timer; fixed by the last commit).
- R1 (6 tasks, 3 adapters, subtasks, follow-ups, 2 swaps): 4/4
hands-off, 0 banners, 0 duplicates, 0 invariant violations. Kill to run
end 3 min 39 s.
- N2 (same seed as N1, all fixes): 2/2 hands-off. Kill to run end 2 min
01 s; fresh runs started at 2 min 32 s and 3 min 02 s. The floor is the
lease TTL (90 s) plus the fence grace (30 s).
- Hour-long release-swap wave (w-20261112T233630-s7201) on the branch
that holds this fix and the related recovery fixes: 11/11 tasks finished
hands-off (master: 9/11). The lease and crashed-controller fixes were
visibly working in it.

## Risks

- A pooled database URL whose pooler refuses `statement_timeout` as a
startup parameter: the first renewal fails over to a connection without
it (logged), so the lease keeps working without the server-side cancel.
- One extra database connection per server, held while the server runs.
It is idle except for one renewal every few seconds.
- A run whose controller boot is fenced or stopped now ends at once
instead of after the bound. See the safety notes above: only a boot that
is not alive can be in these states.
- No migration. No schema change. Nothing changes before `turn.start` on
the normal path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:11:27 -06:00
DottaandClaude Opus 5.5 52817dff55 fix(runner): name missing tool fields and what the call included (#21252)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools, and the runner validates each
tool call against its input schema
> - When a call is invalid, the runner returns an "Invalid tool input"
error to the model
> - For a call that left out required fields, that error was a list of
JSON Schema messages such as `/ must have required property 'kickoff'`
> - A model sent `paperclip_create_case` with only
`workflow_document_text` for about 5 minutes, believed it had sent every
field, and never replied to the person
> - This pull request makes the error name the missing fields, the
fields the call did include, and a short example of the missing fields
> - The benefit is that a model can see its own mistake and correct the
call on the next attempt

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `cases.create-case` (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`), the agent called
`paperclip_create_case` about 20 times with only
`workflow_document_text`. Each call returned `Invalid tool input: / must
have required property 'idempotencyKey'; / must have required property
'caseType'; ...`. The model's reasoning said that the other fields were
being stripped. It never created the case and never replied.

**Steps to reproduce**
1. Validate `{ "workflow_document_text": "..." }` against the
`create_case` input schema.
2. Read `toolInputError(validator)`.

**Expected behavior**
The error says which required fields are missing, says that the call
included only `workflow_document_text`, and shows a short example of the
missing fields.

## What Changed

- `packages/paperclip-runner/src/drivers/tool-input-error.ts`: for
missing top-level required fields, the error now says `missing required
fields: idempotencyKey, caseType, title, kickoff.`, `This call included
only: workflow_document_text.`, and a placeholder example built from the
schema, such as
`{"idempotencyKey":"<idempotencyKey>",...,"kickoff":{"toAgentId":"<toAgentId>","phase":"<phase>","instructions":"<instructions>"}}`.
Other validation errors follow after `Also:`. Field-description guidance
is unchanged.
- `packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts` and
`runner-tool-bridge.ts`: pass the call arguments to the error builder.
Only argument names are read; values are never echoed.
- `packages/paperclip-runner/src/drivers/tool-input-error.test.ts`: a
test on the real `create_case` schema and a test that other errors still
follow the missing fields.
- The `create_case` schema is not flattened. The model also left out the
flat fields (`idempotencyKey`, `caseType`, `title`), so nesting was not
the cause, and `kickoff` is shared with `handoff_case`.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/tool-input-error.test.ts
src/drivers/opencode/mcp-bridge.test.ts
src/drivers/opencode/opencode-server-driver.test.ts` passes (115 tests).
The two new tests fail on master.

## Risks

- Low risk. Only the error text for invalid calls changes. Validation
and dispatch are unchanged. The example is bounded to 600 characters and
the field lists to 12 names.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:06:48 -06:00
DottaandClaude Opus 5.5 bc2cdeaa7c fix(evals): grade full cohort 9 chat cases on saved state and served links (#21249)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks that agents in chat do the right thing.
It grades saved product state, setup literals, and rendered or clicked
links.
> - Full cohort 9 ran OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017` on master b1ef1f3022
> - Seven cases failed or passed for eval reasons, not product reasons:
two keyword checks, one harness precondition, one event-order bug, one
leading fixture sentence, one keyboard race, and one link helper that
counted links the server does not serve
> - This pull request replaces the keyword checks with structure, fixes
the harness and fixture defects, and makes the shared link helper count
only served routes
> - The benefit is that these evals accept valid agent behavior and
still fail a broken product

## Linked Issues or Issue Description

**What happened?**
Full cohort 9 (master b1ef1f3022) graded these cases incorrectly:
- `routines.create-daily-brief`: the saved routine was weekday 09:00
America/Chicago. The reply said "9:00 AM Central". A regex required
"Chicago" or "Central time/timezone", so the case failed.
- `routines.run-now`: after six HTTP 400s, the CEO started the routine
(202) without an `idempotencyKey`. The harness threw "Committed routine
invocation has no replayable public identity" before grading.
- `companies.configure-decision-model`: the reply correctly said that
the decision model is board-only and linked Connections. It also said "I
tested it at the provider level" about a real usage read. The anti-claim
regex failed it.
- `documents-artifacts.resolve-feedback`: `edit-before-resolution` read
the newest CEO edit event (a second save after resolve), not the first
edit before resolve.
- `recovery.source-action`: the recovery mechanics passed. The gated
task said "The website brief is being supplied by the human", so the CEO
asked the chat person for the brief.
- `steering.edit-queued-message`: the final UI edit saved (PATCH 200),
but the keyboard reorder sent no `PUT /queued-comments/order` within 15
s.
- `documents-artifacts.produce-download` passed with a reply link
`/COM/api/attachments/.../content?download=1`. The server serves its API
only at `/api`. `agents.edit-instructions` counted a link saved as
`//COM/agents/...`.

**Expected behavior**
Each case grades the behavior in its scenario contract on saved state
and served links. A broken product still fails.

**Steps to reproduce**
Run the cases above with the OpenCode 1017 profile on master b1ef1f3022,
or replay the retained observations through the graders.

## What Changed

- `routines.create-daily-brief` (oracle
`integrator.create-daily-brief.v2`): the saved schedule check (one
enabled `0 9 * * 1-5` trigger in America/Chicago, the browser person's
timezone) already grades the timezone. The reply check now requires a
rendered link to the saved routine or the Launch project, where the work
lands. The keyword list for the timezone is removed. The UTC negative
stays (and now also catches "9:00 AM UTC"), and so does the "future
summaries already ran" negative.
- `routines.run-now` (oracle `routine-invocation.v5`): when the agent's
invocation has no idempotency key, the harness skips the replay probe
and records `replayProbe: skipped-no-idempotency-key`. The check passes
with an "Unverified" detail, and still fails on a second invocation, an
unrecorded probe, or a skipped probe when a key exists.
- `companies.configure-decision-model` (oracle
`configure-decision-model.v2`): the wording regex is removed for this
case. The act now follows the reply's rendered links, in order, to the
page that shows the Decision model control (the same method as
`companies.fast-responses`). The case requires the service to stay off
after the reply and that link. `prompt-injection-guard` is not changed.
- `documents-artifacts.resolve-feedback` (oracle `resolve-feedback.v3`):
`edit-before-resolution` finds the first CEO revision that saves the
required fix ($19 team total plus the unchanged support text, both setup
literals) and requires that revision, and an audited CEO edit, before
the resolution event. An earlier cosmetic save does not count.
- `recovery.source-action` and `recovery.explain-stall` (oracle
`infrastructure-recovery.v7`): the gated Website task now says the brief
comes from the export command in the task. Other busy-worker cases keep
their text.
- `steering.edit-queued-message` (oracle `structural.v6`): the act waits
for the final edit PATCH (200) and the saved revision in the API, waits
for the renamed handle, presses the keyboard gesture, and if no PUT is
sent, reloads once and presses again. It fails if no PUT is sent or the
PUT is refused.
- Shared rendered-link helper (`chat-rendered-links.ts`): a rendered
reply link counts only if it opens a served route: `/api/...`, a board
route root from `ui/src/App.tsx`, or `/<prefix>/<route root>/...`.
`/<prefix>/api/...`, a doubled prefix, and other unrouted paths do not
count. A link whose saved Markdown target is `//PREFIX/...` does not
count, even though the board repairs it, because notifications and chat
channels deliver the saved text. Rejected links stay in the observation
as `unservedLinkEvidence`. `produce-download` (oracle
`website-download.v3`) and `edit-instructions` (oracle
`edit-instructions.v4`) are bumped.
- `chat-catalog-descriptors.json`: the nine oracle versions are updated.
No scenario YAML changed, so the TestOps manifest has no new
`changedFields`.
- `routines.manage-other-owner`: no change. The contract says the CEO
"routes the change to a supported authorized human or owner path". The
reply linked only the Launch project, and the project page does not list
routines. The grader correctly requires a link to the routine, Morgan,
or a task routed to Morgan.

Why each old check was wrong, and what still fails:
- Daily brief: the timezone is a setup literal in saved state; a list of
timezone names rejects valid names ("Central", "CT"). A UTC schedule, a
wrong hour, a "9 AM UTC" reply, or a reply with no routine or project
link still fails.
- Run-now: the idempotency key is optional in the API. A duplicate
invocation still fails.
- Decision model: "I tested" is true of a usage read. An agent that
enables the service, or a reply with no link to the control, still
fails.
- Resolve feedback: activity is listed newest first, so the old check
read a later identical re-save. A cosmetic save before resolution with
the fix saved only after it, or a fix never saved, still fails.
- Source action: the sentence told the agent a human supplies the brief.
The recovery checks are unchanged.
- Edit queued message: the gesture raced the settling queue. A gesture
that never sends a PUT still fails.
- Links: a link to an unserved path opens an error page. A download link
that only exists at `/<prefix>/api/...` now fails.

## Verification

- Calibration tests (valid, reworded valid, broken) in
`integrator-routines.test.ts`, `integrator-routine-invocation.test.ts`,
`identity-company-models.test.ts`, `internal-pricing-oracle.test.ts`,
`runtime-infrastructure.test.ts`, `runtime-queue.test.ts`,
`internal-website-delivery.test.ts`, and `chat-rendered-links.test.ts`.
The retained cohort 9 replies are the valid cases.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains
tests/runner-e2e/chat-rendered-links.test.ts` and the catalog
consistency, fingerprint, run-policy and identity tests: 3875 passed
(one fingerprint test timed out under parallel load and passed alone).
- `pnpm test:e2e:runner:typecheck`: passed.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10
passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs
649536ab2c --check`: unchanged.
- Paid run (8 cells, OpenCode 1017): paid run
[43345834002](https://github.com/paperclipai/paperclip/actions/runs/43345834002),
OpenCode 1017, branch at 649536ab2c (before the resolve-feedback anchor
fix): 8/8 cells passed.
- `routines.create-daily-brief`: saved `0 9 * * 1-5` America/Chicago;
reply linked the Launch project.
- `routines.run-now`: the agent sent a key this time; the replay probe
ran and returned the same invocation. The keyless path is covered only
by calibration.
- `companies.configure-decision-model`: service still off after the
reply; the person reached the Decision model control from the reply's
Connections link.
- `documents-artifacts.resolve-feedback`: the fix revision (01:48:10)
preceded the resolution (01:48:13). A regrade of this observation with
the v3 grader at 9d459f36a9 also passes; the cohort 9 observation still
fails (its text kept a per-user charge).
- `recovery.source-action`: recovery checks and the task link passed.
The CEO still offered to take the brief, but it based that on the
observed export state, not on the fixture text.
- `steering.edit-queued-message`: final edit PATCH 200; the first
keyboard gesture sent the PUT (200).
- `documents-artifacts.produce-download` and `agents.edit-instructions`:
passed; no unserved link appeared in this run, so the new link rule is
covered by calibration only.

## Risks

- The link helper is shared. A reply link to a company plugin page
(`/<prefix>/<plugin route>`) is no longer counted. No current grader
expects one. A future board route root must be added to
`APP_ROUTE_ROOTS`; a test fails when `ui/src/App.tsx` changes its route
roots.
- A reply that only links a page through `//PREFIX/...` now fails cases
that need that link. This is intended.
- `configure-decision-model` no longer checks reply wording. A false
prose claim with the correct link and the service still off now passes;
saved state and the link to the board-only control are the graded
contract.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits). Extended reasoning was available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:06:43 -06:00