Commit Graph
868 Commits
Author SHA1 Message Date
Nicky LeachandClaude Fable 5.2 23c2343015 feat(server): read a retained provider pack from its Runner image (#21308)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner executes agent work in a sandbox. It needs a provider
pack of agent tools, and the server uploads that pack when the sandbox
does not have it
> - A retained session keeps the pack it started with. When that pack is
not the server's own, the server downloads the pack's transport archives
from a registry (#19779)
> - Today the server looks for them in the application image tagged
`sha-<commit that built the pack>`
> - `master` now publishes the Runner one time for each set of inputs,
as `ghcr.io/paperclipai/paperclip-runner`. A pack from that image names
the commit that published the image, and few commits get an application
image, so that tag often does not exist
> - This pull request reads such a pack from its Runner image, and keeps
the old source for every other pack and for a Runner image that cannot
be read
> - The benefit is that a retained session can still get its pack when
the pack came out of the published Runner image

## Linked Issues or Issue Description

Refs #19779 (the retained pack source), #21228 (the pack field
`runnerInputsDigest`), and #21204, #21229, #21230 (the Runner image). No
issue exists for this change.

**Problem or motivation**

`retained-provider-pack-oci.ts` reads
`ghcr.io/paperclipai/paperclip:sha-<runnerSourceRevision>`. It needs
that image to exist, to carry the pack, and to have one layer with a
known build-history line. For a pack from a published Runner image,
`runnerSourceRevision` is the commit that published the image.
`docker.yml` writes `sha-<commit>` only when its `master` run for that
commit completes, and a newer push can replace a waiting run. The
restart-chaos harness has the same assumption in another place: it
requires the pack of a sandbox image to be built by the commit in the
image's `org.opencontainers.image.revision` label.

**Proposed solution**

A pack that names its Runner image in `payload.runnerInputsDigest` is
read from `ghcr.io/paperclipai/paperclip-runner:inputs-<digest>`. A
sandbox image can name the commit that built its Runner in a label of
its own. No application image carries a pack with that field yet, and no
image carries that label, so behaviour today does not change.

**Alternatives considered**

- Keep the application image as the only source and require an
application image for every Runner publication. Publication would then
depend on `docker.yml`, which serves one ref at a time.
- Take a fixed number of layers ("the image has four, take the last
two"). A server in the field cannot be changed when the image gets
another layer. The server now uses the rule that holds for any number of
layers: the full image is the slim image plus transport layers.
- Fall back to the application image for every failure of the Runner
image. A Runner image that exists and does not match the pack is a
fault, and a second source would hide it.

**Roadmap alignment**

This is build and runtime infrastructure. It adds no product feature and
does not overlap an item in `ROADMAP.md`.

This is part 3 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image (this
pull request)
4. #21310: a registry build cache for the Runner image build

This pull request uses no code from parts 1 and 2. It is stacked on them
because all four parts edit `doc/runner-image-contract.md`.

## What Changed

- `server/src/services/native-runtime/retained-provider-pack-oci.ts`,
for a pack whose `payload.runnerInputsDigest` is 64 hexadecimal
characters:
- The server reads the full index `inputs-<digest>` of the Runner image.
The index must carry `io.paperclip.runner.inputs-digest` equal to the
pack's value, `org.opencontainers.image.revision` equal to the pack's
`runnerSourceRevision`, and a valid digest in
`io.paperclip.runner.slim-digest`.
- It reads the slim index by that digest, never by the slim tag, and
checks the bytes it gets against the digest.
- For the pack's platform, the layer digests of the slim image must be
the first layer digests of the full image, in the same order. The layers
after them are the transport layers. There must be at least one and at
most eight, and their total size has the existing limit.
- The server downloads only those layers. It checks each against its
descriptor and refuses every entry that is not a transport archive, the
receipt or the `provider-pack.archives` directory. A file may not be in
two layers.
- When the Runner image cannot be read, the server reads the application
image as before. That covers a registry error status for the token, for
either index or for either manifest: for example 404 for a tag or a slim
index that does not exist. Nothing is written before that point.
- A Runner image that can be read and does not match fails with
`runner_remote_provider_source_invalid`. The server does not try the
application image then.
- The same file, for every other pack: the old path, unchanged in what
it checks. Its code moved into functions that both paths use (`request`,
`pullToken`, `platformImage`, `receiveTransportLayer`).
- The downloaded layer file is removed as soon as it is unpacked.
Before, the importer removed it after its checks, so
`retained-provider-pack-source.ts` loses that one line.
- The cache record gets an optional `runnerImage` field with the inputs
digest and the digests of the transport layers. `transportLayerDigest`
is the last of them and `transportLayerBytes` their total. An older
server ignores the new field.
- `tests/restart-chaos/daytona-image.ts`: when the image has a non-empty
label `io.paperclip.runner.source-revision`, the pack must be built by
that commit. Otherwise the pack must be built by the commit in
`org.opencontainers.image.revision`, as before. The receipt field
`providerPackRunnerSourceRevision` records the pack's commit.
- `doc/runner-image-contract.md` has a new section, "Reading the
transport archives alone". `tests/restart-chaos/README.md` describes the
label.

One choice needs a reviewer's eye. When the full index can be read and
the slim index cannot (a registry error status), the server falls back
to the application image; it does not fail. Without the slim manifest
the server cannot tell which layers are transport layers, so it has not
accepted any byte of the Runner image, and the old source has its own
complete checks. A slim index that is read and has other bytes than its
digest still fails.

## Verification

Linux (`node:24.22.3-bookworm`, GNU tar 1.34, zstd 1.5.4, the worktree
mounted read-only):

- In `server/`: `retained-provider-pack-source.test.ts` (34 tests, 15 of
them new), `provider-pack-archive.test.ts`,
`remote-provider-pack-verification.test.ts`,
`remote-provider-pack.test.ts`: 127 tests pass in 4 files.
- `node node_modules/vitest/vitest.mjs run --config
tests/restart-chaos/vitest.config.ts
tests/restart-chaos/daytona-image.test.ts`: 25 tests pass, 5 of them
new.

macOS (Node 26.12.0):

- `daytona-image.test.ts`: 25 tests pass.
- `retained-provider-pack-source.test.ts`: the 15 new tests pass. Six
older tests fail, with and without this change. They run the real
uploader, which needs GNU tar, and this host has bsdtar.
- `tsc` on the two changed server files and the test, and on the two
restart-chaos files, with the settings of their test projects: no error.

The new cases:

- Read from the Runner image with 6 registry requests (two indexes, two
manifests, two layers), none to the application image and none for the
slim tag.
- Take every layer after the slim image's: one transport layer, and
three.
- Fall back when the tag does not exist, when the token is refused, and
when the slim index cannot be reached.
- Refuse, with no request to the application image: an index for other
inputs, an index for another revision, no slim annotation, a slim
annotation that is a tag, a slim index with other bytes than its digest,
a slim image that is not a prefix of the full image, a slim image equal
to the full image, nine transport layers, and a transport layer that
holds a pack file.
- Four deliberate faults in the reader each make a test fail: the slim
index bytes are not checked, no prefix check, the slim annotation is not
validated, and no bounds on the number of transport layers.

A real run, in the same Linux container (`linux/arm64`), with no
credential:

- The script took `provider-pack/provider-pack.json` of `linux/arm64`
from the published image
`ghcr.io/paperclipai/paperclip-runner:inputs-0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`.
Pack digest
`sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f`,
`runnerSourceRevision` `25e9a9c2e9277f3be8649be8c809cc81bde6f0c5`.
- `resolveProviderPackUploadSource`, with another pack as the server's
own, returned `{ archiveOnly: true }` after 23 seconds. That function
calls `receiveRetainedProviderPack` and then the importer's checks: the
manifest inside the overlay archive is the selected manifest, every
archive matches the receipt, and the content of every file archive
matches the digest of its file.
- The server code made 9 requests: one anonymous token request, the full
index, the `arm64` manifest, the slim index `sha256:f63ea436…4f3c` by
digest, its `arm64` manifest, and two layers, each with one redirect. No
request went to the application image.
- The cache record:

  ```json
  {
    "schema": "paperclip.retained-provider-pack-source.v1",
    "sourceRevision": "25e9a9c2e9277f3be8649be8c809cc81bde6f0c5",
"providerPackDigest":
"sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f",
"imageIndexDigest":
"sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162",
"imageManifestDigest":
"sha256:73dbe864fbab6208616f0aa6f1610a7ece4ef22f68a73d30901cbaea60139589",
"imageConfigDigest":
"sha256:740e48463d5caf0bfead4b923cdbd8b220acb93d058419c3db9d00c7f1831d65",
"transportLayerDigest":
"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf",
    "transportLayerBytes": 426602696,
    "runnerImage": {
"inputsDigest":
"0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca",
      "transportLayerDigests": [

"sha256:6a340088ca2cfa85f10f0385d9c8f5cdccb22c3a2bbb2062620b1d90616ba5b5",

"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf"
      ]
    },
"receiptSha256":
"2afdd26645845369d9e851b1e6e2884eb6c3884ea7c4866a1eeb2efb9fb230d7",
    "verifiedTarPaths": true
  }
  ```

- `uploadVerifiedProviderPack` then installed the pack from that source
through a local stand-in for the sandbox, with no further registry
request, in 4 seconds. The installed tree is 1,249,060,841 bytes. Its
`provider-pack.json` is the selected manifest, all 7 artifacts of the
manifest have the digest the manifest names, and the pack's own Node
prints `v24.22.3`.

Never run:

- The same real run for `linux/amd64`, whose transport layers are about
1 GB.
- A session in a real sandbox. The real run used a local stand-in for
the sandbox.
- The fallback against the real registry. Fixtures cover it.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The old path was moved into shared functions. Its checks are the same,
and its 19 existing tests pass on Linux, but a review must read that
code as changed.
- The layer file of the old path is now removed before the importer's
checks, not after them. The importer does not read it.
- The reader depends on two rules of the image contract: the full index
names its slim index, and the full image starts with the slim image's
layers. A Runner image that breaks either rule is refused, with no
fallback.
- The fallback covers a registry error status only. A network error ends
the import, as it does today.
- The label `io.paperclip.runner.source-revision` is new and no image
carries it yet. The change to the sandbox image that sets it must use
this name.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:02:19 -08:00
DottaandClaude Opus 5.5 0477f622d6 fix(recovery): keep the sandbox that holds abandoned unsaved files out of replacement and cleanup (#21304)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in remote sandboxes, and the workspace files live in
those sandboxes until a save
> - When a run cannot resume exactly, the fresh continuation fallback
(#21135) abandons the old workspace epoch and keeps its unsaved files
"recorded and unapplied" on the old sandbox
> - The abandonment clears the dirty flag, the writer and the location,
so the workspace looks clean. No code reads the abandonment receipt
> - A later failed resume can then replace the sandbox, and cleanup can
destroy it. That sandbox holds the only copy of the unsaved files
> - This pull request adds one shared hold. The task continues on a new
sandbox, and the old sandbox is parked until its files are saved,
discarded, or kept for 7 days
> - The benefit is that the fallback keeps its promise. Abandoned files
are not lost

## Linked Issues or Issue Description

Refs #21135
Refs #21297

**What happened?**
After the fresh continuation fallback abandoned a workspace epoch, the
receipt in `remoteWorkspaceAbandonedEpochs` was write-only.
`reusableSandboxReplacementIsSafe` saw a clean workspace with a
committed save and allowed replacement.
`retireReplacedReusableSandboxLease` then moved the old lease to
`pending_cleanup`, and the cleanup sweep destroyed the sandbox with the
unsaved files. `nativeWorkspaceExportPendingForLease` (the check that
the sweep, the retained orphan settlement and the scoped destroy paths
use) also answered "nothing pending".

**What did you expect?**
A sandbox that holds abandoned, unsaved files stays until the files are
saved, a person discards them, or a stated retention period ends.

**Steps to reproduce**
1. Abandon a remote workspace epoch whose location names sandbox L.
2. Let the next run fail to resume L.
3. The replacement decision says safe, L goes to `pending_cleanup`, and
the sweep destroys it.

## What Changed

- New `server/src/services/remote-workspace-abandoned-files-hold.ts`
holds the single predicate for "this lease holds abandoned files". It is
one SQL fragment (`abandonedWorkspaceFilesHeldSql`) plus a TypeScript
wrapper (`abandonedWorkspaceFilesHeldOnLease`). A lease is held while a
workspace in its company has an abandoned-epoch receipt whose
`location.providerLeaseId` equals the lease's provider lease id, and
that receipt has no `released` object. A receipt written by the current
release (no new fields) counts as unreleased.
- Replace and park. A failed resume of a held sandbox still lets the run
replace it, so the task continues at once on a new sandbox restored from
the last committed save. `retireReplacedReusableSandboxLease` parks the
old lease as `retained` with the `abandonedWorkspaceFilesParked` marker
(failure reason `abandoned_workspace_files_held`) instead of
`pending_cleanup`. `reusableLeaseCanBeResumed` and
`reusableLeaseCanBeCleanedUp` skip a parked lease, so later runs never
select it again. The check is a metadata read with no query, so the run
path does no extra work.
- Every path that can destroy a sandbox now honours the hold:
- `nativeWorkspaceExportPendingForLease` answers pending for a held
lease. This covers the pending cleanup sweep (defer), the orphaned
active lease sweep, the retained orphan settlement `heldBy`,
`destroyClaimedSandboxLease` (terminal workspace reaper
`destroyRemoteWorkspaceLease`, `destroyReusableSandboxLeases` and the
environment delete path, and run lease release with `destroy`), the
`destroyRunLease` dispatcher, and the plugin sandbox teardown retry.
- The run lease settlement (`environment_run_lease_active`) is the one
caller that ignores the hold (`{ destructive: false }`). It never
destroys a sandbox. It stops the sandbox and keeps it, or it moves a
lease without a sandbox to `pending_cleanup`, where the sweep parks a
held lease. If it waited for the hold, a dead run's lease would stay
`active` and block the retry for the whole retention period.
- `destroyReusableSandboxLease` parks the lease and returns without a
provider call. This covers obsolete reusable lease cleanup, the destroy
after an expired or mismatched resume, release with `expired`, and the
sandbox driver's `destroyRunLease`.
- The `retryPendingSandboxTeardown` dispatcher refuses a destructive
teardown of a held lease. Stop-only cleanup still runs, because a stop
keeps the filesystem.
- The retained orphan listing (`alwaysHeldSql`) uses the same SQL
fragment, so held orphans are not polled.
- Release. `releaseAbandonedWorkspaceFiles` marks a receipt `released`
with a reason (`recovered`, `discarded`, `retention_expired`) and an
actor. It moves the parked leases for that sandbox to `pending_cleanup`
when no other unreleased receipt holds them, and it writes a
`workspace.abandoned_files_released` activity entry on the task in plain
language. No export path for abandoned epochs exists today. The
`recovered` reason is the hook for one; this pull request adds no new
export mechanism.
- Retention expiry. New receipts carry `retainUntil` (7 days). An old
receipt uses `abandonedAt` plus 7 days. `nextEnvironmentCleanupAt`
includes the earliest retention end of any unreleased receipt. The
shared cleanup worker therefore wakes at that real deadline, and a
restart rebuilds it from the database. At the deadline the sweep runs
`expireAbandonedWorkspaceFileHolds`. This releases each due receipt as
an attributed system decision (actor `workspace-retention`) and logs the
held receipts and parked leases for each company. Each expiry always
writes a task entry: "Unsaved files from an interrupted run were kept
for N days, and their retention period has ended." The entry adds "They
are scheduled for removal." only when a parked lease was queued for
cleanup. A separate entry, "... kept for N days and have now been
removed.", is written only after the provider confirms the delete. Each
receipt is released in its own transaction. A failed receipt is logged
and skipped, and it is retried after a backoff (5 minutes, doubling, at
most 6 hours). The other receipts are still processed. The entries
reference a run or a task only if it still exists. If the task was
deleted, the decision goes to the server log only.
- Upgrade. A `pending_cleanup` lease (not in flight, not stop-only) that
an unreleased receipt holds goes back to `retained` and is parked by the
sweep. A sandbox that a release before this change already destroyed
cannot be recovered. Its receipt ends at its retention time like any
other receipt.
- Receipt trim (review P1). `abandonUnverifiedRemoteWorkspaceEpoch`
trims only released receipts and keeps the last 20 of them. It never
drops an unreleased receipt. A new abandonment of the same sandbox and
root replaces the unreleased receipt for that root. So a workspace holds
at most one unreleased receipt for each distinct sandbox and root
abandoned in the last 7 days.

## Verification

- `server/src/__tests__/abandoned-workspace-files-hold.test.ts` (19
tests). It follows the lease through the real acquisition path with a
failed resume, the real `abandonUnverifiedRemoteWorkspaceEpoch`, the
real `sweepPendingCleanupLeases`, and `settleRetainedOrphanSandbox`:
- `gives the next run a new sandbox at once and parks the held one,
which cleanup and later runs leave alone`
- `pulls a held lease already queued for cleanup back to retained, and
leaves it on the provider`
- `treats a receipt written before release fields existed as unreleased`
  - `records a retention expiry on the receipt`
- `hands the parked lease to normal cleanup once the files are
recovered` / `... discarded`
- `removes the files only after the retention period, and says removed
only after the provider confirms it`
  - `does not report removal when the provider delete fails`
- `records the end of retention without scheduling removal when another
receipt still holds the sandbox`
- `records the end of retention when it expires before the lease was
ever parked`
- `wakes the cleanup worker at the receipt's retention end, also after a
restart, and then goes quiet`
- `releases a receipt whose run was deleted and records it without the
run reference`
- `releases a receipt whose task was deleted without writing a task
entry`
- `keeps expiring later receipts when one fails, and retries the failed
one after a backoff`
- `keeps a held sandbox when the retained orphan settlement reviews it`
- `lists the sandbox for the retained orphan settlement again once the
receipt is released`
- `keeps holding the first sandbox after many later abandonments on
another sandbox`
- `replaces and cleans the sandbox as before when the receipt names a
different sandbox`
- `replaces and cleans the sandbox as before when no epoch was
abandoned`
- Red on the base revision (62d673cc3f): the park, upgrade, old-receipt,
expiry, many-abandonments and scheduling tests fail. The controls pass.
- Green: 19 of 19. Also
`environment-runtime-reusable-resume-replacement.test.ts` 17/17,
`environment-retained-orphan-sandbox.test.ts` 32/32,
`heartbeat-pending-cleanup-sweep.test.ts` 51/51,
`fresh-continuation-fallback.test.ts` 58/58.
- `tsc --noEmit -p server/tsconfig.json` and a tsc pass over the new
test file: clean. `environment-cleanup-work.test.ts` +
`heartbeat-pending-cleanup-sweep.test.ts`: 80/80.

- No existing test expectation was changed. Three CI failures on bb43f52
came from this change. Two came from the hold blocking the run lease
settlement: `ownership-reconciliation-report.test.ts > releases the
lease and frees the workspace for the retry once a preparation-only
epoch ends` and `remote-workspace-save-concurrency.test.ts > lets a
retry prepare the workspace again after its first writer was killed
during preparation`. The third, `native-session-executor.test.ts >
restores a warm backup into a created lease only when a settled restore
hold retained the stopped source`, came from a mocked database that has
no `execute`; the predicate now runs through `select`. All three pass
locally now. So do `heartbeat-orphaned-active-lease-sweep`,
`environment-run-lease-ownership`, `native-workspace-warm-reuse` and
`remote-workspaces.integration`.

## Risks

- Cost of a parked sandbox. Parking makes no provider call and changes
only the lease row. A sandbox that could not be resumed is not running.
A reusable sandbox that was released is already stopped and kept. In
both cases the cost is stopped storage for up to 7 days. One case is
different: if a destructive path finds a held warm sandbox that is still
running (`retained` after a keep-running release), the sandbox stays
running until the receipt is released. The expiry log line counts parked
leases for each company, so this cost can be seen.
- Run lease release for an `ephemeral` lease calls the provider's
release. Some providers delete the sandbox on that call. A held sandbox
normally has a reusable lease, so this pull request does not gate that
call.
- No schema change. The receipt is JSON in
`execution_workspaces.metadata`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:00:20 -06:00
DottaandClaude Opus 5.5 bcadbde449 feat(runner): record the OpenRouter host that served each OpenCode model call (#21298)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agents can use models through OpenRouter, and the Runner
proxies those model calls
> - OpenRouter routes one model across many hosts, and these hosts use
different quantizations (fp4, mxfp4, fp8)
> - Chat evaluations with identical prompts behave very differently
between runs, but the run log does not record which host served a call
> - The run usage shows `provider: "deepseek"` and `biller:
"openrouter"`, and `providerRequestId` is always `null`
> - This pull request records the serving host and the OpenRouter
generation ID for each model call in the run log
> - The benefit is that we can compare evaluation cohorts by host, and
the cost event names a real OpenRouter generation

## Linked Issues or Issue Description

**Subsystem affected**
Runner OpenCode provider proxy, runnerd usage normalization, and native
run usage accounting.

**Problem or motivation**
OpenRouter's default routing sends a request to one of many hosts. Each
host can use a different quantization. Chat evaluation results change a
lot between runs with identical prompts. Paperclip does not record which
host served a call, so we cannot compare results by host.
`usageJson.providerRequestId` is always `null` for OpenCode runs.

**Proposed solution**
Every OpenRouter completion response has a top-level `provider` field
and an `id` (the generation ID). Each streamed chunk has them too. The
Runner proxy reads them from the first chunk after it forwards the
bytes. The Runner attaches them to the usage of that call. The values go
to the run log, and the run's first generation ID goes to
`usageJson.providerRequestId`.

**Alternatives considered**
A new PRP event type such as `provider.upstream.served` would need a new
event enum value. An older control plane rejects an unknown event type.
A `harness.diagnostic` event shows in the task chat as a diagnostic. The
usage payload already flows to every consumer, and older consumers
ignore extra fields. Thus this change uses the usage payload.

**Additional context**
Run-log change only (AGENTS.md section 7). The data stays in
`heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch
`packages/shared/src/telemetry/` or OpenTelemetry.

## What Changed

- New `createUpstreamServedTap`
(`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`).
It is a pass-through stream. It pushes each chunk downstream first, then
it reads `provider` and `id` from the first SSE `data:` event (or from a
JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of
a JSON body. It skips encoded bodies and unknown content types. Any
parse problem or recorder error stops the scan and leaves the response
unchanged.
- `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The
OpenCode session keeps the most recent served call. Each
`item.completed` usage payload and `session.usage()` now carry
`upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint
does not include these fields, so the number of usage events does not
change.
- The OpenCode app-server proxy adds the two fields to
`thread/tokenUsage/updated`. Rust `usage.reported` now fills its
existing `providerRequestId` field and adds an optional
`upstreamProvider`. The TypeScript runnerd transport keeps both fields
when it rehydrates the usage notification. `usage.schema.json` documents
`upstreamProvider` (the protocol manifest and schema bundle are
regenerated).
- Server native turn accounting keeps the first generation ID the run
reports. The usage checkpoint and the native adapter result carry it as
`providerRequestId`, so `usageJson.providerRequestId` and the final cost
event name it.
- `doc/run-log-events.md` documents the fields in a new "OpenRouter
serving host" section.

## Verification

- Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's
stream through unchanged and records the host that served it on the
turn's usage". The test sends a streamed fixture with a comment line, a
host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks
that the bytes are identical and that the turn's usage names `Novita`
and the generation ID. The test fails without the proxy wiring.
- `upstream-served.test.ts`: streamed fixture, chunk split inside a
multibyte character, non-streamed JSON, malformed chunks, a truncated
JSON body, an encoded body, the 64 KiB limit, and a recorder that
throws.
- `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts`
(rehydration), Rust
`provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`,
and server `native-usage-accounting.test.ts` (first generation ID stays
stable across turns).
- Local: OpenCode driver, OpenCode proxy, and runnerd transport suites
(31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes
(one timing test failed once under load and passed alone). Runner and
server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks
pass.
- Latency: a local benchmark proxied a 400-chunk SSE stream 200 times
with the tap and 200 times without it. The median time to first byte was
292 µs with the tap and 259 µs without it (+33 µs, which is the extra
stream stage). The total stream time changed from 491.4 ms to 492.5 ms.
The tap spends about 5.5 µs of CPU to inspect the first chunk, after it
has forwarded that chunk. OpenRouter time to first token is hundreds of
milliseconds.
- Paid evidence: one single-case run of
`chat-features.runner-opencode.local.skills.fork-read-only` on this
branch. The result is in a comment on this pull request.

## Risks

- Low risk. All new fields are optional and additive. An older control
plane ignores the extra usage fields. `usage.schema.json` is not bound
by the event schema, and no validator enforces it on `usage.reported`.
- Saved state from the previous release: this change adds no persisted
run, session, checkpoint, or launch state. A saved session resumes
unchanged. A run that is in flight during a deploy reports usage without
the new fields until its Runner is new. `usageJson.providerRequestId`
then stays `null`, as before.
- `providerRequestId` on the run and its cost event is the run's first
generation ID, not each call's ID. The value is stable, so a late usage
receipt still matches the posted charge. Per-call IDs and hosts are in
the run log.
- If OpenCode starts a new model call before it reports the usage of the
previous call, a usage event can name the newer call. OpenCode reports a
message's usage when its step ends, so this is not expected.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code as an agent with tool use (shell, file edits, test runs). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:15:53 -06:00
DottaandClaude Opus 5.5 c730872974 fix(recovery): a chat turn waiting on the person is not stranded work (#21302)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In a chat, the agent's turn ends and the conversation waits for the
person's next message. The stranded-work sweep must leave that wait
alone
> - The sweep recognizes a finished chat turn by
`finalizationReasonCode: "conversation_turn_finished"` on the run record
> - Run completion re-reads the run through the 64 KiB read projection.
When the stored result is larger, that projection drops the finalization
fields, and completion wrote the reduced copy back
> - The chat turn then lost its finalization reason. Its reply was not
presented, and the sweep queued
`issue.productive_terminal_continuation_recovery` 30 s later. That is an
extra run while the agent waits on a person
> - This pull request keeps the full stored result through run
completion, and makes the sweep read the committed decision from its
durable rows
> - The benefit is that a turn waiting on a person is never treated as
stranded work

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `app-permissions.new-quarantined-action`, the
CEO's chat turn yielded with a `response_wake` continuation and waited
for a person to allow an app action. The stored run result was 113,937
bytes. Its saved record had the read projection's `truncated`,
`truncationReason: oversized_result_json` and `originalSizeBytes` fields
and no `finalizationReasonCode` or `decisionId`. The run's presentation
decision was `interaction_owns_conversation_response`, so no reply
comment was written. 30 s later the sweep queued
`issue.productive_terminal_continuation_recovery`, and the case failed
its run limit.

**Steps to reproduce**
1. Finish a native chat turn that yields `response_wake`, so
finalization records `conversation_turn_finished`.
2. Let the stored run result be larger than 64 KiB when run completion
reads it.
3. Run completion writes back the reduced copy without
`finalizationReasonCode`.
4. Run the stranded-work sweep. It queues a productive continuation run.

**Expected behavior**
The run record keeps its finalization fields, and the sweep leaves the
waiting chat turn alone. No new run starts until the person replies.

## What Changed

- `server/src/services/heartbeat/run-completion.ts`: run completion
reads the run with `unsafeFullResultJson` before it merges the native
result. It no longer writes the read projection's reduced copy back.
- `server/src/services/native-runtime/native-response-wake-wait.ts`:
`hasCommittedNativeConversationTurnWait` checks the run's committed
finalization decision (`conversation_turn_finished`) through
`native_run_finalizations` and `status_decisions` for the conversation's
own agent.
- `server/src/services/recovery/service.ts`: the sweep's
finished-chat-turn skip also uses that durable check, so runs already
saved with the reduced copy are recognized.
- Tests: `run-completion.test.ts` keeps the finalization record when the
stored result is larger than the read projection.
`heartbeat-process-recovery.test.ts` finalizes a real chat turn with the
eval's yielded result, saves the reduced run record, and expects two
sweeps to start no run.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts -t "larger than the read
projection"` — fails on master (the saved result is the truncated copy)
and passes here.
- `cd server && npx vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "chat turn that
yielded"` — fails on master (`continuationRequeued: 1`) and passes here.
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts
src/__tests__/heartbeat-process-recovery.test.ts` (614 passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- Run completion now reads the full stored result for one run. This is
one row; the read projection still serves every API read.
- The sweep check adds one indexed query for chat issues whose latest
run succeeded and whose run record lacks the reason.
- Previous-release saved state: no persisted shape changes. Runs already
saved with the reduced copy are covered by the durable decision check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:11:30 -06:00
Nicky LeachandClaude Opus 5.5 e65a3095a7 fix(runner): quiesce the Core-dialed route once the Runner commits its suspend (#21245)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Core runs agent sessions through the Paperclip Runner. With provider
ingress, Core dials the Runner over a PRP WebSocket.
> - When Core closes a session, it asks the Runner to suspend. The
Runner suspends, reports the result, and exits.
> - Core's ingress connection did not know that this exit was expected.
It treated the drop as a transport loss and redialed the Runner. Each
redial got a 404 until the close released the route.
> - The release (park) path already quiesces the connection before the
Runner goes away. The suspend path did not.
> - This pull request makes the suspend close quiesce the route in the
same way, once the Runner commits its suspend.
> - The benefit is that a clean suspend no longer looks like a lost
Runner. A drop before the suspend is committed still redials and
recovers as before.

## Linked Issues or Issue Description

**What happened?**
After a provider-ingress session closed cleanly, Core redialed the
Runner's ingress endpoint several times and got 404 each time. The
Runner state was `suspended`, `runner.drain` and `runner.suspend` had
both completed, and the Runner had exited with code 0. The redials
followed the outbound reconnect backoff (about 250, 500, 1000 and 2000
ms) and stopped when the registration release closed the outbound.

**Expected behavior**
After Core commits to `runner.suspend` and the Runner reports it
completed, the Runner's exit is expected. Core must not redial or
classify the drop as a transport loss. The release path already works
this way: it calls `outbound.quiesce()` before it releases the task.

**Steps to reproduce**
1. Open a session through a Core-dialed route (plugin provider ingress
or the `listen_ws` driver).
2. Close the session. Core drains, queues `runner.suspend`, and the
Runner suspends and exits.
3. See that the outbound reconnect loop redials the gone Runner until
the registration release closes it.

**Paperclip version or commit**
`master` at b4ca19f3a8.

**Agent adapter(s) involved**
Not adapter-specific (core bug). It affects any Core-dialed Runner
route: plugin provider ingress and the `listen_ws` driver.

**Additional context**
The redials do not fail the session. The plugin outbound uses a 7-minute
recovery grace, and `#failTransport` ignores failures after
`#closeAndSettle` sets `#closed`. The new test confirms this: without
the fix, the close still settles, but the route redials. This PR removes
the misleading redial. It does not change how a run's own failure is
classified.

## What Changed

- `RunnerExecutionDriver.controlPlaneRegistration` results can now
include an optional `quiesce()`. Core calls it when it has committed the
Runner's exit. The route then stops liveness and reconnect, and it keeps
the established receive/ACK path.
- `DurablePrpControlPlane.onRunnerExitCommitted(listener)` is new. The
authority calls its listeners after it saves a completed
`runner.suspend` or `runner.shutdown` result and before it sends the
acknowledgement. The Runner exits when it gets that acknowledgement, so
a route quiesced here never sees the exit as a loss.
- `awaitRunnerSuspensionBarrier` calls a new `onExitCommitted` callback
once, when its suspend command is `completed`. It also calls the
callback when it reuses a completed receipt and the durable state is
still `suspended`. A rejected, failed or unanswered suspend never calls
it.
- `DurablePrpCodexTransport` registers each control-plane route through
one `#registerControlPlane` helper. The helper records each route's
`quiesce` until that route is released. The suspend close subscribes to
`onRunnerExitCommitted` while its barrier runs and quiesces all of these
routes from that notification. The barrier's own `onExitCommitted`
covers a completed receipt that was acknowledged before this close.
Quiescing clears the tracked routes, so it runs once per close. The
helper copies property descriptors, so getters such as the `listen_ws`
driver's lazy `failure` stay live.
- The plugin task registration (`plugin-runner-task.ts`) and the
`listen_ws` driver registration (`native-runner-execution-driver.ts`)
implement `quiesce` with `outbound?.quiesce()`. This is the same call
that the release path makes.

## Verification

- New transport test with a real runnerd:
`packages/paperclip-runner/src/live/runnerd-codex-transport-suspend-route.test.ts`.
The test route acts like a Core-dialed provider ingress: on a socket
drop that it was not told to expect, it records a redial into a 404.
Before the fix the events were `connected, redial: 404, release`. After
the fix they are `connected, quiesce after runner.suspend completed,
release`. The close settles, and the durable Runner state is
`suspended`. A second case delays the durable-state read by 1.5 s, so
the Runner exits while the barrier still reads. That case fails without
the authority hook and passes with it.
- New barrier unit test in `runnerd-codex-transport.test.ts`. It shows
that the exit is committed only after the retried suspend completes,
never on a rejected or unanswered suspend, and on a reused receipt only
when the state is `suspended`.
- New server tests for the plugin task and `listen_ws` registrations.
`quiesce` reaches the active outbound and does not close it. Release
still closes it.
- Existing tests pass. These include the outbound release-path quiesce
test, the half-open and lease-expiry failure tests (a drop before the
commit still fails after its grace), and the plugin release/resume
facade tests.
- `pnpm verify` exited 2 (the change touches files outside the resolved
test graphs), so I ran focused suites. Locally: 8 runner transport files
and the 4 control-plane test files, 667 tests
(`runnerd-codex-transport*.test.ts` for suspension, release/resume,
remote retention, settlement, warm attachment and attachment recovery,
plus the new suspend-route test), and 5 server files, 148 tests (both
plugin Runner integration tests, the plugin task, the execution driver
and the outbound). Required CI supplies the broad coverage.
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Low risk. `quiesce` is optional. Registrations that do not supply it
behave as before.
- The quiesce happens only after the Runner reports a completed suspend.
A socket drop before that point still redials and recovers.
- The quiesced route keeps its receive/ACK path, so the Runner's last
frames before exit are still acknowledged.
- No persisted state or protocol change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:19:24 -08:00
DottaandClaude Opus 5.5 8a0f95849d perf(native-runtime): start the first turn without re-copying launch files, and never fail a run on a slow copy (#21287)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode in a remote sandbox must have a durable,
verified copy of the provider launch files before their first turn (the
rule from #19622)
> - Today each new run copies these files out of the sandbox, uploads
them and verifies them again, even when the company already has a ready
artifact with the same files
> - The whole copy runs under one five-minute clock. When eight runs
start together on small sandboxes, the copies compete, and slow copies
fail the run before its first turn
> - This makes cold starts slow (p50 273 s from run start to turn
submitted) and makes half of the first runs fail in a burst
> - This pull request reuses a ready artifact with the same digests,
shares one in-flight copy per company and manifest, and bounds each
network step instead of the whole copy
> - The benefit is that first turns start in about one minute on a cold
burst and in about 20 s when the artifact is already ready, and a slow
copy no longer fails the run

## Linked Issues or Issue Description

Refs #19622

**What happened?**
When 8 OpenCode runs started together on 1 vCPU Daytona sandboxes, 4 of
8 first runs failed before their first turn. Each run copied, uploaded
and verified the same provider launch files. The single 300 s clock over
the whole copy expired. Run start to turn submitted was p50 273 s and
p95 314 s.

**Expected behavior**
A run starts its first turn without a new copy when a ready, verified
artifact with the same files exists. A slow copy does not fail the run.

**Steps to reproduce**
1. Use a company with remote OpenCode agents on Daytona sandboxes with
durable storage.
2. Start 8 runs at the same time on new sandboxes.
3. See the time before the first turn and the failed runs.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- `retainReadyNativeProviderArtifacts` (`native-provider-artifacts.ts`):
reuse a ready, verified company artifact with the same paths and SHA-256
digests through a lookup only. No file is copied, uploaded or verified
again. The lookup (`readyNativeProviderArtifactCandidates`) matches
target, paths and digests in Postgres with jsonb containment on the
company's ready rows (existing partial index), newest first, limit 4. It
does not load or hash other retained versions. No migration.
- `shareNativeProviderArtifactCapture`
(`native-provider-artifact-capture.ts`): one in-flight capture per
company and manifest digest per server. Other runs wait for it, then
reuse its result.
- The single 300 s clock over the whole capture is replaced by a bound
for each network step (upload, verifying download): max(120 s, size at
256 KiB/s). A timeout has the typed cause
`native_provider_artifacts_transfer_timeout`. The capture makes up to 3
attempts from the local copy.
- Stop now interrupts the capture and the wait for a shared capture
(`ActiveNativeSession.admissionAbort`).
- `doc/opencode-cold-recovery.md` describes the new behavior.
- Tests: new `native-provider-artifact-gate.test.ts` (7 tests), 5 new
tests in `native-harness-backup-storage.integration.test.ts`, and new
executor variants `opencode-new-durable-slow-capture` and
`opencode-new-durable-ready` in `native-session-executor.test.ts`.

## Verification

Local (this branch, on master 9b613562b7):

- `npx vitest run
src/services/native-runtime/native-provider-artifact-gate.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
3 files, 1511 passed, 7 skipped.
- `npx tsc --noEmit -p server`: pass.
- Lookup scale: the integration test "loads only the ready artifact
whose launch files match, however many other versions the company keeps"
adds 200 ready non-matching versions. It returns only the matching
artifact. Without the SQL filter it fails (4 rows returned).
- Red/green: with master's `native-session-executor.ts` and the new
tests, `opencode-new-durable-slow-capture` fails with `TimeoutError: The
operation was aborted due to timeout`, and `opencode-new-durable-ready`
fails because the ready artifact is not looked up. With this change,
both pass.

Daytona (same commit content, cherry-picked onto the baseline commit
84a1ee3e11 with #21262; image `sha256:9461cc75…`; 1 vCPU sandboxes;
OpenCode):

| Wave | Shape | First runs OK | Run start → turn submitted | Other |
|---|---|---|---|---|
| Baseline `w-20261113T011027-s1702` | 8 runs started together | 4/8 (4
failed) | p50 273 s / p95 314 s | "only a getter" TypeError ×8,
`bootstrap_not_proven_incomplete` ×8 |
| Fix `w-20261113T031045-s1702` | same seed and shape | 8/8 | p50 66.0 s
/ p95 85.7 s | 8/8 tasks hands-off, 0 Retry banners; first agent event
p50 72.9 s / p95 90.0 s; exactly 1 upload, 7 runs waited on it (~65 s
shared capture); all failure codes 0 |
| Fix `w-20261113T034033-s1702` | 16 tasks, cap 8 | 16/16 | p50 19.1 s /
p95 24.3 s | second batch (artifact already ready, cold sandboxes) gate
time 24, 25, 26, 27, 31, 31, 45, 119 ms; first agent event p50 25.9 s |

## Risks

- The rule from #19622 still holds: a durable, verified artifact exists
before the first turn. Reuse only accepts a ready, verified artifact of
the same company with the same paths and digests.
- The first run per company and manifest per server still pays the full
transfer.
- Two servers can still upload duplicate artifacts for the same company
and manifest. Sharing is per server.
- No test drives Stop through the executor during a capture. Signal
handling is unit-tested only.
- A capture that fails all 3 attempts still ends blocked with a working
Retry. A follow-up PR (pre-turn failures continue automatically) removes
that.
- The documented five-minute deadline per capture is replaced by bounds
per network step. A very large artifact on a slow link can take longer
than five minutes in total, but each step still has a bound.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Default context
window; extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:48:55 -06:00
DottaandClaude Opus 5.5 0c04ff76eb fix(native-runtime): a follow-up that arrives as a warm session closes continues the task (#21279)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent session on a remote sandbox stays warm after a run,
and an idle timeout closes it after about 5 minutes
> - When a person's follow-up comment arrived just as that idle close
began, the new run failed with `native_session_supervisor_busy`
> - The idle close of the finished owner could also fail its final
checkpoint with `native_runner_checkpoint_owner_lost`, because the task
already pointed at the queued follow-up run
> - Then the retry was held for workspace repair and the task stayed
blocked; the environment could not be deleted (409) because its warm
runner could not be stopped
> - This pull request makes the follow-up wait for the idle close that
is in flight, and lets the finished owner save its final checkpoint
while its successor still waits in the queue
> - The benefit is that a follow-up that lands at the warm session's
idle close continues the task with no person involved

## Linked Issues or Issue Description

Refs #18011 (the checkpoint ownership fence this change extends).

**What happened?**

A follow-up comment arrived 295–340 s after the previous run ended,
while the warm session's idle close was in progress. The new run failed
with `native_session_supervisor_busy`. In another case, the idle close
failed its final checkpoint with `native_runner_checkpoint_owner_lost`,
the next run failed
`native_instruction_materialization_handoff_unavailable`, and the retry
was cancelled with `execution_reconciliation_required`. The task stayed
blocked. A hosted instance showed the same class as "previous run's
session on this sandbox could not be retired".

**Expected behavior**

The follow-up run waits for the idle close (inside the existing bounded
release wait) and then starts. The finished owner saves its final
checkpoint and closes. The task completes hands-off.

**Steps to reproduce**

1. Run an opencode agent on a Daytona sandbox with a warm session (idle
timeout 300 s).
2. Let a run finish.
3. Post a comment on the task 298–306 s after the run ended.

**Paperclip version or commit**

master `444dc18d98`.

**Deployment mode**

Local server with Daytona remote sandboxes (restart-chaos harness, light
lane). The same class was seen on a hosted staging instance.

## What Changed

- `claimWarmNativeInstructionCopy` now also waits when an idle close is
in flight for the session scope (`closingWarmNativeSessions`). The wait
uses the same deadline as the existing release wait
(`warmSessionReleaseWait`), so it adds no new wait and nothing before
`turn.start` on the normal path.
- `validateNativeRunnerCheckpointOwner` accepts the finished owner's
final checkpoint (accepted result, no new work) when
`issues.executionRunId` points at a successor run that is `queued` or
`scheduled_retry` for the same company and task and has no
`native_run_finalizations` row.
- Tests: a server test for a follow-up that arrives while the idle
timeout closes the warm session; four new scenarios (`queued_successor`,
`scheduled_retry_successor` × `runner.suspend`/`turn.stop`) and four new
negatives (`queued_wrong_issue`, `queued_new_work` × both commands) in
the early-retirement checkpoint matrix.
- Restart-chaos harness:
`RESTART_CHAOS_MIX_FOLLOW_UP_AFTER_END_MS=<min>-<max>` and
`RESTART_CHAOS_MIX_FOLLOW_UP_COUNT` post the staging-mix follow-up a set
delay after the task's last run ended, so a cell can aim the follow-up
at the warm idle close. The restart-chaos README describes both
variables. A failed post is retried with the same `clientRequestId`, the
grade check `after_end_follow_ups_served` needs an agent reply after
each post and the ledger marker, and the task budget includes the
planned waits.
- Harness: the wave database cleanup
(`tests/runner-e2e/controller-database.ts`) no longer waits forever when
a group signal (for example `timeout` or Ctrl-C) already stopped the
embedded Postgres. Before, `embedded-postgres` `stop()` waited for an
exit event that had already fired, and an interrupted wave hung after
the Daytona R6 sweep.

## Verification

- Fix 1, red without the product hunk: `native-session-executor.test.ts
-t "idle timeout is closing"` fails with `promise rejected "Error:
native_session_supervisor_busy"`. Green with the fix: 1 passed.
- Fix 2, red without the product hunk:
`native-runner-restart-recovery.integration.test.ts` fails 4 of 208
(`queued_successor` and `scheduled_retry_successor` ×
`runner.suspend`/`turn.stop`) with
`native_runner_checkpoint_owner_lost`. Green with the fix: 208 passed,
14 skipped. The two new negative cases pass in both states.
- Harness hang fix, red without the fix: the new
`controller-database.test.ts` case ("finishes cleanup when the database
already stopped because a group signal reached it") times out on cleanup
(`expected 'hung' to be 'cleaned'`). Green with the fix: 5 passed, 1
skipped.
- After the rebase onto master `7154030ef6`, both server test files
together: 1639 passed, 21 skipped.
- Restart-chaos harness tests (`staging-mix`, `staging-mix-retry`,
`observer`): 75 passed, including a test for the after-end retry, the
`after_end_follow_ups_served` grade check and the added budget. `tsc
--noEmit` passes for `server`, `tests/runner-e2e` and
`tests/restart-chaos`.
- `pnpm verify` exits 2 because `tests/restart-chaos/observer.ts` is a
global CI input. Required CI supplies the broad coverage.
- Daytona cell on this branch (4 opencode tasks, warm sandboxes, 1 vCPU,
follow-up comment posted 300.8–308.2 s after each task's last run ended,
image built for master `444dc18d98`): 4/4 tasks done hands-off, 8/8 runs
succeeded, 0 Retry banners, 0 `owner_lost` / `supervisor_busy` / "could
not be retired" / `execution_reconciliation_required` lines in the
server log, environment delete clean, warm-shutdown grade pass, 0
sandboxes left. The harness exited by itself after 1076 s.
- Same cell before the fix (master, 8 tasks): 1/8 tasks blocked
(`native_session_supervisor_busy` →
`native_runner_checkpoint_owner_lost` →
`execution_reconciliation_required`). With fix 1 only: 1/8 still blocked
(`owner_lost` on the idle close). With both fixes: 8/8 done and 0 failed
runs (cut by the 20-minute cap before cleanup), then 4/4 above.
- The cell grade `T:completion_once` fails on all 4 tasks by design: the
timed follow-up comment arrives after the task is done, so the agent
posts a second completion comment.

## Risks

- Fix 1 can delay a follow-up by up to the existing warm release wait
(30 s) while an idle close finishes. Before this change, that run
failed. If the close fails or the wait expires, the claim returns null
and the run uses a fresh instruction copy, as before.
- Fix 2 widens the checkpoint ownership fence from #18011. It does not
reopen what #18011 closed:
- The exemption applies only to the finished owner's final save: the
checkpoint must hold an accepted result and no new work, so no provider
turn can be granted through it.
- The successor must still be unclaimed (`queued` or `scheduled_retry`)
and must have no native coordinator (`native_run_finalizations`) row. No
controller has started it, so nothing can have consumed or depended on
the predecessor's checkpoint. Once the successor is claimed, the
existing preparing/claimed rules from #18011 apply unchanged.
  - The successor must belong to the same company and the same task.
- Cases that must still reject, with tests in the same matrix:
- `queued_wrong_issue`: a queued successor for a different task is
rejected with `native_runner_checkpoint_owner_lost` and the stored
checkpoint does not change.
- `queued_new_work`: a `turn.start` (new work) while the successor is
queued is rejected with `native_runner_checkpoint_owner_lost` and the
stored checkpoint does not change.
- All earlier rejections stay: unclaimed native coordinator,
dispatching, expired or foreign preparation, foreign native preparation,
wrong task, new work, unaccepted result, replaced controller.
- Not changed: an owner whose close fails still blocks environment
delete (409). That is a separate product decision.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Earlier commits on
this branch were written by the same model in an earlier session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:42:50 -06:00
DottaandClaude Opus 5.5 497ccba8f1 fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner that the server
controls over the durable PRP channel
> - When a run is interrupted, a same-run retry reattaches the live
Runner and reads its session (`session.snapshot`) before it continues
> - On staging, the reattached Runner authenticated and restored its
session, and then did not answer the read
> - The recovery kept that Runner, so each later attempt adopted the
same Runner and failed the same way until the attempt budget ended the
run
> - This pull request makes the next attempt replace that Runner with a
fresh Runner over its saved journal, and makes a failed Codex history
read say what failed
> - The benefit is that an interrupted Codex run continues without a
manual Retry, and its provider turn is not sent again

## Linked Issues or Issue Description

Refs #21161, Refs #21053, Refs #20853

**What happened?**

A Codex agent on Daytona ran recovery runs (wake
`issue_recovery_action_restored`). In one hour, 11 of about 35 finished
runs failed. All failed the same way:

- `native_session_recovery_failed: HarnessCapabilityUnavailableError:
read is unavailable: Error: runnerd exited while waiting for
session.snapshot` (8 runs)
- `native_session_recovery_failed: HarnessReconciliationError:
codex_history_read_failed: thread/turns/list requires supported
paginated history; Error: provider_transport_failed: PRP command
session.snapshot timed out (runner connected; pending ahead 0; last
runner event seq 338 76s ago)` (3 runs)

The run log of each run shows one sequence:

1. Attempt 1 fails with `runnerd did not report the provider turn
identity`. The close leaves the Runner alive and unsettled for the
same-run reattachment.
2. Attempt 2 adopts that Runner (same PID). The Runner authenticates.
The log shows `runnerd restored its durable PRP session and provider
thread`. Then the driver's recovery read (`thread/read` or
`thread/turns/list`, which both send `session.snapshot`) gets no answer.
The Runner still answers WebSocket pings, but it sends no command result
and almost no events (`pending ahead 0`, last event 76 s to 391 s
earlier). The recovery treats this as transient and detaches. The Runner
stays alive.
3. Attempt 3 adopts the same Runner again and fails the same way. In
some runs a protocol fault (`thread_binding_mismatch`) closes the
transport during the read, so the message is `runnerd exited while
waiting for session.snapshot`.
4. The provider attempt budget ends the run.

#21161 did not help. Its wait extension applies only while a queued
answer is being committed. Here the Runner sent no answer at all.

The `paginated history` text was also wrong. `codex-history.ts` called
every failed history read unsupported, also a transport timeout.

**Expected behavior**

A Runner that left a recovery read unanswered is not adopted again. The
next attempt starts a fresh Runner from the saved journal, resumes the
provider session from the saved checkpoint, and does not send the
provider turn again. A failed history read reports the real failure. A
Codex build that refuses paginated history falls back to one complete
thread read.

**Steps to reproduce**

1. Start a Codex run in a remote (Daytona) Runner.
2. Let the first attempt fail at turn start, so that the close keeps the
Runner alive.
3. Let the reattached Runner stop answering commands after it
authenticates (for example, stop the Runner process after its first
answer).
4. See `PRP command session.snapshot timed out (runner connected;
pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3,
with the same Runner PID.

**Paperclip version or commit**

Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on
`master` after `53608bd1d6`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `nativeRecoveryCommandUnanswered` (executor) finds a
`native_session_recovery_failed` whose Runner did not answer a recovery
command (`PRP command <type> timed out` or `runnerd exited while waiting
for <type>`). Identity, authority, integrity and incompatibility
refusals do not match.
- When such a failure ends a `reattach_remote_runner` attempt on a
sandbox target, the failure detail records `adoptedRunnerUnanswered:
true`.
- `claimNativeRestartRecoveries` copies it to the next claim as
`replaceUnansweredRunner: true`, in the same way as
`exitedBeforeSessionOpen`.
- `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps
all identity checks. When the old Runner is still alive, it returns no
adopted process, and the attempt launches a fresh Runner from the saved
journal. The existing launch fence stops the old `paperclip-runnerd` on
that state folder and its whole lineage, and fails the launch closed
when it cannot prove them gone. A Runner that already exited mid-turn
keeps the existing fail-closed path, because its provider or tool
processes can survive where the fence cannot find them. The task then
continues in a fresh session from the saved work. The attempt budget is
unchanged.
- `codex-history.ts`: a history read that fails without a provider
answer is now `codex_history_read_failed: <method> read failed;
<error>`. A provider refusal of the method (`-32601` or `-32600`) on the
first page falls back to one `thread/read` with `includeTurns: true`.
That fallback fails closed when the thread differs, or when turns, items
or turn identities are missing. A refusal on a later page still fails
closed.
- `doc/execution-semantics.md` describes both rules. The launch-fence
code comment now names the replacement case.

### Changed existing test

`codex-history.test.ts` "does not turn an interrupted page or
unsupported method into idle history" used a plain `Error("method not
found")` on page 2 and expected the `supported paginated history` text.
A plain error is now a failed read, not an unsupported method. The test
now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2
and still expects a fail-closed rejection. A new test covers the
failed-read message.

## Verification

- Red before the fix, green after:
- `server/src/services/native-runtime/native-session-executor.test.ts` ›
"a same-run reattachment whose adopted Runner never answered its
recovery read": the replacement and staging-message tests failed on the
old code (the Runner was adopted again, the three staging messages were
not recognized). Two further tests ("never resumes beside an exited
unanswered Runner's lineage") failed on the first revision of this PR,
which relaunched beside an exited mid-turn Runner, and pass now. All 15
pass.
- `native-runner-restart-recovery.integration.test.ts` › "asks the
same-run reattachment to replace a Runner that left its recovery read
unanswered" (embedded Postgres): the claim had no
`replaceUnansweredRunner` on the old code. Now passes.
- `codex-history.test.ts`: 7 new tests failed on the old code. All 16
pass.
- Real `paperclip-runnerd` and fake Codex:
`runnerd-codex-transport-remote-retention.test.ts` › "continues the same
provider thread from a fresh Runner over the journal of a retained
Runner that stopped answering, without replaying its turn". It runs the
staging sequence: turn started, close keeps the stopped Runner, a second
controller adopts it, the Runner stops answering and the controller
detaches, the Runner and provider are killed (the launch fence), and a
fresh Runner over the same journal returns the same thread. The fake
Codex call log shows one `thread/start` and one `turn/start`.
- Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus
the remote-retention, adoption and cold-restore transport suites (442
tests). Two suites timed out under parallel load and pass alone. Server:
`native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation` (1714 passed, 21 skipped).
- `pnpm typecheck` in `server` (all layers) and `tsc` for
`paperclip-runner` source and tests pass.
- Not run locally: the container-gated remote Runner suites
(`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the
full suite.

## Risks

- Medium. The replacement relaunches a Runner over a journal that is not
settled, but only while the old Runner is alive. Safety comes from three
facts: the launch fence stops the old Runner and its lineage with proof
or fails the launch; the Runner restores its own durable journal (as
local `resume_dead_runner` does); and the real-runnerd test shows that
the turn is not sent again. An exited Runner is never relaunched beside
its possible survivors.
- If the previous close left a `runner.suspend` pending, a fresh Runner
can execute it and exit. The attempt then fails as it does today. The
usual detach path after an unanswered read sends no suspend.
- The replacement uses the existing attempt budget. It does not add
attempts.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events
and logs), extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:06:24 -06:00
Nicky LeachandClaude Fable 5.2 9b613562b7 feat(docker): split the Runner image into stable layers and add a slim variant (#21229)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Part 2 of this stack builds the Runner alone as the image
`runner-artifacts`, in one layer of about 850 MB
> - Most of those bytes are provider binaries and Node. They change only
with a provider version or the Node pin, but one layer means that every
Runner source change makes all 850 MB new
> - About half of the image is the pack's transport archives. A consumer
that copies the pack as files does not need them
> - This pull request splits the image into layers by how often their
content changes, and adds a slim variant without the transport archives
> - The benefit is that a new image shares about 93% of its bytes with
the one before it, and a consumer that needs no transport archives pulls
half as much

## Linked Issues or Issue Description

Refs #21204 (part 2 of this stack) and #21167 (provider pack layers). No
issue exists for this change.

**Problem or motivation**

The image from part 2 is one layer. A registry and a builder store and
move layers, not files. A change to one Runner source file therefore
makes a new 850 MB layer, although 790 MB of its content is the same as
before. The image also holds the provider pack two times: as files, and
as transport archives of those files. Some consumers need only the
files.

**Proposed solution**

Write the image as four layers: the pack's large dependency files,
everything else, the transport archives of those large files, and the
other transport content. Make the two large layers byte-identical from
build to build. Add the target `runner-artifacts-slim`, which is the
first two layers. The full image is the slim image plus the two
transport layers, as the same layer blobs. Check all of this on the
built images.

**Alternatives considered**

- Two unrelated images. Then the registry stores the shared content two
times unless the layers happen to match, and nothing ties the two
together.
- Split by path only, with all files of 1 MiB or more in the first
layer. Six of those files are the Runner's own bundles under `dist`.
They change with the Runner's source, so the first layer would change
with it too.
- Leave the transport archives as they were written. Their bytes then
depended on timing (see "What Changed"), and the archive layer never
repeated.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 3 of 4. Each part is on top of the one before it:

1. #21228 (merged): an optional field in the provider pack manifest
2. #21204: build and check the Runner image, with no publication
3. #21229: layers by change rate, and the slim variant (this pull
request)
4. #21230: publish both variants from `master`

## What Changed

- `Dockerfile`: two final stages. `runner-artifacts-slim` is two `COPY
--link` layers. `runner-artifacts` starts from it and adds two more. The
build stage sorts its output into the four layers. Every path is where
it was in part 2: consumers copy by path and see no difference.

  | Layer | Content | Changes with |
  | --- | --- | --- |
| 1 | The pack's dependency files of 1 MiB or more: provider binaries
and Node | A provider version or the Node pin |
| 2 | Everything else, with `runner-image.json` | Every Runner source
change |
| 3 (full only) | The transport archive of each file in layer 1 | The
same as layer 1 |
| 4 (full only) | The other transport archives, the overlay archive and
the receipt | The same as layer 2 |

- Stable bytes. The files of layers 1 and 3 get timestamp 0, and the
large dependency files get it before their transport archives are
written. A file under the pack's `dist` is the Runner's own output, so
it and its archive are in layers 2 and 4.
- `scripts/create-provider-pack-archive.mjs`, two separate commits.
**This script also writes the application image's archives.**
- The same file now gives the same archive every time. The script piped
tar into Node's zstd compressor. When the pipe ended while the
compressor still had its last chunk queued, Node finished the zstd
stream two times, and the archive got an empty frame of nine bytes at
its end. That depended on timing. In the pinned Linux base, 37 of 60
runs of the old code had the frame. The compressor is now ended only
after it has taken every chunk.
- Archive N holds file N of the receipt, and the receipt listed the
files in path order. `dist` sorts before `node_modules`, so one more
large file in the Runner's own output gave every dependency file another
archive name, and layer 3 became a new blob with the same contents. The
receipt now lists the files outside `dist` first. The readers
(`provider-pack-layers.ts`, `retained-provider-pack-source.ts`) pair
file N with `N.tar.zst` and do not use the path order, so they need no
change. For the application image, the archive numbers change one time,
and the bytes of each archive do not change.
- `runner-image.json` is the same file in both variants. Its new field
`fullImageOnly` names the two paths that only the full image has.
`paths` did not change.
- `scripts/runner-image-layout.mjs check` takes `--variant slim|full`.
It is required and has no default. A slim tree that holds a transport
path fails. A full tree without them fails.
- `scripts/runner-image-oci.mjs` (new, 585 lines). It is not an input of
the image, because the image build does not run it.
- `unpack <oci archive> --out <tree>` writes the tree of an image from
the layers of its archive, in order. It checks every blob against its
digest. It refuses a whiteout, an absolute path, a path with `..`, a
path that is written two times, an entry behind a link, and every entry
that is not a file, a directory or a link.
- `check-layers --slim <oci archive> --full <oci archive>` checks the
table above on the built images. It reads the receipt to decide which
archive belongs in which layer. It fails when the full image does not
start with the slim image's layer blobs in the same order, when a path
that is not a directory is in two layers, and when a shared layer holds
transport content under any name or directory.
- The script reads the archive and its layers itself. It does not use
the system's `tar`, so every platform reads them the same way. It finds
a member only under the name that buildx gives it.
- `.github/workflows/runner-image.yml`: each leg builds each variant one
time, as an OCI archive and nothing else. It unpacks both archives, runs
`check` on each tree with its variant, runs `check-layers`, and runs the
smoke matrix on the slim tree. The tree that is checked is therefore the
bytes of the image, not a second export of the same build. Every step
that builds has a time limit. The workflow still has no registry access.
- Why one export: in the first runs of this pull request, the slim build
stalled on all four legs. Each build command had two exports, a tree and
an archive. The slim command, with every stage from the cache, wrote its
archive and then made no progress on the tree for 50 minutes. A test now
holds the rule of one export and a time limit for each build command.
- The smoke matrix runs on the slim tree. The daemon is in a layer that
both variants share, so one run covers both.
- `server/src/__tests__/runner-image-identity.test.ts`: the test that
holds the rules for the Runner stages now finds a stage wherever
BuildKit starts one. It fails for `FROM x as name`, a flag on `FROM`, a
lowercase or indented `COPY`, an `ADD` or `ONBUILD`, a `COPY` with
another flag, and a `COPY` that continues on the next line. Nine samples
show each case.
- Two comments are corrected. The comment on
`PAPERCLIP_RUNNER_INPUTS_DIGEST` in `build-provider-pack.mjs` said that
no check reads the field. The `Dockerfile` comment on the publication
argument named a workflow "that builds the image for publication".
- `doc/runner-image-contract.md` (159 lines): the two variants, the
section "Layers", how the checks get their tree, and the new checks.
`doc/DOCKER.md` names the slim target.

Script sizes: `runner-image-layout.mjs` 969 lines,
`runner-image-oci.mjs` 585, `runner-image-identity.mjs` 166,
`create-provider-pack-archive.mjs` 114.

## Verification

Image identity at this commit:
`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`.

Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and
zstd, the worktree mounted read-only):

- `runner-image-identity.test.ts` (41), `runner-image-layout.test.ts`
(100), `runner-image-oci.test.ts` (50), `runner-image-workflow.test.ts`
(8), `provider-pack-archive.test.ts` (47, none skipped),
`retained-provider-pack-source.test.ts` (19),
`remote-provider-pack-verification.test.ts`,
`docker-image-layer-reuse.test.ts`, `docker-build-stamp.test.ts`: 293
tests pass in 9 files.
- The same files pass on macOS. The tests of `check-layers` failed on
Linux in the first runs of this pull request: their fixtures named the
archive members `./index.json`, which bsdtar finds and GNU tar does not.
The fixtures now name the members as buildx does.

`linux/arm64`, on a local Docker host, with the new export path:

- Each variant builds as an OCI archive only: the full target in 156
seconds without a cache, the slim target in 5 seconds from the cache.
- `unpack` writes the full image (10,519 entries that are not
directories) in 4 seconds, and the slim image in 4 seconds.
- `check <tree> --variant <variant> --expect-inputs-digest <digest>
--expect-source-revision <sha> --expect-architecture arm64` passes on
each unpacked tree. `runner-image.json` is the same file in both.
- `check-layers --slim <slim> --full <full>` passes.
- The smoke matrix passes on all five base images, from the unpacked
slim tree.

| Layer | Digest | Bytes | In |
| --- | --- | --- | --- |
| 1 | `sha256:3f21b15b3a99…70ac1c` | 379,609,589 | slim and full |
| 2 | `sha256:d5567acc977f…b4f12a` | 43,906,956 | slim and full |
| 3 | `sha256:6a340088ca2c…6ba5b5` | 410,938,259 | full |
| 4 | `sha256:789e97f5db88…3eab44` | 15,656,249 | full |

- The slim image is 423,516,545 bytes in two layers. The full image is
850,111,053 bytes in four.

Stability, with three builds that share no cache for the build step:

| Build | Source revision | Runner build output | Layer 1 | Layer 2 |
Layer 3 | Layer 4 |
| --- | --- | --- | --- | --- | --- | --- |
| A | `aaaa…` | this commit: 6 files of 1 MiB or more under `dist` |
`3f21b15b…` | `d5567acc…` | `6a340088…` | `789e97f5…` |
| B | `bbbb…` | this commit | `3f21b15b…` | `a2f8a4fe…` | `6a340088…` |
`4e6c492d…` |
| D | `dddd…` | one new source file that adds two large files under
`dist` | `3f21b15b…` | `33b0bdaf…` | `6a340088…` | `0ea322bf…` |

- Layers 1 and 3 are byte-identical in all three: 790,547,848 of
850,111,053 bytes, 93%. Build D has two more large files under `dist`,
eight in place of six, and another image identity. Its dependency files
keep the archive numbers 0 to 20. With the earlier numbering, those
numbers moved by two, and layer 3 was a new blob with the same contents.
- Layer 1 has the same digest as before this round. Layer 3 has a new
digest one time, because the archives have new numbers.

Other checks:

- `tests/runner-e2e` workflow tests: 39 pass. `node --test
.github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image
contract scripts: 33 pass. `check-docker-deps-stage.mjs`,
`check-node-version-policy.mjs` and `docker buildx build --check` report
nothing.

GitHub, at this head:

- The `Runner image` workflow passes on both architectures with the new
export path: `Build and check (amd64)` in 14m41s and `Build and check
(arm64)` in 8m41s. Each leg builds both variants, checks both layouts
and the layer rule, and runs the smoke matrix. The first runs of this
pull request stalled in the slim export before any check.

Never run:

- `native-session-executor.test.ts`. It does not load on this host, with
or without this change, because `@paperclipai/plugin-sdk` is not built
here. It imports the reader of the retained pack and no archive number.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The two changes to `create-provider-pack-archive.mjs` change the
application image's transport archives: about half of them lose nine
bytes, and all of them get new numbers one time. An archive's content
does not change. A reader pairs file N of a receipt with archive N of
the same build.
- The large dependency files in the pack, and in their transport
archives, now have the modification time 0 (1970). Before, they had the
build time. Nothing in this repository reads those times.
- `unpack` is a reader of tar and gzip that this repository owns. It
reads the layers that BuildKit, GNU tar and bsdtar write (tested with
each), and it fails for a header it does not know.
- The check on the built images, not the recipe, is what proves the
layer rule; it runs in each leg of the workflow.
- `check` now needs `--variant`. A caller that does not pass it fails
with a clear message. No caller exists outside this repository yet.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 19:52:02 -08:00
DottaandClaude Opus 5.5 3b4c3265ed fix(recovery): offer Try again on a model-unavailable failure and recheck the model first (#21244)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native run fails, the task thread shows a failure marker, and
some markers offer Try again
> - A `native_provider_model_rejected` failure blocks the task for the
person and shows "Model unavailable", but it has no Try again control
> - Recovery must just work: after the person fixes the model, they need
one obvious control that continues the task, and that control must not
start a run that is sure to fail
> - This pull request shows Try again on the "Model unavailable" marker
and makes the retry recheck the selected model before it starts a run
> - The benefit is that the person fixes the model, presses Try again,
and the task completes, and a too-early press explains which setting to
change instead of failing again

## Linked Issue or Issue Description

Refs #21210

**What happened?**
Paid run 43321773136, `recovery.retry-exhausted` (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`). The run failed with
`native_provider_model_rejected`. The task stayed blocked and the thread
showed "Model unavailable", but no Try again control appeared
(`task-chat-run-failed-try-again` not found in 30 s). Only the "Run
failed" and "Usage limit reached" markers offer Try again.

**Expected behavior**
The "Model unavailable" marker offers Try again. Pressing it checks the
selected model first. If the provider-rejected model is still selected,
no run starts, and the person sees which setting to change with a link
to it. After the person changes the model, Try again starts one retry of
the original run.

**Steps to reproduce**
1. Set an agent's model to an id the provider rejects.
2. Assign it a task. The run fails with
`native_provider_model_rejected`.
3. Open the task. Before this change, the marker has no Try again.

## What Changed

- `server/src/services/native-runtime/model-rejected-retry.ts`:
`rejectedModelRetryRefusal` reads the model the failed run sent
(`runnerProfileJson.nativeExecutionInput.provider.model`) and the model
a new run would use (the task's model override for its assignee, else
the agent's model). If they are the same, it returns a 409 refusal with
code `native_provider_model_still_unavailable`, the setting to change
(`agent_model` or `task_model_override`) and the rejected model.
- `server/src/routes/agents.ts`: the exact failed-run retry (`POST
/agents/:id/wakeup`, `reason: retry_failed_run`) applies this check
before it queues anything.
- `ui/src/components/task-chat/adjacent-errors.ts`,
`TaskChatThreadView.tsx`: one `isRetryableFailureMarker` rule, now
including "Model unavailable".
- `ui/src/lib/model-retry-refusal.ts`, `ui/src/pages/IssueDetail.tsx`: a
refused retry shows "Model still unavailable" with the server's message
and an "Open <agent> model settings" link to `/agents/<agent>/runtime`.
A task override refusal names the override in the message.
- Eval `recovery.retry-exhausted` (oracle `runtime.retry-exhausted.v7`):
the rejected model stays selected after the failure. The person presses
Try again (409, toast names the model, its link opens the agent's model
settings, no run starts in 20 s, task stays blocked), restores the
working model, presses Try again (202), and the one retry of the
original run uses the working model, saves the brief and completes the
task. 3 runs: failed 1, succeeded 2. The budget-hold and two follow-up
chat phases are removed; the case now tests the model recheck.
`recovery.yaml` context and expected outcome updated, recorded in
`chat-full-catalog-testops-manifest.json` `changedFields`, pinned in its
test, descriptors and inventory digest updated.

## Verification

- RED on master: `TaskChatThread.test.tsx` "offers Try again on a
model-unavailable failure and retries that run" fails (no control).
`agent-live-run-routes.test.ts` "refuses a retry while the rejected
model is still selected and starts no run" fails (the route queues the
run).
- GREEN: both pass. `agent-live-run-routes.test.ts` 101/101.
- `server/src/services/native-runtime/model-rejected-retry.test.ts`
(embedded PostgreSQL), 6/6: refuses while the agent still selects the
rejected model (names the agent setting); names the task override;
allows the retry when the agent or the override now selects another
model, or the run failed for another reason; ignores another company's
run.
- `ui/src/lib/model-retry-refusal.test.ts`: names the setting and links
to `/agents/morgan/runtime`; no link for a task override; other errors
fall through to the generic toast.
- Server `tsc --noEmit` clean. UI `tsc` clean apart from the
pre-existing generated storybook catalog. `node
scripts/check-token-gates.mjs`: clean.
- Eval: `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog tests/runner-e2e/chat-catalog-domains
tests/runner-e2e/chat-*.test.ts`: 339 files pass. New calibration
rejects an accepted early retry, a run started while the model is
rejected, a refusal without the model, setting link or blocked task, a
retry that used the old model, and duplicate or missing retries. `pnpm
test:e2e:runner:typecheck`: pass. Manifest test 10/10.
`source-sdk-public-inputs --check`: `changed: false`.
- Paid run: Runner Full-Stack E2E 43338695850 on throwaway branch
`chat-eval/qa-mr-paid`. Result to follow in a comment.

## Risks

- The recheck compares the selected model with the model the provider
rejected. It does not call the provider. A model the provider rejected
stays refused until the person selects another one; a different but also
unsupported model starts a run that fails with the same marker again.
- The refusal message is a toast (15 s). The marker detail still says to
choose a supported model.
- No persisted shape changes; runs saved by older builds carry the same
`runnerProfileJson` field.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:42:52 -06:00
DottaandClaude Opus 5.5 db802d1230 fix(heartbeat): deliver a queued comment again unless the reading run cites it as handled (#21243)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can add a comment to a task while the agent's run on that
task is still working
> - #21175 made a `get_task_history` read of a queued comment's current
revision count as delivery
> - A read is not handling. In `agent-chat.multiple-handoffs`, Morgan's
running turn read the "Launch sooner" comment, set it aside, and saved
"Launch later". No continuation ran, so the instruction was lost.
> - This pull request counts a read as delivery only when the run's
accepted result cites the comment as handled. Otherwise one continuation
still delivers it, and that continuation is told the comment may already
be applied.
> - The benefit is that a person's comment is never silently lost, and
the #21175 protection against repeated completed work stays in place

## Linked Issue or Issue Description

Refs #21175
Refs #21203
Refs #21210

**What happened?**
Paid run 43321773136 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`), case
`agent-chat.multiple-handoffs`. Morgan's running turn called
`get_task_history` and received the CEO's queued "Launch sooner" comment
at its current revision (`task_history.read` at seq 75). The model
judged the comment as an unauthenticated agent instruction and saved
"Launch later" from the brief. Its accepted result did not mention the
comment. `deliveredQueuedCommentIds` counted the read as delivery, the
release drain dropped the held comment, and no continuation ran.

**Expected behavior**
A read counts as delivery only when the run records the comment as
handled. A comment that was read but not recorded as handled is
delivered once in a new turn. The new turn's prompt says that an earlier
turn already read the comment and may have applied it.

**Steps to reproduce**
1. Start a native run on a task.
2. Queue a comment behind the run.
3. Let the run read `get_task_history` and finish without applying the
comment.
4. Before this change, no continuation runs and the comment is lost.

## What Changed

- `server/src/services/queued-comment-delivery.ts`: the history-read
delivery rule now also requires the reading run's `run.result.accepted`
event to cite `comment:<id>` (top-level `evidence[].ref` or a
criterion's `evidenceRefs`, prefix match so an annotated ref counts).
The read revision, succeeded-run and finished-after-revision rules are
unchanged. All consumers (release drain, stale-continuation save,
ownership reconciler, first-turn queued input, remote-Stop resume) use
this one rule.
- New `readByEarlierTurnCommentIds`, `commentsArrivedDuringRun` and
`taskHistoryHandlingGuidance`.
- `get_task_history` (both tool authorities) returns
`arrivedDuringThisTurn` (the `comment:<id>` refs of comments other
authors posted or edited after the run started) and a `handling`
instruction to cite each applied comment in `paperclip_finish` /
`paperclip_block` evidence. The comment list itself is unchanged. A
failure to compute the list returns no guidance, so the comment stays
owed.
- `buildPaperclipWakePayload` marks a wake comment `readByEarlierTurn`
when another finished run of the agent read its current revision.
`renderPaperclipWakePrompt` adds one line after that comment: check the
current task state first, and do not repeat work that is already done.
- `doc/run-log-events.md`: the delivery rule and the new fields.
- Evals: `agent-chat.multiple-handoffs` (oracle
`runtime-multiple-handoffs.v6`) and `steering.edit-queued-message`
(oracle `edit-queued-message.structural.v6`) accept in-turn consumption
only with a current-revision read and a citation. A read without a
citation must get exactly one continuation whose saved wake marks
`readByEarlierTurn`, and the final saved revision must hold the change
("Launch sooner"; blue). `runtime-queue.ts` reads citations from the
run's `run.result.accepted` event. No scenario YAML changed.

## Verification

- RED on master `b1ef1f3022`,
`server/src/__tests__/heartbeat-held-comment-wake.test.ts`: 10 of 41
fail. The key one: "settles a held message as soon as its run closes the
task (read mid-turn: true, cited: false)" delivered 0, expected 1.
- GREEN: 41/41. The three required shapes:
- read, not cited: one continuation, and its wake marks
`readByEarlierTurn` (`...read mid-turn: true, cited: false` and the
stale-queued-run variant).
  - read and cited: no continuation (`...cited: true`).
  - never read: one continuation, as before.
  - Edited after the read, or a failed reader: still owed.
- `server/src/services/queued-comment-history-read.test.ts` (remote-Stop
resume): succeeded+cited not owed; succeeded+uncited owed; failed+cited
owed. 4/4.
- `packages/adapter-utils/src/server-utils.test.ts`: the prompt line
renders only for `readByEarlierTurn`.
- Neighbouring suites: `muse-control`, `dot-runner`,
`paperclip-runner-tool-authority`, `paperclip-control-plane-port`,
`muse-native-follow-up`, `runner-prp-coordinator`,
`native-session-handoff`, `execution-continuation-history-read`,
`heartbeat-comment-wake-batching`, `ownership-reconciliation-report`,
`run-preparation`: all pass. Server and adapter-utils `tsc --noEmit`:
clean.
- Eval graders: `npx vitest run --config
tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog
tests/runner-e2e/chat-catalog-domains`: 283 files pass. New calibration:
the observed lost-instruction case (read, not cited, no continuation)
fails; read-not-cited with one warned continuation passes; an unwarned
continuation, a missing final change, and a cited read plus a
continuation fail.
- `pnpm test:e2e:runner:typecheck`: pass. `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10.
`node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
`changed: false`.
- Paid run: Runner Full-Stack E2E 43338695850 on throwaway branch
`chat-eval/qa-mr-paid` (this change plus the model-retry fix on master
19f8ed1ea2). Result to follow in a comment.

## Risks

- A run that applies a queued comment but does not cite it gets one more
turn on that task. That turn is told the comment may already be applied.
In the restart-chaos t005 shape (#21175) this can reopen a done task
when the agent does not cite the comment. This is the chosen tradeoff: a
lost comment is worse than one extra checked turn.
- Each `get_task_history` call adds one small indexed read of the run's
`started_at`.
- Upgrade-safe: runs saved by older builds have no citation, so a
read-only delivery from before this change stays owed and is delivered
once more. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:07:08 -06:00
DottaandClaude Opus 5.5 04b50e6007 fix(native-runtime): a timed-out step records its real cause instead of leaving the run unrecoverable (#21262)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs start a Runner and a provider session, then send the
first turn
> - When a startup step fails, the native session executor redacts the
error, classifies it and records the failure
> - A timed-out step fails with a `DOMException` (`TimeoutError` or
`AbortError`). Its `message` property has only a getter
> - The redaction assigned to `error.message`. The assignment threw a
`TypeError`, so the classifier did not run and the real cause was lost
> - The run then fell into a restart-recovery chain that ended in
"requires reconciliation" wording for an operator
> - This pull request makes the redaction safe for errors with a
read-only message
> - The benefit is that a timed-out step records its real cause and
settles through the normal failure path

## Linked Issues or Issue Description

**What happened?**

In a local chaos cell with 8 opencode tasks on Daytona sandboxes, four
first runs failed before their first turn. A step before the first turn
ran under a five-minute `AbortSignal.timeout`. The step took longer than
five minutes. The run log shows `TypeError: Cannot set property message
of which has only a getter` at the startup failure handler, then
`native_finalization_missing`. The run never recorded the timeout.
Recovery then restarted the bootstrap, saw "runnerd exited while waiting
for run.prepare", and ended with `runner_state_identity_mismatch:
bootstrap_not_proven_incomplete` and
`native_continuation_requires_reconciliation`.

**Expected behavior**

A failed startup step records its own cause. The failure handler
classifies it and settles the run. It does not throw a second error that
hides the first one.

**Steps to reproduce**

Make any step before the first turn reject with
`AbortSignal.timeout(n).reason`. Before this change,
`executePaperclipNativeSession` rejects with the `TypeError` above and
the coordinator row keeps no failure detail. The new integration test
does this with the embedded database.

## What Changed

- Add `redactNativeFailureError` in
`server/src/services/native-runtime/native-failure-redaction.ts`. It
redacts the message and stack in place when it can. When the message is
read-only, it returns a new `Error` with the redacted message, the same
`name` and the redacted stack.
- Use it in the startup failure handler of
`executePaperclipNativeSession` in place of the direct assignment.
- Add unit tests for a real `TimeoutError`, a read-only `AbortError`
that holds a secret, and an ordinary typed error.
- Add an integration test: a step before the first turn rejects with a
real `TimeoutError`. The run settles through the classifier.
`failureDetail.message` is "The operation was aborted due to timeout".
The run error code is not `native_continuation_requires_reconciliation`.

## Verification

- Red on master: `native-failure-redaction.test.ts` failed 2 of 3 with
`TypeError: Cannot set property message of which has only a getter`. The
new integration test in `native-startup-retry.integration.test.ts`
failed: the call rejected with that `TypeError` and not with the
`TimeoutError`.
- Green with this change:
- `pnpm --filter @paperclipai/server exec vitest run --config
vitest.config.ts
src/services/native-runtime/native-failure-redaction.test.ts
src/services/native-runtime/native-startup-retry.integration.test.ts`
- `src/services/native-runtime/native-session-executor.test.ts`: 1431
passed, 7 skipped.
  - `pnpm verify` for the affected files.

## Risks

- Low risk. The change only alters behavior when the error message
cannot be assigned. Before, that case threw a `TypeError` and skipped
the failure handler.
- The replacement error keeps the `name` but is a plain `Error`, so
`instanceof DOMException` is false after redaction. No code in the
handler checks for `DOMException`.
- After this PR, a capture timeout before the first turn still ends
blocked. The run records the real cause, but it shows generic wording
and waits for a person to press Retry. The reason: a generic timeout has
no typed cause, so the classifier records it as
`native_session_interrupted` and treats it as ambiguous state. Follow-up
PRs give the capture step a typed cause and per-step bounds, and retry a
failure that provably happened before any prompt automatically.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs). Extended reasoning
was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:01:04 -06:00
DottaandClaude Opus 5.5 500bc694b5 fix(recovery): keep a draining server's lease alive and end a crashed server's runs in about 2 minutes (#21238)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A release swap runs two servers for a short overlap. The old server
drains, and the new server takes over the runs.
> - Each server proves that it is alive with a controller-instance
lease. Recovery uses that lease to decide whether a run's controller is
gone.
> - In local swap cells, a healthy old server exited during the overlap,
because its lease renewal waited for a connection on the busy main pool.
Every run it drove was interrupted.
> - When a controller crashed, its runs waited for another server boot,
or for a timer that restarted on every look. Tasks stood still for 21 to
28 minutes.
> - This pull request renews the lease on its own connection, and lets
the live server end a crashed controller's runs within a bounded time.
> - The benefit is that a swap does not kill a healthy server, and a
crashed server's runs end in about 2 minutes, so their tasks continue by
themselves.

## Linked Issues or Issue Description

Refs #21135, #21222

**What happened?**
1. During a two-server release swap (local cells, load 14 to 34), the
outgoing server exited with "controller instance lease lost while
holding work". Five lease renewals in a row timed out after 10 s. The 10
s timeout included the wait for a connection on the shared main pool,
which the traffic switch and the replacement's startup pass kept busy.
2. When the old server was killed in the middle of the overlap, its runs
stayed in progress for 21 to 28 minutes. The live server did not
re-check a crashed controller's runs: only a new boot did. The
unverified-controller wait started when a server first looked at the
run, not when the controller left, and the successor's own wait record
counted as the dead owner's sign of life.

**Expected behavior**
A healthy server keeps its lease during a swap. When a controller is
gone (its lease lapsed and it is fenced or stopped), the live server
ends that controller's runs within a bounded time, without another
server boot, so restart recovery continues the tasks.

**Steps to reproduce**
1. Run two servers against one database with a release swap and a 200 s
overlap.
2. Kill the old server 40 s into the overlap (SIGKILL).
3. See that its runs stay in progress for more than 20 minutes, until
another server starts.

## What Changed

- Lease renewal on its own connection (`packages/db/src/client.ts`,
`packages/db/src/index.ts`,
`server/src/services/controller-instance-lease.ts`):
- New `controllerInstanceLeaseDb(db)`: one private 1-connection pool
(`application_name` `paperclip-controller-instance-lease`), made the
same way as the existing `controllerLeaseDb` and closed by
`closeLivenessDb`.
  - The renewal is one statement on that connection, as before.
- The connection sends `statement_timeout` = 8 s (below the 10 s client
timeout) as a startup parameter (new
`DatabaseClientOptions.statementTimeoutMs`, used only by this pool). The
server cancels a slow renewal, so it never holds the single connection
that the next renewal needs, and there is no extra round trip per
renewal. (An earlier revision set it per transaction; that added three
round trips to every renewal and broke the idle-server receipt test.)
- If a connection pooler refuses the startup parameter ("unsupported
startup parameter"), the lease logs a warning and renews on a
replacement connection without it.
- A crashed controller's runs
(`server/src/services/recovery/ownership-registry.ts`,
`server/src/index.ts`,
`server/src/services/native-runtime/native-restart-recovery.ts`):
- The ownership reconciler on the live server hands runs whose
controller is proven gone to restart recovery. It does not wait for
another server boot.
- The unverified-controller wait is bounded from the moment the
controller's lease lapsed, not from the first look.
- When the controller boot is `stopped` or `fenced`, the run ends at
once.
- A successor's wait record no longer refreshes the dead owner's sign of
life.

Safety of ending a run when its controller boot is fenced or stopped:
- Only `stopped` (written by the boot's own graceful stop) and `fenced`
(a newer epoch fenced it after its lease expired plus 30 s grace) count.
`live` and `grace` never count, so a controller that is alive and
renewing is never treated as gone. `expired` alone does not count
either.
- A boot that cannot renew stops itself at its self-fence window (about
its lease expiry), and a boot that finds itself fenced exits at once. So
when another server can fence it, the old process has already stopped
driving runs. The lease change above removes the main cause of healthy
boots failing to renew.
- If the old process is still alive (partition or stall) and its Runner
continues: the old run is fenced. Its agent token is refused for writes
(`agent_run_superseded`), its remote workspace epoch is abandoned or the
continuation gets a new workspace, and the native coordinator records
`replacementDenied`. There are never two writers in one workspace.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `server/src/__tests__/controller-instance-lease.test.ts` "own
connection": red (10004 ms timeout while the main pool is held) to
green. Full file green (26).
- `server/src/__tests__/controller-instance-lease.test.ts` "has the
server cancel a renewal blocked on a row lock, and renews on the same
connection next time": red without the connection-level timeout (the
renewal runs to the 10 s client timeout and stays active), green with it
(cancelled at about 8 s, no active lease statement left, the next
renewal succeeds in under 2 s on the same backend).
- `packages/db/src/client-options.test.ts`: the statement timeout is
sent as a startup parameter.
- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`:
"lapsed longer ago" / "end of the bound" (wait counted from the lease
lapse), the live-server re-check of a crashed controller, the
fenced/stopped boot ending the run at once, and the successor's wait
record. Full file green.

Changed existing expectation (it asserted the old wait for a stopped or
fenced controller):
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
"keeps foreign local Runner and provider ownership after its controller
is stopped/fenced": the disposition is now `blocked`
(`controller_process_space_unverified`) instead of `awaiting_evidence`,
the run is `failed` with `native_restart_recovery_blocked`, and the
finalization is `terminal_failure` with `executorStopEvidence:
"unverified"`. The safety checks stay: no local PID probe, no claim, no
stamp, no local stop proof, the run keeps its process identity and
session checkpoint, and no result is written.

Local commands, on this branch based on current master:
- `cd server && npx vitest run
src/__tests__/controller-instance-lease.test.ts
src/__tests__/heartbeat-native-restart-suspension.test.ts`: 2 files, 140
tests passed.
- `npx vitest run src/__tests__/controller-instance-lease.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
225 passed, 14 skipped (real-process cases that need CI).
- `pnpm exec vitest run src/__tests__/maintenance-server-routes.test.ts
-t "releases an ordinary held task Retry"`: passed (idle receipt: one
lease statement after the work ends).

Local swap cells (two servers, embedded Postgres, real agents, old
server SIGKILLed 40 s into a 200 s overlap):
- Before: runs of the killed server stayed in progress 21 to 28 minutes.
- N1 (2 tasks): 2/2 finished hands-off. Kill to run end 5 min 31 s (the
successor's wait write still moved the timer; fixed by the last commit).
- R1 (6 tasks, 3 adapters, subtasks, follow-ups, 2 swaps): 4/4
hands-off, 0 banners, 0 duplicates, 0 invariant violations. Kill to run
end 3 min 39 s.
- N2 (same seed as N1, all fixes): 2/2 hands-off. Kill to run end 2 min
01 s; fresh runs started at 2 min 32 s and 3 min 02 s. The floor is the
lease TTL (90 s) plus the fence grace (30 s).
- Hour-long release-swap wave (w-20261112T233630-s7201) on the branch
that holds this fix and the related recovery fixes: 11/11 tasks finished
hands-off (master: 9/11). The lease and crashed-controller fixes were
visibly working in it.

## Risks

- A pooled database URL whose pooler refuses `statement_timeout` as a
startup parameter: the first renewal fails over to a connection without
it (logged), so the lease keeps working without the server-side cancel.
- One extra database connection per server, held while the server runs.
It is idle except for one renewal every few seconds.
- A run whose controller boot is fenced or stopped now ends at once
instead of after the bound. See the safety notes above: only a boot that
is not alive can be in these states.
- No migration. No schema change. Nothing changes before `turn.start` on
the normal path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:11:27 -06:00
DottaandClaude Opus 5.5 2d1c2db64a feat(mcp): host the MCP Skills extension for connected apps (#21219)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use connected apps through MCP. Paperclip already hosts MCP
Apps views and speaks stateless MCP 2026-07-28.
> - The MCP Skills extension (SEP-2640, Final) lets an app publish
skills (instructions and files) next to its tools. Hugging Face already
serves 25.
> - Paperclip did not read them, so agents missed each app's own
workflow guidance.
> - Skills are a prompt-injection surface. The spec makes manifest
binding, verification, origin tagging and approval revocation host
requirements.
> - This pull request makes Paperclip a Skills host: discover, review,
verify, and deliver app skills to the agents that use the app.
> - The benefit is that connected apps can teach agents their workflows,
under board control.

## Linked Issues or Issue Description

**Problem**
Connected MCP servers can publish skills with the
`io.modelcontextprotocol/skills` extension. Paperclip ignored them. A
board could not see, review or approve them, and agents never received
them.

**Proposed solution**
Implement the host side of SEP-2640. List skills during catalog refresh,
record them per connection and company, bind board approval to the full
file manifest, and load approved skills into agent runs with verified,
lazily fetched content.

**Why it matters**
Apps ship workflow guidance with their tools. Without the extension,
each team must copy that guidance into company skills by hand, and it
goes stale.

Refs #21119, Refs #21140 (the MCP Apps and stateless MCP client work
this builds on).

## What Changed

- **Client capability.** Every stateless request declares
`io.modelcontextprotocol/skills`. Legacy handshake sessions do not. The
capabilities that `server/discover` advertises are remembered beside the
protocol era.
- **Discovery.** Catalog refresh calls `skills/list` (paged) only when
the server declared the extension. A listing inside its `ttlMs` is
reused. A skills failure never fails the tool refresh. Nothing runs
while a run starts.
- **Schema.** New `tool_mcp_skills` (company, connection, skill URI;
frontmatter, sorted manifest, manifest digest, review state) and
`tool_mcp_skill_files` (verified bytes keyed by connection, URI and
digest). Both cascade with the connection.
- **Governance.** New skills wait for review. Approval binds to the
order-free digest of every `{uri, digest, size}`; a request for any
other digest gets 409. A changed manifest or frontmatter revokes
approval and logs `tool_mcp_skill.approval_revoked`. Dynamic, oversized
and invalid skills are declined with a reason. Approve and decline use
the company skill policy (`skills.install`, `external_package`,
`mcp-connection:<id>`).
- **Run time.** Approved skills of connections the agent can use load in
parallel with other skill preparation, inside a 3 s budget. `SKILL.md`
is read with the agent's grant and checked for size, SHA-256 and
frontmatter. A failure calls `skills/get` and takes the skill out of
use. Each skill becomes a per-run bundle on the connector skill path,
named in its app's namespace, tagged with its origin. Paperclip writes
the frontmatter: `allowed-tools` only if the board approved it.
- **Supporting files.** `GET
/api/agents/me/mcp-skills/:skillId/files?path=` serves manifest-listed
files to an agent in an active run, through the skill's own connection.
The sandbox bridge allows the route.
- **Board API and UI.** `GET/POST /api/companies/:companyId/mcp-skills…`
(list, read instructions, approve, decline). The Skills page shows
"Skills from connected apps" with app, state, size, requested tools and
plain-language notes. It renders nothing when no app publishes skills.
- **Docs.** `doc/connections/MCP-SKILLS.md`, linked from
`doc/connections/README.md`.
- **Shared frontmatter parser.** It now decodes YAML single-quoted
scalars. Before, 5 of Hugging Face's 25 skills kept their quotes and
would fail the frontmatter check; now all 25 match. The generated runner
copy is regenerated.

File size notes: `server/src/services/tool-gateway.ts` grows by about
200 lines. The run-time reads must use its grant resolution and
`callAssignedConnectionProtocol`, like MCP Apps view reads. Splitting
the gateway is a separate refactor. `CompanySkills.tsx` and
`CompanySkills.production.tsx` gain 2–4 lines each; the section lives in
its own component.

## Verification

- `server/src/__tests__/mcp-skills.test.ts` (15 tests) uses a fake
stateless skills server and embedded Postgres. It covers: discovery only
after the server declares the extension; the capability in `_meta`;
`ttlMs` reuse; lazy fetch (no reads at listing or approval; `SKILL.md`
at load; supporting files on read); a stale-digest approval refused; a
reorder that keeps approval; a changed or added file that revokes it;
dynamic skills declined; a same-size tampered `SKILL.md` rejected,
refreshed with `skills/get`, and taken out of use; mismatched
frontmatter rejected; reads outside the manifest refused; the same URI
and name from two servers kept apart; company scoping; a slow app
skipped within the budget and cached later; HTTP authorization (board
only, agent needs an active run, run header must match) and activity
logging.
- Red/green: with the digest check disabled, the tamper test fails; with
the manifest comparison disabled, the revocation test fails. Both pass
with the code.
- `server/src/__tests__/mcp-protocol-versions.test.ts`,
`mcp-apps.test.ts`, `connection-instructions.test.ts`,
`tool-gateway-discovery-http.test.ts`: 47 pass (two capability
assertions updated for the new extension).
- `packages/shared/src/frontmatter.test.ts` adds a single-quoted scalar
case; the runner contract test passes (23 tests).
- `ui/src/components/skills/ConnectedAppSkills.test.tsx` (3) and
`ui/src/pages/CompanySkills.test.tsx` (44) pass.
`packages/adapter-utils/src/sandbox-callback-bridge.test.ts` (89)
passes.
- `pnpm --filter @paperclipai/server typecheck` passes. UI `tsc` passes
except the known missing generated Storybook catalog files.
- `pnpm verify` returned exit 2 (doc change is a global CI input);
required CI supplies broad coverage.
- Live check against `https://huggingface.co/mcp` without credentials,
through Paperclip's own services and the real egress guard: 25 skills
discovered as pending; three approved; cold load 135 ms, cached load 13
ms; a supporting file read through the manifest; the bundle tagged with
its origin.
- Storybook: `Agents and reusable capabilities/Skills and Skill
Studio/Connected app skills` (needs review, all states, tool request,
name collision, reading instructions).

## Risks

- **Migration:** two new tables only. No changes to existing tables.
Safe with the previous release running.
- **Run start:** companies without an approved app skill pay one indexed
read. With approved skills the wait is capped at 3 s; late or failing
skills are left out with a `skills.mcp_unavailable` run warning and
never fail the run.
- **Agent harness enforcement:** Paperclip cannot gate each
code-execution tool call an agent makes while following an app's skill.
Approval is explicit and per skill, the approval note says agents may
run commands the skill describes, and `allowed-tools` is never granted
without a separate tick.
- **Shared-home adapters** receive app skills like company skills. A
revoked skill leaves on the next skill sync.
- Not in this PR: per-connection opt-in for dynamic skills; serving
Paperclip's own skills from its public MCP endpoint;
`resources/directory/read`; app skills on an agent's skills tab.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs, live HTTP checks).
Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:18:06 -06:00
DottaandClaude Opus 5.5 b0233b5b33 fix(native): count a parent run's read of a finished child as delivery of its result (#21227)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task gets a wake when a child task completes. If the parent
is running, that wake waits behind the run.
> - At finalization, a parent's done claim stays open
(`native_child_completion_pending`) while such a wake is still owed, so
the parent never finishes without the child result.
> - The server had no record of what a running parent actually read. A
parent that read its finished children mid-turn and then finished the
task still kept the wake owed.
> - The task stayed `in_progress`, the wake started a second parent
turn, and the agent wrote a second completion summary.
> - This pull request records a run's read of a finished child and
counts that read as delivery of the child result.
> - The benefit is one completion turn when the parent already read its
children, while unread child results still get a fresh turn.

## Linked Issues or Issue Description

Refs #21189 (child results saved before the run started). Refs #21175
(the same pattern for comments: `task_history.read`). Refs #15372 (the
pending child-completion guard).

**What happened?**

Restart-chaos wave `w-20261112T223803-s7101`, on a master build that
includes #21189, task t006 (OpenCode, native runner, staging-mix parent
with two subtasks):

- Parent run `bfe9941c` started at 22:55:56.
- Its children CHA-11 and CHA-12 finished at 23:07:53 and 23:11:38.
- The child-completion intent and its deferred dispatch row were saved
at 23:11:38, after the run started. So the #21189 rule counted them as
new input.
- At 23:26:41 the run's reasoning says "Both subtask ledgers exist …
Both subtasks are done". The run then saved the CHILD hashes and posted
DONE at 23:33:39.
- At finalization, the done claim was committed as
`native_child_completion_pending`. The deferred wake then started run
`40928116`, which wrote a second completion summary.

Root cause: `hasPendingNativeChildCompletion()`
(`server/src/services/native-runtime/native-child-completion-delivery.ts`)
proved delivery only by wake binding and timestamps. No durable record
showed that the finishing run had read the completed child.

**Expected behavior**

A finishing run's recorded read of a completed child counts as that
child result being delivered. A read while the child was open, a read by
another run, or a failed event write does not count. An unread child
result still keeps the task open and gets one fresh parent turn.

**Steps to reproduce**

1. Start a native parent run.
2. Complete a child while the run is active. The child-completion wake
waits behind the run.
3. Let the run read the child (`get_task_context`, or `GET
/api/issues/:childId`), then claim `done`.
4. Finalize and release the run. On `master`, the decision is
`native_child_completion_pending` and the wake becomes a second run.

The same wave's t001 grade ("duplicate completion") is not this bug.
That run only said it *will* post the DONE marker later, and the grader
counts any mention. Its `subtasks_once` failure comes from the
restart-recovery continuation. This PR does not change either one.

## What Changed

- `native-child-completion-delivery.ts`:
- New `recordChildTaskRead()` writes a `child_task.read` run-log event.
The payload is `{ parentIssueId, children: { id: { status, updatedAt } }
}`, and it lists only children in `done`.
- `hasPendingNativeChildCompletion()` no longer counts a
child-completion wake as pending when the finishing run recorded a read
of that wake's completed child in `done` at or after the child's
completion decision (`status_decisions.created_at`, or the wake's
`createdAt` when the decision row is missing).
  - The #21189 start-time rule is unchanged.
- `paperclip-runner-tool-authority.ts`: `get_task_context` records the
finished child tasks it returns. A failed write is logged and leaves the
result owed.
- `routes/issues.ts`: `GET /issues/:id`, `GET /issues/:id/documents` and
`GET /issues/:id/documents/:key` record the read when the request has
all of these:
  - an agent actor with a run token
  - a child issue in `done`

This runs after the existing access and company checks. So a board user,
another company's agent, or a read of an open task writes nothing. A
qualifying read adds one statement: the existing
`appendHeartbeatRunEvent` fast path, a single `update … returning` /
`insert` CTE bound to the run's id, company and agent. Every other read
adds no query.
- `doc/run-log-events.md`: documents `child_task.read` and how delivery
uses it.
- Tests: a finalization-to-release regression in
`heartbeat-process-recovery.test.ts` and a new
`native-child-task-read-routes.test.ts`.

## Verification

- RED on `master`: in `heartbeat-process-recovery.test.ts` "counts a
child result as read by the finishing parent run (saved after the run
started, read: after the child finished)", the run reads the child
through the real `get_task_context`, then finalization and the
wake-queue release run.
- On master: `native_child_completion_pending`, task `in_progress`, and
the wake is promoted into a second run, the same as t006.
- With this change: `done`, the wake is cancelled, and there is one run.
- The other cases in the same test behave the same before and after this
change:
  - saved before the run started: done
  - not read: pending, fresh turn with the child results
  - read while the child was open: pending
- `native-child-task-read-routes.test.ts` (5 tests):
- An agent read of a done child through each of the three routes records
the read and clears the pending result.
  - An open child records nothing.
  - A board read records nothing.
  - Another company's agent gets no 200 and records nothing.
- Neighbouring suites after merging `origin/master`:
`heartbeat-process-recovery`, `muse-control`,
`heartbeat-held-comment-wake`, `native-status-arbiter-corpus`,
`heartbeat-dependency-scheduling`, `paperclip-runner-tool-authority`,
`agent-task-hierarchy-depth-routes`, and the two new test files: 8
files, 982/982 tests pass.
- `pnpm --filter @paperclipai/server typecheck` passes.

**Not verified**

- No chaos wave has run on this change yet.
- In t006, the parent's tool calls fell in the 4,220 run events that the
evidence omitted. Its reasoning text proves the read, but the exact tool
(`get_task_context`, `get_task`, `read_task_document` or `call_api`) is
not visible.
- A read through `search_tasks`, or through a shell command that calls
the API without the run token, is not recorded. Such a result stays owed
and gets one fresh turn, which is the old behavior.

## Risks

- A recorded read of a done child now lets the parent finish without
another turn. If the agent read the child but ignored it, no second turn
reminds it. A read while the child was still open never counts.
- Each qualifying agent read of a done child appends one run-log row.
Board reads and reads of open tasks write nothing.
- No migration. Wakes saved by older builds work unchanged. Without a
recorded read, the old behavior applies.
- This is a run-log change only (instance database). It adds no
Telemetry or OpenTelemetry export.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:07:31 -06:00
Devin FoleyandPaperclip ad320aeb71 fix(native): wait for a dated provider limit reset and keep it out of Sentry (#19775)
Keep native quota cooldowns tied to valid dated reset messages. Parse explicit years and strict month names while preserving already-expired dates. Keep typed native quota and rate-limit failures on their tasks in unexpected Sentry reporting mode. Preserve setup, process, malformed and all-mode reporting.

Verified current-head CI, Greptile 5/5 at Medium impact, local parser/report red-green tests and PostgreSQL integration coverage.

Co-Authored-By: Devin Foley <devin@devinfoley.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-12 17:06:22 -08:00
Nicky LeachandClaude Fable 5.2 2dcfdabc79 feat(runner): let a provider pack record the Runner image it was built in (#21228)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - A provider pack has a manifest, `provider-pack.json`. It records the
commit the pack was built from as `payload.runnerSourceRevision`
> - The next pull requests build the Runner alone as an image that has
an identity of its own: a digest of the Runner's inputs
> - A pack from that image must be able to say which image it came from.
The commit does not say that, because many commits have the same inputs
> - This pull request adds one optional field to the pack manifest for
that digest
> - The benefit is that a reader of a pack can find the exact image that
built it. Packs that are built today do not change

## Linked Issues or Issue Description

Refs #21204 (the Runner image, part 2 of this stack). No issue exists
for this change.

**Problem or motivation**

The provider pack manifest records a source revision. It has no place
for the identity of a Runner image. The Runner image in #21204 is named
by a digest of its inputs, and one image serves every commit that has
those inputs. Without a field in the manifest, a pack cannot name the
image that built it.

**Proposed solution**

Add the optional field `payload.runnerInputsDigest`. The pack builder
writes it only when the environment variable
`PAPERCLIP_RUNNER_INPUTS_DIGEST` is set. The value must be 64 lowercase
hexadecimal characters. The pack builder fails for another value.
Nothing sets the variable in this pull request.

**Alternatives considered**

- Put the digest in a file beside the pack. Then the pack digest does
not cover it, and a copy of the pack loses it.
- Reuse `runnerSourceRevision`. That field is a commit, and code
compares it with other commits.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 1 of 4. Each part is on top of the one before it:

1. `runner/pack-inputs-digest`: an optional field in the provider pack
manifest (this pull request)
2. `runner/runner-image-build` (#21204): build and check the Runner
image, with no publication
3. `runner/runner-image-layers`: layers by change rate, and the slim
variant
4. `runner/runner-image-publish`: publish both variants from `master`

## What Changed

- `packages/paperclip-runner/scripts/build-provider-pack.mjs`: reads
`PAPERCLIP_RUNNER_INPUTS_DIGEST`. When it is set, the script checks its
form and writes `payload.runnerInputsDigest` after
`runnerSourceRevision`. When it is not set, the manifest is the same as
before, byte for byte.
-
`server/src/services/native-runtime/remote-provider-pack-verification.ts`:
the manifest type has the optional field `runnerInputsDigest`. No check
reads it.
-
`server/src/services/native-runtime/remote-provider-pack-verification.test.ts`:
one test. A sealed manifest with the field passes verification. A
manifest whose field was changed after it was sealed fails, because the
pack digest covers the field.

The change is 20 added lines in 3 files.

## Verification

- `cd server && node node_modules/vitest/vitest.mjs run
src/services/native-runtime/remote-provider-pack-verification.test.ts`:
19 tests pass.
- Part 2 builds a pack with the variable set and a pack without it, on
`linux/arm64`. The first has the field and the second does not.

Never run:

- The full test suite and the full typecheck. CI must supply them.

## Risks

- Low risk. The field is optional and additive, and nothing sets the
variable yet.
- The pack digest covers the payload. A pack that has the field
therefore has another digest than the same pack without it. No pack has
the field until part 2 builds one.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 16:58:14 -08:00
DottaandClaude Opus 5.5 79334f58ab test(runtime): stop one slow plugin Runner release/resume case from failing the cases after it (#21213)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The plugin Runner release/resume integration test proves that a
released Runner can be resumed by the next run
> - The test failed on unrelated PR #21171 in the "Verify Paperclip
Runner (vitest native server)" lane
> - One slow case caused 5 more cases to fail, and the PR retry could
not run
> - #21199 fixed the same class of failure in the plugin Runner eviction
test
> - This pull request applies the same isolation to the release/resume
test, and also isolates its shared spies
> - The benefit is that one slow case no longer fails the cases after it

## Linked Issues or Issue Description

Refs #21199

**What happened?**

`server/src/services/native-runtime/plugin-runner-release-resume.integration.test.ts`
failed on unrelated PR #21171. "completes codex runs through a retained
cached process" timed out at 60s. Then 5 more cases failed, some at 60s
and some in under 3s with `RangeError: Maximum call stack size
exceeded`. Every retry failed on `duplicate key value violates unique
constraint "issues_identifier_idx"`.

Root cause: every case changed process-wide state and restored it in its
own `finally` block. That state was the default TLS trust,
`https.globalAgent`, a `vi.spyOn` on
`PaperclipRunnerToolAuthority.prototype.definitions`, and a `vi.spyOn`
on `inspectRemoteWarmSessionRecovery`. Vitest continues to the next case
when a case times out, but the timed-out function continues to run.
While it ran, the next case read the timed-out case's spy as the
"original" `definitions`. When the timed-out case restored its spies,
the two spies called each other, and the stack overflowed. Its late
`finally` also removed the next case's TLS trust, so that case stayed in
`connecting` with no wire frames until its own 60s timeout. It also
closed the next case's warm sessions with
`closeIdleWarmNativeSessionsForRestart()`. The retry could not help,
because the issue identifier came only from the case name.

**Expected behavior**
A case that times out fails alone. Later cases and the PR retry run
normally.

**Steps to reproduce**
1. Make the `codex cached` case sleep 62s once, after its first run.
2. From `server/`, run `npx vitest run
src/services/native-runtime/plugin-runner-release-resume.integration.test.ts
-t "codex runs through a retained (cached|adopted|adopted restart)
process"`.
3. Before this change, 3 of 3 cases fail, with `Maximum call stack size
exceeded`. With `--retry=1`, 3 of 3 still fail.
4. After this change, only the slow case fails. With `--retry=1`, 3 of 3
pass.

**Paperclip version or commit**
origin/master at the base of this branch (after #21199)

**Deployment mode**
CI and local test only (embedded Postgres, real `paperclip-runnerd`)

**Agent adapter(s) involved**
paperclip_runner (codex and claude ACPX harnesses through the plugin
task driver)

## What Changed

- Create one fixture certificate in `beforeAll`. Set the TLS trust and
`https.globalAgent` once for the file, and restore them in `afterAll`.
- Read the original `PaperclipRunnerToolAuthority.prototype.definitions`
once at module load, before any case spies on it.
- Move the per-case teardown into an `afterEach` that is registered
before the database fixture. The teardown closes idle warm sessions, the
host control, and the TLS proxy, and it restores both spies. Each case
binds its own teardown holder before its first await.
- Add a random suffix to the issue identifier of each attempt, so the PR
retry can run the case again.
- Increase the idle timeout of the adopted cases from 5s to 10s. The
product limits a release to `min(idleTimeoutMs, 30s)`, and #21199
measured a loaded release that took more than 4.8s.
- Probe the Runner port on `0.0.0.0`, the address that the Runner binds.

No product code changed.

## Verification

- Injection test (one case sleeps 62s): see the steps above. Before: 3
of 3 failed, also with `--retry=1`. After: 1 of 3 failed without retry,
and 3 of 3 passed with `--retry=1`.
- Before/after loops of the full file (14 cases), run in parallel on the
same loaded macOS host: see the loop result comment on this PR.
- `tsc --noEmit -p server/tsconfig.json` reports no errors in the
changed file.

## Risks

Low risk. Only the test changed. The adopted (non-restart) cases take
about 5s more each. The file creates one certificate instead of 14. A
case that hangs inside Core execution can still leave a busy warm
session in module state.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with
tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:19:06 -06:00
DottaandClaude Opus 5.5 57aeda190b test(runtime): stop one slow plugin Runner eviction case from failing the cases after it (#21199)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The plugin Runner eviction integration test proves that a retained
Runner continues after the host evicts it
> - The test failed on many unrelated pull requests in the "Verify
Paperclip Runner (vitest native server)" lane
> - One slow case caused other cases to fail, and the PR retry could not
run
> - This pull request isolates each case from a case that timed out, and
gives the retry a fresh issue
> - The benefit is that one slow case no longer fails the cases after it

## Linked Issues or Issue Description

**What happened?**

`server/src/services/native-runtime/plugin-runner-eviction.integration.test.ts`
failed in CI on at least 5 unrelated branches in 48 hours, most recently
on #21149. The errors were `Test timed out in 60000ms` and `Timed out
waiting for automatic release A`. Each retry then failed at once with
`duplicate key value violates unique constraint
"issues_identifier_idx"`. The failed cases logged a Runner in
`connecting` with no wire frames.

Root cause: every case changed process-wide state. It set the default
TLS trust and `https.globalAgent` for its own certificate, and it
restored them in its own `finally` block. Vitest continues to the next
case when a case times out, but the timed-out function continues to run.
Its late `finally` then removed the TLS trust of the next case and
closed the warm sessions of the next case with
`closeIdleWarmNativeSessionsForRestart()`. The next case could not open
its Runner connection, waited for the 60s startup deadline, and also
timed out. That case then did the same to the case after it. In CI run
43266976387, one timeout caused 3 more failures. Locally, one timeout
caused 5 more failures.

Two more problems made this worse:
- The issue identifier came only from the case name. A retry of the same
case therefore failed on the unique identifier, so the PR retry never
ran the test again.
- Detached cases used a 5s idle timeout. The product limits a release to
`min(idleTimeoutMs, 30s)`, so these cases had 5s for the drain and the
park. Under load, one park took more than 4.8s, and the release expired.

**Expected behavior**
A case that times out fails alone. Later cases and the PR retry run
normally.

**Steps to reproduce**
1. Make one case slower than 60s. For example, add a 62s sleep after
release A in `codex cached retired`.
2. Run `npx vitest run
src/services/native-runtime/plugin-runner-eviction.integration.test.ts
-t "codex with a (cached|detached) Core owner"` from `server/`.
3. Before this change, 3 of 4 cases fail: the slow case, and 2 cases
that it disturbs. With `--retry=1`, the slow case also fails on the
duplicate identifier.
4. After this change, only the slow case fails. With `--retry=1`, all 4
cases pass.

**Paperclip version or commit**
master at ef6d347f79, rebased on the current origin/master

**Deployment mode**
CI and local test only (embedded Postgres, real `paperclip-runnerd`)

**Agent adapter(s) involved**
paperclip_runner (codex and claude ACPX harnesses through the plugin
task driver)

## What Changed

- Create one fixture certificate in `beforeAll`. Set the TLS trust and
`https.globalAgent` once for the file, and restore them in `afterAll`. A
case no longer changes process-wide trust.
- Move the per-case teardown (close idle warm sessions, close the host
controls, close the TLS proxy) into an `afterEach`. It is registered
before the database fixture, so it runs before the database clients
close, also when a case times out.
- Add a random suffix to the issue identifier of each attempt, so the PR
retry can run the case again.
- Increase the detached idle timeout from 5s to 10s, and wait 200ms past
it as before. This gives the release the time that it needs under load.
- Probe the Runner port on `0.0.0.0`, the address that the Runner binds.

No product code changed.

## Verification

All loops ran on a shared, loaded macOS host (load average 15 to 20 on
18 cores), plus 2 CPU busy loops. Each iteration ran the 12 cases that
match `with a (after inspection|missing state|reused lease) Core owner`.

- Before: 20 iterations, 2 failed (10%). In iteration 10, `claude reused
lease retired` timed out at 60s, and 5 later cases failed on
`readNativeSessionIdleWork() === "none"` because of the case that timed
out. In iteration 17, `claude after inspection retired` failed with
`Timed out waiting for automatic release A`.
- After: see the loop result comment on this PR.
- Injection test (one case sleeps 62s): before, 3 of 4 cases failed, and
with `--retry=1` the retry failed on the duplicate identifier. After,
only the slow case failed, and with `--retry=1` all 4 passed.
- Full file after the change: 40 of 40 passed in 320s locally (335s
before, because the file now creates one certificate instead of 40).
- `tsc --noEmit -p server/tsconfig.json` reports no errors in the
changed file.

## Risks

Low risk. Only the test changed. The detached cases take about 5s more
each (12 cases, about 60s in all), which is partly offset by creating
one certificate per file. A case that hangs inside Core execution can
still leave a busy warm session in module state. This change does not
prevent that. It removes the causes of the cascade that this PR found.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with
tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:18:01 -06:00
DottaandClaude Opus 5.5 de675d924a fix(heartbeat): stop a comment the closing run already read from reopening its finished task (#21175)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can add a comment to a task while the agent's run on that
task is still working
> - The comment must reach the agent once: in the running turn, or in
one later turn
> - In a restart-chaos wave, a comment that the running turn already
read reopened the finished task, and the agent did the completion turn a
second time 15 minutes later
> - This pull request makes a read by the running turn count as
delivery, and holds the comment behind the running run so the run's own
release settles it
> - The benefit is no reopened done task, no repeated completion turn,
and an unread comment that is delivered when its run finishes, not after
a slot wait and a stale cancellation

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261112T181948-s7101`, task t005 (OpenCode,
native Runner), during a two-server overlap deploy (K15).
- 18:26:56 run A started on the old server. 18:26:55 traffic switched to
the new server.
- 18:27:03.953 a board user posted a follow-up comment. The new server
handled it. Run A was not in that process's `activeRunExecutions`, so
`filterZombieCoalesceTarget` treated it as a zombie. Admission returned
`proceed`, and the comment got its own queued run B (18:27:04.252). The
queued run is not part of the release drain.
- 18:33:32 run A called `get_task_history`, received the comment,
applied it, and finished the task (`done`, 18:34:20).
- Run B waited for an agent slot until 18:41:49 and was cancelled as
`continuation_task_ownership_changed`.
- `deferUndeliveredCommentsOfStaleRun` then saved the comment again. Its
"unread" test is only "a run started before the comment and finished
after it". `deliveredQueuedCommentIds` had no record of run A's read.
The release drain reopened the task (done to todo, 18:41:51), and run C
(18:49:22) posted a second "Task complete" summary. The harness grade
`done_comment_once` failed.

**Expected behavior**
- A comment that the running turn received is consumed. No queued work
outlives that run, and the task stays done.
- A comment that the running turn did not receive is delivered once,
through the normal comment-on-done-task path, when that run finishes.
- A stale continuation is never re-issued for a comment that a completed
run already read.

**Steps to reproduce**
1. Start a native run on a task.
2. Post a person's comment while another server instance executes the
run (or seed a queued comment run behind it).
3. Let the run read `get_task_history`, then finish the task.
4. Before this change, the queued comment run is cancelled as stale, the
task reopens, and the agent runs a second completion turn.

**Paperclip version or commit**
`master` at ef6d347f79.

## What Changed

- `server/src/services/queued-comment-delivery.ts`: new run-log event
`task_history.read` and `recordTaskHistoryRead`.
`deliveredQueuedCommentIds` also counts a comment as delivered when a
succeeded run of the assignee on that task read the comment's current
revision (`historyReadDeliveredSql`). All existing consumers use this
one check: the release drain liveness check, the stale-continuation save
(`saveUndeliveredUserComments`), the ownership reconciler, and
first-turn queued input.
- `paperclip-runner-tool-authority.ts` and
`runner-semantic-authority.ts`: `get_task_history` records the returned
comment IDs and `updatedAt` values. The tool output does not change. A
failed record is logged and leaves the comment owed, so it is delivered
again.
- `server/src/modules/wake-queue/application/use-cases.ts`: a comment
follow-up for the same agent's `running` run is held as a deferred wake
even when this process does not track that run. Before, a run executed
by another server instance was treated as a zombie, and the comment got
its own queued run that bypassed the release drain. Coalescing into an
untracked run is still refused.
- `doc/run-log-events.md`: documents `task_history.read`.
- Regression tests in
`server/src/__tests__/heartbeat-held-comment-wake.test.ts`.

Intended behaviour chosen: in t005 run A received the comment mid-turn
through its own `get_task_history` call (run events at 18:33:32 show the
comment ID in the tool result, and the result says it added the
follow-up line), so (a) applied. The fix makes that read the durable
proof of delivery. A comment that no run read still follows the existing
outbox: it is held behind the run and delivered once at the run's
release, reopening the done task as a person's comment does. Envelope
task history of another message's turn still does not count (unchanged
doctrine and test).

## Verification

- RED on master (`ef6d347f79`), `npx vitest run
src/__tests__/heartbeat-held-comment-wake.test.ts -t "mid-turn|another
server|held message"`: 3 failed, 1 passed.
- "consumes a message the closing run read mid-turn, so its stale queued
run reopens nothing": delivered 1, expected 0.
- "holds a message behind a run another server is executing instead of
queueing a run of its own": a queued run was created.
- "settles a held message as soon as its run closes the task (read
mid-turn: true)": delivered 1, expected 0.
- The unread variant (delivered once, no cancelled stale run) passes on
both, as intended.
- GREEN with this change: all 4 pass.
- Review follow-up (f43e79c3): a history read counts as delivery only
for a succeeded run on the read revision. Tests cover an edit after the
read (the stale queued run delivers the comment once) and reads by an
edited or failed run (the comment stays owed). Full file: 37/37 pass.
- Security-scan follow-up (239f0597): the delivery check probes each run
binding (`native_issue_id`, `context_snapshot->>'issueId'`) through its
own existing index with `offset 0`. It reads only the assignee's
succeeded runs of the task with `finished_at >= ` the comment's current
revision. It does not use `started_at`, because the t005 reader started
before the comment. It stops at the first matching `task_history.read`
event of each run through `heartbeat_run_events_company_run_idx`. No
migration is needed.
- Seeded plan (local PostgreSQL: 200 tasks, 4,000 runs, 1.2M run events,
one comment read by one run), `EXPLAIN (ANALYZE, BUFFERS)`:
- Before: 1.202 ms, 6,097 shared buffers. It scanned every event of all
21 runs of the task.
- After: 0.030 ms, 23 shared buffers. It reads one index-matched run and
one event. The full `deliveredQueuedCommentIds` call takes 1.55 ms.
- New case: a succeeded run's read is not delivery when the comment was
edited after that run finished. Full held-comment file: 38/38 pass.
- `muse-control.test.ts` compared the whole run row after a
`get_task_history` call. It now allows that read's own run-log append
(`nextEventSeq` +1, `updatedAt`), and every other column must be
unchanged. That CI failure (General tests server 2/12) came from this
PR. "e2e shard (5/8)" failed because the self-hosted runner lost
communication; that is not related to this PR.
- Runner and Muse suites (`dot-runner`, `paperclip-control-plane-port`,
`runner-prp-coordinator`, `historical-task-questions`,
`paperclip-runner-tool-authority`, `muse-native-follow-up`,
`muse-control`): 509/509 pass. Server `tsc` is clean.
- Neighbouring suites: `heartbeat-held-comment-wake`,
`src/modules/wake-queue/**`, `execution-continuation`,
`paperclip-runner-tool-authority`, `first-turn-queued-input`,
`heartbeat-comment-wake-batching`, `ownership-reconciliation-report`,
`heartbeat-zombie-guard`: 555/556 passed in one loaded run. The 2
timeouts (one batching test, one reconciler `afterAll` hook) pass when
run alone (1/1, 44/44).
- `server` `tsc --noEmit`: clean.
- The restart-chaos harness already grades this class
(`done_comment_once` in staging-mix). No new cell is necessary.

## Risks

- Product choice: a comment counts as delivered when a succeeded run
read it through `get_task_history`, even if the model ignored it. This
is the same trust that applies to a comment delivered as wake input. The
alternative fails worse: it reopens a done task and repeats a completed
turn. An edit after the read, or a reader run that fails, leaves it
owed.
- Legacy (non-native) adapters that read comments through the REST API
do not write the record. Their behavior does not change.
- A comment follow-up behind a truly dead running run now waits as a
deferred wake until recovery releases that run. Before, its queued run
also could not start until the lock was released.
- Each `get_task_history` call appends one small run-log row. This is
not in the path before `turn.start`.
- Upgrade-safe: continuations queued by older builds still go through
the dispatch stale check, and that path now reads the same delivery
record.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:31:11 -06:00
DottaandClaude Opus 5.5 f92f91f0fb fix(native): finish a parent claim when its child results arrived before the run started (#21189)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task gets a wake when one of its child tasks completes. The
wake carries the child results.
> - When the parent already has a run, the child result wake waits
(`deferred_issue_execution`) behind that run.
> - At finalization, a parent completion claim stays open
(`native_child_completion_pending`) while such a wake is still queued,
claimed or deferred. This guard protects child results that arrive while
the parent works.
> - The guard did not check when the child result arrived. A child
result saved before the parent run started was already visible to that
run, but it still blocked the claim.
> - So the task stayed `in_progress`, the deferred wake started a second
run, and the agent repeated its completion summary.
> - This pull request counts a child result as undelivered only when it
was saved after the finishing run started.
> - The benefit is that a parent that already read its child results
finishes in one turn, and child results that arrive during the run still
get a fresh turn.

## Linked Issues or Issue Description

Refs #15372 (added the pending child-completion guard). Refs #20935
(same family: a self-wake after a settled run).

**What happened?**

Restart-chaos wave `w-20261112T181948-s7101`, task t001 (OpenCode,
native runner, staging-mix parent with two subtasks). Both subtasks were
done before parent run `8006a205` started at 18:45:29. The run read
them, posted its final comment and claimed `done` at 18:46:15.860. A
child-completion wake for the parent had been deferred since about 18:42
behind that run. The finalizer found that wake and committed
`native_child_completion_pending`, so the task stayed `in_progress`. 63
ms later the release promoted the deferred wake into run `9282de34`
(`invocationSource: automation`, `wakeCommentId: null`). Its
`paperclip.native-continuation.v1` turn carried the subtask results that
the first run had already used, plus the agent's own two comments
(resume delta since the base run). That run posted a second "complete"
summary and only then moved the task to `done`.

Root cause: `hasPendingNativeChildCompletion()`
(`server/src/services/native-runtime/native-child-completion-delivery.ts`)
treated every queued, claimed or deferred child-completion wake for the
issue as unread, except a wake bound to the finishing run itself. It did
not compare the wake's save time with the run's start.

**Expected behavior**

A child result that was saved before the finishing run started is not
new input for that run. The completion claim completes the task. The
deferred wake is cancelled at release as a wake on a terminal task. A
child result saved during the run still keeps the task open and gets a
fresh parent turn.

**Steps to reproduce**

1. Create a parent task with a done child. Save a deferred
`issue_children_completed` wake for the parent.
2. Start a native parent run after that wake was saved. Let it claim
`done`.
3. Finalize the run. On `master` the decision is
`native_child_completion_pending` and the task stays `in_progress`; the
deferred wake then starts a second run.

## What Changed

- `native-child-completion-delivery.ts`:
`hasPendingNativeChildCompletion()` counts a child-completion wake only
when its `createdAt` is at or after the finishing run's `startedAt`. A
run without a recorded start keeps the old behavior: every such wake
stays pending. The finalizer, the status-commit recheck and status
reassessment all use this function.
- `heartbeat-process-recovery.test.ts`: a regression test finalizes a
real native done claim with a deferred child-completion wake saved
before (task becomes `done`) or after (task stays `in_progress` as
`native_child_completion_pending`) the run started. It then releases the
issue execution through the wake queue. In the `before` case the
deferred wake is cancelled and there is one run total. In the `after`
case the wake is promoted into a fresh run that carries the child
results.
- `doc/architecture/native-status-arbitration.md`: describes when a
child result counts as undelivered.

## Verification

- RED on `master` (`ef6d347f79`): `heartbeat-process-recovery.test.ts`
"treats a child result saved before the parent run started…" fails with
decision `reasonCode: "native_child_completion_pending"`, `toStatus:
"in_progress"`. This is the same decision as the wave. The "after" case
passes on `master`.
- GREEN: both cases pass with the fix.
- Neighbouring suites pass: `native-status-arbiter-corpus` (includes the
queued / claimed / deferred / consumed child-delivery cases from
#15372), `heartbeat-dependency-scheduling` (revised child completion
delivered after a running parent), `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, and `src/services/native-runtime`.
- `pnpm --filter @paperclipai/server typecheck` passes.
- The regression test runs finalization and release end to end. The
cancel-at-release step itself is existing behavior, also covered by
`modules/wake-queue/application/use-cases.test.ts` ("cancels stale
assignee continuations before claiming promotion on %s tasks", `done`
case).

## Risks

- A child result saved before the run started no longer blocks the
claim. The child status was already `done` when the run started, so the
run could read it. If the clocks of two server instances differ, a child
result saved during the run could look older than the run start; the
window is the clock skew.
- A re-dispatched native status wake gets a new row with a later
`createdAt`. Such a row stays pending, which is the old behavior.
- No migration. Saved wakes from older builds work unchanged, because
the check reads only existing columns.
- Scope: this change does not alter the comment-wake promotion path or
the resume-delta builder.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:19:32 -06:00
DottaandClaude Opus 5.5 82c6810107 fix(runtime): post the agent's blocker reply when a retried finalization blocks the task (#21169)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime finalizes an agent's run, applies the agent's
status decision to the task, and posts the agent's reply as a task
comment
> - When the first workspace copy-back fails, or the run reconnects, a
background retry commits the result instead of the live heartbeat
> - That retry path applies the "blocked" status but never posts the
agent's reply, so the task turns blocked with no visible reason
> - The retry path already shares one recovery step for replies, but
that step accepted only "done", "needs review" and conversation replies
> - This pull request lets the shared recovery step present a blocked
task's reply, so the retry path and the normal path converge on one
presentation step
> - The benefit is that a blocked task always tells the user why,
exactly once

## Linked Issues or Issue Description

**What happened?**

An agent reports that it is blocked. The task moves to "blocked". No
comment from the agent explains why. This happens when the run finishes
through a retried workspace copy-back or a reconnect loop instead of the
live heartbeat. The run succeeds and its result has the agent's blocker
summary, but the run has no presentation decision and the task has no
reply. On a staging company, 58 of the 150 most recent agent-blocked
tasks for one agent had no agent comment.

**Expected behavior**

Every finalization path that applies an agent's status decision also
presents the agent's reply. A blocked task always shows the agent's
explanation once. The status decision does not post a second copy.

**Steps to reproduce**

1. Run a native agent on a task. The agent reports `blocked` with a
summary and a task-wide blocker.
2. Make the first workspace copy-back fail
(`native_workspace_sync_out_failed`), so the heartbeat leaves
finalization to the retry.
3. Let the retry finish the copy-back and commit the result.
4. The task is blocked and the run succeeded, but the task has no agent
comment.

**Paperclip version or commit**

`master` at b8207c4ea9

**Deployment mode**

Hosted (staging cloud). The defect is in server finalization, so it
applies to all deployment modes that use the native runtime.

**Agent adapter(s) involved**

Native runtime (`paperclip_runner`), seen with the Codex-based agent.
The defect is adapter-independent.

## What Changed

- The response presentation recovery step
(`recoverCommittedNativeChatResponse`) now presents a non-conversation
task's reply when the agent reported `blocked` and the task is still
blocked.
- For a blocked reply, recovery uses the agent's final message when one
exists. Otherwise it uses the accepted result summary, the same as the
live completion path.
- A blocked reply on a channel task (an external chat run) uses the same
rule as live run completion. Recovery resolves the chat publication
authorization, prefers the final response, and posts with that
authorization. The worker does not refuse these runs, so it does not
record a false publication failure.
- The presentation work query now finds these runs, so the background
worker also presents them and the bounded publication retry applies to
them.
- A blocker reply is not presented when the task is no longer blocked or
when a later status decision was applied. This keeps old blockers from
posting stale comments.
- Presentation stays idempotent: the step records
`presentationDecision`, reuses a reply that the run already posted, and
never posts twice.
- Existing affected tasks need no separate backfill. The presentation
worker loads all pending candidates at startup, so each task that is
still blocked gets its saved reply once after deploy.

## Verification

- Red/green test:
`server/src/services/native-runtime/native-blocked-reply-recovery.test.ts`.
It completes a blocked native run, fails the first copy-back with
`native_workspace_sync_out_failed`, then commits the retry with
`finalizeNativeRun`.
- Red on `master`: the retry-path tests fail (no comment is posted, and
the worker finds no candidate). The channel task test also fails on the
first PR revision (5eb8cdab95).
- Green with this change: 4/4 pass. The tests check that the blocker
summary is posted once, that the final reply is preferred, that a
repeated retry and a repeated worker check do not double-post, and that
a stale blocker is not posted.
- `cd server && npx vitest run
src/services/native-runtime/native-blocked-reply-recovery.test.ts
src/services/native-runtime/native-response-publication-retry.test.ts
src/services/native-runtime/agent-chat-response-recovery.test.ts
src/__tests__/idle-native-finalization.test.ts
src/services/presentation-ownership-read.test.ts
src/services/chat-monitor-continuation.test.ts` passes (172 tests).
- `cd server && npx tsc --noEmit -p .` passes.

## Risks

- After deploy, tasks that are still blocked from earlier affected runs
get the agent's saved reply once. This is the intended backfill. Tasks
that moved on are skipped.
- The candidate query has one more branch with a `NOT EXISTS` on
`status_decisions`, keyed by the existing `(company_id, issue_id,
decision_version)` unique index. The cost is low.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, browser reads of staging API).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 14:30:32 -06:00
Nicky LeachandPaperclip 7f69ef8b44 Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path

> - Paperclip lets plugins own managed Git workspaces.
> - A portable Runner task can move to a different physical allocation.
> - The checkout must be present on the allocation that launches the
Runner.
> - Workspace preparation and finalization need exact placement when
they retry.
> - This change checks the placement at Runner admission and records it
before finalization.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Workspace preparation and finalization for portable Runner tasks.

**Current behavior**
A task replacement can change the allocation after Git preparation. A
lost finalization reply can leave retry without the original placement.

**Proposed behavior**
Prepare the checkout on the actual allocation before admission. Record
finalization placement so an exact retry can use it.

Portable ownership and controller recovery are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066).

## What Changed

- Reserve the managed workspace before acquiring the portable task
lease.
- Pass the task placement to the workspace provider. Reuse the described
task while placement stays the same and prepare again before admitting a
replacement.
- Record the initial preparation's run, agent, generation, and optional
task placement before dispatch. Inspection and retries on the same
Launcher replay the same request; VM and sandbox preparation use the
same recovery path.
- Keep provider request identity separate from Git credential authority:
reservation reconciliation uses the existing project-consent path in a
host-only scope bound to the plugin and workspace. Foreground
preparation still requires an active run; consent, repository, and
revocation checks are unchanged.
- Acquire the current Launcher before retrying preparation. If placement
changes, use ordinary preparation on the replacement so the provider can
settle the prior writer and recover the retained checkout. No additional
provider discovery call is added.
- Select finalization placement before publishing the operation record,
then dispatch and recover using that saved choice. Retain the Launcher
during bounded finalization retries.
- Preserve interrupted pre-upgrade finalization requests without
placement, including their original operation IDs and provider routing.
No migration or new durable mode flag is required.

## Verification

- Final revision `74ea3e6712`: server typecheck and exact SDK source-pin
verification (538 inputs) passed.
- All 185 tests passed across the full plugin worker manager, project
Git authorization, and plugin workspace integration suites (including 56
workspace tests). Independent verification reran the three lost-reply
cases: inspection, same-Launcher retry, and replacement after deleting
the original Launcher. The tests assert the correct request identity,
durable intent, and clearing that intent after success.
- Before the final retry-accounting fix, all 323 tests passed across
workspace admission, runtime execution, native workspace finalization,
plugin workspace integration, and terminal cleanup retry. Regressions
exercise crashes after operation insertion but before dispatch in both
finalization paths, exact replay after lease release, and retry after
ambiguous placement clears without publishing an incomplete operation.
Pre-upgrade interrupted requests retain their operation IDs despite an
available portable lease.
- Final retry-accounting fix: all 84 native workspace finalizer,
failure, and supersession tests passed. A real missing-plugin regression
first failed on the prior commit, then verified healthy-run progress,
three-attempt exhaustion, terminal cleanup, and absence of a misleading
workspace operation. Independent verification also passed the legacy and
crash-before-dispatch cases.
- Worker-process tests cover both preparation RPC methods: the provider
still receives the original run and agent, maintenance receives
project-scoped credential authority, and foreground or mismatched-scope
requests remain denied. Database tests confirm a stopped run cannot
authorize foreground Git access while valid project consent can
authorize maintenance.
- Existing metadata protection coverage proves callers cannot overwrite
the private preparation intent or read it through workspace responses.
- Two existing Launcher Git tests passed: retry preserves dirty files
and the original base commit without fetching, and retry resumes the
saved checkout intent while rejecting a changed pending branch.
- Implementation was delegated to GPT-6.1 Sol and independently
reviewed. The GitHub stack's automatic rebase preserved the original
workspace-placement patch.

- [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706)
passed on `74ea3e6712`, including server and workspace tests, Runner,
end-to-end tests, typecheck, build, and canary dry run. Ordinary
Greptile and
[Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388)
both scored `74ea3e6712` 5/5, with no unresolved findings.

## Risks

- Live Daytona end-to-end qualification is pending. The checkout record
persists on the mounted volume; replacement preparation relies on the
provider confirming the previous writer is settled before using that
retained checkout. Local tests cover Core request routing and Launcher's
durable checkout recovery separately.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 12:12:41 -08:00
DottaandClaude Opus 5.5 4363a3115a fix(recovery): continue any unfinished run in a fresh run from saved work (#21135)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs are interrupted when a server restarts, a deploy swaps
servers, a controller lease is lost, or a sandbox Runner cannot be
reached again
> - Exact recovery (resume the same session, replace, retry) refuses in
many ways, and each refusal leaves the task blocked or idle for a person
> - On staging, 577 of 1,000 runs in 11 hours failed or were
interrupted, across 25 error codes. About 900 earlier PRs each fixed one
refusal code, and the rate did not converge
> - This pull request adds one universal fallback: when an unfinished
run cannot be resumed exactly, for any reason that is not the person's,
the task continues in a fresh run from its last saved workspace and its
conversation
> - The benefit is that a new failure code next week falls back without
code changes, and tasks stop waiting on a person for infrastructure
faults

## Linked Issues or Issue Description

**What happened?**
After a server restart or deploy swap, many runs end with codes such as
`native_continuation_requires_reconciliation`
(`runner_remote_process_identity_unavailable`), `controller_lease_lost`,
`process_lost`, `server_shutdown_interrupted` and
`native_session_interrupted` (including `runner_harness_state_mismatch:
resumed_not_suspended` / `resumed_unavailable`). The disposition worker
then settles the recovery hold as "blocked, no replay". The task stays
Blocked until a person presses Retry or reconciles the run.

**Expected behavior**
A task waits on a person only for reasons that are the person's: an
approval, a question, a credential to share, a budget stop, a provider
usage limit, or a Stop they asked for. For every other failure, the
system starts a new run for the same task from the last verified saved
workspace, without a person.

**Steps to reproduce**
1. Start a task on a sandbox agent.
2. Restart or swap the server while the run is in progress, so the new
server cannot re-reach the Runner.
3. See that the run fails with
`native_continuation_requires_reconciliation` and the task stays Blocked
with no successor run.

**Paperclip version or commit**
`master` at 9e910ff9ae.

## What Changed

- New module `server/src/services/fresh-continuation-fallback.ts`. One
entry point, `ensureFreshContinuation` / `continueFreshInTransaction`,
takes a run that ended without finishing and guarantees one of:
something already follows it (a newer run, a linked replacement, an
operator decision), exact recovery is still under way (checked again
later), the cause is the person's, or one fresh continuation is
recorded.
- Classification, not an allowlist: `PERSON_OWNED_FAILURE_CODES` plus
task-level waits (pending approval, pending question, paused or
unavailable agent, paused company, review in progress, a Stop that a
person asked for, a Blocked status that another decision owns). Every
other code, including unknown future codes, falls back.
- Idempotent per source run. The task row lock serializes callers on
every server; the recorded `freshContinuation.sourceRunId` makes later
calls a no-op.
- Isolation, never waiting on proof. A local process still running on
this host is settled by its own owner first. If the old local writer is
proven gone, the task workspace is kept. Otherwise the continuation gets
a new isolated workspace (`fresh_isolated_workspace`), so two runs never
write one directory. The decision records the stop proof honestly
(`providerStopProof: isolated`).
- Claim admission no longer waits on the cleanup of a predecessor that a
fresh continuation replaced (`heartbeat/queue.ts`, predecessor loop).
Without this, the successor stayed queued behind the unreachable lease,
which is the staging failure itself.
- Only reconciliation holds that this decision owns are settled.
Workspace restore holds that workspace recovery still owns, and resolved
history, are untouched.
- `continuation_user_authorization_missing` is the system's failure (a
Retry's authorization was not carried forward), not the person's.
- Safety: the old run keeps its fencing. When it still owns its remote
workspace epoch, the epoch is abandoned
(`abandonUnverifiedRemoteWorkspaceEpoch`), so the old writer can never
publish and the next run restores the last committed save into a new
root on a new allocation. The unverified files stay in the abandoned
root, unapplied. The native coordinator records `replacementDenied:
fresh_continuation`, so no exact replay also starts.
- The replaced run loses its write authority. The fallback records
`freshContinuationSupersededAt` on the old run. The agent-JWT middleware
(`loadAgentRunAuthentication` reads one more field in its existing row
read) and `assertAgentRunWriteAllowed` refuse every write of the old
token with `agent_run_superseded`. Before this change, a failed or
interrupted run's token could still write. So an agent that is still
alive in an unreachable sandbox cannot act on the task next to its
continuation.
- The opening context (`recoveryOutcomes` in the execution continuation)
says that the previous attempt was cut off and lists the actions with
unverified outcomes, so the agent checks before it repeats a side
effect. No copy asks a person to inspect or repair a sandbox, workspace
or run. The hold text written at run end no longer says "Inspect the
original failure and reconcile".
- Delivery reuses the existing path: the settled hold gets an
`executionReconciliation` decision and `continuationDelivery: pending`,
and `deliverReconciledExecutions` starts the run (fresh session,
`retryOfRunId` = source).
- Budget per task, not per restart: at most 6 fresh continuations per
task per rolling hour (the server-restart lane's cap). At the cap the
hold stays with the system and is checked again when the window has
room. The failure allowance is never charged.
- Call sites:
- `settleUnrecoverableExecutions` (execution-recovery-resolution.ts):
every hold whose exact recovery is ruled out now goes through the
fallback before it would block the task. Interrupted preparations keep
their own replay lane first.
- `continueHeldExecutions` (same worker): picks up holds saved by older
releases (settled-but-blocked no-replay holds, finished or exhausted
saved-work recoveries) for the task's latest unfinished run.
- Wake admission (heartbeat/queue.ts): an ordinary system continuation
wake (sweep, monitor, timer) that a hold would defer now calls the
fallback in the same transaction and becomes the fresh continuation
itself. This is the path the idle-task sweep uses. A person's Retry,
message or queued send keeps its exact admission; the disposition worker
settles the hold, and the saved input then continues.
- `executionControlSchedule.disposition` wakes the worker for held tasks
and for the next due recheck.

### Human gate versus system uncertainty

One documented rule (`isHumanGate` plus `personOwnedTaskWait`) decides
what the fallback never touches. The rule goes by cause, never by owner
type:
- Human gates: a pending approval, a pending question, a user-owned
hold, an owner-gone attestation, a recorded explicit continuation, an
operator reconciliation, a person's Stop.
- System uncertainty: a no-replay record ("outcome unknown after an
interruption"), a hold that is `board`-owned only by default, an
unverifiable controller process space or host.

For uncertainty the fallback starts a fresh run, never a replay. The
interrupted turn is not re-sent. No tool call or write is re-executed.
Governed actions go through their approval gate again. The earlier
`automaticRecovery` decision is kept on the hold as
`automaticRecoveryHistory`, beside the attributed
`freshContinuationSettlement` and an activity entry.

### Restart recovery refused for an unverifiable controller

The case is `controller_process_space_unverified`, its remote variant,
or `controller_host_unrecorded` / `controller_host_unprovable`. Its
lease has lapsed; a live, renewing lease is still waited for (server
overlap). After `NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS` (2 minutes) the
run ends (failed, `native_restart_recovery_blocked`). That frees the
task's execution lock and the agent's slot, and a system-owned hold lets
the fallback continue the task. Recording the stops is optional. The "a
board operator must record both stops" text is replaced.

Board/operator attention sites in native restart recovery:
- `surfaceNativeControllerNamespaceAttention`: copy replaced; the run
now ends after the bounded wait.
- `requestNativeControllerStopConfirmation`
(native-controller-host-recovery.ts): copy says confirmation is
optional; the same bounded end applies.
- The blocked restart-recovery hold (`nextAction` "Inspect the preserved
checkpoint and reconcile…"): copy replaced; the fallback settles the
hold by cause.
- `owner-gone-attestation.ts` (generic resource records from older
releases): unchanged. It is not a run-recovery path, and it is listed as
a follow-up.

### Every wait and its bound

| Wait | Owner that rechecks | Worst-case total before the fresh
continuation |
| --- | --- | --- |
| `exact_recovery_pending` (execution settling, native finalizer,
controller lease, interrupted-preparation lane, saved-work recovery,
restore hold, retained-source fallback, live local process) |
Disposition worker, woken at each deadline; rechecks at 1, 2, 3 and 4
minutes, then every 4 | About 10 minutes from the first pending check.
Up to 15 while the owner shows progress (a new event, output, lease
renewal or coordinator update since the last check). |
| Saved-work (workspace) recovery that is `scheduled`,
`awaiting_evidence`, or `running` without progress | Held-run scan, now
selected and scheduled like the other holds | About 10 minutes; 15 at
most |
| Saved-work recovery that is `running` and advancing (revision,
attempt, state or hold update since the last check) | Held-run scan |
Its own 30-minute budget |
| Unverifiable lapsed controller in restart recovery | Restart recovery
(`retryAt`) | 2 minutes, then the run ends and the fallback continues it
within one disposition pass |
| Hourly cap (`backoff`) | Disposition worker, woken at the deadline |
Until the rolling hour frees a slot, at most 1 hour. This is the
deliberate restart-lane bound, not a stuck state. |
| Person-owned or dependency wait | The person's own action wakes the
task; rechecked hourly | Not bounded: it is the person's |

Restart-recovery classifications (`claimNativeRestartRecoveries`, which
`recoverNativeRunsAfterRestart` calls):

| Classification | Who rechecks, when | Bound |
| --- | --- | --- |
| Claims: `reattach_existing_runner`, `reattach_remote_runner`,
`resume_dead_runner`, `bootstrap_incomplete` | The claiming server
resumes exactly | Exact resume; no fallback |
| `already_finalized` | Nobody; it is terminal | None needed |
| `blocked` | The run is terminal and its hold goes to the fallback
(disposition worker) | Fallback bounds above |
| `awaiting_evidence` / `current_controller_preparing` | This process's
own executor | Ends with that executor |
| `awaiting_evidence` / `recovery_rows_busy` | Restart dispatch after 1
second | Self-resolving: the row lock clears |
| `awaiting_evidence` / a live lease (`controller_lease_held`,
`controller_lease_grace`, `controller_still_alive`) | Restart dispatch
at the lease-renew interval | Never ended while the old controller
renews its lease |
| `awaiting_evidence` / unverifiable lapsed controller (process space or
host) | Restart dispatch every 2 minutes | Ends 2 minutes after the
lease lapsed |
| `awaiting_evidence` / any other reason (startup cleanup, finalizer or
cleanup owner, recovery failure, routine busy, future reasons) | Restart
dispatch at the owner's `retryAt`, or within 2 minutes when it has none
| Ends 2 minutes after the lease lapsed, or after 10 minutes, whichever
comes first. It never ends while a lease is live. |

Ending a run sets it failed (`native_restart_recovery_blocked`),
releases the task lock and the agent's slot, and records a system-owned
hold. The fallback then continues the task once, isolated.

No wait needs longer. Saved-work recovery and retained-source checks
keep their own attempt budgets. After 10 to 15 minutes the fallback
continues anyway in a new root, without the unverified files, which stay
recorded and unapplied.

### When the fallback preempts saved-work recovery

The unverified files are not destroyed. They stay where recovery found
them: in the abandoned remote epoch's root, or in the old local
workspace that the task is no longer bound to. They are recorded and
never applied. Settling the hold records the decision on it, so the
retained-workspace worker stops scheduling it (its query requires no
`executionReconciliation` unless recovery already succeeded). If an
attempt already in flight finishes later, it cannot publish into the
fresh run's workspace. For a remote workspace, the epoch it targets is
abandoned and the writer is cleared. For a local one, the task now
points at a new workspace. Whenever a wait is preempted, the
continuation is isolated: a fenced remote epoch or a new local
workspace.

### What people see

The task's interruption presentation shows "working" while the primary
fresh-continuation record is pending and has not started (coalesced
holds never count, and they follow the primary's final state), and a
timed wait with no Retry while a check is scheduled. The Retry banner
appears only when the system has given up or the cause is the person's.
Holds settled together with the primary record are `coalesced`; they no
longer show "closed or reassigned". A pending continuation is never a
second run: if a person's Retry or saved input started a run after it
was recorded, delivery links to that run instead. A task that finished
before delivery gets no run.

### Readers of what the fallback writes

- Recheck deadline (`freshContinuationCheck`):
- `unresolvedExecutionDispositions` and
`heldExecutionsAwaitingFreshContinuation` read it to skip work that is
not due.
- `nextFreshContinuationCheckAt` and
`executionControlSchedule.disposition` read it to schedule the worker.
- Tests: the cap deadline is scheduled, and a cancelled task goes quiet.
- Isolation decision:
- The task's workspace binding is cleared. Every path that starts the
next run (system continuation, saved input, Retry, sweep wake) resolves
the workspace from that binding at run start (heartbeat.ts
`issueContext.executionWorkspaceId`).
- `freshContinuationIsolation` in the run context is read by
`readFreshWorkspaceReadiness` for the shared-directory case.
- `freshContinuationSupersededAt` on the old run is read by the auth
middleware, `assertAgentRunWriteAllowed` and the claim gate.

## Verification

- `pnpm vitest run src/services/fresh-continuation-fallback.test.ts`
(server): every test passes (it covers the cases below).
- The continuation starts (leaves the queue) while the unreachable old
lease is `active` or `pending_cleanup`. This was red before the
claim-admission fix.
- Isolation: a writer on another host gets a new isolated workspace; a
process alive on this host waits for its owner; a proven-gone writer
keeps the workspace.
  - Owned holds only: a restore hold and resolved history are untouched.
  - The recheck schedule goes quiet when the task is cancelled.
- `continuation_user_authorization_missing` continues under the task's
responsible user.
- One test per top-five staging code (native and legacy hold shapes):
the task gets exactly one fresh successor run from saved state.
- `runner_harness_state_mismatch` (`resumed_not_suspended`,
`resumed_unavailable`) falls back.
  - An unknown code falls back.
- Two concurrent callers plus the disposition worker create one
successor.
- Old-saved/new-resumed: a settled-but-blocked hold from the previous
release, with its coordinator in `awaiting_runner_reattach`, continues
on the first pass.
- An ordinary system continuation wake is admitted as the fresh
continuation.
- Fencing: after the continuation is recorded, the old run's token is
refused for writes (`agent_run_superseded`) and the continuation's token
is not. The old run's remote workspace epoch is abandoned: the writer is
cleared, the location and provider root are cleared, and the epoch
advances. So the next preparation restores the last save into a new root
(`prepareRemoteWorkspace`: `sameAllocation` is false when the location
is null, server/src/services/remote-workspaces.ts:437-470).
- The opening context names the cut-off run and an unverified tool
action.
- Person-owned causes (pending approval, credential not shared, budget
stop) are not continued.
  - The hourly cap backs off without blocking, then continues.
- Red/green: with the wiring reverted to `master`, 9 of these tests fail
(the task stays Blocked with no successor).
- Affected suites pass locally: `execution-recovery-dead-controller`,
`completed-turn-cleanup-hold-settle`, `execution-control-work`,
`heartbeat-process-recovery`, `issue-recovery-actions`,
`native-safe-replacement`, `native-failed-workspace-recovery`,
`task-retry`, `runner-lost-mid-turn-continuation`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
wake-queue `postgres`, `agent-conversations`, `native-startup-retry`,
`governed-wait-runner-loss`, `native-preclaim-continuation`.
- `pnpm verify --explain` selects the full suite (the wake queue is
shared), so CI supplies full coverage.

Existing expectations changed because they asserted the "wait for a
person" behavior this PR replaces:
- `issue-monitor-scheduler.test.ts`: "explains held monitor dispatch
without replaying uncertain execution" is now "continues a held task in
a fresh run when its monitor is due". A due monitor on a task held by a
`process_lost` hold now starts one fresh run instead of deferring.
- `agent-run-authentication.test.ts`: the auth row also reads
`freshContinuationSupersededAt` (still under 512 bytes), and a
superseded run is refused.
- `assigned-work-invariant.test.ts` (#21139): "with the real wake queue,
a hold that defers the continuation leaves a violation…" is now "…a hold
reached by the sweep's wake becomes one pending fresh continuation". The
sweep's wake now yields one pending fresh continuation, and the next
sweep adds no attempt.
- `heartbeat-native-restart-suspension.test.ts` and
`native-runner-restart-recovery.integration.test.ts`: awaiting-evidence
dispositions now carry a `retryAt`, because every wait is rechecked. A
lapsed unverifiable controller no longer waits forever.
- `native-remote-retry-admission.test.ts`: the blocked-hold text is now
"This run stopped before it finished. Your saved work is kept." The old
text asked people to reconcile.

Holdless legacy runs: in the 1,000 staging runs, 0 tasks end on
`controller_lease_lost` or `process_lost` (every one has a later run).
12 end on `server_shutdown_interrupted` with no retry scheduled (attempt
0). On staging now, 9 of those 12 sit `in_progress` with no run and no
hold, 1 is done and 2 have a live run. Those 9 belong to the idle-task
sweep (#21139).

## Risks

- Speed: nothing is added to the normal path before `turn.start`. The
fallback runs only on the disposition worker after exact recovery is
ruled out, or at wake admission when a hold already defers the wake. The
auth middleware reads one more JSON field in the row read it already
does.

- Behavior shift: tasks that today stay Blocked for a person after an
infrastructure failure now continue automatically. A side effect with an
unknown outcome is not replayed; the new run is told about it and must
check first.
- After deploy, the disposition worker continues tasks already blocked
by these codes, at most 25 per pass and at most 6 per task per hour. No
migration. No manual Retry is needed.
- The old executor is not proven stopped before the successor starts.
Fencing is: its API writes are refused, its workspace epoch is
abandoned, the native `replacementDenied` marker is set, and the
successor has a separate run identity.

Known gaps (follow-ups):
- No quarantine attachment: unverified files stay unapplied in the
abandoned workspace root (recorded on the workspace in
`remoteWorkspaceAbandonedEpochs`). They are not read into a task
attachment, because the sandbox is usually unreachable in these
failures.
- Agent Chat conversations are excluded at wake admission. They keep
their owner-bound continuation.
- Under a `reuse_by_environment` lease policy the new allocation can be
on the same provider sandbox. It always uses a new root and a new epoch,
and the old run cannot publish or write.
- Tasks left `in_progress` with no run and no hold (for example after
`server_shutdown_interrupted` with no retry scheduled) belong to the
idle-task sweep. That sweep's wake reaches this fallback when a hold
exists.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:34:34 -06:00
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00
DottaandClaude Opus 5.5 b8207c4ea9 fix(runner): evict finished API snapshots instead of blocking reads at the company quota (#21133)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs read the Paperclip API through `call_api`. Large
responses are saved as "API snapshot" assets so the agent can read them
in pages.
> - Each company has a 20 GiB snapshot quota. Before a large read, the
server reserves 1 GiB of worst-case space against that quota.
> - Snapshot assets were never removed. Reservations from interrupted
captures (crash, restart, ambiguous upload) stayed at 1 GiB each until
"an operator reconciles" them. Nothing did that.
> - After a morning of high retry volume, the quota filled. Every large
read then failed with `api_response_company_storage_limit`, and agents
blocked their own tasks and asked an operator to free storage.
> - This pull request makes company admission reclaim space by itself:
it releases stale reservations of finished runs and evicts the oldest
snapshots of finished runs, oldest first, only as much as the new read
needs.
> - The benefit is that a read never fails because old snapshots filled
the quota, and no person must clear storage by hand.

## Linked Issues or Issue Description

**What happened?**

On staging, about 97 tasks of one agent had successful runs but ended in
`blocked`. In each run, `GET` calls through `call_api` for large records
(for example heartbeat runs and their events) failed with
`api_response_company_storage_limit`: "Company API snapshot storage
quota is exhausted. Read an existing asset or ask the operator to remove
old snapshots or increase the quota." The agent then reported a
task-wide blocker owned by "the instance storage operator". At least 15
tasks had a final comment such as "Blocked: company API snapshot storage
is exhausted". The error message itself told the agent that an operator
must act.

**Expected behavior**

A read must not fail because old snapshots filled the company quota. The
server must reclaim the space automatically. Only snapshots that active
runs still hold, or that users attached to issues, cases or logos, may
stay.

**Steps to reproduce**

1. Set `PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES` to 2 GiB.
2. Store a 1 GiB `runner-api` snapshot asset from a finished run, or
leave two 1 GiB unattached reservations from finished runs that are
older than 20 minutes.
3. In a running native run, call `call_api` for a response larger than
the inline limit.
4. Before this change, the call fails with
`api_response_company_storage_limit`. After this change, the call
succeeds and the oldest finished snapshot is evicted.

**Paperclip version or commit**

Staging deploy `704b3f4d63`. Fix is based on `origin/master` at
`8bdd53c11f`.

**Deployment mode**

Hosted (staging cloud), native runtime with Daytona sandboxes.

**Agent adapter(s) involved**

Native Paperclip runner (Codex app server, `gpt-6.1-sol`). The defect is
in the server-side `call_api` capture admission, so it affects every
native adapter.

## What Changed

- Added
`server/src/services/native-runtime/runner-api-capture-retention.ts`. It
measures company snapshot usage and reclaims space under the existing
per-company advisory lock.
- Reclaim step 1: release unattached reservations older than twice the
capture deadline (20 minutes) whose run is missing or not `queued`,
`running` or `scheduled_retry`.
- Reclaim step 2: evict snapshot assets oldest first until the new
reservation fits. A snapshot is eligible only when it is older than 1
hour, every run that captured it has finished (succeeded, interrupted,
failed, cancelled or timed out), and no issue attachment, case
attachment or company logo links to it. Each eviction writes an
`asset.evicted` activity-log row with the asset id and the owning run
id.
- A capture upload that finishes after its reservation was reclaimed is
rejected in the publish transaction. The same transaction records the
object for deletion, and the bounded cleanup removes it. The upload has
a 10-minute abort signal.
- Failed deletes are logged per company and counted with the
`runner_api_capture_delete_failed` metric. When cleanup keeps failing,
the read error says so in plain words.
- Eviction credits deletes that are already pending before it chooses
more snapshots. Eviction is durable. In the same transaction that
deletes an asset row, it inserts a reservation row with
`pending_delete_object_key`, so the bytes stay counted. Admission then
deletes the stored objects in pages, under a 30-second abort signal, and
removes each row only after its delete succeeds. A failed or interrupted
delete is retried at the next over-quota admission. No background poller
is added.
- Added the `pending_delete_object_key` column and an index on
`runner_api_response_reservations.asset_id` (migration 0472).
- `#reserveApiResponseCapture` now runs reclaim only when the quota
would be exceeded, so reads below the quota do no extra work. After
marking evictions it finishes the deletes and retries admission once.
- When no space can be reclaimed, the error message now names what holds
the space (runs still working, files attached to tasks, deletes in
progress). It no longer tells the agent to ask an operator.
- Replaced the test that asserted old snapshots block reads with tests
for eviction, multi-page eviction, durable delete retry, a bounded
stalled delete, stale-reservation release, and protection of active and
attached snapshots.

## Verification

- Red: the three new tests in `runner-api.integration.test.ts` fail on
`origin/master` (two get `api_response_company_storage_limit`, one gets
the old "ask the operator" message).
- Green: `cd server && npx vitest run
src/services/native-runtime/runner-api.integration.test.ts` passes 70/70
with the fix.
- The multi-page test failed first with
`api_response_company_storage_limit` because pending deletes ran in one
200-row batch. The paged delete loop fixed it.
- `cd server && npx tsc --noEmit -p .` passes.

## Risks

- An agent that reads a saved snapshot of a finished run after it was
evicted gets a not-found response. It can read the API again. Eviction
happens only under quota pressure, oldest first.
- A released stale reservation can leave an unknown storage object from
an ambiguous upload. These bytes were never linked to an asset, so they
were already unreachable. This is the same storage the old code kept
counted forever.
- Eviction runs inside the company capture lock. The candidate query is
bounded to pages of 200 rows and uses the new `asset_id` index.
- Migration 0472 adds one nullable column (`ADD COLUMN IF NOT EXISTS`)
and builds one index with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It
is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. It is additive and
safe for old and new servers during a rolling deploy.
- During a storage outage, an over-quota read still fails until deletes
succeed. The bytes stay counted, so the quota is never exceeded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser reads of staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:30:38 -06:00
DottaandClaude Opus 5.5 90ae3c4337 fix(native): process a superseded export's late stop receipt and stop its sandbox when none arrives (#21143)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run can be retired as superseded while its workspace export
fails. Its files stay unsaved, and the run stays the writer of the
task's workspace.
> - #21136 records those files for the automatic workspace recovery.
That record needs the sandbox's stop-and-retain receipt.
> - The native run projection worker checked for that receipt for one
hour after the run ended. After that, it dropped the run. Nothing woke
it when the receipt arrived later.
> - If the receipt came late or never, the run kept the task's workspace
fenced. Every queued run on the task waited and then failed setup.
Nobody owned the problem.
> - This pull request makes the worker process a late receipt, stop and
keep the sandbox itself when no receipt arrives, and never drop the run
silently.
> - The benefit is that the task's queue moves again by itself, and two
writers never use one workspace.

## Linked Issues or Issue Description

Refs #21136 (review thread on
`native-superseded-workspace-export.ts:84`)

**What happened?**
`preserveSupersededWorkspaceExport` returned `recheckAt: null` when the
sandbox stop receipt had not arrived within `finishedAt + 1h`. The
projection worker then deleted its entry. A later stop receipt
(`settleStopOnlyCleanup`) woke only the workspace recovery queue, not
the projection queue. `preserveNativeFailedWorkspace` requires
`hasConfirmedSandboxStopAndRetain`, so no hold was recorded. The run
stayed the dirty writer, and queued runs kept failing setup.

**Expected behavior**
A late receipt is processed at once. When no receipt arrives, Paperclip
stops and keeps the sandbox itself and waits for the provider's
confirmation. The workspace stays held while the sandbox may still run.
When the provider confirms that the sandbox is gone, the task continues
from its last save. When the stop stays unconfirmed for hours, the task
shows one plain notice, and the check continues.

**Steps to reproduce**
1. Retire a native run as superseded after its workspace export fails
(the #21136 shape).
2. Let the run's sandbox stop not settle (for example, the server that
owned the release is replaced).
3. Wait more than one hour after the run ended.
4. See that the projection worker drops the run, and that a later stop
receipt records nothing. The workspace stays held.

## What Changed

- `sandbox-stop-and-retain.ts`: a confirmed stop-only receipt now also
notifies `DELIVERY_QUEUES.nativeRunProjection` (scoped to the company
and run). New `prepareUnconfirmedSourceSandboxStop` puts a
stop-and-retain intent on the source lease from its current status. It
skips a destroyed sandbox, a stop already in progress, and a cleanup in
flight.
- `native-superseded-workspace-export.ts`:
- Within one hour of the run ending, the worker checks every minute
(unchanged).
- After that hour, it requests the stop-and-retain itself. The existing
pending-cleanup sweep dispatches the stop and retries it until the
provider confirms. The worker checks again every 15 minutes, and the
receipt wakes it at once. The workspace stays held.
  - A sandbox that another lease holds is never stopped.
- When the provider confirms that the sandbox was destroyed, the files
cannot be saved. In one transaction (task, run, then workspace locks),
the worker sets aside only that run's workspace epoch, wakes queued-run
dispatch, and records `issue.previous_run_files_lost` on the task.
- When the stop is still unconfirmed six hours after the run ended (or
the sandbox is held by another lease), the task gets one
`issue.previous_run_files_waiting` activity entry. The worker keeps
checking. It never drops the run.
- A stop-only dispatch now settles when the provider confirms that the
exact sandbox is already gone. Before, the runtime accepted only a
`stopped` reply and threw on `destroyed`, so a gone sandbox kept the
stop pending forever.
- `sandbox-stop-and-retain.ts`: new `acceptedStopOnlyReceipt` accepts
`stopped`, or `destroyed` for a stop-and-retain intent (a native
workspace export resume still needs `stopped`). `settleStopOnlyCleanup`
stores a `destroyed` receipt as `released` / `success` /
`workspace_source_unavailable`, removes the stop intent, never writes a
retain receipt, and wakes the projection and workspace recovery queues.
- `environment-runtime.ts`: the plugin `environmentStopLease` and
built-in `stopLease` checks use `acceptedStopOnlyReceipt`.
- `workspace-recovery.ts`: when its own stop settles as `destroyed`, the
source is unavailable (the existing alternative-strategy path) instead
of a binding failure.
- Daytona plugin: stop of a sandbox that Daytona reports as not found
returns `destroyed` (destroy already did this) instead of throwing. boxd
already returns `destroyed`. Other sandbox plugins have no stop-only
method, so nothing changes for them.
- New shared `SANDBOX_ALLOCATION_HOLDER_STATUSES` (`active`, `retained`,
`pending_cleanup`). The stop-only cleanup ownership check and the
worker's "held by another lease" guard both use it, so a retained warm
sandbox is never stopped and the two checks cannot drift.
- `ui/src/lib/activity-format.ts`: plain labels for the two task
activity entries. They do not ask a person to fix anything.
- `doc/execution-semantics.md`: document the rule.
- Not changed: the shared owner prover still refuses to treat a stopped
instance on another host as proof that its child processes stopped.

## Verification

- `server/src/services/superseded-workspace-export-recovery.test.ts` (4
new tests, red/green):
- "checks again every minute while the run's own release may still stop
its sandbox": the recheck branch. Nothing is recorded or stopped, and
the workspace keeps its writer.
- "stops and keeps the sandbox itself when no stop receipt arrived, and
records the files once the late receipt lands": after the hour, the
lease gets a stop-only intent. A second pass changes nothing. The late
receipt wakes the projection queue. The next check records the hold, and
the system recovery frees the workspace. Fails on master (`expected {
…(19) } to match object { status: 'pending_cleanup' }`).
- "names the unconfirmed stop on the task once, and keeps checking, when
the provider never confirms it": the expiry branch. One notice, the
check continues, no hold, and the workspace stays held. Fails on master.
- "continues from the last save when the provider confirms the sandbox
is gone": epoch 1 is set aside, the workspace is free, the task says so,
and the result is not applied. Fails on master.
- Review-round tests (all 5 fail on the first head `a208bfa58f`, pass
now):
- `environment-runtime.test.ts` "settles a plugin stop whose provider
reports the sandbox gone (boxd, Daytona not found) through the cleanup
sweep": provider dispatch through `releaseRunLeases` and
`sweepPendingCleanupLeases`. The lease settles `released` with a
`destroyed` receipt and no retain receipt.
- `environment-runtime.test.ts` "settles a built-in stop whose provider
reports the sandbox gone, without claiming retained files": the built-in
`stopLease` path. No destroy call.
- `superseded-workspace-export-recovery.test.ts` "continues from the
last save when Paperclip's own stop finds the sandbox already gone": the
worker's stop request gets a `destroyed` reply, the projection queue
wakes, and the workspace is freed at epoch 2.
- `superseded-workspace-export-recovery.test.ts` "never stops a sandbox
a retained warm lease still holds, says so once and keeps checking": the
source lease is unchanged, a recheck is returned, and `sandbox_in_use`
is recorded.
- Daytona `plugin.test.ts` "explicit stop confirms a sandbox Daytona no
longer has as destroyed".
- `environment-runtime.test.ts` (207) and Daytona `plugin.test.ts` (421)
pass.
- Related suites pass: `native-run-projection-work`,
`retained-native-session-cleanup-work`, `workspace-recovery-stop-proof`,
`native-failed-workspace-recovery`, `legacy-workspace-restore-recovery`,
`environment-retained-orphan-sandbox`,
`workspace-recovery-retry-copyback`, `workspace-recovery-fallback` (398
tests), and `ui/src/lib/activity-format.test.ts`.
- `tsc --noEmit` for `server`.

## Risks

- Low to medium risk. The new behavior acts only on an ended superseded
run that failed in its workspace export, still holds a dirty workspace,
and has no retained-file hold.
- The worker now stops an ended run's sandbox one hour after the run
ended if no stop receipt arrived. The stop uses the existing stop-only
contract and keeps the sandbox. A sandbox held by another lease is never
stopped.
- A destroyed sandbox's unsaved files are not recoverable. The task
continues from its last save and says so.
- If a provider never confirms a stop, the workspace stays held (by
design: no second writer). The task shows one notice, and Paperclip
keeps retrying.
- Shared cleanup code now treats a confirmed `destroyed` stop-only reply
as settled. Every caller was checked: the cleanup sweep and run release
count it as settled; the workspace recovery treats the source as
unavailable; the fallback treats it as stopped (nothing can run there).
No caller reads it as retained files, because no retain receipt is
written. A native workspace export resume still requires `stopped`.
- Each extra delivery notification per settled stop-only cleanup is
cheap. The projection worker's scoped load finds nothing for runs that
do not match.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:54:47 -06:00
DottaandClaude Opus 5.5 3fb35fcf13 fix(native): restart an interrupted bootstrap whose runner connected before the deploy (#21120)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. A deploy stops the server and a new server recovers
each run.
> - When the server stops during a run's bootstrap, recovery restarts
that bootstrap on the same run (`bootstrap_incomplete`). The claim is
only granted when the controller checkpoint in Postgres proves that no
provider turn was sent.
> - Host preparation then reads the sandbox Runner state. A Runner that
already authenticated is `ready`, not `suspended`. The restart is
allowed only with a stop proof for that Runner.
> - The stop proof was taken in two places only: from a restored local
controller root, or for a Runner that never connected. A new server
after a deploy can have neither, so the run failed with
`runner_harness_state_mismatch: resumed_not_suspended` and needed a
manual Retry.
> - This pull request lets host preparation take the same Postgres
checkpoint proof that granted the claim, stop that exact Runner, and
restart the bootstrap.
> - The benefit is that a deploy during a bootstrap no longer fails the
run. No provider turn can be replayed, because the proof shows that none
was sent.

## Linked Issues or Issue Description

**What happened?**
On staging, three Codex runs on Daytona failed with
`native_session_interrupted` and `runner_ingress_unavailable: runnerd
process failed: runner_harness_state_mismatch: resumed_not_suspended`
(12:48Z twice and 13:19Z on 2026-11-12). In each run the Runner
authenticated 1 to 4 seconds before a deploy stopped the server
("controller authority detached for restart; durable runner left
alive"). The new server chose "Restarting an incomplete native runner
bootstrap on the same heartbeat run". Its host preparation found the
sandbox Runner `ready`, had no stop proof, and failed the run. The run
log has no "Stopped the interrupted runner" line, so the earlier stop
step did not run on the new server.

**Expected behavior**
The run restarts its bootstrap on the same sandbox and continues without
a manual Retry. A provider turn that may have run is never replayed.

**Steps to reproduce**
1. Start a Codex run on a Daytona sandbox.
2. Deploy the server after the Runner authenticates and before the first
turn starts.
3. Let the new server recover the run with a `bootstrap_incomplete`
claim on a server that has no local controller root for the run.
4. Host preparation fails with `runner_harness_state_mismatch:
resumed_not_suspended`.

**Paperclip version or commit**
Staging at 1d153f62b6. The code is the same on master at 704b3f4d63.

**Deployment mode**
Hosted staging (paperclip.staging.paperclip.app), remote sandbox
execution on Daytona.

**Agent adapter(s) involved**
Paperclip Runner with Codex (`codex_app_server`).

## What Changed

- `native-runner-checkpoint.ts`:
`nativeRunnerCheckpointRemoteBootstrapEvidence` returns the bootstrap
proof together with the runner instance that the checkpoint names.
`nativeRunnerCheckpointRemoteBootstrapProof` now wraps it and behaves as
before.
- `native-session-executor.ts`:
- State migration runs before this server registers its own controller
for the run. Under a `bootstrap_incomplete` claim for a remote sandbox,
it now reads the interrupted controller's checkpoint proof there
(`captureClaimTimeRemoteBootstrapProof`). A later read is not used,
because the new controller rewrites the checkpoint with its own
bootstrap commands.
- Host preparation: when this run's sandbox Runner is not suspended, it
runs the same-run restart proof. The proof accepts a Runner that never
connected (as before), or the claim-time proof for the exact runner
instance in the sandbox state. It then stops that exact Runner by its
identity marker and records the stop proof. A backup, a different run
id, another runner instance, or a failed stop still fails closed.
- `native-session-executor.test.ts`: new
`interrupted_bootstrap_connected_checkpoint_proven` case. The Runner is
`ready` and connected, and no stop proof exists yet. It fails before the
fix with `resumed_not_suspended` and passes after it.
`interrupted_bootstrap_unproven` still fails closed.
- `native-bootstrap-claim-time-proof.integration.test.ts` (embedded
Postgres, no mocks of the proof): an interrupted controller that had
connected registers its checkpoint, the claim-time proof is read, and
then a replacement controller registers its own checkpoint. A live read
after that returns `bootstrap_incomplete` even when the interrupted
controller had sent `turn.start`. The claim-time proof stays false for
that case and true for the bootstrap-only case, and it refuses another
runner instance.

## Verification

- Red: with the source change reverted, `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session"` fails on the new case.
- Green: the same command passes (65 tests after merging master).
- `pnpm exec vitest run
server/src/services/native-runtime/native-bootstrap-claim-time-proof.integration.test.ts`:
2 passed.
- `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts
server/src/services/native-runtime/native-restart-recovery.test.ts`:
1474 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.
- Merged `origin/master` (including #21134) without conflicts and reran
the tests above.

## Risks

- Low. The new path runs only under a `bootstrap_incomplete` claim for
the same run id, with no failover backup, and only after the Postgres
controller checkpoint proves that no provider turn was sent. It stops a
Runner only when the identity marker and process fingerprint match,
which is the same check the reattach path uses.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21134 (the never-launched branch of the same helper), #21079
(stops a leftover Runner at launch time; this change acts earlier, in
host preparation).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:38:06 -06:00
DottaandClaude Fable 5.2 2174f2dfa5 perf(server): stop re-reading and re-decrypting unchanged checkpoint headers on every upgrade readiness probe (#21130)
Cloud repeats the upgrade readiness probe. Every probe loaded, decrypted and parsed the checkpoint header of each active native run and of each retained task session's last run, even when nothing had changed.

The statement that reads a run now also reads its header's row revision (xmin, ctid, updated_at, stored size) and returns the material only for a revision the process has not inspected. A bounded memo in the checkpoint owner keeps the four identifiers the answer compares. It reuses an answer only for the same binding, row revision and secrets key, loads the key again for each answer it serves, never remembers a header that failed to decrypt, and forgets a header the latest probe did not inspect.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 10:45:44 -06:00
DottaandClaude Opus 5.5 d5bd38468d fix(runner): stop a leftover Runner on the same state folder before a relaunch (#21079)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. The Runner keeps its saved state in one folder per
session and listens on one port in the sandbox.
> - During an automatic upgrade, the old server can launch a Runner a
few seconds before it stops. That Runner stays alive in the sandbox and
keeps the state folder and the port.
> - The new server then restarts the run on the same sandbox. Each new
Runner fails at once with "Address already in use"
(`runner_ingress_bind_conflict`). Each launch also overwrites the
process marker that pointed at the old Runner, so the next check cannot
find the old Runner to stop it. The run restarted about 12 times in 19
minutes and then stopped, and the task stayed blocked.
> - Two Runner processes must never use the same saved state at the same
time.
> - This pull request makes each Runner launch stop any live Runner that
still names the same state folder, before the new Runner starts.
> - The benefit is that a restart after an upgrade gets the port and
continues the run, instead of looping on the bind conflict.

## Linked Issues or Issue Description

**What happened?**
On staging, a Runner launched by the old server just before an upgrade
kept the Runner port. Every relaunch by the new server exited within
seconds with "runner_ingress_bind_conflict: failed to bind listener:
Address already in use (os error 98)". The run looped for about 19
minutes and then failed with `native_runner_process_exited`. The task
was blocked and needed a person.

**Expected behavior**
The relaunch stops the leftover Runner of the same session, gets the
port, and continues the run.

**Steps to reproduce**
1. Launch a Runner for a run in a sandbox.
2. Stop the server before the Runner is adopted (an upgrade). The Runner
stays alive.
3. Let the new server restart the run's bootstrap. The new Runner cannot
bind the port, and the next restart cannot see the old Runner because
the marker now names the dead new process.

## What Changed

- `native-session-executor.ts`: the detached Runner launch script now
runs a short fence before it writes the process marker.
- It finds processes whose argv[0] is `paperclip-runnerd`, that are not
a `reap-lineage` command, and whose `--state-dir` is followed by this
exact folder. Runners on another folder, a folder named by a different
flag, and other programs are not touched.
- It records the stale Runner's lineage by PID **and start time**: its
descendants, and members of a process group or session that a recorded
process leads while that leader is alive (provider tools start their own
sessions). It signals a PID only while its start time still matches, so
a reused PID is never signalled, and it never signals a whole process
group.
- TERM, up to 5 s wait, then KILL, up to 2 s wait. Then it runs
`<runnerd> reap-lineage --state-dir … --runner-pid …` for the stopped
Runner, which stops the tools its lineage journal recorded that have
already left the process tree.
- It fails closed: a survivor (`runner_stale_state_holder_alive`) or a
reap that leaves a recorded process alive or gives no report
(`runner_stale_lineage_unproven`) makes the launch exit 75 **before**
the marker is written, so the existing marker is kept. The reason goes
to the process stderr log only when that path is a plain file (same rule
as the Runner stderr log). The whole stop is bounded at about 14 s,
under the 20 s marker wait.
- Two reap refusals let the launch continue with a
`runner_stale_lineage_partial` warning: `journal_unavailable` (the
Runner kept no journal; its visible lineage is already gone) and
`unattributed_orphan` (reported only after every recorded process is
gone, for orphans the reaper can neither claim nor signal) **only when
every orphan it names is proven to have another owner**: this launch's
own detached shell, a `paperclip-runnerd` serving a different
`--state-dir`, or that Runner's stderr trimmer. Any other orphan could
be an old tool that escaped between samples, so the launch fails closed.
- When `/proc` is not readable, or `grep` is missing, the fence does
nothing (the old behavior).
- `native-session-executor.test.ts`: Linux tests (a stale Runner whose
tool runs in its own session, plus four look-alikes that must survive;
fail-closed with the marker kept, with a plain and a symlinked log; reap
refusals for orphans owned elsewhere, an unknown orphan, and no journal;
a normal launch starts no extra node process) and a cross-platform check
that the embedded script parses and has no shell-special characters.

**When a launch happens vs. a reattach.** The launch is reached only
after the server chose not to adopt a Runner. A same-run reattachment
(`reattach_remote_runner`) first verifies the marker's Runner: if it is
alive it is adopted and nothing launches; a transient verification
failure (for example the read timeout that #21053 keeps the Runner alive
for) throws and never reaches a launch; only a Runner proven not running
falls through to a fresh launch. Warm recovery likewise throws instead
of launching when revalidation fails. Every launch overwrites the
marker, so after a launch no reattachment could adopt a leftover Runner
anyway, and its port would make the new Runner fail with
`runner_ingress_bind_conflict`. Stopping it therefore never removes a
Runner that the same run could still re-adopt.

## Verification

- Linux (Debian bookworm container, node 24): the Linux tests pass on
this head. The same tests against the previous head (8df6a9099e) fail:
it killed a `paperclip-runnerd reap-lineage --state-dir <same folder>`
command, and it launched although the stale lineage was not proven gone.
On master there is no fence, so the stale Runner stays alive.
- Boot time: a normal launch took 19–35 ms for the whole launch script
in the container with the fence; node is not started by the fence
(asserted by a test). With a stale holder, the stop and launch took
about 160 ms.
- macOS: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts` passes
(1412 passed, 4 skipped: the Linux-only tests).
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- A launch now stops an older Runner on the same state folder. See "When
a launch happens vs. a reattach" above: this only happens when that
Runner is already impossible to adopt and would block the new one.
- A normal launch pays one `grep` over `/proc` plus reading argv[0] of
the processes that name the folder (its own subshell). No node start, no
extra round trip, no wait.
- If `paperclip-runnerd reap-lineage` cannot verify the stale lineage,
the launch fails closed instead of starting a second Runner next to old
tools.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:34:22 -06:00
DottaandClaude Opus 5.5 df89950eb6 fix(native): save or set aside a superseded run's unsaved workspace so its task queue moves (#21136)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result. Then a finalizer exports the
run's workspace files and applies the result to the task.
> - When a newer run already decided the task, the finalizer retires the
older result as superseded. Since #21106, this also ends the run.
> - If that retirement interrupts a failed workspace export, the run's
files stay unsaved. The run also stays the writer of the task's
workspace. Nothing records those files for recovery, so nothing ever
releases the workspace.
> - Every run queued on the task then waits behind a save that never
settles. After one hour, each queued run fails setup instead.
> - This pull request records those files for the existing automatic
workspace recovery. Recovery saves them from the retained sandbox, or,
when every bounded attempt fails, continues the task from its last save
with the files attached.
> - The benefit is that the task's queue moves again by itself, and the
run's unsaved work is saved or kept findable. The superseded result is
never applied.

## Linked Issues or Issue Description

Refs #21106

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. Its workspace export failed once with
`remote_workspace_archive_fs_eacces`. The finalizer then retired it as
`native_finalization_superseded`, because another run had already
decided the task. #21106 ended that run at 14:59:17Z (`failed`). The
task still could not move. The next queued run was cancelled with
`workspace_busy` ("ended, but its workspace save has not settled"). Its
retry re-armed every minute. Four more runs stayed queued. The task
showed no recovery action.

**Expected behavior**
The ended run's unsaved files are saved from its retained sandbox, and
the workspace is released. If they cannot be saved within the bounded
retries, the task continues from its last committed save, and the files
are attached to the task. The queued runs proceed in both cases. The
superseded result is never applied.

**Steps to reproduce**
1. Accept a native result for a run that writes a remote workspace.
2. Let the workspace export fail once, while a newer run has already
committed the task's decision.
3. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is `failed`.
4. See that the workspace still names the run as its dirty writer, and
that no retained-file recovery action exists.
5. Queue another run on the task. It defers with `workspace_busy`, and
after one hour it fails setup with "The workspace still has an
unfinished writer or save operation."

**Root cause**
`recordRetryableFailure` (`native-run-finalizer.ts`) retires a
superseded finalizer without preserving the export.
`preserveNativeFailedWorkspace` records a retained-file hold for an
accepted result only when the failure code is
`native_workspace_sync_out_retry_exhausted`. A
`native_finalization_superseded` coordinator never qualifies.
`readRemoteWorkspaceWriterHold` keeps an ended dirty writer as the
holder until its save settles. Without a hold, no worker ever settles
it.

**Paperclip version or commit**
Staging Core with #21106 (`81d2e4d8d9`). Master `ef2b53b675` has the
same behavior.

## What Changed

- `native-superseded-workspace-export.ts` (new): identifies a superseded
finalizer that failed in the workspace export. It selects ended runs
that still hold their workspace with unsaved files and no retained-file
hold. It records the hold through `preserveNativeFailedWorkspace`. It
checks again only while a just-ended run's sandbox stop receipt is still
arriving (at most one hour).
- `native-failed-workspace-recovery.ts`: `preserveNativeFailedWorkspace`
treats a superseded workspace export like an exhausted copy-back, with
the same proofs (run ended, no live controller, accepted result,
confirmed sandbox stop-and-retain, source still bound to the workspace).
It marks the hold `supersededWorkspaceExport`.
- `native-run-projection-work.ts`: the native run projection worker also
selects these runs. It loads them at startup and wakes when a finalizer
changes. Rows written by older releases are therefore settled after
deploy without a person.
- `workspace-recovery-fallback-discovery.ts` and
`workspace-recovery-fallback.ts`: a marked hold whose bounded re-export
is spent (at most 5 retries in 30 minutes) is eligible for the existing
continue-from-last-save fallback. That fallback sets aside only this
run's workspace epoch and keeps its location on the workspace record. It
attaches the readable files to the task unapplied. The task message says
Paperclip "could not save" the previous run's unsaved files.
- `preserved-result-finalization.ts`: a superseded result is never
resumed for application, even after its files are saved.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/services/superseded-workspace-export-recovery.test.ts`
(new). The first two tests fail on master and pass on this branch:
- "records the files for automatic recovery, saves them from the
retained sandbox and frees the workspace". It uses the staging row
shape. The projection worker records the hold, and a second pass changes
nothing. The workspace waits for that save. The system recovery saves
`saved.txt` into a committed snapshot. The workspace becomes free. The
finalizer, the result and the task's decision do not change, and the
result is not resumed. On master the worker selects nothing (`expected 0
to be 1`).
- "continues from the last save with the files attached when every
re-export fails". After an exhausted re-export budget, the hold is
resolved `restored`. The files are attached (`quarantine.state =
captured`). The workspace moves to epoch 2 with no writer, and the
abandoned epoch keeps its lease location. The message has no request for
a person to fix anything.
- "leaves a superseded result alone when it never failed in the
workspace export".
- Related suites pass: `native-run-projection-work`,
`native-finalization-recovery`, `workspace-recovery-fallback`,
`native-failed-workspace-recovery`, `workspace-recovery-retry-copyback`.
- `tsc --noEmit` for `server`.

## Risks

- Low to medium risk. The change acts only on runs whose finalizer is
already terminal and superseded, that failed in the workspace export,
and that still own a dirty workspace. All existing preservation proofs
still apply.
- The saved files of a superseded run become the workspace's next saved
version. The run's result (the task decision) is still never applied.
- When the export can never succeed, the task continues from its last
save. Files that cannot be read are not attached. The workspace record
keeps the set-aside epoch's location. Saving the readable part of a
partly unreadable workspace is not in scope.
- Runs that already failed setup before this deploys are not restarted
by this change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:20:54 -06:00
DottaandClaude Opus 5.5 fb8b61bc3e fix(native): restart a bootstrap whose closing server staged files but never launched its Runner (#21134)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. A release swap stops the old server and a new server
recovers each run.
> - When the old server stops during a run's bootstrap, the new server
restarts that bootstrap on the same run (`bootstrap_incomplete`). The
claim is granted only when the Postgres controller checkpoint proves
that no provider turn was sent.
> - The old server can stop after it staged launch files (Codex home,
config) into the sandbox session folder, but before it launched the
Runner. Its detach fence stops it there.
> - The new server then finds no Runner state in the sandbox. A fresh
bootstrap must create the session folder itself, and the folder already
exists, so the run failed with `runner_harness_state_mismatch:
resumed_unavailable`. The failure is an ambiguous session, so the task
waits for a person.
> - This pull request lets host preparation prove that no Runner was
ever launched in that folder, move the folder aside, and restart the
bootstrap.
> - The benefit is that a release swap during this part of a bootstrap
no longer blocks the task. No provider work can be lost or replayed,
because no Runner ever ran.

## Linked Issues or Issue Description

Refs #21120 (same release-swap class: the Runner was launched and
connected before the old server stopped; that pull request stops it in
host preparation. This pull request covers the case where no Runner was
launched.)
Refs #21079 (launch-time fence for a stale Runner on the same state
folder.)

**What happened?**
On hosted staging, during a release swap on 2026-11-12, a Codex run on
Daytona failed with `native_session_interrupted` and
`runner_ingress_unavailable: runnerd process failed:
runner_harness_state_mismatch: resumed_unavailable`. The old server was
in `harness_state.launch_material` when its shutdown drain detached the
run. It recorded no Runner process identity. The new server chose
"Restarting an incomplete native runner bootstrap on the same heartbeat
run". Its host preparation found the sandbox resumed and no Runner
state. Then the claim of the session folder failed, because the old
server had already created that folder. The run needed a person.

**Expected behavior**
The run restarts its bootstrap on the same sandbox and continues without
a person. A provider turn is never replayed and two Runners never share
the run.

**Steps to reproduce**
1. Start a Codex run on a Daytona sandbox.
2. Stop the server after the bootstrap stages launch files into the
sandbox session folder and before it launches the Runner.
3. Let a new server recover the run with a `bootstrap_incomplete` claim.
The new server has no local controller root for the run.
4. Host preparation fails with `runner_harness_state_mismatch:
resumed_unavailable`.

**Paperclip version or commit**
Staging release `704b3f4d63`. The code is the same on master at
`ef2b53b675`.

**Deployment mode**
Hosted staging, remote sandbox execution on Daytona.

**Agent adapter(s) involved**
Paperclip Runner with Codex (`codex_app_server`).

## What Changed

- `native-session-executor.ts`: under a same-run `bootstrap_incomplete`
claim with no failover backup, when host preparation finds no Runner
state, a new check asks the sandbox in one command whether the session
folder exists and holds no Runner trace. A Runner trace is the live
`runner/runner-state.json`, the `runner/runner-process.identity` launch
marker (the launch script writes it before the Runner starts), an
`authority-epochs` archive, or any other `runner-state.json` or launch
marker anywhere under the folder. If the folder holds no Runner trace,
no Runner was launched there, so host preparation records the restart
proof (`absent_confirmed`). It then uses the existing path: move the
folder aside (it is kept for inspection), claim a new folder, and
bootstrap again. If the folder is missing, the existing fresh-claim path
runs unchanged. If any Runner trace exists, the run stays fail-closed as
before. An archived session is therefore never discarded.
- `native-session-executor.test.ts`: new
`interrupted_bootstrap_unlaunched` cases. Resumed and created sandboxes
restart the bootstrap. A launch marker, an archived authority epoch,
Runner state elsewhere in the folder, or a failover backup keeps the run
fail-closed with `resumed_unavailable`.

## Verification

- Red: with the source change reverted, `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session" --maxWorkers=2` fails on
both `interrupted_bootstrap_unlaunched` restart cases with
`runner_harness_state_mismatch: resumed_unavailable`, which is the
staging error.
- Red: without the archived-state guard, the `_archived` and
`_nested_state` cases fail (the folder is moved aside). With the guard
they pass.
- Green: the same command passes (64 tests).
- `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts
--maxWorkers=2`: full file, 1422 passed before the guard commit. The
focused 64 pass on the final revision.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.
- `git merge-tree` against #21120's head (`0d18f18928`) merges without
conflict. The new check runs only when no Runner state exists, so
#21120's new call site (it runs only when a Runner state exists) gets
the same results as before.

## Risks

- Low. The new path runs only in host preparation, under a
`bootstrap_incomplete` claim for the same run id, with no failover
backup. That claim is granted only after the previous controller's lease
has ended. With no Runner state or launch marker anywhere in the folder,
no Runner and no provider session existed there. The old folder is moved
aside, not deleted.
- A controller that is still launching at the same moment would have to
hold a lease that the claim already took over. The detach fence stops
such a launch, and #21079 adds a launch-time fence for a leftover Runner
on the same state folder.
- No added wait before a normal turn starts: the check runs only on this
failure path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser for read-only run evidence, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:06:55 -06:00
DottaandClaude Fable 5.2 5b698b68cf perf(run-events): append a native event in one transaction of two statements (#21131)
One appended native run event cost eight statements on the database
connection (BEGIN, SAVEPOINT, run lock, replay probe, sequence update,
insert, count/max read, COMMIT). The query builder built five of them
again for every event, and each event created two transaction objects.

A native source event now takes the run lock and then runs one statement
that finds a replayed event or reserves the sequence and inserts, and
reads the retained count and maximum for the acknowledgement. The lock
stays a statement of its own so the append reads a snapshot taken after
the lock was granted. The control-plane port passes its controller check
to the append instead of opening an outer transaction, so there is no
SAVEPOINT. The hot statements are fixed SQL text.

Events that wake a queue still register the wake before the write and
only when the event is new. Order, sequence numbers and durability do
not change: every event still commits before its append returns.

Local benchmark (server/scripts/benchmark-run-event-append.ts), one
native event: 8.3 -> 4.6 statements, 589 -> 222 KB allocated,
1.09 -> 0.53 ms CPU, 28 -> 11.5 garbage collections per 1,000 events.
One ordinary event: 105 -> 27 KB allocated.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 09:58:31 -06:00
DottaandClaude Opus 5.5 9e910ff9ae fix(chat): keep raw IDs and internal codes out of chat replies (#21081)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and the agent replies in plain
text
> - In a full chat-eval cohort, 47 of 158 passing replies showed raw
UUIDs, error codes, field keys, tool names or server paths
> - Users must not see internal jargon or raw IDs, so these replies are
a product defect, even when the facts are correct
> - Agents copy these terms from API error messages and saved fields;
the chat directive did not tell them to translate
> - This pull request fixes the sources: the chat directive, and API
denial messages that put IDs and policy codes in the human message
> - The benefit is plain replies for every model, while machine codes
stay available in separate fields

## Linked Issues or Issue Description

**What happened?**

Chat replies quote internal names and raw IDs. Examples: run, handoff,
revision, interaction and trigger UUIDs in prose;
`native_session_retry_exhausted`, `board_access_required`,
`review_policy_denied`, `local-board`, `enableCases`; a raw user ID
inside the inbox denial message.

**Expected behavior**

Replies name records the way the person sees them (task identifier,
title or name, with a link) and explain codes in plain words. IDs stay
only in link targets, in commands the person must run, or when the
person asks for an ID.

**Steps to reproduce**

Run the chat-features OpenCode cohort (for example `cases.close-case`,
`routines.webhook-trigger`, `inbox-search.archive-without-grant`) and
read the agent replies.

**Paperclip version or commit**

`master` at f30005440b.

**Agent adapter(s) involved**

Not adapter-specific (core bug). Seen with the Paperclip runner and
OpenCode.

**Deployment mode**

Local dev server (embedded PGlite) and the CI chat-eval runner.

**Installation method**

Source checkout (`pnpm dev`).

## What Changed

- `AGENT_CHAT_DIRECTIVE`: add one rule. Name records by their
identifier, title or name and link them. Do not write raw IDs, internal
codes, field keys, tool names, process names or server paths, even in
inline code. The rule has exceptions for link targets, commands the
person runs, and IDs the person asks for. The rejection rule now asks
for the platform's reason "in plain words".
- Authorization denials (inbox management, membership, responsible user)
no longer put user or agent IDs in the explanation. The `reason` and
`code` values do not change.
- Review-policy denials explain the rule in plain words. The
`review_policy_denied` code and the `policy` and `allowedActor` fields
do not change.

## Verification

- Red/green: `server/src/__tests__/agent-conversations.test.ts`
(directive rule, and it reaches the chat task prompt),
`server/src/__tests__/authorization-service.test.ts` (inbox denials
carry no IDs), `server/src/__tests__/issue-review-policy.test.ts` and
`server/src/services/issue-thread-interaction-resolution.test.ts` (plain
review messages). Each new assertion failed before the fix.
- `pnpm verify` and the server typecheck pass locally.
- A paid OpenCode run on 8 affected chat cases is in progress; results
will be added here.

## Risks

- Low risk. Only human-readable messages and prompt text change. No
client matches the old message text; codes and machine fields stay the
same.
- The directive grows by one paragraph. A model can still copy an ID;
the rule reduces this but cannot prevent it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:37:07 -06:00
DottaandClaude Opus 5.5 81d2e4d8d9 fix(native): end a run whose finalizer a newer run superseded, so it stops holding its task (#21106)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result, then a finalizer exports the
workspace and applies the result to the task.
> - When a newer run already committed the task's decision, the
finalizer retires the older result as `native_finalization_superseded`.
> - That retirement set the finalizer terminal, but it did not end the
run. The run stayed `running` and held the task and every run queued
behind it, with no way out except a person.
> - A stuck run must never hold a task and its queue forever.
> - This pull request ends the run when its finalizer is retired as
superseded, and it ends runs that an older release left in that state.
> - The benefit is that the task continues with its queued runs by
itself.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. The workspace export then failed
(`remote_workspace_archive_fs_eacces`). The task's last decision came
from another run that was created 42 seconds later but ran earlier. The
finalizer judged this run superseded: it set the finalizer to
`terminal_failure` with `native_finalization_superseded`. The run stayed
`running` with "Recovery needed". Over one hour later it still held the
task, and five queued runs waited behind it. A server upgrade at 12:02Z
did not change it.

**Expected behavior**
A run whose finalizer is retired ends. The task keeps the newer run's
decision, the retired result is never applied, and the queued runs
proceed.

**Steps to reproduce**
1. Accept a native result for a run.
2. Commit a status decision on the same task from a run that was created
later.
3. Let the older run's finalization fail once, so the finalizer records
the failure.
4. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is still `running`.

**Root cause**
`recordRetryableFailure`
(`server/src/services/native-runtime/native-run-finalizer.ts`) projects
a terminal run status only when `exhausted && !supersededByNewerRun`. A
superseded retirement therefore never ended the run. No other path ends
a run whose finalizer is already `terminal_failure`: the reconciler
skips `terminal_failure` rows.

**Paperclip version or commit**
Staging Core `3eca7a6112`. The behavior is the same on master
`482063aeca`.

## What Changed

- `native-run-finalizer.ts`: a superseded retirement now ends the run
with the same status mapping as an exhausted finalization (workspace
failure: `failed`; otherwise the provider's terminal state). It does not
change the task, does not apply the result, and does not keep the
workspace export lease. The run's error says in plain words that a newer
run already decided the task.
- `native-run-finalizer.ts`: add `endSupersededNativeFinalizationRuns`.
It ends a native run that is still `running` while its finalizer is
`terminal_failure` / `native_finalization_superseded`. This covers rows
that an older release wrote. It locks in the finalizer's order
(coordinator, task, run), and re-checks the row under the lock.
- `native-run-projection-work.ts`: the registered native run projection
worker (it loads at startup, pages through all candidates, and wakes on
finalization changes) also selects such runs and ends them with
`endSupersededNativeFinalizationRuns`.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/__tests__/native-finalization-recovery.test.ts`, red on
master and green on this branch:
- "retires an older failed finalizer when a newer run already committed
the issue": the retired run now ends (`succeeded`, `finishedAt` set).
Master: the run stays `running`.
- "ends a run an earlier release left running after retiring its
finalizer as superseded": the shape of the staging row (workspace export
failed, superseded finalizer, run `running`). The repair ends it as
`failed` with the plain message. The task, its decision and the
finalizer stay unchanged, and no status decision is written for the run.
A second pass changes nothing.
- `server/src/__tests__/native-run-projection-work.test.ts`, "ends, at
startup, a run an earlier release left running after retiring its
finalizer as superseded": the registered worker ends the staging-shaped
row at startup and settles its claimed wake. Red without the worker
change.
- Related suites pass: `native-finalization-recovery`,
`native-run-projection-work`, `native-finalization-reconciler-timing`,
`native-reconciliation-timing`, `native-workspace-finalizer`,
`native-workspace-finalizer-failure`, `native-run-finalizer-telemetry`.
- `tsc --noEmit` for `server`.

## Risks

- Low risk. The change only ends runs whose finalizer is already
terminal and superseded. It never applies a result and never changes the
task.
- A superseded run whose workspace export failed now releases its
sandbox lease through the normal settlement instead of waiting forever.
Its result was not applied, because a newer run decided the task.
- Not changed here: the superseded check compares run creation times. A
run created earlier but executed later is still judged superseded. This
PR makes that verdict end the run; it does not change the verdict.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:07:18 -06:00
DottaandClaude Opus 5.5 0a2f618cae fix(native): leave a run another live server is preparing alone, and re-adopt a run a shutdown handed off (#21107)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade replaces the server. The old and the new server serve
together for a few minutes, then the old server gets SIGTERM and hands
its native runs to the next controller.
> - #21046 taught the legacy reaper to leave a run with its owner while
the owner's server process is proven alive. Native runs have a similar
gap.
> - Runtime selection writes a native run's coordinator with no
controller and no lease, and stops the run lease. The executor claims
the coordinator about 20 seconds later. In that window the other
server's sweep sees no owner and marks the run "ownership could not be
verified".
> - When the old server then hands the run off before any Runner
checkpoint exists, no sweep looks at it again. Only the next server boot
does, 15 to 40 minutes later. The task and its queued runs wait.
> - This pull request leaves such a run with its live preparing server,
and lets the serving server re-adopt a handed-off run through the
ordinary restart claim.
> - The benefit is that a run started during an upgrade continues within
one sweep, with no false "blocked" mark and no wait for another upgrade.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, an upgrade started a new server at
12:19:57Z. The old server started a native run and selected its runtime
at 12:20:32Z. At 12:20:35Z the run logged "Native execution ownership
could not be verified; automatic recovery is blocked"
(`observed_owner_unverified`). The old server's executor claimed the run
at 12:20:51Z. The old server got SIGTERM and suspended the run at
12:20:54Z (`awaiting_runner_reattach`). After that the run showed
"Confirming execution" for 29 minutes, and two queued runs waited behind
it. At 12:49:45Z the next upgrade's startup recovery restarted its
bootstrap, and the run succeeded at 12:58Z.

**Expected behavior**
- A server does not mark a run unverified, or recover it, while another
live server is preparing it.
- A run that a graceful shutdown hands to the next controller is
re-adopted by the serving server at once, not at the next boot.

**Steps to reproduce**
1. Seed a native run whose coordinator is `observed` with no controller
and no lease, and whose `controllerProcess` names another server with a
live controller instance lease.
2. Run the recovery sweep on this server. See that the run gets
`native_execution_ownership_unverified`.
3. Claim the coordinator for the other server, then write what its
graceful shutdown writes (`restart_suspended`,
`awaiting_runner_reattach`, lapsed lease) and stop its instance lease.
Give the run a predecessor's Runner PID.
4. Run the sweep again. See that nothing claims the run.

**Root cause**
- `recoverActiveExecutions`
(`server/src/services/heartbeat/recovery.ts`) treats an `observed`
coordinator without a live controller lease as an owner it cannot
verify. Between runtime selection and the claim, the only owner on
record is the run's `controllerProcess`, and the sweep did not check it.
- `listExpiredNativeRunnerAuthorities` (`native-runner-checkpoint.ts`)
selects runs for the restart claim only when they have a controller
checkpoint, or no process identity at all. A run handed off just after
its claim has no checkpoint, and a follow-up carries its predecessor's
Runner PID. The sweep then skips `awaiting_runner_reattach` runs, so
only a server boot looked at them again.

**Paperclip version or commit**
Staging Core `814fe5d20f` (old server) and `3eca7a6112` (new server).
The behavior is the same on master `849f109f62`.

## What Changed

- `recovery.ts`: before it lists expired native runs, the sweep checks
runs in the runtime-selection window (coordinator `observed`, no
controller, no lease, not tracked here). When the run's
`controllerProcess` names another server process that the shared owner
prover proves alive, the sweep leaves the run with that server: it does
not recover it and does not mark it unverified. It re-checks within 30
seconds. After 5 minutes without a claim
(`LEGACY_LIVE_OWNER_SETTLE_GRACE_MS`), or once that server is gone, the
existing rules apply.
- `native-owner-stamps.ts`: add `preparingControllerOwnerStamp` for that
proof.
- `native-runner-checkpoint.ts`: `listExpiredNativeRunnerAuthorities`
also selects a run that a graceful shutdown handed off and whose lease
lapsed: recovery state `awaiting_runner_reattach` or
`awaiting_evidence`, with a `restart_suspended` history entry that no
restart claim followed. It stays selected through ownership waits (the
departing server can still be alive at the first look). `recovery.ts`
schedules a 30-second re-check for an ownership wait that has no retry
of its own. The ordinary restart claim then proves the takeover and
picks the disposition, exactly as the startup pass of a new server does.
A row that an older release already marked unverified is re-adopted too:
the claim clears the mark.
- `doc/execution-semantics.md`: document both rules.
- No schema change. Nothing changes on the path that starts a run or a
turn: the new proof runs only in the recovery sweep, and only for runs
in the window.

## Verification

- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`,
block "a run the old server started during an upgrade overlap". Red on
master, green on this branch:
- "leaves the run with its live preparing server, then re-adopts it
after that server hands it off": two concurrent sweeps leave the run
`running` with no error and do not touch the coordinator. After the old
server claims, hands the run off and stops, one sweep of the new server
re-adopts it and reaches provider dispatch. Master: the run is marked
`native_execution_ownership_unverified`, and after the handoff nothing
claims it.
- "keeps a handed-off run eligible while the departing server is still
alive, and re-adopts it once it exits": the first sweep waits
(`awaiting_evidence`), the sweep after the departing process exits
re-adopts the run. Red with a filter that matches only the last history
entry.
- "re-adopts a handed-off run that an older release already marked
ownership-unverified": the stuck staging shape. Master: nothing claims
it.
- Related suites pass: `heartbeat-native-restart-suspension` (all 109),
`services/heartbeat/recovery.test.ts` (all 103).
- `tsc --noEmit` for `server`.

## Risks

- A run that another live server prepares, but never claims, now stays
`running` for up to 5 minutes before the existing rules apply. That is
the same bound #21046 uses.
- The guard trusts the shared owner prover. It reads "alive" only from a
live controller instance lease or a live PID in the same PID space.
- More handed-off runs reach the restart claim earlier. The claim keeps
all its fencing and evidence checks; this change only makes the serving
server ask the same question a new boot asks.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:06:57 -06:00
DottaandClaude Opus 5.5 67403baaa4 fix(native): an agent termination Stop with no turn result stays a cancellation (#21093)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person terminates an agent, Paperclip stops the agent's
running native turn
> - Two paths can record that Stop: the terminate route ("Cancelled
because the agent was terminated ...") and the lifecycle worker ("Agent
lifecycle stop requested"). Both are system-initiated and expected
> - When the stopped turn ends without a semantic result, the native
executor keeps the run cancelled only for a person's Stop, a
reassignment, or a governance pause. A termination Stop fell through, so
the run ended `failed` with `native_session_interrupted`
> - Whether the run ended cancelled or failed depended on which path won
the race
> - This pull request treats an acknowledged termination Stop like a
person's Stop in the executor and in the lease-release terminalization
> - The benefit is that a terminated agent's interrupted run ends
cancelled, whichever path records the Stop first

## Linked Issues or Issue Description

Refs #20186 (the same treatment for a governance pause Stop)

**What happened?**
A person chose Terminate for an agent while its native turn ran. The
lifecycle Stop recorded an acknowledged native cancellation (initiator
system). The turn ended without a semantic result. The executor then
wrote the run as `failed` with `native_finalization_missing: session
returned no semantic result`.

**Expected behavior**
The run ends `cancelled`. A termination is an intentional Stop, not a
failure to retry.

**Steps to reproduce**
Terminate an agent while its native turn runs, so that the Stop
interrupts the turn before it records a result. Before this change, the
run became `failed` when the lifecycle worker recorded the Stop first.

## What Changed

- `governance-pause-stop.ts`: adds `AGENT_TERMINATED_STOP_REASON`,
`COMPANY_ARCHIVED_STOP_REASON` and `isAgentLifecycleStopRecord`. The
predicate accepts a system-initiated, expected control-plane
cancellation whose reason is a lifecycle, termination or company-archive
Stop.
- `acknowledged-native-stop.ts`: adds
`hasAcknowledgedNativeLifecycleStopIntent`, which requires that record
and an acknowledged, dispatched native Stop receipt.
- `native-session-executor.ts`: the no-semantic-result branch accepts
that intent, so it settles the coordinator as `native_retry_cancelled`
and raises `NativeCancellationPendingRecoveryError`, as for a person's
Stop.
- `heartbeat/run-lifecycle.ts`: lease-release terminalization keeps such
a run `cancelled`.
- The terminate route and the lifecycle driver use the shared reason
constants. The reason text does not change.

## Verification

- `native-session-executor.test.ts`: a new case for each reason
(lifecycle Stop, terminate route, company archive). Each was red before
the change and is green now. The earlier control case now uses a system
Stop that is neither a pause nor a termination, and it still takes the
retry path. Full file: 1414 passed.
- `heartbeat-run-lease-release-terminalization.test.ts`: a new case for
a lifecycle Stop when teardown wins the finalizer race. It was red
before the change (`interrupted`) and is green now.
- Product evidence: a paid chat-eval run of `agents.retire-agent` on a
branch that combines this change with three related fixes. The result is
in a PR comment.

## Risks

- A termination or company-archive Stop that interrupts a turn no longer
schedules an exact resume of that turn. The agent is leaving service, so
no resume can run.
- Other system Stops keep their current behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:19:21 -06:00
DottaandClaude Opus 5.5 3eca7a6112 fix(runner): log which check refuses a saved provider-pack admission on reattach (#21085)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode turns run in a remote Runner. When a server restarts, the
new server reconnects to the Runner and reuses the provider pack
admission that the run saved earlier.
> - On staging, a run reconnected cleanly after an upgrade but then
failed with `native_provider_artifacts_admission_unverified`. Its saved
admission state stayed "pending".
> - The reuse step refuses a saved admission for several reasons. It
returns only false and logs nothing, so we cannot tell which check
refused it.
> - This pull request adds diagnostics only. It names the refused check
in the run log and the server log. It also logs when the admission stays
unverified for a session that needs it.
> - The benefit is that the next upgrade audit shows the real cause, so
the next fix can target it.

## Linked Issues or Issue Description

**What happened?**
After an automatic staging upgrade, a run reconnected to its live Runner
without a new process. The next checkpoint failed with
`native_provider_artifacts_admission_unverified`, which used one of the
run's 3 attempts. The run log had no line that explained why the saved
admission was not reused.

**Expected behavior**
When a saved admission is not reused, the run log and the server log
name the check that refused it.

**Steps to reproduce**
1. Start an OpenCode run on a remote sandbox.
2. Restart the server, so the run reconnects to its Runner.
3. If the saved admission is refused, the run fails later with
`native_provider_artifacts_admission_unverified` and no reason in the
logs.

## What Changed

- `native-provider-pack-admission.ts`:
`rememberRecoveredNativeProviderPack` accepts an optional `onRefused`
callback. It is called with one reason: `source_state_pending`,
`source_state_unknown`, `source_state_missing`, `receipt_missing`,
`runner_instance_differs`, `environment_lease_differs`,
`receipt_invalid`, or a session identity reason. The result of every
check stays the same. An error in the callback is ignored.
- `native-session-executor.ts`: the reattach path logs the reason to the
run log and the server log. When `recordNativeProviderPackAdmission`
returns false for a session that requires admission, it logs that too.
- `native-provider-pack-admission.test.ts`: each refusal names its
reason, and a reused admission reports nothing.

## Verification

- `npx vitest run
src/services/native-runtime/native-provider-pack-admission.test.ts`
passes 46/46. The new reason tests fail on the old code because
`onRefused` is never called.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts` passes
1412/1412.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. This change adds diagnostics only, and admission decisions
do not change. The new log lines contain only a fixed reason code and
run ids. They do not contain secrets or file paths.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:38:23 -06:00
DottaandClaude Opus 5.5 2e593bf3cd fix(native): drop a repeated identity-record check and prove an unsettled predecessor stays an ordinary refusal (#21086)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21001 sends a re-wake to a fresh sandbox when live adoption of a
crashed predecessor's warm Runner is refused, and lets an inherited
Runner identity pass the "workspace unchanged" proof
> - The reviewer found two gaps on #21001 that were not answered before
it merged
> - The unchanged proof checked for a `native.process_identity_recorded`
event twice, and no test proved that an unsettled predecessor stays an
ordinary refusal
> - This pull request removes the repeated check and adds that proof
> - The benefit is simpler proof code and a test that stops the
fresh-sandbox fallback from widening to unsettled predecessors

## Linked Issues or Issue Description

Refs #21001, #20863

**What happened?**
Two review warnings on #21001:
1. `carriesOnlyInheritedRunnerIdentity` in
`remote-workspace-unchanged-proof.ts` looked for the run's own
`native.process_identity_recorded` event.
`executionStartedEventPredicate()`, later in the same proof, already
refuses a run with that event (`execution_recorded`).
2. The existing denial cases in
`native-controller-recovery.integration.test.ts` use `toThrow(denied)`.
That also matches the wrapped
`NativePredecessorRetirementUnprovenError`, so a regression that wraps
an unsettled predecessor's refusal would pass.

**Expected behavior**
The proof has one check per fact, and the tests fail when an unsettled
predecessor goes to the fresh-sandbox fallback.

**Steps to reproduce**
1. Read `proveRemoteWorkspaceUnchanged` and
`executionStartedEventPredicate`.
2. Change `remote-warm-session-recovery.ts` to wrap every refusal; the
old denial cases still pass.

## What Changed

- `server/src/services/remote-workspace-unchanged-proof.ts`: delete
`carriesOnlyInheritedRunnerIdentity` and its `PROCESS_IDENTITY_RECORDED`
import; call `carriesInheritedRunnerIdentity(tx, run)` directly. A run
with its own identity record is still refused, now by
`execution_recorded`.
-
`server/src/services/native-runtime/native-controller-recovery.integration.test.ts`:
- The denial cases for an unsettled predecessor (other company, other
execution, finalization lease held, run still `running`) now assert the
error is not a `NativePredecessorRetirementUnprovenError`.
- The pending-command case after the predecessor is `succeeded` and
`committed` now asserts it is one (the #21001 behavior).
- The crash-restart case adds a predecessor run that is `failed` or
`running`: an unsettled journal is refused with
`warm_recovery_refused=owner_unsettled`, not wrapped.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts
src/services/remote-workspace-unchanged-release.test.ts`: 50 passed. The
three inherited-identity cases from #21001 return the same results
(released / refused / refused).
- `cd server && npx tsc --noEmit -p tsconfig.json`: passes.

## Risks

- Low risk. No behavior change: the removed check is covered by
`executionStartedEventPredicate()`. The rest is tests.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:35:15 -06:00
DottaandClaude Opus 5.5 5820d43197 fix(runner): relaunch a settled OpenCode Runner with its admitted artifact paths (#21076)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agent turns run in a remote Runner in a Daytona sandbox.
The Runner saves its state, so a retry of the same run can start a new
Runner from that saved state.
> - The first launch of a run starts OpenCode from the admitted provider
files under one folder in the sandbox. The new Runner replays that first
launch and checks that its own launch profile names the same files.
> - When a retry reconnects to a Runner that already stopped, Paperclip
starts a new Runner. On that path, Paperclip did not restore the
admitted provider files and gave the new Runner the provider pack folder
instead.
> - The new Runner then refused the replayed launch ("does not match the
runner-owned qualified profile"). That used the last retry, and the task
stayed blocked.
> - This pull request restores the admitted provider files and gives the
new Runner the same folder as the first launch.
> - The benefit is that a retry after a server upgrade continues the
same run instead of blocking the task.

## Linked Issues or Issue Description

**What happened?**
On staging, after an automatic upgrade, a run used all 3 attempts. The
last attempt started a new Runner from its saved state. It failed in the
drain step with "OpenCode run.prepare launch does not match the
runner-owned qualified profile". The task was blocked and needed a
person.

**Expected behavior**
The new Runner gets the same OpenCode launch files as the first launch
of the run, and the run continues.

**Steps to reproduce**
1. Start an OpenCode run on a remote sandbox with durable provider
artifacts.
2. Let the server restart, so the retry reconnects with
`reattach_remote_runner`.
3. The Runner is found settled, so a new Runner starts from the saved
state. Its launch profile points at the provider pack folder, not at the
admitted artifact folder.

## What Changed

- `native-session-executor.ts`: when a reconnect relaunches a settled
Runner and the run has admitted provider artifacts, restore those
artifacts and prepare the same artifact folder as the first launch.
Other reconnects (live adoption) do not change. Normal turn starts do
not change.
- `native-session-executor.test.ts`: the existing settled-relaunch case
now also covers a run with admitted provider artifacts. It expects the
restore and the admitted folder in the launch profile.

## Verification

- Red: without the fix, the new test case fails
(`restoreNativeProviderArtifacts` is not called, and the launch uses the
provider pack folder).
- Green: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-opencode-artifacts.test.ts` passes
1417/1417.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The change applies only when a reconnect must relaunch a
settled OpenCode Runner and the run has admitted provider artifacts.
- That relaunch now downloads the admitted artifacts from storage, as a
cold continuation already does. This adds time only on that recovery
path, not on a normal turn start.
- The artifact folder check (same account and same digest) still
applies, so a different sandbox account still fails closed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:24:17 -06:00
DottaandClaude Opus 5.5 559551e977 fix(native): wait for a busy agent directory while retiring a predecessor (#20991)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a reused warm sandbox first retires the previous
run's warm session, and that retirement collects the old session's agent
files
> - On a server without a persistent volume, agent-file collection takes
a per-agent Postgres lock, so runs of one agent collect one at a time
> - #20863 sends a run whose predecessor cannot be retired to a fresh
sandbox. It classified every retirement error as unprovable, including a
busy lock held by another run of the same agent
> - So ordinary lock contention failed the run, dropped a healthy
sandbox and rested the task
> - This pull request makes retirement wait for a busy directory lock in
the same run, and keeps the fresh-sandbox fallback for retirements that
stay unprovable after the lock is obtained
> - The benefit is that busy agents no longer fail follow-up runs on
their warm sandboxes

## Linked Issues or Issue Description

**What happened?**
A no-fault restart-chaos wave with 8 concurrent runs of one Claude agent
on Daytona, with a volume-free server, failed one monitor-due follow-up
after 6 seconds. The error was `adapter_failed`, cause
`native_predecessor_retirement_unproven`: "The previous run's session on
this sandbox could not be retired (Agent directory operation is already
active)". The run also logged `INSTRUCTION_COLLECTION_UNAVAILABLE`. The
stack shows the error came from `claimWarmNativeInstructionCopy` during
instruction preparation. The previous session had closed; only its
agent-file collection met the per-agent lock that another run held.

**Expected behavior**
A busy agent-directory lock is transient. The run waits for it and then
retires the predecessor on the same sandbox. Only a retirement that
stays unprovable after the lock is obtained takes the fresh-sandbox
fallback.

**Steps to reproduce**
1. Run several warm native tasks of one agent at once on Daytona with a
volume-free server.
2. Re-wake one task while another run of the agent collects its files.
3. The follow-up fails with `native_predecessor_retirement_unproven` and
its sandbox is dropped.

Refs #20863, #20829, #20957.

## What Changed

- `retireWarmPredecessor` (native executor): a busy agent-directory lock
(`AgentDirectoryOperationBusyError`) is waited out in the same run,
polling every second for up to five minutes
(`warmRetirementDirectoryWait`). The usual holder is another run's short
collection, so retirement then finishes on the same sandbox with the
copy this run adopted. The run is not deferred: deferring after adoption
would release the adopted copy and could skip saving the predecessor's
edits.
- Past that budget the run fails with its own error,
`native_predecessor_retirement_directory_busy`, not as an unprovable
retirement. Before failing it files the adopted copy's retirement
receipt, so the copy is kept as `pending_collection` and run cleanup
does not mark it `unavailable`. The owner is not marked unretired and
stays registered for its sandbox, so a later run retires it there.
- An idle-timeout close that meets the lock retries after the
agent-directory retry delay instead of leaving the session registered
with no retry.
- Closing idle sessions for environment delete (#20957) reports such an
owner as `busy`, not `failed`.

## Verification

- RED on master, then GREEN:
- `native-session-executor.test.ts -t "directory lock"`: two runs of one
agent contend on the lock while a follow-up retires the predecessor. On
master the claim rejects with `NativePredecessorRetirementUnprovenError`
(`Agent directory operation is already active`). Now a short-lived lock
is waited out and the predecessor is retired and collected on the same
sandbox; a lock that outlasts the budget fails with
`native_predecessor_retirement_directory_busy`, environment delete
reports `busy: 1`, a fresh-sandbox attempt does not forget the owner,
and once the lock is free a later run retires it.
- `native-session-executor.test.ts -t "idle close whose file
collection"`: on master the idle close never retries; now it retries and
closes.
- Existing suites pass: native executor, agent-directory busy heartbeat,
native pre-dispatch retry, native controller recovery and plugin runner
eviction. Server `tsc --noEmit` passes.

## Risks

- Under contention a follow-up can wait up to five minutes before its
first turn. The normal path adds no wait.
- A lock held for more than five minutes fails the run visibly instead
of dropping the sandbox.
- An idle owner whose collection stays busy keeps retrying at the 5 to
15 second agent-directory delay until the lock frees, a controller
restart, or the next admission.
- No persisted state changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:08:15 -06:00
DottaandClaude Opus 5.5 533f135780 fix(native): send a re-wake whose crashed predecessor's warm Runner cannot be adopted to a fresh sandbox instead of failing it (#21001)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a warm remote sandbox leaves its Runner `ready` in
the sandbox. The next run of the task adopts that Runner live, after it
proves that the previous owner finished
> - When the server is killed after a run parked on a question, the warm
owner lived only in the killed server's memory. The Runner in the
sandbox was never suspended
> - A re-wake on the replacement server resumed that sandbox and tried
live adoption. A proof failed, and the new run failed with
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate` and needed a Retry, although
it did no provider work
> - #20863 already sends such a run to a fresh sandbox when the
predecessor's Runner cannot be retired, but the live-adoption path of a
resumed sandbox (`remote-warm-session-recovery.ts`) was not connected to
it, and the failed run's inherited Runner identity blocked the
"workspace unchanged" proof that the fallback needs
> - This pull request connects the two: a refusal of live adoption after
the predecessor is durably terminal with an accepted result goes to the
#20863 fresh-sandbox path, and an inherited Runner identity no longer
counts as a process of the follow-up
> - The benefit is that a re-wake seconds after a crash restart
continues without a Retry

## Linked Issues or Issue Description

Refs #20863, #20844, #20928, #20979, #20953, #20972

**What happened?**
Restart-chaos wave `w-20261112T012841-s7000` (master `4b24352712`,
volume-free mode, Daytona warm, Codex, 10 tasks, K1 and K15 with a fresh
home and overlap), task t08 (W2Q):
- Run `58f57bf1` asked its question and succeeded at 01:29:56. Its warm
owner stayed in the server's memory.
- K1 killed the server at 01:30:06. The replacement was ready at
01:30:15.
- A board comment woke the task at 01:30:17. Run `8f812df6` started at
01:30:19 and resumed the same sandbox (`run.startup.reuse`
`verified_reuse`).
- The first live-adoption inspection passed. The second check
(`revalidate`, in backend setup, before the Runner attach) refused at
01:30:21 (stack: `remote-warm-session-recovery.ts:217` from
`revalidate`). The run failed `native_session_interrupted` with
`INSTRUCTION_COLLECTION_UNAVAILABLE`.
- The failed run kept its full save because the unchanged-files proof
refused with `process_recorded`: the run carried its predecessor's
Runner process identity. Retry (`6968dc0f`) worked.

**Expected behavior**
The predecessor finished with an accepted result before the kill, and
the new run did no provider work. The task must continue without a
Retry.

**Steps to reproduce**
1. Finish a warm remote run that parks on a question.
2. Kill the server (SIGKILL) and start a replacement.
3. Wake the task within seconds. The new run resumes the warm sandbox.
4. Any live-adoption proof fails, and the run fails terminally with
`prior_owner_terminal_state_indeterminate`.

**Which proof failed**
The old refusal had no reason. The wave evidence proves that it was
`revalidate`, after a passed inspection, and not the SIGTERM of the next
K15 fault (that server logged "shutdown signal received" only at
01:31:09). `revalidate` checks the predecessor's coordinator row, the
current run's coordinator row and lease, and the task. The evidence
cannot tell which one changed in the first seconds after the restart.
This pull request names the failed proof
(`warm_recovery_refused=<reason>`), so the next occurrence identifies
it. The fix covers the class: every refusal that concerns the
predecessor or the sandbox.

## What Changed

- `server/src/services/native-runtime/remote-warm-session-recovery.ts`:
- Each refusal names its proof from a closed list:
`identity_incomplete`, `target_not_resumed_sandbox`, `owner_unsettled`,
`execution_mismatch`, `checkpoint_unsettled`, `journal_unsettled`,
`controller_not_stopped`, `prior_lease_mismatch`, `owner_changed`,
`current_lease_changed`, `current_run_not_admitted`,
`current_owner_changed`.
- After the predecessor's run is `succeeded` and its coordinator is
`committed` with a result and no lease, a refusal about the predecessor,
its journal or the sandbox throws
`NativePredecessorRetirementUnprovenError` (predecessor run, sandbox).
The existing #20863 path then marks the sandbox `replaceBeforeReuse` and
re-queues the run in the uncharged `interrupted_preparation` lane. The
fresh sandbox is used only when the workspace save is clean and the
harness backup is verified (unchanged #20863 safety rules); otherwise
the sandbox is resumed as before.
- A refusal because the new run is no longer admitted (stopped,
reassigned, paused, company inactive) or no longer owned by this
controller stays an ordinary failure, so a fallback cannot become a path
around Stop or reassignment.
- `server/src/services/remote-workspace-unchanged-proof.ts`: a run whose
recorded Runner process identity is exactly a settled predecessor's
identity of the same session (`carriesInheritedRunnerIdentity`, from
#20979), with no `native.process_identity_recorded` event of its own, is
no longer refused with `process_recorded`. A provider session of its own
(`sessionIdAfter`) and every other check still refuse.
- `doc/execution-semantics.md`: document both rules.

## Verification

- Red on `origin/master` (`8818402e7a`):
- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts
-t "crash restart"`: `AssertionError: expected Error:
runner_state_identity_mismatch: pr… to be an instance of
NativePredecessorRetirementUnprovenError`.
- `npx vitest run
src/services/remote-workspace-unchanged-release.test.ts -t "follow-up
that carries"`: `expected false to be true` for the inherited-identity
case; the own-identity and own-record cases passed (still refused).
- Green with the fix: `native-controller-recovery.integration`,
`remote-workspace-unchanged-release`, `native-recovered-owner-followup`
and `heartbeat-native-predispatch-retry` pass (149 tests). Also run:
`native-session-executor.test`, `explicit-native-continuation.test`,
`task-workspace-binding.test`,
`native-runner-restart-recovery.integration.test` (see the PR comment
for the result). `tsc --noEmit` for the server passes.
- The new integration case is the staging shape: a predecessor that
parked on a question (unanswered `semantic_tool.input` before its
terminal), a replacement controller, a resumed sandbox. It checks the
journal refusal, a predecessor row taken by another worker between
inspection and revalidation, a changed current lease, and that a
cancelled current run is not sent to the fallback.
- Chaos: the before/after rerun of seed 7000 is in a PR comment.

## Risks

- The fallback starts the next attempt on a fresh sandbox instead of
reusing the warm Runner. That costs one cold start, only after live
adoption is refused.
- The unchanged-files proof accepts an inherited Runner identity. The
identity must match a settled predecessor of the same session exactly
(PID, group and start time), and the run must have no own identity
record, no execution event, no runner checkpoint, no workspace operation
and no result. These are the same facts #20979 uses for restart
recovery.
- Error text of this refusal gains `; warm_recovery_refused=<reason>`.
Callers match the `runner_state_identity_mismatch: ` prefix only.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:08:03 -06:00
DottaandClaude Opus 5.5 814fe5d20f fix(runner): issue the remote bootstrap ticket at dispatch and retry unauthenticated starts (#21073)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox start `runnerd` in the sandbox, and
Paperclip connects to it through provider ingress and authenticates it
with a one-use bootstrap ticket
> - The ticket lives for 60 seconds, but Paperclip issued it before the
remote launcher staged its files in the sandbox
> - Under load, staging took 60 to 81 seconds, so `runnerd` presented an
expired ticket and every connection was closed. The run failed with
`runner_ingress_auth_failed`, and the task stopped behind a manual Retry
> - This pull request issues a fresh ticket right before dispatch,
retries these failures automatically when the controller checkpoint
proves no runner ever authenticated, and adds spans that attribute the
slow startup window
> - The benefit is that a slow but healthy start authenticates, a failed
start recovers on its own, and the remaining startup delay becomes
measurable

## Linked Issues or Issue Description

**What happened?**

On a hosted instance, 9 runs of one Codex agent on Daytona failed in
about 70 minutes with `native_session_interrupted` and the message
`runner_ingress_auth_failed: Runner ingress did not complete PRP
authentication before the deadline.` In the same window only 6 runs
succeeded company-wide. The run log showed Paperclip reconnecting for 60
seconds while `runnerd` reported that its peer closed each WebSocket. In
294 remote runs with timing spans, the time between the bootstrap ticket
issue (end of `runner.transport.connect`) and the process dispatch
(start of `runner.process.launch`) decides the outcome without
exception:

- 60 s or more: 23 of 23 runs failed authentication.
- Under 60 s: 271 of 271 runs authenticated, in 0 to 1 s.

The tasks did not recover by themselves, because the failure was
classified as ambiguous state.

**Expected behavior**

A runner that starts slowly but correctly authenticates. If ingress
authentication still fails before any runner authenticated, Paperclip
retries the start automatically. The time between the start request and
the runner connection is visible in run spans.

**Steps to reproduce**

1. Run a Codex agent on a Daytona sandbox environment that uses provider
ingress.
2. Make remote staging (`runner.runtime.stage`, inside the remote
launcher) take 60 seconds or more. Many concurrent runs of the same
agent do this on staging.
3. See that `runner.prp.authenticate` fails after 60 seconds and the run
ends with `runner_ingress_auth_failed`.
4. The regression test `runnerd-bootstrap-ticket-refresh.test.ts`
reproduces this locally with a 75-second staging delay.

**Paperclip version or commit**

`4e519bd4` (deployed), reproduced on `origin/master` at `4228bcf89c`.

**Deployment mode**

Hosted (cloud), controller on Railway, sandboxes on Daytona.

**Agent adapter(s) involved**

`paperclip_runner` with the Codex app-server provider (`gpt-6.1-sol`),
remote sandbox target, provider-ingress transport.

## What Changed

- `RunnerProcessLaunchSpec` and `spawnRunner` accept
`refreshBootstrapTicket`. A restart does not inherit it.
- The runnerd Codex transport passes a refresh that issues a new
bootstrap ticket and flushes the controller checkpoint. It does not do
this for a warm-transition launch.
- The remote sandbox launcher calls the refresh after `ensureArtifact`
and immediately before the launch RPC, and puts the new ticket in the
launch environment.
- `inspectUnauthenticatedBootstrapCheckpoint` proves from a controller
checkpoint that no runner authenticated: no connections, no leases, no
used tickets, no events, and only pending commands.
- `nativeSessionFailureSourceCode` classifies the outbound ingress
startup failures as `runner_ingress_startup_failed`.
`nativeProviderRecoveryEvidence` treats it as pre-session
(`bootstrap_retry`) only when that proof holds for the exact run
identity and there is no provider session or provider event. Restart
recovery and failed-run retry accept the new code with the same rules as
similar startup codes.
- Remote startup now records `runner.control_plane.open`,
`runner.launch.prepare`, `runner.checkpoint.register` and
`stage.asset.codex_skills.verify`. The 6 to 11 minute gap between
`native.process_start_requested` and `runner.transport.connect` had no
spans before.

## Verification

- Red/green:
`packages/paperclip-runner/src/live/runnerd-bootstrap-ticket-refresh.test.ts`
starts a real `runnerd` with a 75-second staging delay. Without the fix
it fails with `runner never connected`. With the fix it opens the
session.
- Red/green: `native-session-executor.test.ts` "launches runnerd with a
bootstrap ticket issued after remote staging". Without the fix the
launch RPC gets the stale ticket.
- Red/green: `native-controller-recovery.integration.test.ts` "retries a
fresh remote start whose launched runner never authenticated through
provider ingress" (embedded Postgres). Without the fix the failure is
`native_session_interrupted` and `ambiguous_state`.
- `durable-prp-control-plane.test.ts`: the new checkpoint proof, its
negative cases, and the restart rule for the refresh.
- `native-run-trace.test.ts`: remote startup spans are recorded without
duplicate launcher spans.
- Focused suites pass: runner package 5 files and 235 tests; server 9
files and 1,607 tests, including `native-session-executor`,
`native-runner-execution-driver`, `remote-launch-assets`,
`native-restart-recovery`, `runner-prp-outbound` and the recovery
integration tests.
- `tsc --noEmit` passes for `packages/paperclip-runner` and `server`.

## Risks

- A launch now issues two bootstrap tickets. The first one is never sent
to `runnerd` and expires in 60 seconds. Bootstrap counters already allow
more than one issued credential.
- The new retry path needs a checkpoint proof that no runner
authenticated, so a run that did provider work cannot be retried this
way. The existing attempt budget still bounds the retries.
- This change does not remove the 6 to 11 minute controller-side delay
before `runner.transport.connect`. The new spans make it attributable
for a follow-up change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, code search) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:02:30 -06:00
DottaandClaude Opus 5.5 f30005440b fix(recovery): reconnect a live remote Runner with its admitted skills instead of re-downloading them (#21064)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner sandbox. A server upgrade
detaches each running turn, and the new server must reconnect to its
Runner (restart recovery).
> - A reconnect runs the full turn preparation before it contacts the
Runner. On a new server, skill preparation downloads every saved skill
version again, under one lock for the whole company.
> - After an upgrade, many reconnects wait behind each other's
downloads. The Runner stops answering after about 2 minutes, so a late
reconnect loses the turn.
> - This pull request lets a reconnect to the same run's live remote
Runner use the exact saved skill versions, without downloading them and
without the lock.
> - The benefit is that skill preparation no longer adds to the
reconnect delay, and it does not grow with the number of reconnects.
Normal turn starts do not change.

## Linked Issues or Issue Description

**What happened?**
After an automatic staging upgrade, runs that the new server reached
late failed. The measured time from detach to reconnect was 114–152 s
for the failed runs. Turn preparation took 30–130 s, and skill
preparation alone took 46 s on one run.

**Expected behavior**
A reconnect to a live Runner reaches it quickly. Its speed must not
depend on how many other runs prepare skills at the same time.

**Steps to reproduce**
1. Run a remote Runner turn for each of many tasks in one company.
2. Upgrade the server while those turns run, so the new server starts
with an empty skill cache.
3. See that each reconnect waits in
`heartbeat.prepare.configuration_skills` for the company skill cache
lock and downloads the skills again.

**Paperclip version or commit**
master at 4228bcf89c

**Additional context**
Root cause: `withRuntimeSkillEntries` holds the company-wide skill cache
lock (a Postgres advisory lock) while it materializes each saved skill
version. A new server has an empty skill cache, so each reconnect
downloads its skills again, one run at a time. The Runner already has
these files, and the reconnect path does not read them.

## What Changed

- `company-skills.ts`: new `admittedVersionsOnly` option. It still
checks that each saved version exists and that the agent can read the
skill. It does not materialize the files or take the company skill cache
lock.
- `heartbeat/live-runner-adoption.ts`: `adoptsLiveRemoteRunner` is true
only for a `reattach_remote_runner` claim for this run, with the claim's
own lease owner, on a native run.
- `heartbeat/execution-configuration.ts`: uses the option only when all
of these are true: the reconnect gate is true, the run has its saved
native input with a runtime context, and the run has its saved skill
versions. In that case it times the phase as
`configuration_skills_adopted` and adds one lifecycle run event
(`skillPreparation: "reused_admitted"`).
- `heartbeat.ts`: passes the gate to execution configuration.
- `native-runtime/runtime-context.ts` and `native-session-executor.ts`:
on the same reconnect to a remote Runner filesystem, the Runner
controller describes the Runner's skills by their admitted digests. It
rehydrates the pinned skill files (which takes the same lock) only if it
must launch a Runner process and stage them. All other starts rehydrate
as before.
- `doc/execution-semantics.md`: describes the new behavior.

Not in this PR (by design): a separate "dispatch-only" reconnect path,
and a new startup scheduling lane. The native executor needs about 40
values from preparation (runner environment, MCP token, managed AI
credentials, the reattached execution target, run output). Connector and
instruction preparation also feed the session fingerprint. That cut is
not mechanical, so this PR removes the largest measured single cost
only.

Other `heartbeat.prepare.*` costs that stay on the reconnect path (use
post-upgrade timings to choose the next cut):
- `configuration_secrets`, `configuration_connector_assignments`,
`configuration_connection_instructions`,
`configuration_connector_skills`/`_delivery`,
`configuration_agent_identity`, `configuration_ai_runtime`
- `environment_selection`, `workspace` (retained lease resume and
realization), `session`, `task_context`, `run_identity`
- `github_environment`, `github_launchers`, `run_output`,
`runtime_services`, `instructions`, `runtime_selection` (including
`runtime_selection.skill_assets` and `runtime_selection.workspace_sync`)
- The wait from detach to "run started" (queue and controller-preparing
waits) is outside `heartbeat.prepare.*`.

## Verification

- New test `reconnects to a live remote Runner with its admitted skills
without re-downloading them or waiting on the cache`
(`server/src/__tests__/heartbeat-execution-configuration.test.ts`):
- It admits a run, deletes its skill cache (like a new server), and
holds the company cache lock.
- The reconnect preparation finishes while the lock is held. It returns
the same key, runtime name and version. The session fingerprint is
identical. The cache is not created again.
- Normal preparation of the same saved selection still waits for the
lock and materializes the files ("normal start unchanged").
- Red on master source: `A live Runner reattach must not queue behind
skill downloads: expected null not to be null`. Green with this change.
- New test in `managed-skill-content.test.ts`: with the cache lock held
and an empty cache, `prepareRuntimeContextSource` for a reconnect
returns at once with the admitted context. Its staging step and a normal
start still wait for the lock and then rehydrate the same digests.
- New `server/src/services/heartbeat/live-runner-adoption.test.ts`: the
gate is false for a normal start, a scheduled resume, a dead-Runner
resume, an incomplete bootstrap, another run's claim, a different lease
owner, and a legacy run.
- `npx vitest run
src/__tests__/heartbeat-execution-configuration.test.ts
src/__tests__/skill-version-cache.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/heartbeat/live-runner-adoption.test.ts`: 289 passed, 14
skipped. After the controller change: restart-recovery, diskless
restart, runtime skill preservation, managed skill content, session
executor and execution configuration suites: 1686 passed, 14 skipped.
- `tsc --noEmit -p server/tsconfig.json`: pass.
- `pnpm verify` asks for broader coverage (exit 2). Required CI supplies
it.

After deploy, a reconnect run shows
`heartbeat.prepare.configuration_skills_adopted` and the
`reused_admitted` event, and no `skills.cache.acquire` or
`skills.version.materialize` phases.

## Risks

- Low. The option applies only to a same-run reconnect of a retained
remote Runner that has a saved runtime context and saved skill versions.
Every other start keeps the current path.
- Access revocation and deleted versions still fail the reconnect, as
before. A failure to download the files on this server no longer fails a
live reconnect, because the Runner already holds them.
- Nothing new is saved, so rows from older releases, two live servers,
and a deploy during a reconnect behave as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code search, editing, shell, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:58:01 -06:00
DottaandClaude Fable 5.2 fd9985b571 fix(native): say what a Runner command waited behind when it times out (#21056)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes talk to a Runner through durable PRP
commands. The controller waits for each command's result
> - After a server replacement, the new controller reattached to live
Runners. For runs it reached later than about 115 s after detach,
`session.snapshot` got no answer within 30 s
> - The failure said only `provider_transport_failed: PRP command
session.snapshot timed out`. The run log could not tell whether the
Runner was disconnected, whether the command waited behind older
commands, or when the Runner last sent an event
> - This pull request adds that evidence to the timeout message. It
names types, counts and durations only, never payloads
> - The benefit is that the next upgrade's failures explain the stall,
so the runner-side cause can be fixed

## Linked Issues or Issue Description

Refs #21053, #21052, #21039

**What happened?**
At the 07:10Z staging swap, 9 remote native runs that the new controller
reached 114–152 s after detach failed. Each recovery read ended with
`PRP command session.snapshot timed out`. The transport had
authenticated, and in some runs the same Runner process was verified.
The Runner journal recorded no diagnostic other than "WebSocket peer
closed the connection". Runs reached within 39–116 s succeeded.

**What did you expect to happen?**
A command timeout should say what the command waited behind, so the
cause can be found from the run log.

**Steps to reproduce**
1. Start a remote native run and replace the server while a provider
turn is active.
2. Let the new controller reach the Runner more than about two minutes
after detach.
3. Read the run log. The failure says only "timed out".

## What Changed

- `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`:
`waitForRunnerCommand` accepts an optional `describeStall`. When a
command times out, the message ends with a bounded description, for
example `(runner connected; pending ahead 1 [semantic_tool.result]
oldest 188s; last runner event seq 152 181s ago)`. The description
covers whether a Runner is connected or how long it has been
disconnected, the pending commands ahead of this one (types and count,
up to 6 types, and the age of the oldest), and the last committed Runner
event's sequence and age. A failing or empty description keeps the plain
message.
- The transport passes this description for every command it waits on.
The last event's age comes from the PRP payload's `emittedAt`.
- `server/src/services/native-runtime/native-session-executor.ts`: the
startup `session.open` timeout classifier accepts the optional
description, so a startup timeout stays `provider_session_open_timeout`
and keeps its bounded same-run retry. A regression case covers each
startup code with the suffix.
- Tests: the exact message for the staging shape, the plain message when
there is nothing to describe or the description throws, and a
disconnected Runner with no events.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/live/runnerd-codex-transport.test.ts -t "a Runner command that gets
no answer"` passes 3 of 3. These tests fail on `origin/master`, which
has no `describeRunnerCommandStall` export and no description in the
message.
- The full `runnerd-codex-transport.test.ts` file passes 307 of 308. The
one failure (`stops a lost runnerd's provider and escaped tool job from
its lineage journal`) is a process-timing test that passes alone and
does not touch this code.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes.
- Message matchers: the only matcher anchored on the end of the timeout
text was the startup `session.open` classifier. It now accepts the
suffix. `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"classifies a startup session.open timeout"` passes. Without the
classifier change it fails, because the suffix turns the timeout into
`native_session_interrupted`.

## Risks

- Low. Only the text of a timeout message changes, by appending text. No
control flow, persisted state or Runner behavior changes.
- The description holds command types, counts and durations only. It
holds no payloads, identifiers or provider text.
- No wait is added before a turn start.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, with tool use (shell, file edits,
test runs, read-only staging API reads). Model ID as attributed in the
commit trailer: Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 03:00:32 -06:00
DottaandClaude Opus 5.5 4e519bd428 fix(chat): send writes with an empty responseText and point card declines to resolve-from-comment (#21050)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents call the Paperclip API through `call_api` and find
routes with `search_api`
> - #21035 added a pre-send refusal for writes that carry
`responseText`, and new summaries for the question cancel and withdraw
routes
> - Full chat cohort 5 (OpenCode with DeepSeek v4.1 flash) showed that
both changes went too far
> - The refusal blocked a valid write that had an empty `responseText`,
and the summaries hid the route that records a person's decline
> - This pull request narrows the refusal and corrects the guidance text
> - The benefit is that valid writes go through on the first try, and
agents settle a declined card instead of leaving it pending

## Linked Issues or Issue Description

Refs #21035, #21002

**What happened?**
- inbox-search.restore-archive: DeepSeek sent `responseText: {}` with a
valid body on `POST /api/issues/{id}/inbox-archive`. The refusal from
#21035 rejected 24 POSTs in 3 minutes. The model did not drop the key,
and the run was cancelled at the 240s window. In cohort 4 the same POSTs
went through on the first try.
- questions-and-approvals.reject-and-revise: the person said "Do not
publish it; just make a draft I can review." The CEO quoted the cancel
summary ("an agent withdraws its own pending card with POST
.../withdraw"), saw that the person created the publish card, and left
the card pending. The plan decision did not settle. In cohorts 3 and 4
the CEO declined the card with `resolve-from-comment` (resolver policy
`anyone`).

**Expected behavior**
- A write with an empty or default `responseText` is sent once. A write
whose `responseText` sets a window is still refused before it is sent.
- The guidance says that withdraw is only for the agent's own obsolete
card, and that a person's explicit decline or answer goes through
`resolve-from-comment` when the card's resolver policy allows.

**Steps to reproduce**
Run the two chat cases above with OpenCode and DeepSeek v4.1 flash on
master.

## What Changed

- `call_api` (`assertRunnerApiDispatchable`): the check reads the raw
`responseText` value. `{}` or `{ offsetBytes: 0 }` on a write is dropped
and the write is sent. A `responseText` with `limitBytes` or a later
offset is still refused. `saveResponse: false` on a write was already
accepted; a test now covers it.
- OpenAPI: the cancel and withdraw summaries say that withdraw is only
for the agent's own obsolete card, and that a decline or answer on the
person's explicit instruction uses `POST .../resolve-from-comment` when
the resolver policy allows. The `resolve-from-comment` summary says it
also records a decline of a card the person created.
- `AGENT_CHAT_DIRECTIVE`: a person's explicit decline in chat answers a
pending confirmation with reject, whoever created the card. Withdraw
never records the person's decision.
- `doc/runner-api-tools.md`: describes the narrowed refusal.

## Verification

- `server/src/services/native-runtime/runner-api.test.ts`: new test
sends `POST /api/issues/{id}/inbox-archive` with a valid body and
`responseText: {}`, `{ offsetBytes: 0 }`, `saveResponse: false`, or
both. The request is sent exactly once. Red before the fix (3 of 4 cases
refused), green after. The refusal test for a `responseText` with
`limitBytes` stays.
- `server/src/__tests__/openapi-routes.test.ts`: new test for the
cancel, withdraw and resolve-from-comment summaries. Red before, green
after.
- `server/src/__tests__/agent-conversations.test.ts`: new directive test
for the decline rule. Red before, green after.
- Also passed: `runner-api.integration.test.ts`,
`native-case-tools.integration.test.ts`, server `tsc --noEmit`. SDK
public input pins: `--check` reports no change.
- Paid run (OpenCode + DeepSeek v4.1 flash) on this branch for
inbox-search.restore-archive, questions-and-approvals.reject-and-revise,
gateways-profiles.rotate-token and budgets-costs.repair-cost-recording:
4 of 4 cells passed on 3f20fb2bcc (run 43177772183); details in a PR
comment.

## Risks

- A write with a populated `responseText` is still refused, as in
#21035. Only the empty and default shapes change. A dropped empty
`responseText` changes nothing, because the server already removed it
from writes before sending.
- The directive grows by two sentences, which adds a small number of
prompt tokens per chat turn.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 02:09:14 -06:00
DottaandClaude Opus 5.5 0dfbbdeb31 fix(chat): route company creation, decline retries and question cancellation correctly (#21035)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents follow the chat directive and discover API routes with
search_api and call_api
> - Full chat cohort 4 (OpenCode with DeepSeek v4.1 flash) showed four
guidance gaps
> - The missing-name rule overrode the board-only company-creation
boundary; a repeated request after a decline did not post a new approval
card; search_api showed the board-only question cancel route as open to
agents (and withdraw as board-only); write calls carried `responseText`
beside or instead of `body`
> - This pull request fixes the guidance, the route authorization
metadata and the pre-send refusal
> - The benefit is that agents take the supported path the first time
instead of asking for the wrong input, refusing a valid retry, or
calling a route that returns 403

## Linked Issues or Issue Description

**What happened?**
- onboarding.new-company: asked to "set up a new company for my website
business", the CEO cited the missing-name rule (#21007) and asked for a
business name with a question card. Creating a company is a board
action, so the agent should send the person to the creation flow.
- app-permissions.ask-before-action: the person declined a write card
and then repeated the same request. The CEO asked the person to tell it
to resubmit instead of posting a new approval card. The gateway already
allows a fresh request after a new human message
(`isNewHumanIntentAfterToolRejection`), but the decline text said "Do
not retry the same call".
- questions-and-approvals.multi-part-question: search_api listed `POST
/api/issues/{id}/interactions/{interactionId}/cancel` as
`board_or_agent`. The route rejects agents. The agent route,
`/withdraw`, was listed as board-only although its route admits the
card's creator agent and the task's assignee.
- Several cases: write calls through call_api carried `responseText`
with no body (for example "Received arguments: operationId, pathParams,
responseText").

**Expected behavior**
The agent links the company-creation flow; it posts a fresh approval
card for a repeated request; search_api shows the true authorization of
cancel and withdraw; a write with `responseText` is refused before it is
sent, with the same guidance #21002 gave `saveResponse`.

**Steps to reproduce**
Run the four chat cases above with OpenCode and DeepSeek v4.1 flash on
master.

Refs #21002, #21007

## What Changed

- `AGENT_CHAT_DIRECTIVE`: the missing-name rule applies to work the
agent creates itself and never comes before a board-only boundary. For a
new company the agent links [Create organization](/onboarding) (no
company prefix) and repeats the business details. A repeated request
after a declined approval card is an explicit retry: make the same call
so a fresh card is posted. A clarification question card puts candidate
links in its description, because labels show as plain text.
- Decline instructions in `tool-action-delivery.ts` and
`tool-gateway.ts` say that a later request from the person for the same
action is an explicit retry.
- OpenAPI: `POST /api/issues/{id}/interactions/{interactionId}/cancel`
is board-only and its summary names the `/withdraw` route for agents.
`/withdraw` is no longer marked board-only, which matches its route.
- `call_api`: a new write that carries `responseText` is refused before
it is sent; the message says `responseText` never carries request fields
and names the allowed body fields. Older saved calls still validate for
recovery. `doc/runner-api-tools.md` is updated.

## Verification

- `server/src/__tests__/agent-conversations.test.ts`: new directive
tests for the board-only boundary order, the decline retry and card
links.
- `server/src/__tests__/openapi-routes.test.ts`: cancel is `board`,
withdraw is `board_or_agent`, the cancel summary names withdraw.
- `server/src/services/native-runtime/runner-api.test.ts`: a write with
`responseText` is refused before sending (three window shapes), the same
write without it is sent once, and saved calls still validate.
- `tool-gateway.test.ts`, `tool-gateway-service.test.ts`,
`issue-thread-interaction-routes.test.ts`,
`tool-action-replay-intent.test.ts`: passed. `server` `tsc --noEmit`:
passed.
- Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) together with the
eval PR: see the PR comment for results.

## Risks

- A model that used to send a write with `responseText` and a valid body
now gets one refusal and must resend without it. Nothing is sent before
the refusal, so there is no duplicate side effect.
- The directive grows by a few sentences, which adds a small number of
prompt tokens per chat turn.
- Agents that read `/withdraw` as board-only before now see that they
can withdraw their own card; the route already enforced who may.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:27:35 -06:00
DottaandClaude Opus 5.5 aa94939988 fix(native): continue a run whose remote Runner died mid-turn after a restart, without replaying the turn (#21039)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a server restart, recovery reattaches to that
Runner on the run's own sandbox lease
> - On staging, a deploy detached the controller and left the Runner
alive. The restart's reattachment then found the Runner process gone
with a provider turn in flight. The run failed with
`runner_remote_process_identity_unavailable` and the task waited for a
person
> - The run log said `startup reuse: environment (fresh) created`, so
the sandbox looked re-created. It was not. The reattachment resumed the
exact retained lease. The receipt reported the lease's original creation
> - `requestInterruptedNativeTurnRetry` refused every unsettled turn
(`provider_turn_unsettled`), and refused every system wake unless the
journal proved that no turn was issued. So this run got no automatic
continuation
> - This pull request continues such a run once in a fresh provider
session when the refused reattachment proved the Runner process gone. It
records that the turn's outcome is unknown and never replays the turn.
It also makes the environment receipt tell the truth
> - The benefit is that a restart that kills a Runner in the middle of a
turn no longer needs a Retry click

## Linked Issues or Issue Description

Refs #20853, #20923, #21023, #20278, #20627, #20828

**What happened?**
A staging deploy landed in the middle of a remote native Codex run on
Daytona. Shutdown detached the controller and logged "durable runner
left alive". The new server reattached through the run's original lease
(same lease row, same provider sandbox id, same directory). Its process
check then found the Runner gone: `process_verification` /
`process_not_running`, `runnerLifecycle: ready`, `runnerSettled: true`,
last diagnostic "WebSocket peer closed the connection", empty stderr.
The provider checkpoint still had an active turn. The run failed with
`runner_remote_process_identity_unavailable` /
`native_continuation_requires_reconciliation`, and the task went to
`blocked`.

**Root cause**
- `server/src/services/heartbeat/restart-interrupted-turn.ts`
(`requestInterruptedNativeTurnRetry`): an unsettled last turn always
returned `provider_turn_unsettled`, also when the refusal proved the
Runner process gone. A system wake (here `transient_failure_retry`)
continued only with `providerTurnIssued: false`.
- The sandbox was not re-created. The environment and lease listings
show one lease for the run, never replaced, and the final workspace
snapshot used the same provider sandbox. `environmentStartupReuse` read
`sandboxLeaseAcquisition: created` from the lease's first acquisition,
so the reattachment's receipt said `fresh / created`. The 46 s acquire
was the provider resume of that exact lease. The evidence available to
GET requests cannot say what stopped the Runner process (the resume
result is not kept). This pull request does not change that path.
- #21023 named this run as not covered. #20853 resumes a Runner only at
a settled, idle `ready` cut. Here the provider checkpoint has an active
turn, so #20853 stays fail-closed, as it should.

**Expected behavior**
The run continues once by itself in a fresh provider session over the
saved work. The turn whose outcome is unknown is recorded and is not
replayed.

## What Changed

- `restart-interrupted-turn.ts`: read the last
`native.remote_runner_recovery_failed` refusal once. When it proves the
Runner process gone, an unsettled last turn no longer blocks the
automatic Retry. Before it requests the Retry, the server records
`native.interrupted_turn_outcome_unknown` once (turn id, last turn
event, `replayed: false`, closed `runnerExit` fields). The Retry is the
existing one: saved-work recovery and stop proof first, then a fresh
provider session (`forceFreshSession: true`), once per chain.
- The same file: a system wake whose run is an automatic retry of owed
work (`retryOfRunId` set) also continues under the run's responsible
user once the refusal proved the Runner gone. Other system wakes (for
example a monitor) keep their policy.
- `packages/adapter-utils/src/startup-reuse.ts` and
`native-session-executor.ts`: a `reattach_remote_runner` recovery
reports its `environment` receipt as `reused / verified_reuse`. That
path only resumes the run's exact retained lease and never acquires one.
- Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`.

Previous-release story: no persisted shape changes. The rule reads the
closed `remoteRunnerRecovery` fields that the previous release already
writes; the new test uses the staging refusal payload as written. A run
that a previous release already failed this way continues on the next
completion or restart hook call only; runs already settled stay as they
are and keep their Retry. The new event is additive.

## Verification

- Red on `origin/master` (51d77648bc), with only the new tests: `cd
server && npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts` fails 4 of 6.
The three continuation cases return `{ requested: false, reason:
"provider_turn_unsettled" }`, and the monitor case returns that reason
instead of `not_user_request`.
- Green with the fix: 6 of 6. The cases cover a user's assignment
through the restart claim, the staging shape (system
`transient_failure_retry` with `retryOfRunId`), and a same-run retry.
Each one checks one recorded unknown-outcome event, an idempotent second
call, saved-work recovery first, one queued successor with
`forceFreshSession: true`, and one Retry intent. Negative cases: a
refusal that did not prove the process gone, no refusal, and a monitor
wake. Each keeps the visible Retry and records nothing.
- `cd packages/adapter-utils && npx vitest run
src/startup-reuse.test.ts`: the new receipt case fails before (`fresh /
created`) and passes after.
- Focused suites pass: `workspace-recovery-retry-copyback.test.ts`,
`governed-wait-runner-loss.test.ts`,
`heartbeat-native-remote-failure-retention.test.ts`,
`native-runner-restart-recovery.integration.test.ts` (302 passed, 14
skipped), and `native-session-executor.test.ts -t "startup
reuse|restart|reattach"` (149 passed).
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/adapter-utils typecheck` pass.
- `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global
CI input. CI supplies that coverage.
- Not run: a live Daytona restart. The change is in the server's
recovery decision; the tests use the fixture's sandbox confinement.

## Risks

- Medium. A run whose Runner died in the middle of a turn now continues
by itself. The turn's outcome is unknown, so tool work from that turn
can be partly done. The successor is a fresh session over the saved
workspace; it does not replay the turn. Saved-work recovery stops and
keeps the sandbox before the successor starts, so the old provider
cannot still run.
- Without positive proof that the Runner process is gone, an in-flight
turn keeps the visible Retry, as before.
- An automatic retry of owed work now continues after a lost Runner.
Monitor and other system wakes without `retryOfRunId` do not. The bound
is still one automatic Retry per chain.
- The environment receipt changes only for the reattachment path. No
execution decision reads it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:13:25 -06:00
DottaandClaude Opus 5.5 940d681974 fix(heartbeat): hold a queued native run while its task's previous session finishes cleanup (#21031)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner runs keep a provider session per task. When a run
stops, its session close continues in the server process
> - The runner refuses every new session of the task until that close
settles. A close that cannot prove the old runner suspended needs a
stopped-process proof before it is released
> - A reassignment stops the old assignee's run mid-turn. The new
assignee's run was dispatched one second later into that gate, and it
failed with `native_session_cleanup_quarantined`
> - Only a Retry continued the task. Recovery must work without a
person, and the needed-retry rate must be 0%
> - This pull request keeps such a run queued until the close settles or
its old owner is proven stopped
> - The benefit is that the new run starts by itself after cleanup,
without a failed run or a Retry

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `tasks.edit-properties` (and
`tasks.assignment-conflict` in an earlier cohort), the CEO reassigned a
task while the old assignee's native run was mid-turn, with a Paperclip
tool call in flight. The reassignment cancelled that run. Its session
close then failed with `native_session_close_unrecoverable: runner did
not durably suspend before checkpoint`, and the runner kept an
operator-required cleanup gate for the task. The new assignee's run had
started 1 second after the cancel. The runner's admission waited for the
in-flight close. When the close escalated, admission refused the run
with `native_session_cleanup_quarantined`. No run was queued again.

**Steps to reproduce**
1. Run a native task and stop its run while its session close cannot
settle at once.
2. Queue a run of another agent for the same task.
3. The run is dispatched into the cleanup gate and fails before it
launches.

**Expected behavior**
The run waits for the cleanup, and starts once the close settles or the
old owner is proven stopped. It never fails, and no Retry is needed.

## What Changed

- `server/src/services/native-runtime/native-cleanup-admission-hold.ts`:
`nativeCleanupHoldsAdmission` reads the task's cleanup gate. For an
operator-required gate it first tries
`retireStoppedSessionCleanupGates`, which retires the gate when the old
owner's runner and provider processes are proven stopped. This is the
same proof that Retry uses.
- `server/src/services/heartbeat/queue.ts`: `claimQueuedRun` keeps an
unstarted `paperclip_runner` run queued while its task has a settling or
unproven close. This matches the existing
`predecessor_settling_in_process` hold, which covers only the same
agent. A reassignment changes the agent.
- `server/src/services/native-runtime/native-cleanup-release-wake.ts`: a
released close now also wakes queued-run dispatch. A close that
escalates to operator-required also wakes it, so admission tries the
stopped-process proof at once. The queued-run liveness redrive covers
any later proof.
-
`server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts`:
an assembled heartbeat test. It is red on master (the run is claimed
into a pending close and into an operator-required close). It is green
here (the run stays queued, then is admitted after the close settles). A
control case shows that a run for another task is not held.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts` (3
passed; 2 red on master)
- `cd server && npx vitest run
src/services/quarantined-task-retry.test.ts
src/services/heartbeat/queue.test.ts
src/__tests__/heartbeat-workspace-busy.test.ts
src/services/queued-run-dispatch-liveness.test.ts` (180 passed)
- `cd server && npx tsc --noEmit` passes.
- Paid E2E run on a combined branch:
https://github.com/paperclipai/paperclip/actions/runs/43161593494
(results in a comment below).

## Risks

- A run whose task has an operator-required gate that is never proven
stopped now stays queued instead of failing. Before, that run failed,
and Retry was blocked by the same proof. The liveness redrive re-checks
it with back-off.
- The hold applies only to unstarted `paperclip_runner` runs of the
gated task. Other tasks and agents are not affected.
- Previous-release saved state: no persisted shape changes. The gate
lives in process memory, and runs saved by the previous release are
claimed by the same rule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:53:54 -06:00
Nicky Leach d68088465c Reuse retained Runner descriptions during startup (#21004) 2026-11-11 21:20:08 -08:00