Commit Graph
4030 Commits
Author SHA1 Message Date
DottaandClaude Fable 5.2 39ef43a73e fix(retry): move a recorded continuation forward before its delivery (#21379)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run cannot resume, the server records a continuation for the
task and delivers it later as a new run
> - #21378 made a person's Retry deliver a recorded, undelivered
continuation as the one run, with its workspace isolation
> - #21373 added the `move_forward` Retry action, which does nothing
when the observed run is not queued or running
> - Together, a `move_forward` request before delivery returned "already
moved on" and started no work
> - That state can last a long time when the agent has a deep queue, and
it is when a person wants to move the task forward
> - This pull request accepts the request when the task has a recorded,
undelivered continuation
> - The benefit is one action that moves waiting work forward in each
state, with one run and the recorded isolation

## Linked Issues or Issue Description

**What happened?**

A task has a continuation that is recorded but not delivered. The latest
run is the failed source run. A Retry request with `retryAction:
"move_forward"` returns `accepted: false, alreadyMovedOn: true`. No run
starts.

**Expected behavior**

The request is accepted. Wake admission delivers the recorded
continuation as the one run for the task. The run keeps the recorded
isolation. A later delivery pass creates no second run.

**Steps to reproduce**

1. Record a fresh continuation for a task and do not run the delivery
pass.
2. Call `requestTaskRetry` with `observedRunId` set to the source run
and `retryAction: "move_forward"`.
3. The receipt says `alreadyMovedOn: true`.

Refs #21373, #21378, #21348.

## What Changed

- `admitTaskRetry` in `server/src/services/task-retry.ts` does not
return `alreadyMovedOn` when `undeliveredContinuation` finds a recorded,
undelivered continuation for the task. The check runs under the task row
lock.
- A new test in `fresh-continuation-retry-delivery.test.ts` covers the
three record kinds: `fresh_isolated_workspace`, `writer_ended` and
native.
- The presentation code is unchanged. The interface does not offer the
action in this state yet.

## Verification

- The new test failed before the change for the three kinds (`expected {
requestId: null, … } to match object { accepted: true }`).
- `npx vitest run
src/services/fresh-continuation-retry-delivery.test.ts`: 15 of 15
passed.
- `npx vitest run src/services/task-retry.test.ts`: 117 of 117 passed.
- `npx vitest run src/__tests__/recovery-wake-herd.test.ts -t move`: 2
of 2 passed. The case where the run ended and no continuation is pending
still returns `alreadyMovedOn`.
- `npx tsc --noEmit -p .` in `server/`: no errors.

## Risks

Low risk. The change adds one read under a lock that the function
already holds. The read runs only for a `move_forward` request when the
observed run is not queued or running. A request without a pending
continuation behaves as before.

## Model Used

Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 07:31:16 -06:00
DottaandClaude Opus 5.5 ed16a5b1a8 fix(recovery): do not continue from a latest run read while it was still finishing (#21365)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent's run ends without a live next step, a background
sweep finds the stranded task and queues a continuation
> - The sweep reads the task's latest run first, and checks for a live
execution path several queries later
> - A run that finishes between those two reads looks neither live nor
failed. The sweep fell through to its generic continuation
(`issue.continuation_recovery`) with the stale "running" snapshot
> - In a chat, that woke the agent about a second after its turn had
answered the person, and the agent posted a redundant "nothing new is
pending" reply
> - This pull request makes the sweep act only on a latest run it
observed as finished. A run it read while still active is left to its
own completion and to the next sweep
> - The benefit is that a chat turn that already answered is never woken
again by recovery, with no change to real recovery

## Linked Issues or Issue Description

**What happened?**
In a chat, the person asked a follow-up and the agent answered. The
answering run's success was written at 10:09:28.840. A continuation run
was queued at 10:09:28.841 with `wakeReason: issue_continuation_needed`,
`source: issue.continuation_recovery`, and `retryOfRunId` set to the run
that had just succeeded. The agent ran again and posted an extra reply.
The generic continuation source is only reachable when the sweep's
latest-run snapshot is neither succeeded nor live, so the snapshot was
read while the run was still `running`, and the run had finished by the
time of the live-path check.

**Steps to reproduce**
1. Have an assigned `in_progress` task whose latest run is `running`.
2. Start the stranded-work sweep. It reads the latest run as `running`.
3. Before the sweep's live-path check, let the run finish with
`succeeded` and release the task lock.
4. The sweep finds no live path and queues `issue.continuation_recovery`
for the finished run.

**Expected behavior**
The sweep queues nothing for a run it read while still active. Its own
completion settles it, and the next sweep decides from the committed
outcome.

## What Changed

- `server/src/services/recovery/service.ts`: after the live-path check,
the stranded sweep skips a non-review task whose latest-run snapshot was
not terminal. Review tasks keep their participant-run check, which
already does this.
- `server/src/services/recovery/service.ts`: an optional test hook,
`afterStrandedLatestRunRead`, runs after the sweep reads the latest run,
alongside the existing `beforeStaleLockClear` and
`beforeOrphanedRunTerminalWrite` hooks.
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: a
regression test finishes the run between the two reads and expects no
continuation.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "still finishing"`
fails without the fix (one `issue.continuation_recovery` wake with
`retryOfRunId` set to the finished run) and passes with it.
- `cd server && npx vitest run
src/__tests__/heartbeat-process-recovery.test.ts
src/__tests__/issue-recovery-actions.test.ts
src/__tests__/heartbeat-issue-liveness-escalation.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/durable-chat-wakeup.test.ts
src/__tests__/live-recovery-work.test.ts`: 790 passed. One unrelated
test timed out under parallel load and passes on its own (its latest run
is already terminal, so the new check does not apply).
- `cd server && npx tsc --noEmit` passes.

## Risks

- Behavior: for any task whose latest run is `queued`, `running` or
`scheduled_retry`, the live-path check already skipped it, so the only
changed outcome is the race above. A run stuck in `running` with no
owner is still settled by the existing orphaned-run reaper; the sweep
then sees its terminal status.
- Latency: the check is in memory on an already-read row. It adds no
query and nothing before a turn starts; recovery after a real stall is
at most one sweep later only inside the race window.
- Saved state: no persisted shape changes, so old saved runs and mid-run
deploys behave as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:28:44 -06:00
DottaandClaude Fable 5.2 3f030971dc fix(recovery): deliver a recorded continuation as the person's Retry run, keeping its isolation (#21378)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When Paperclip cannot resume a cut-off run exactly, the
fresh-continuation fallback records a continuation for the task. A
delivery pass later starts it as a run. Staging shows this pending state
can last 30 to 110 minutes for backlog items.
> - The fallback also decides how the next run is isolated from the old
writer. For `fresh_isolated_workspace`, the next run must prepare a new
workspace.
> - A person's Retry pressed while the continuation was recorded but not
yet delivered started its own run at once. That run did not carry the
recorded isolation, so it could use the shared directory next to a
writer that is not proven stopped. It also did not carry the
continuation's recovery context.
> - This pull request makes wake admission deliver the recorded
continuation as the person's run, under the task lock. It also folds a
saved Retry into a delivered continuation for every record kind.
> - The benefit is one run per interruption, whatever arrives first, and
the recorded isolation decision is never lost.

## Linked Issues or Issue Description

Refs #21348

**What happened?**
A person's Retry (`requestTaskRetry` + `resumeTaskRetry`), pressed while
a fresh continuation was recorded but not delivered, was admitted at
once as an ordinary `task_retry` run. That run had no
`freshContinuationIsolation` and no `recoveryActionId`. On a projectless
task the run used the shared directory that the fallback had decided to
isolate from. The later delivery pass marked the continuation as
superseded by that run. The reproduction showed exactly one run, so the
reported "two runs" did not happen in that order. The isolation was lost
for all three record kinds tested. For `writer_ended` and native
records, a Retry saved before delivery was not folded into the delivered
continuation, because the fold only covered `fresh_isolated_workspace`.

**Expected behavior**
With a recorded, undelivered continuation, the next run admitted for the
task is that continuation. It has the recorded isolation and the
person's Retry is recorded as answered by it.

**Steps to reproduce**
1. Record a fresh continuation (`ensureFreshContinuation`, delivery
`recorded`) for a projectless task whose legacy writer is on another
host.
2. Call `requestTaskRetry` and then `resumeTaskRetry`.
3. Before this change: one `task_retry` run without
`freshContinuationIsolation` or `recoveryActionId`, while the record is
still `pending`.

## What Changed

- `fresh-continuation-retry-fold.ts`:
- `undeliveredContinuation` reads the task's pending fresh-continuation
record under the task lock (`for update`). It returns nothing when a run
of the task was created after the record.
- `undeliveredContinuationContext` gives the run the continuation's
`recoveryActionId`, `retryOfRunId`, `forceFreshSession` and, for
`fresh_isolated_workspace`, `freshContinuationIsolation`.
- `deliverContinuationAsRun` marks the record `delivered` to the new run
in the same transaction, settles its coalesced holds, and folds other
waiting Retries.
- `foldWaitingRetriesIntoRecordedContinuation` folds a waiting Retry
into any delivered fresh continuation, not only an isolated one.
- `heartbeat/queue.ts` (wake admission, issue transaction): a
non-system, non-delivery wake with no execution blocker takes the
pending continuation. The run is inserted as before, then delivered as
the continuation. The fold after a delivery wake now applies to every
fresh-continuation kind. No change to ranking or to the claim loop.
- `execution-recovery-resolution.ts`: the delivery pass folds a saved
Retry for every fresh-continuation kind. The superseded-run fold keys on
`recoveryActionId`.
- The person's run keeps its `task_retry` wake reason and the user's
wake receipt, so the existing #21349 rules rank it as person work.
- Limits: the person-triggered delivery does not spend an automatic
attempt. It adds no `issue.continued_automatically` entry and no
scheduled transient retry, and it is not gated by the hourly cap or by
delivery-pass pacing. The record itself was already counted when it was
recorded. The agent's concurrency limit still applies at claim time.

## Verification

New file
`server/src/services/fresh-continuation-retry-delivery.test.ts`, with 12
tests. All cases cover `fresh_isolated_workspace`, `writer_ended` and
native (`remote_epoch`) records, through the real `requestTaskRetry` +
`resumeTaskRetry` path:
- A Retry during pending delivery gives exactly one run. That run is the
continuation, carries the recorded isolation (the workspace readiness
reads `fresh_isolated_workspace`), coalesces the Retry, and ranks as
person work. Later delivery passes add nothing.
- Delivery first, then a saved Retry: one run, and the Retry is
coalesced. A Retry on the delivered run joins it through the busy path.
- A Retry and two delivery passes race (concurrent transactions): one
run.
- After the continuation's own failure, a Retry gets its own run.
- The delivery pass is held by pacing (backlog source, agent queue
full), and the hourly cap and the shared attempt ledger are exhausted.
The person's Retry still starts the run.

Red before the fix: 8 of 12 failed. Green after: 12 of 12.

Local runs, one file at a time:
- `fresh-continuation-fallback.test.ts`: 71 of 72. The failure is "runs
a person's Retry pressed before delivery…", a deadlock in `afterEach`
truncate. It passes alone, and it fails the same way on unmodified
master under this machine's load.
- `__tests__/recovery-wake-herd.test.ts`: 29 passed
- `__tests__/execution-control-work.test.ts`: 27 passed
- `__tests__/issue-recovery-actions.test.ts`: 112 passed
- `workspace-recovery-fallback.test.ts`: 19 passed
- `heartbeat/queue.test.ts`: 101 passed
- `task-retry.test.ts`: 117 passed;
`legacy-preparation-retry-hold.test.ts`: 5 passed
- `server` `tsc --noEmit`: clean

## Risks

- A person's message, chat input or manual wake that arrives while a
continuation is pending also becomes that continuation, with a fresh
session and the recorded isolation. Before, such a wake ran without
isolation and superseded the record. This is the same rule for every
admission path, and it closes the shared-directory gap there too.
- A wake that takes the context and is then deferred by a later gate can
keep the isolation marker in its saved context. If the record is
delivered meanwhile, that later run prepares a new isolated workspace.
That is the safe direction.
- No schema change and no migration. Nothing is added before turn start
for healthy runs: the new read runs only at wake admission.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent with tool use (shell, file edit,
test runs). Model ID `claude-opus-5-5`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 07:13:49 -06:00
DottaandClaude Opus 5.5 5ba26600ef fix(native-runtime): a remote Runner reattachment continues a turn the Runner can recover (#21362)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime runs agent turns through a Runner, often in a
remote sandbox, and must recover runs after a restart without help from
a person
> - When a Runner process is lost, a new Runner restores the
conversation. If the provider turn is gone, the Runner sends
`turn.failed` with `provider_turn_lost_on_restore` and `recoverable:
true`. The Runner sends this only after it proves that the old process
ended
> - The runtime can continue such a turn in the same conversation, but
only when the server sets `resumeInterruptedTurn`. The server set it
only for a local `resume_dead_runner` claim or a scheduled resume, and a
scheduled remote resume is refused. So on a remote Runner reattachment
the runtime did not continue the turn, and the run ended
> - This pull request sets `resumeInterruptedTurn` for a
`reattach_remote_runner` claim too
> - The benefit is that a remote run continues the lost turn in the same
conversation immediately, instead of failing and then restarting 3 to 6
minutes later

## Linked Issues or Issue Description

**What happened?**
On hosted staging (build `d1d71a7755`, read-only trace), 10 runs in 3
hours ended with `native_provider_terminal_failed` and the message "The
runner restored the conversation after process loss, but the previous
turn is no longer active." The Runner marked each of these events
`recoverable: true`. Example: run 646e2efb. A new Runner process started
at 08:32:04. `session.resumed` came at 08:34:49. `turn.failed
provider_turn_lost_on_restore` (recoverable) came at 08:34:51. The run
failed at 08:35:21. Example: run 11fe2cdc. `session.resumed` came at
09:23:24. `turn.failed` came at 09:23:25. The run failed at 09:23:52. In
each case, a separate recovery path continued the task only 3 to 6
minutes later.

**Expected behavior**
When the Runner marks a lost turn as recoverable after a restore on a
remote reattachment, the run continues the turn in the same
conversation.

**Steps to reproduce**
1. Start a native run on a remote sandbox Runner.
2. Lose the Runner process while a turn is active, so that a new Runner
reattaches through a `reattach_remote_runner` claim.
3. The new Runner restores the conversation and reports
`provider_turn_lost_on_restore` with `recoverable: true`.
4. See that the run ends with `native_provider_terminal_failed` and does
not continue the turn.

**Agent adapter(s) involved**
Not adapter-specific (core bug). Seen with Codex on a Daytona remote
sandbox.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`:
`resumeInterruptedTurn` is now also true for a `reattach_remote_runner`
restart-recovery claim. The runtime still continues the turn only when
the Runner itself sends `turn.failed provider_turn_lost_on_restore` with
`recoverable: true` for the checkpointed active turn.
- `doc/execution-semantics.md`: one paragraph describes this behaviour,
and says that a lost turn that still ends the run keeps
`native_provider_terminal_failed`.
- Tests:
- `native-scheduled-resume-continuation.integration.test.ts`: "continues
a lost turn on a same-run remote Runner reattachment" (2 cases: a plain
reattachment and a `replaceUnansweredRunner` reattachment).
- New `native-recoverable-lost-turn.integration.test.ts`: a guard
through `executePaperclipNativeSession`. The runtime saves a
failed-terminal checkpoint and then raises the lost turn (recoverable
and not recoverable). The run ends as `native_provider_terminal_failed`,
and the hold cause stays one that the replacement lane continues.
- An earlier revision of this PR also mapped a recoverable lost turn to
`native_session_interrupted`. Review showed that this does not make the
run retryable, and the hold it writes is not picked up by any automatic
continuation. That hunk is removed.

## Verification

- Red without the product change: the two integration cases fail with
`passed` = `[false]` (expected `[true]`).
- Green with the change:
- `cd server && npx vitest run
src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts
src/services/native-runtime/native-recoverable-lost-turn.integration.test.ts`
— 14 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
— passed (with the scheduled-resume file: 224 passed, 14 skipped,
pre-existing skips).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts` — 1469
passed, 7 skipped.
  - `cd server && npx tsc --noEmit` — exit 0.
- Staging evidence (hosted staging, build `d1d71a7755`, read-only
trace): 10 runs in 3 hours ended with `native_provider_terminal_failed`
although the Runner marked the lost turn `recoverable: true`. Run
646e2efb: new Runner pid 08:32:04, `session.resumed` 08:34:49,
`turn.failed provider_turn_lost_on_restore` (recoverable) 08:34:51, run
failed 08:35:21. Run 11fe2cdc: `session.resumed` 09:23:24, `turn.failed`
09:23:25, run failed 09:23:52. The task continued only 3 to 6 minutes
later through a separate recovery path.

## Risks

- This change has not run live on staging or on a Daytona sandbox yet.
The evidence is the staging trace and the local tests.
- This PR does not introduce or widen the gap where a recoverable lost
turn still ends the run. On master, that run already ends as
`native_provider_terminal_failed`, and the replacement lane continues it
after its decision (3 to 6 minutes on staging). This PR removes the
run-ending case whenever the Runner reports the turn recoverable for the
checkpointed active turn on a remote reattachment. Where the run still
ends (for example, when the continuation itself is lost), the behaviour
is unchanged, and the guard test
`native-recoverable-lost-turn.integration.test.ts` proves that it stays
`native_provider_terminal_failed`.
- The remaining case is tracked with the owner of the recovery path:
recovery evidence will accept `provider_turn_lost_on_restore` /
`recoverable` directly, after their ownership-deadline PR #21364. An
earlier hunk here tried to shorten that case. An integration test
through the real path showed that it would create a board-owned
ambiguous hold that no automatic pass continues, so the hunk is removed.
- This change does not fix why the turn is lost. In staging, a retry
replaced a live Runner. A separate fix for that is in progress.
- The runtime continues a turn only when the Runner sends the
recoverable lost-turn signal. The Runner sends it only after it proves
that the old process ended. So no provider turn runs twice.
- No Runner image change. No schema change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edit, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:09:20 -06:00
DottaandClaude Opus 5.5 d9b42b50fd feat(presentation): let a person move a waiting continuation forward without starting a second run (#21373)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, Paperclip records a fresh continuation
and continues the task by itself. Since #21353 the task reads as
continuing (`working`, no Retry) for as long as that continuation waits
> - That is truthful, but a person has no control while it waits. On
hosted staging one agent had 21 running and 209 queued runs, and
continuations waited 30 to 110 minutes. Pacing (#21234, #21354) admits a
waiting continuation into the agent's queue within minutes, so most of
that wait is in the queue, behind other work
> - #21353 removed Retry from this state because a Retry started its own
run, which raced the continuation and could give the task two runs
> - A Retry that observes the queued continuation run cannot start a
run: admission sees a queued run and joins the Retry to it (`coalesced`,
`runId` = that run). Under #21234's ranking, a queued run that a person
joined starts ahead of system recovery and ordinary work
> - This pull request keeps the continuing state and, once the queued
continuation has waited 15 minutes, also offers Retry labelled "Move it
forward now"
> - The benefit is that a person can move a long-waiting task to the
front of its agent's queue, and the click can never create a second run

## Linked Issues or Issue Description

Refs #21353, #21348, #21234, #21354, #21264, #21320, #21200

**What happened?**

On hosted staging, tasks with a recorded continuation waited 30 to 110
minutes behind a deep agent queue. The task showed "continuing" with no
action. The person could only wait.

**Expected behavior**

The task still never shows "failed". After a long wait the person can
move the waiting continuation forward. The click must not start a second
run.

## Which states a Retry joins (established before building)

I read #21348's fold (`foldWaitingRetriesIntoContinuation`) and its
tests, and drove the real click path (`requestTaskRetry` followed by
`resumeTaskRetry`, as the route does) in a probe against each record
state:

| Continuation state | What a Retry does today | Offer Retry? |
|---|---|---|
| Recorded, waiting for delivery (`continuationDelivery: "pending"`, no
run yet) | The Retry is admitted at once as its own `task_retry` run,
for every isolation (`fresh_isolated_workspace`, `writer_ended`,
native). The delivery pass then marks the record delivered (superseded).
#21348's fold never runs: it folds only a Retry that is still
`deferred_issue_execution` when the continuation run is created, and
only for `fresh_isolated_workspace`. Nothing keeps the Retry deferred,
so the click races the delivery pass. | **No** (unchanged) |
| Delivered, continuation run `queued` | `requestTaskRetry` sees the
latest run is busy and records the Retry as `coalesced` into that run in
the same transaction. No run is created; `resumeTaskRetry` acts only on
deferred Retries. The run is now ranked as a person's (`joinedByUser` in
`queue.ts`). | **Yes, after 15 minutes** |
| Continuation run `running` | Same join, but the run already started,
so there is nothing to move forward. | No |
| Continuation ended | A Retry gets its own run (the existing failed-run
path). | Existing behaviour |

So the Retry offered here is joined by the existing busy-run admission
in `requestTaskRetry`, not by #21348's fold. Its source run is the
queued continuation itself (the run the presentation names as
`observedRunId`), not the run the continuation was created for.

## What Changed

- `server/src/services/task-interruption-presentation.ts`: for a task
whose latest run is `queued`, created 15 minutes ago or more, and named
as `continuationRunId` by a delivered fresh-continuation record, the
projection returns `state: "working"`, `canRetry: true`, `retryAction:
"move_forward"`, responsibility `system`, `observedRunId` = that queued
run. It does not offer this when a person's wake already joined the run,
when the run's agent is not the assignee, or when a person-owned wait
applies (approval, reconnect, budget, pause and the others are checked
first). Two small reads run only for queued runs older than 15 minutes.
The existing restore-hold check that withdraws `canRetry` also withdraws
`retryAction`.
- `packages/shared/src/types/execution-projection.ts`: new optional
`retryAction?: "move_forward"` on `TaskInterruptionPresentation`. It
only changes the label; `canRetry` still decides whether Retry is
offered.
- `ui/src/components/ExecutionBlockerNotice.tsx`: a `working`
presentation with `retryAction: "move_forward"` shows "Waiting for the
agent to finish other work. Paperclip continues this task by itself."
and a "Move it forward now" button that calls the same task Retry.
Token-only classes; `pnpm check:token-gates` is clean.
- One join-only action, `move_forward`, carried from the offer to
admission (contracts kept in sync):
- `packages/shared/src/validators/issue.ts`: `taskRetrySchema` accepts
`retryAction: "move_forward"`. The route already passes the body to
`requestTaskRetry`.
- `server/src/services/task-retry.ts` (`admitTaskRetry`): right after
the stale check, still under the task row lock and before any other
Retry path, a `move_forward` Retry whose observed run is no longer
`queued` or `running` returns `{ accepted: false, alreadyMovedOn: true,
requestId: null }` and writes nothing. This block was cleared with the
owner of Retry admission. The owner will later extend it so that
"continuation recorded, not yet delivered" also counts as actionable,
after their pending-delivery change lands (about two lines on their
side). Until then this presentation never offers the button in that
state, so the early return cannot misfire from our UI.
- `ui/src/api/issues.ts` and `ui/src/hooks/useTaskRetry.ts`: the click
sends `retryAction: "move_forward"` when the offer has it.
`alreadyMovedOn` shows "The task already moved on, so nothing new was
started. It now shows its current state."
- What a click does at each timing: continuation `queued` → joined to
that run (moves it forward); `running` → joined (nothing more to do);
ended → "already moved on", nothing started.
- A joined click cannot become a run later. The busy path writes it as
`coalesced` with `runId` set to the continuation and `finishedAt` set
(`task-retry.ts:456-461`). Promotion after a run ends
(`heartbeat/run-control.ts:944` `releaseIssueExecutionAndPromote`;
`modules/wake-queue/adapters/postgres.ts:89` `DEFERRED_WAKE_STATUS` and
the selects near `:1358`), `resumeTaskRetry` (`run-control.ts:833-839`)
and `resumeExecutionWaitComments` (`:876`) pick up only
`deferred_issue_execution` wakes.
- `ui/src/pages/Inbox.tsx`: the failed-run row does not show its Retry
for this state. That row sends the failed run as the observed run, which
is no longer the latest run, so its Retry would be refused as stale.
- `doc/execution-semantics.md`: one paragraph on the offer, when it
ends, and why there is none before delivery.

## Verification

- Red (before the product change, new tests only), in `server/`: `pnpm
exec vitest run src/__tests__/task-interruption-automatic-path.test.ts
src/__tests__/recovery-wake-herd.test.ts`: 2 failed, 58 passed. Both
failures are the missing offer (`canRetry: false`, no `retryAction`) for
a continuation queued 15+ minutes.
- Green after merging master 8e41247b65, in `server/`: `pnpm exec vitest
run src/__tests__/recovery-wake-herd.test.ts
src/__tests__/task-interruption-automatic-path.test.ts
src/services/task-retry.test.ts
src/services/task-interruption-presentation.test.ts
src/__tests__/task-interruption-query.test.ts`: 5 files, 279 tests pass.
UI: `useTaskRetry.test.tsx` and `ExecutionBlockerNotice.test.tsx`: 35
tests pass.
- New tests:
- (b, integration, `recovery-wake-herd.test.ts`) A continuation
delivered by the real delivery pass waits 16 minutes behind older
ordinary work. The task offers "move forward". The click goes through
the real `requestTaskRetry` and `resumeTaskRetry`. Result: the task has
exactly one run after the source run, the continuation; the Retry wake
is `coalesced` with `runId` = the continuation; the offer disappears;
the next delivery pass adds nothing; the real dispatcher
(`resumeQueuedRuns`) starts the continuation ahead of the older work;
after it ends, still only that one run.
- (Greptile P1, red then green) "starts nothing when the continuation
ended between showing Move it forward now and the click": the
continuation ends after the offer is read. Before the admission block
the click was accepted (`accepted: true`, a new deferred Retry that
becomes a turn). Now it returns `alreadyMovedOn`, the task has one run,
and there is no `task_retry` wake.
- "never turns a joined move-forward click into a new run after the
continuation it joined ends": the click joins the queued continuation.
The run is then cancelled through the real `cancelRun` release. Two
rounds of `resumeExecutionWaitComments`, `resumeTaskRetry`, the delivery
pass and `resumeQueuedRuns` follow. The joined wake stays `coalesced` on
the ended run. No run comes from the click, and the task's later runs
equal those of an identical task with no click (the system's own
`issue_assignment_recovery`).
- UI hook (`useTaskRetry.test.tsx`): the click sends `retryAction:
"move_forward"`, and the `alreadyMovedOn` outcome shows the plain
message.
- (a) Under 15 minutes: no Retry. Control in the herd file: without a
click the older work starts first.
- (b, presentation) Queued 15, 45 and 110 minutes: `working`, `canRetry:
true`, `retryAction: "move_forward"`.
- (c) No Retry while the continuation waits for delivery (16, 45, 110
minutes), when it is running, when a person already joined it, for a
queued run that is not a recorded continuation, and a paused agent still
asks the person to resume.
- UI (`ExecutionBlockerNotice.test.tsx`): plain wording with one "Move
it forward now" button that calls Retry; no button for a read-only user;
nothing shown for a plain `working` state.
- (d) The #21200, #21264, #21320 and #21353 guard tests pass unchanged,
including "follows the new run once a recorded continuation is
delivered" (a just-queued continuation shows no Retry).
- `node scripts/typecheck-layers.mjs` in `server/`, `tsc --noEmit` in
`packages/shared`, `pnpm run typecheck` in `ui/`: clean.
- Not related to this change:
`src/services/fresh-continuation-fallback.test.ts` has one failure
("starts the continuation while the unreachable old lease is active", a
deadlock in the `afterEach` truncate) that also fails on master
18fb6ee571 without this change.

## Risks

- Moving a waiting continuation forward lets it start before other
queued work of the same agent. That is the purpose, and it is the
ranking #21234 already gives any run a person joined.
- No Retry is offered while a continuation is still waiting for
delivery. The delivery pacing admits a waiting continuation into the
queue within minutes, and a fresh interruption at once (#21354), so the
offer appears after the continuation reaches the queue. For the recovery
owner (not changed here): offering Retry before delivery would need the
Retry admission to hold or deliver the pending continuation itself under
the task lock, instead of starting its own run beside it.
- Admission changes only for a request that carries `retryAction:
"move_forward"`. Every other Retry is unchanged. No schema, migration or
dispatch change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:03:23 -06:00
DottaandClaude Opus 5.5 eb54831d28 fix(native): relaunch a remote Runner that exited for a fresh bootstrap (#21374)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs often use a remote Runner in a sandbox. After a restart,
a deploy or a same-run retry, the server reattaches to that Runner or
starts a new Runner over its saved state
> - A Runner that no controller reaches before its connection lease
expires stops its provider, saves its state as `recoverable_failure`
with `lease_expired_requires_bootstrap`, and exits. The Runner protocol
says that only a new process with a fresh bootstrap ticket can
reconnect, and that the new process continues from the same cursor
> - Reattachment started a new Runner over saved state only for a
settled `suspended` exit (or an exit before `session.open`). So it
refused this exit, and the run ended with
`runner_remote_process_identity_unavailable`
(`native_continuation_requires_reconciliation`)
> - This pull request lets reattachment launch a Runner over the exact
saved state of a Runner that exited to ask for a fresh bootstrap
> - The benefit is that the run continues by itself after a long
controller gap, with no Retry and no manual sandbox repair

## Linked Issues or Issue Description

Refs #21329, #21362, #20853

**What happened?**
On hosted staging (build `2b69cdfcd6`, read-only trace), Codex runs on
remote Daytona Runners ended with
`native_continuation_requires_reconciliation` and the error
`runner_remote_process_identity_unavailable`, about 5 per hour, after
#21329 was deployed. All five sample runs (c6fabe9a, 31d54b02, 995b63df,
dd595934, 50c6e6e6) ended the same way. The last
`native.remote_runner_recovery_failed` event recorded
`process_verification` / `process_not_running`, `runnerLifecycle:
recoverable_failure`, `runnerRecoverableFailure:
lease_expired_requires_bootstrap`, and the Runner diagnostic "connection
lease expired; a fresh bootstrap is required". Since 10:30 UTC, 8 of the
15 runs that ended with this error had this exact exit.

Timeline of run c6fabe9a:
- 10:46 to 10:54: `runner.session.startup` took 476 s in a loaded
sandbox.
- 10:54:02: the controller sent `turn.start`. At 10:54:47 the command
timed out: "PRP command turn.start timed out (runner connected; pending
ahead 0; last runner event seq 2 91s ago)". The run resumed in the same
run and the turn then ran (tools completed from 11:14).
- 11:39: a graceful server shutdown suspended the controller. At 11:43
the next controller started the reattachment, but it did not reconnect
to the Runner until the server restarted again at 11:58. In that gap the
Runner's lease expired and the Runner exited.
- 11:58:53: the reattachment proved the process gone and refused the
`recoverable_failure` state. The run failed at 11:59.

Why `turn.start` timed out while the Runner was connected: the Runner
answers `turn.start` only after Codex answers its `turn/start` JSON-RPC
request. While it waits, the Runner buffers Codex notifications and
commits no events, so the controller sees a quiet connected Runner with
nothing pending ahead. The controller's bound for the `turn.start`
command result is a fixed 30 s. #21329 changed only the later wait for
`turn/started`. In a sandbox that needed 8 minutes to start, Codex
answered later than 30 s. That timeout is recoverable, and the run did
recover from it. This pull request does not change it (see Risks). The
failure that ended the runs is the refused reattachment.

**Expected behavior**
When a remote Runner exits because its lease expired (or its reconnect
grace or deadline ended) and its saved state is intact, reattachment
starts a new Runner from that state and the run continues.

**Steps to reproduce**
1. Start a native Codex run on a remote sandbox Runner.
2. Keep every controller from reaching the Runner for longer than its
connection lease (on staging: a deploy that stalled the reattachment for
about 15 minutes).
3. The Runner saves `recoverable_failure` /
`lease_expired_requires_bootstrap` and exits.
4. The next reattachment fails with
`runner_remote_process_identity_unavailable`, and the run ends.

**Agent adapter(s) involved**
`paperclip_runner` with Codex on a Daytona remote sandbox. The rule is
not provider-specific.

## What Changed

- `remote-runner-recovery-diagnostics.ts`: new
`exitedRunnerRequestedFreshBootstrap` (diagnostic) and
`runnerStateRequestsFreshBootstrap` (saved state). They accept lifecycle
`recoverable_failure` with `lease_expired_requires_bootstrap`,
`transport_reconnect_grace_exceeded` or
`transport_reconnect_deadline_exceeded`. These are the three codes the
Rust Runner records after `shutdown_preserving_cleanup`. The state check
also requires no pending terminal delivery and no warm transition.
- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): when
the exact Runner process is proven gone and its diagnostic names one of
these exits, the server reads the saved state again. If the state has
the same six identity fields and still names that exit, the reattachment
launches a Runner from it instead of failing. The outbox does not need
to be empty and a turn may be active: the new Runner replays from the
same cursor. The attempt bound is the one the reattachment already
admits (3, or 4 for an unanswered-Runner replacement claim). Other exits
are unchanged.
- `native-session-executor.ts` (host preparation in
`createRunnerdBackend`): a resumed sandbox for a reattachment that did
not adopt a process re-reads the Runner state. When it still has the
same six identity fields and names the fresh-bootstrap exit, preparation
keeps it in place, as for an adopted live Runner. It no longer fails
with `runner_harness_state_mismatch: resumed_not_suspended` (found by
Greptile).
- `doc/execution-semantics.md`: one paragraph describes the rule and the
staging evidence.

## Verification

- Red on `origin/master`: the 5 new "launches a new Runner over the
saved state after ..." cases failed (the reattachment threw
`runner_remote_process_identity_unavailable`). The 7 new fail-closed
cases passed.
- Green: `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "remote
controller restart adoption"` passes 118 tests.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
1712 passed, 21 skipped, 1 failed. The failure ("verifies a live warm
owner before refreshing run authority", warm-session supervision, not
touched here) passes when run alone. It is a timing failure under
full-file load.
- Launch path: "keeps the saved mid-turn state of a remote Runner that
exited for lease_expired_requires_bootstrap when the reattachment
prepares its sandbox" runs `createRunnerdBackend` and
`prepareExternalRunnerState` with a mid-turn Codex checkpoint. Red
before the host-preparation change (`resumed_not_suspended`), green
after. An unknown recoverable failure stays fail-closed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`
on the final revision: 1494 passed, 7 skipped.
- `cd server && node scripts/typecheck-layers.mjs` (includes test
programs): passes.
- Fail-closed cases covered: an unknown recoverable failure, a warm
transition, a pending terminal delivery, a state that changed after the
exit, a different Runner instance, attempt 4 on an ordinary claim, and
attempt 5 on a replacement claim.
- Staging read (read-only): the 5 sample runs and 69 runs with this
error in the last 9 hours were classified from their
`native.remote_runner_recovery_failed` events.

## Risks

- Medium. Reattachment now starts a new Runner over a saved state that
can be mid-turn. The Runner itself stops its provider before it saves
this state. The new Runner restores the session. It continues the turn,
or it reports the lost turn as recoverable only after it proves that the
old provider process ended. A recoverable lost turn on a remote
reattachment ends the run as `native_provider_terminal_failed` until
#21362 merges; the replacement lane then continues the task. That is
still better than the current reconciliation hold.
- The attempt bound does not grow. A Runner that exits this way again on
a later reattachment is bounded by the same provider attempt budget.
- This does not change the 30 s `turn.start` command bound or the deploy
gaps that let the lease expire. Those are separate follow-ups. 57 older
runs with this error had a different exit (`ready`, not settled). This
pull request keeps that case fail-closed; #20853 covers the
settled-`ready` part.
- No schema, API, UI or Runner image change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edit, code execution, read-only browser
reads of staging run data).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:53:38 -06:00
DottaandClaude Fable 5.2 e3faf579a1 test(recovery): drain background runs before the fresh-continuation test cleanup (#21375)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server continues an interrupted task in a new run from saved
work
> - `fresh-continuation-fallback.test.ts` tests that behavior against a
real PostgreSQL database
> - Some tests start runs that continue in the background after the test
body returns
> - The cleanup truncated its tables while those runs still held locks,
and PostgreSQL sometimes reported a deadlock
> - The failure appeared on unrelated pull requests in server shard 9/12
> - This pull request makes the cleanup wait for in-flight runs and
retry the truncate on a deadlock
> - The benefit is that this file no longer fails CI at random

## Linked Issues or Issue Description

**What happened?**

The `afterEach` in
`server/src/services/fresh-continuation-fallback.test.ts` sometimes
fails with `PostgresError: deadlock detected` (code 40P01). The truncate
waits for an `AccessExclusiveLock`. A background run that a test started
holds an `AccessShareLock` and waits for another table. No assertion
fails. The test that receives the failure changes from run to run.

**Expected behavior**

The cleanup completes after each test. The file passes on each run.

**Steps to reproduce**

1. Check out master at 18fb6ee571.
2. Run `npx vitest run src/services/fresh-continuation-fallback.test.ts`
in `server/` on a loaded machine, several times.
3. One test fails in `afterEach` with the deadlock error in some runs.

**Relevant logs or output**

`truncate table issue_recovery_actions, ... heartbeat_runs, issues
cascade` → `PostgresError: deadlock detected (40P01)`

Refs #21348, #21135.

## What Changed

- The `afterEach` awaits
`heartbeatService(db).drainActiveRunExecutions()` before the truncate.
The registry of in-flight runs is shared by all service instances.
- The `afterEach` truncates through the existing
`truncateTablesWithDeadlockRetry` helper.
- No product code changed.

## Verification

- `npx vitest run src/services/fresh-continuation-fallback.test.ts` in
`server/`: 72 of 72 passed, two times in sequence.
- `npx tsc --noEmit -p .` in `server/`: no errors.
- The deadlock is not deterministic. Two passing runs do not prove that
it cannot occur. The retry helper covers a late writer that the drain
cannot see.

## Risks

Low risk. The change is in one test file. The drain adds a short wait to
each test cleanup.

## Model Used

Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 06:46:18 -06:00
DottaandClaude Opus 5.5 7b27fc453f fix(presentation): a provider failure the system is still recovering shows as working, not as needing Retry (#21369)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a provider ends a run's session with a terminal failure
(`native_provider_terminal_failed`), Paperclip recovers by itself: it
checks whether a safe replacement is possible, saves the remote files,
and then continues the task in a fresh run from saved work
> - These steps take 15 to 60 seconds. Until the continuation is
recorded, the task view did not know the system was still deciding
> - In a restart-chaos wave, two tasks showed a failed attempt with a
Retry button for 13 to 27 seconds, and one of them changed state in a
loop: failed, then working, then failed with Retry, then working
> - Product rule: a task that the system will continue by itself never
shows as failed with a Retry button, and the state does not change back
and forth during one recovery
> - This pull request reads the record that says the system is still
deciding (the native coordinator's terminal-failure row) and shows the
task as working during that time, within the existing 2-minute bound
> - The benefit is one steady "working" state from the failure to the
new run, and Retry comes back at once when the system hands the task to
a person

## Linked Issues or Issue Description

Refs #21264, #21353, #21200, #21320

**What happened?**

Restart-chaos wave `w-20261113T092657-s7302` on master d1d71a7755, tasks
t030 and t039 (Claude through ACPX). The provider returned a temporary
server error ("ACP agent reported a terminal service failure. API Error:
503 ..."). The run failed with `native_provider_terminal_failed`. The
system then continued the task by itself
(`issue_recovery_action_restored`), and a new run completed the task. In
between, the task view showed:

- t030: 10:20:18 working → 10:20:32 failed, Retry offered → 10:20:45
working (13.4 s with Retry).
- t039: 10:21:15 failed, no Retry → 10:21:29 working → 10:21:46 failed,
Retry offered → 10:22:12 working (26.8 s from the failure; it changed
state in a loop).

The wave database is not kept, so the record at each step comes from the
samples (`recoveryOwner`, `executionBlocker`, issue status), the server
log, and the code paths that write those records:

1. The run ends. The native executor saves the coordinator row (`phase:
terminal_failure`, `failureCode: native_provider_terminal_failed`, no
`replacementDenied`) and a board-owned hold with cause
`native_provider_terminal_failed`, and blocks the task. The projection
fell through to "failed". At t039 10:21:15, the execution-blocker check
removed Retry because the remote files were not saved yet.
2. Workspace recovery adopts the hold (`ownerType: system`,
`systemRecovery` running). While it runs, the task shows "working" (t039
10:21:29).
3. The files are saved (`systemRecovery` complete). No record now says
"running", the execution-blocker check no longer removes Retry, and the
task shows "failed" with Retry (t039 10:21:46, t030 10:20:32).
4. Safe replacement is denied (`replacementDenied` on the coordinator
row, and the hold is upserted as board-owned). The settle pass then
records a fresh continuation (log: "Unfinished run continues in a fresh
run from saved work", `trigger: recovery_refused`), sets the task back
to `todo`, and the task shows "working" (t039 10:22:10–12).

**Expected behavior**

From the failure until the continuation is recorded, the task shows
"working" with no Retry, without changing back and forth. When recovery
is handed to a person, or nothing is recorded within the bound, Retry is
offered.

**Steps to reproduce**

1. Let a native run fail with `native_provider_terminal_failed` on a
remote workspace.
2. Read the task before the fresh continuation is recorded.
3. The task shows "failed" with Retry.

## What Changed

- `server/src/services/task-interruption-presentation.ts`:
- The run read selects `providerRecoveryAt`: the coordinator row's
update time while the row says the system is still deciding (a failed
native run, `phase: terminal_failure`, `failureCode:
native_provider_terminal_failed`, no result, no lease owner, no
successor run). This matches the rule that `execution-projection.ts`
already uses to show "Checking recovery" for this failure.
- A denied replacement (`replacementDenied`) does not end this time.
After a denial, the settle pass continues the task in a fresh run unless
the failure code is a person's. `native_provider_terminal_failed` is not
a person's code.
- While the time is inside the existing 2-minute bound
(`AUTOMATIC_DISPOSITION_GRACE_MS`), the task shows `working`, `canRetry:
false`, responsibility `system`, with the saved-files step text when
there is one. The bound counts from the latest of: the run end, the
coordinator row update, and the recovery record update. So each recovery
step extends it, and the steps do not make the state change back and
forth.
- Retry is offered at once when recovery was handed to a person: a
fresh-continuation check that decided it cannot continue (any kind other
than `exact_recovery_pending` or `backoff`), `ownerGoneAttestation`,
`explicitUserContinuation`, or an exhausted saved-work recovery.
- No change to recovery decision code, the fallback, or the Retry path.
No new record or marker. The coordinator row already records "still
deciding".

## Verification

- Red (before the fix), in `server/`: `pnpm exec vitest run
src/__tests__/task-interruption-automatic-path.test.ts`: the new t039
sequence test failed at its first step (`state: "failed"`, `canRetry:
true` right after the failure).
- Green (after the fix), in `server/`: `pnpm exec vitest run
src/__tests__/task-interruption-automatic-path.test.ts
src/__tests__/task-interruption-query.test.ts
src/services/task-interruption-presentation.test.ts
src/__tests__/recovery-wake-herd.test.ts`: 4 files, 148 tests pass.
- New tests in `task-interruption-automatic-path.test.ts`:
- The t039 sequence (failure, files being saved, files saved,
replacement denied with the hold upserted to board) shows `working` with
`canRetry: false` at every step.
  - The bound passed with nothing recorded: Retry.
  - The continuation worker decided it cannot continue: Retry at once.
  - A person took over the hold: Retry at once.
- The #21200, #21264, #21320 and #21353 guard tests pass unchanged. This
includes "still offers Retry when recovery gave up and handed the hold
to the board" and the person-owned hold tests.
- Related suites: `workspace-recovery-preparation-dispatch`,
`explicit-native-continuation`, `workspace-recovery-retry-copyback`,
`workspace-recovery-admission`, `native-safe-replacement` and
`run-failure-report`: 1046 of 1046 tests pass. One run on a loaded
machine had a 30-second `afterAll` cleanup timeout in
`native-safe-replacement` after all its tests passed. An earlier
parallel run had one timing failure in
`workspace-recovery-preparation-dispatch` that did not repeat.
- `node scripts/typecheck-layers.mjs` in `server/`: passes.

## Risks

- A provider failure that the system will not continue (for example, the
task was reassigned) can show "working" for up to 2 minutes before Retry
appears. This is the same bound #21264 uses for a system-owned hold on
an ended run.
- The rule applies only to `native_provider_terminal_failed`, which is
the failure that the safe-replacement evaluation owns. Other terminal
failures are unchanged.
- One more small read per page, only when a listed task has a run in
this state.
- Note for the recovery owner, not changed here: the provider's 503
("credential validation failed. This is a server-side issue") is
classified as a permanent provider failure
(`native_provider_terminal_failed`). Whether a 5xx from the provider
should instead be a temporary retry on the same run is the recovery
owner's decision.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs, reading
restart-chaos wave evidence).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:43:34 -06:00
DottaandClaude Opus 5.5 57b787d3d2 fix(retry): continue from recorded work when saved files cannot be recovered (#21363)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run finishes but copying its files out of the sandbox fails,
Paperclip holds the task and tries to recover the saved files. When
recovery gives up, the task offers Retry.
> - Some saved files can never be recovered: the sandbox that held them
was destroyed, or Paperclip cannot prove which runs used a shared
sandbox. Each Retry ran the same check again, failed again, and parked
the Retry in a wait that never ends.
> - The task told the person to press Retry again or to wait for a
Paperclip update. That breaks the rule that recovery must work and that
a Retry that is shown must work.
> - This pull request makes Retry continue the task in a fresh session
and a new workspace from the work that is recorded, when the saved files
are provably unrecoverable or cannot be proven safe. The task says
clearly that the unsaved files were not recovered.
> - The benefit is that an offered Retry starts a run, and nobody waits
for a fix that another click cannot bring.

## Linked Issues or Issue Description

Refs #21315, #21295, #21267

**What happened?**

On hosted staging, two blocked tasks offered Retry (`canRetry: true`).
Each Retry was accepted, and within about 3 seconds it was parked in a
wait. No run started. Every later Retry did the same.

- Task A: the hold cause was
`native_workspace_sync_out_retry_exhausted`. Recovery ended with
`source_unavailable` (`use_alternative_strategy`). The sandbox that held
the files had a signed absence proof and a `destroyed` termination
receipt. Retry was parked with `saved_work_recovery_failed`: "Saving the
agent's files failed. Retry to try again".
- Task B: the same hold cause. Recovery ended with
`allocation_ownership_ambiguous` (`service_required`), because the
sandbox was a reusable sandbox that other runs also used. Each Retry
opened a new recovery generation, ran the same check, and was parked
with `saved_work_recovery_refused`: "Retry runs the same check again and
succeeds only after Paperclip is updated."
- In both tasks, the agent's turn had finished and its result was
accepted. Its work was recorded outside the sandbox: a plan document and
screenshot attachments for task A, and a pushed branch with an open pull
request and an implementation document for task B.

Root cause: admission had a branch for an accepted result whose files
were not yet saved. When the recovery budget was spent, the branch
always returned a `saved_work_recovery_*` wait. It did not check whether
the files could ever be recovered, so the fresh-workspace path that
admission already has was never reached. The automatic retained-source
fallback did not help either, because it handles only the remote
workspace v1 reference, and these runs use the native workspace sync v3
reference. Also, Retry opened a new recovery generation for an ownership
refusal, which only repeated the same check.

**Expected behavior**

When the saved files are provably unrecoverable or cannot be proven
safe, Retry continues the task in a fresh session and a new workspace
from the recorded work. The old files are never applied and the source
sandbox is not changed. No provider turn is replayed. The task says that
the unsaved files could not be recovered. The Retry offer and admission
agree.

**Steps to reproduce**

1. Let a native run finish with an accepted result, and let its
workspace copy-back fail (`native_workspace_sync_out_retry_exhausted`).
2. Let saved-work recovery end with `source_unavailable` (sandbox
destroyed) or with `allocation_ownership_ambiguous` (shared reusable
sandbox).
3. Press Retry.
4. Observe that the Retry receipt waits with
`saved_work_recovery_failed` or `saved_work_recovery_refused`, and no
run starts. Press Retry again and observe the same result.

**Paperclip version or commit**

Staging on `origin/master` after #21315 (`bbafa78fe2`).

**Deployment mode**

Hosted cloud (staging), remote Daytona sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (native runtime runs).

## What Changed

- `savedFilesUnrecoverable` and `unrecoverableSavedFilesFailure`
(`system-recovery-state.ts`) are the one rule for saved files that no
recovery can apply. The rule is true for a spent recovery whose failure
is a permanent source refusal: `source_unavailable`,
`allocation_ownership_ambiguous` or `historical_source_missing`. It is
false for `source_binding_invalid`, because a new recovery generation
can still verify that source. It is false for `archive_unsafe` and
`snapshot_verification_failed`, because a fresh attempt still imports
earlier saved snapshots and would fail before the agent starts. It is
also false after a fresh attempt is recorded.
- Admission (`explicit-native-continuation.ts`) no longer parks a Retry
of an accepted result in `saved_work_recovery_*` when every live
recovery of the run is unrecoverable. The Retry takes the existing
fresh-workspace path (`fresh_isolated_workspace`). That path keeps all
stop proofs, leaves the source sandbox and workspace untouched, and
starts a fresh session.
- When such a Retry starts, the task gets one system comment. The
comment says that Paperclip could not recover the unsaved files, why,
and that the task continues from its recorded work (comments, documents,
attachments and pushed commits). The settled hold records the same text.
- Retry (`task-retry.ts`) binds to an unrecoverable recovery generation.
It does not open a new generation that repeats the same check.
- The blocker text for unrecoverable files says that Retry continues the
task in a new workspace from its recorded work. It no longer says that
Retry succeeds only after an update.
- `canAdoptNativeSourceRetention` and the permanent source-refusal lists
moved into `system-recovery-state.ts`, so that admission, Retry and
messages share them. The old modules re-export them.

## Verification

- Red/green: `cd server && npx vitest run
src/services/unrecoverable-saved-work-retry.test.ts`. The tests fail on
`origin/master` sources (or on the earlier head for the snapshot case)
and pass with this change. They are built from the two staging shapes
(destroyed sandbox with `source_unavailable`, and a shared reusable
sandbox with `allocation_ownership_ambiguous`):
- offers a Retry that continues the task (blocker and task view agree,
and the text names the fresh workspace)
- starts a fresh run from the recorded work: the Retry receipt has no
wait and a run, the strategy is `fresh_isolated_workspace`, the hold is
settled with the fresh attempt, one system comment explains the lost
files with no raw IDs, the source lease is unchanged, and no new
recovery generation opens
- keeps a Retry out of a fresh attempt that would import a succeeded
snapshot whose archive is unsafe
- `workspace-recovery-warm-handoff.test.ts` › "wakes physical
preservation from HTTP Retry before passive history without replaying
the provider" failed on an earlier head of this PR, which also treated
`source_binding_invalid` as unrecoverable. It passes unchanged now.
- Updated tests that pinned the old repeat-the-check behavior:
`workspace-recovery-preparation-gates`, `task-retry`,
`system-recovery-state`, `workspace-recovery-retry-copyback`.
- Related suites pass: `system-recovery-state`,
`workspace-recovery-preparation-gates`,
`workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`,
`task-retry`, `saved-work-retry-eligibility`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
`workspace-recovery-admission`, `task-interruption-query`,
`native-workspace-sync-retention`.
- `cd server && node scripts/typecheck-layers.mjs` passes (source and
test layers).

## Risks

- Medium-low. A Retry of an unrecoverable hold now starts a new agent
turn in a new workspace instead of waiting. The turn starts only through
the existing fresh-workspace admission, which keeps every stop and
cleanup proof.
- Unsaved files of the old run are not applied. That was already true;
the change is that the task continues and says so.
- `source_binding_invalid`, `snapshot_verification_failed` and
`archive_unsafe` keep today's path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:23:23 -06:00
DottaandClaude Opus 5.5 5504151eae fix(recovery): restart a remote bootstrap again after it waited for a busy agent directory (#21370)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each interrupted native run to restart
recovery. Recovery decides if the run can continue safely.
> - For a remote sandbox run that reached no provider work, recovery
restarts the bootstrap on the same run (`bootstrap_incomplete`).
> - The restarted executor can meet a busy agent directory while it
prepares. It then defers the run for a few seconds and asks restart
recovery to look again.
> - That deferral saves the coordinator as `retryable_failure` while the
run stays `running`. No remote proof accepted this shape, so recovery
blocked the run with `ambiguous_remote_recovery_evidence`.
> - This pull request lets restart recovery restart the bootstrap again
when the previous claim was a bootstrap restart and nothing reached
execution since that claim.
> - The benefit is that a deploy plus agent-directory contention no
longer ends the task as failed. No provider turn is replayed.

## Linked Issues or Issue Description

**What happened?**

On staging, native runs of a Codex agent on Daytona failed about 3 times
an hour with `native_restart_recovery_blocked` and error
`ambiguous_remote_recovery_evidence`. The same failure was seen on the
two days before. I read 12 failed runs. 11 of them have the same event
sequence:

1. The first executor prepares the remote workspace. A deploy restarts
the server.
2. Restart recovery waits for the old controller lease, then records
"Restarting an incomplete native runner bootstrap on the same heartbeat
run" (`bootstrap_incomplete`, controller generation 2).
3. The new executor starts the run again and records "Another run of
this agent is still using its files" (`agent_directory_operation_busy`).
4. About 15 seconds later, restart recovery records "Native restart
recovery blocked ambiguous or conflicting ownership"
(`ambiguous_remote_recovery_evidence`). The run fails.

The busy-directory deferral writes `phase: retryable_failure`,
`failureCode: agent_directory_operation_busy`, `recoveryMode:
bootstrap_retry`, no lease owner, and keeps the run `running`. Every
remote proof in restart recovery needs either `phase: observed` with a
lease owner, or a `failed` run with a specific saved failure code. So
none matched. With no checkpoint and no provider evidence, the
classifier returned `ambiguous_remote_recovery_evidence`. The other
result fields on the failed run (workspace restore failure,
fresh-continuation supersede) were written after this block. They are
consequences, not the cause.

**Expected behavior**

The deferral happens while the executor prepares, before it requests a
runner. The previous recovery claim already proved that the run reached
no provider work. If nothing reached execution after that claim,
recovery must restart the bootstrap again without manual Retry. If a
runner start, runner identity, provider event or checkpoint appears
after the claim, recovery must keep the run held and not replay provider
work.

**Steps to reproduce**

1. Start a native remote-sandbox run and stop the server before the
runner receives a command.
2. Let restart recovery claim the run as `bootstrap_incomplete`.
3. Make the agent directory lock busy when the new executor prepares
instructions, so it defers with `agent_directory_operation_busy`.
4. Let the scheduled restart recovery run at the retry time. The run is
blocked with `ambiguous_remote_recovery_evidence`.

The new integration test reproduces these steps with the real recovery
and deferral functions.

**Paperclip version or commit**

Staging Core `70be453c38`. Fix based on `e23549cd9c`.

**Deployment mode**

Hosted staging (cloud workspace), remote Daytona sandbox, frequent
auto-deploys (every 20 to 45 minutes).

**Agent adapter(s) involved**

`paperclip_runner` with the `codex_app_server` backend on a Daytona
sandbox.

## What Changed

- Restart recovery accepts a new remote proof,
`remote_bootstrap_restart_deferred_before_launch`. It applies when all
of these are true:
- The most recent recovery claim was `bootstrap_incomplete`, with a
recorded event watermark.
- The coordinator is `retryable_failure` with `failureCode:
agent_directory_operation_busy`, `recoveryMode: bootstrap_retry`, no
lease owner, and no successor run. The run is still `running`.
- No `native.process_start_requested`,
`native.process_identity_recorded`, `adapter.invoke`, tool start or
runner-sourced event exists after the claim watermark. No process
identity was ever recorded for the run.
- No provider event exists. A checkpoint, if present, is only the
continued session with no turn, result or request.
- The workspace is a prepared remote workspace with no durable
controller checkpoint and no staged attachment.
- The run then goes through the same resumption admission as the other
settled remote retries (agent invokable, task not paused, dependencies
ready, no competing run).
- Added red/green integration tests built from the staging shape: the
run restarts again (generation 3), and a runner identity, a provider
event, or a checkpoint saved during the wait keeps it held.

## Verification

- Red: on the unchanged source, the new test returns `{ kind: "blocked",
reason: "ambiguous_remote_recovery_evidence" }`, the same error as
staging.
- Green: `npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/native-controller-recovery.integration.test.ts`
passes (251 passed, 14 skipped).
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The change only adds a narrow proof. It requires that
nothing reached execution since a claim that already proved no provider
work. Any runner start, runner identity, provider event or new turn
keeps the run held, so a provider turn is never replayed.
- The restart counts against the existing provider attempt budget and
the no-progress deadline, so a loop of deferrals stays bounded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
code editing, browser read-only API fetches) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Refs #20829

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:12:35 -06:00
DottaandClaude Fable 5.2 18fb6ee571 fix(recovery): continue a native terminal failure whose provider is never proven stopped (#21364)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs that end in a terminal provider failure go through
automatic recovery: an exact replacement first, then a fresh
continuation from saved work
> - The exact replacement waits for proof that the previous provider
stopped. Without that proof it writes neither a successor nor a denial
> - That wait has no end. If the old sandbox or Runner can never be
proven stopped (for example after a server swap), the recovery resolver
keeps waiting for the replacement, and the task sits with no run and no
reason a person can see
> - This pull request bounds the wait. After 5 minutes on the database
clock from the run's terminal time, the replacement pass records a
denial, and the existing fallback continues the task in a fresh run in
an isolated workspace
> - The benefit is that these tasks continue by themselves instead of
stalling forever, and exact replacement still wins when proof arrives in
time

## Linked Issues or Issue Description

**What happened?**
A native run that failed with a terminal provider failure stayed stuck
when its previous provider could not be proven stopped. The replacement
pass skipped the run on every pass. It recorded no successor and no
denial, and it set no deadline. The recovery resolver waits for one of
those two records, so it never continued the task. The replacement queue
also did not wake again on its own, because a completed scan reports
nothing pending.

**Expected behavior**
Exact replacement gets a bounded first chance. After the bound, the task
continues in a fresh run that cannot use the old writer's workspace.

**Steps to reproduce**
1. Create a native run in `terminal_failure` with code
`native_provider_terminal_failed` and a finished time more than 5
minutes ago.
2. Record a process on the run that this server cannot prove stopped
(for example a PID recorded by another server).
3. Run the replacement pass and then the recovery resolver.
4. Before this change: nothing happens, on every pass. After this
change: the pass records a denial and the resolver records one fresh
continuation with isolation `fresh_isolated_workspace`.

**Paperclip version or commit**
`master` at 2b69cdfcd6.

## What Changed

- `native-safe-replacement.ts`: an unverified-ownership decision now
waits only until 5 minutes after the run's terminal time (`finished_at`,
else the finalization's last update), compared with the database
`now()`. After that, the pass records `replacementDenied` and the
existing watchdog hold, with a plain-language next action. The expiry is
checked again inside the locked transaction, so two servers cannot both
act, and a successor that a concurrent pass committed first still wins.
- The deadline applies only to `native_provider_terminal_failed` runs
whose finalization has no lease owner. Other failure codes and
person-owned causes are unchanged.
- The bound reuses `RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 minutes) from
`recovery-continuation-pacing.ts`, as `NATIVE_OWNERSHIP_PROOF_BOUND_MS`.
- The wake has no time window. The wake query returns the deadline of
every marked run that is still undecided (no successor, no denial, no
lease owner), with the same eligibility as the denial. When that
deadline is past, the queue asks for a pass at once. That pass always
removes the run from the query:
  - It denies the run, which ends the wait.
  - Or it starts the exact replacement, when ownership was proven.
- Or it finds the run legitimately undecidable and clears the marker.
This happens when the history has unresolved interactions, or when the
task is not eligible for a successor (for example in backlog). The
existing task and interaction notifications wake the queue again later.
- More than 25 overdue runs: the scan cursor requests the next pass at
once.
- A lost lock or any other error: the pass throws, and the coordinator
retries after its existing `retryMs`
(`EXECUTION_RECONCILIATION_INTERVAL_MS`, 15 seconds). That is the same
backoff as every replacement-decision error today.
- An environment lease that is never released no longer blocks the
decision after the deadline. Before the deadline, the run waits as
before and records the marker. After it, an unreleased lease counts as
"provider not proven stopped", so the pass denies the run and never
starts an exact replacement next to a live lease. The fallback then
treats the unreleased lease as pending cleanup and picks
`fresh_isolated_workspace`. Admission (`heartbeat/queue.ts`, the
wake-queue adapter) does not block a run that a fresh continuation
superseded, so the new run does not wait for or touch the old lease. The
environment orphan-settlement modules are not changed.
- When the pass first sees unverified ownership before the deadline, it
writes `ownershipUnverifiedAt` once into the failure detail. It does not
change `updated_at`. This marker only limits which runs schedule a wake.
The deadline still counts from the terminal time.
- `execution-control-work.ts`: when no replacement is pending, the
replacement queue schedules its next pass at the earliest future
deadline of a marked run. This is a single timer, not a polling loop.
Rows already past their deadline are handled by the pass that runs at
startup, with or without the marker.
- No changes to `execution-recovery-resolution.ts` or
`fresh-continuation-fallback.ts`. The fallback already selects
`fresh_isolated_workspace` when its own owner proof cannot show that the
old local writer ended, `remote_epoch` for a remote workspace, and waits
(bounded) only for a process proven alive on this host.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-safe-replacement.test.ts`: new cases:
- Past the deadline (6 minutes, and 2 days for a row stuck before this
release): denial recorded on the first pass, no exact successor, then
exactly one fresh continuation with `fresh_isolated_workspace` after two
resolver passes.
- Before the deadline: no denial, no hold, no continuation, and the
replacement schedule wakes at terminal time + 5 minutes.
- Ownership proven before the deadline: the exact replacement is
scheduled and no denial is written.
- Red before the fix: 3 of the 4 new cases failed (no denial written; no
wake scheduled). The exact-replacement case passed, as expected, because
it guards existing behavior. Green after: 93 of 93 passed in
`native-safe-replacement.test.ts` and `execution-control-work.test.ts`.
- `tsc --noEmit` for `server`: clean.
- Review fixes, red before and green after:
- A deadline that passed 10 minutes before the schedule was asked: the
queue requests an immediate pass, which denies the run and continues it
isolated.
- An overdue run with an unreleased environment lease: denied, no exact
successor, continued with `fresh_isolated_workspace`.
- An overdue run the pass cannot decide (task in backlog): one wake,
then no second wake.
- 26 overdue runs: 25 decided on the first pass, all 26 on the second,
then no wake. This one also passed before, because the existing scan
cursor covers it. It stays as a guard.
  - On the earlier head, the first three failed.
- Green after all commits: 97 of 97 passed in
`native-safe-replacement.test.ts` and `execution-control-work.test.ts`.
- `pnpm verify` selected 2624 of 3444 test files, because
`execution-control-work.ts` is widely imported. I stopped it on a
heavily loaded machine (load average about 20). Its partial output
showed only unrelated attention-test timeouts near 30 seconds. Required
CI supplies the broad coverage.

## Risks

- After 5 minutes, a run whose old provider is truly still alive on
another server is continued in a new isolated workspace. The old writer
cannot write to it, and its files are not applied. A process proven
alive on this server still makes the fallback wait for its own bound
first.
- Deploy burst: rows stuck before this release are denied on the first
pass, at most 25 per pass (existing scan limit). Continuations then go
through the existing pacing: 6 per task per hour, 5 per company per
pass, one per agent per pass, and the agent's concurrency limit.
- No schema or API change. No change before `turn.start` for healthy
runs.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 06:03:53 -06:00
DottaandClaude Opus 5.5 a090a6a932 fix(runner): stop a model from resending the same rejected tool call (#21360)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools such as `call_api`, and the tool
layer refuses an invalid call before it sends anything
> - Some model hosts drop optional tool arguments. A `PATCH` call then
arrives without `body` every time, even when the model says it added it
> - The refusal already names the missing argument, but nothing stops
the model from resending the same call. In one recorded run the agent
sent the same rejected call 12 times in 80 seconds and never finished
the task
> - This pull request makes the tool layer recognise a repeated
identical validation failure, correct it once with the exact arguments
that arrived, and then stop running that exact call for the rest of the
run
> - The benefit is that a dropped argument ends in a clear correction or
a reported blocker, not a long loop that burns time and tokens

## Linked Issues or Issue Description

**What happened?**
An agent created a task and then tried to update it with `call_api`
`PATCH /api/issues/{id}`. Every call arrived with `operationId`,
`pathParams`, `query` and `responseText`, but no `body`. The tool layer
refused each call with "requires body" and showed the shape to resend.
The model's reasoning said that the body was missing, but each new call
arrived without `body` again. The agent sent the same call 12 times and
was still running when the test ended.

**Expected behavior**
After the same call fails twice in the same way, the agent should get
one clear correction that says what arrived and what is missing.
Paperclip should then refuse to run that exact call again, so the agent
changes the call or reports the blocker.

**Paperclip version or commit**
`master` at ff971149c2.

**Steps to reproduce**
1. Run an agent on a model host that drops optional tool arguments.
2. Ask the agent to create a task and then update it.
3. See the same body-less `PATCH` call refused again and again until the
run times out.

## What Changed

- Add `RepeatedToolFailureGuard`
(`server/src/services/native-runtime/repeated-tool-failure-guard.ts`).
It keys each failure by tool name and a hash of the canonical arguments.
- Count only failures that must repeat for identical arguments:
`400`/`422` refusals and schema errors. Other errors, such as `503`,
pass through unchanged and the call still runs.
- On the second identical failure, keep the original error text and add
the names of the arguments that arrived, a note that a missing argument
did not arrive, and that Paperclip will not run this exact call again.
- On the third and later identical calls, do not run the tool. Return
the original error with an instruction to change the arguments or stop
and report the blocker.
- Any change to the arguments runs normally. A success clears the record
for that call.
- Run identical calls that arrive in parallel one after another, so each
sees the failures before it. Different calls never wait on each other.
- Wire the guard into the per-run `SessionAuthorityEpoch.execute`,
inside the tool trace, so blocked attempts are still recorded. State is
in memory, per run, and capped at 64 entries.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/services/native-runtime/native-tool-trace.test.ts`: 11 passed (6 in
the new file).
- Red/green: with the guard removed from
`SessionAuthorityEpoch.execute`, the recorded-run test fails (the second
failure has no correction and every call runs). With the guard in place,
it passes.
- The test replays the recorded body-less `PATCH` through the real
`executeRunnerApi` validation. It checks that the tool runs only twice
for five identical calls and that no HTTP request is sent.
- Parallel identical calls run at most twice, and a different call is
not held back while an identical one runs.
- Negative controls: changed arguments run again, a `503` keeps running,
and different error messages are not merged.
- `cd server && npx tsc --noEmit`: clean.

## Risks

- Low risk. The guard acts only on identical arguments with an identical
`400`/`422` or schema error, which cannot succeed on resend.
- A refusal that depends on state that changes during the run (for
example, a validation that later passes) would be blocked after two
tries with the same arguments. The agent can still send changed
arguments.
- Latency: the guard adds one SHA-256 of the canonical arguments per
tool call (about 2 µs for a 4 KB argument). It adds no wait and nothing
before turn start. Only identical calls that arrive in parallel wait for
each other.
- No persisted state changes. After a restart, the record starts empty.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:00:26 -06:00
DottaandClaude Opus 5.5 5ede64f247 fix(recovery): settle an older run's saved-work hold, fold a waiting Retry into the isolated continuation, and plain recovery wording (#21348)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, a saved-work hold protects the files it
left. The fresh-continuation fallback then continues the task in a new
run, or asks a person to press Retry.
> - Staging found that an older run's saved-work hold made the dispatch
gate cancel every new run of the task. The hold was never settled, so
the task could not run again, and no person could act on the reason.
> - Recovery notices also showed internal codes and words such as
"board", "inspect", "repair" and "no live execution path". People cannot
act on these.
> - A person's Retry pressed while an isolated continuation was waiting
for delivery was saved, and then ran a second time after the
continuation.
> - This pull request settles an older run's hold through the fallback's
existing machinery, uses plain wording in every remaining recovery
notice (server and UI together), and folds a waiting Retry into the
isolated continuation that answers it.
> - The benefit is that a task held by an older run continues by itself,
or on one Retry, in a fresh isolated workspace. People read only what
happened and the one thing they can do, and a Retry never starts a
redundant second run.

## Linked Issues or Issue Description

**What happened?**
1. An earlier run's saved-work hold (for example "Automatic recovery of
saved work failed … Retry to try again") made the dispatch gate cancel
every new run of the task with `execution_reconciliation_required`. The
fresh-continuation fallback only took holds of the task's latest run.
Each refused run looked like a newer run, so the hold was never settled.
2. Recovery notices and recovery-task comments showed internal codes,
raw IDs, "board", "inspect", "repair" and "no live execution path". The
hold for remote files that could not be restored asked a person to
"verify its exact stop receipt" and to "record workspaceRepairEvidence".
3. On a projectless task whose old legacy writer might still be alive, a
person's Retry pressed after the isolated continuation was recorded, but
before it was delivered, stayed deferred. When the continuation run
ended, task release promoted that Retry into a second, sequential run.
Concurrent writers were already impossible on this path, because the
Retry waited behind the continuation. The second run was redundant: the
continuation was the answer to the person's Retry.

**Expected behavior**
An older run's hold does not block new runs indefinitely. The task
continues in an isolated workspace, by itself while the interruption is
recent, or on a person's Retry. Notices say what happened and the one
thing to do. A person's Retry answered by the continuation never starts
another run.

**Steps to reproduce**
1. Leave a hold from an older run whose saved-work recovery gave up,
then start a new run of the task. Before this change, the run is
cancelled at dispatch every time and the hold stays.
2. Open a task whose run was cut off and its retries exhausted. Before
this change, the notice reads "No live execution path … visible for
intervention".
3. On a projectless task with an isolated continuation recorded, press
Retry before delivery, then let the continuation end. Before this
change, a second run follows it.

## What Changed

- Older-run holds (`fresh-continuation-fallback.ts`,
`modules/run-dispatch/adapters/postgres.ts`):
- A run that was cancelled at dispatch (status `cancelled`, never
started, `execution_reconciliation_required`) never ran and follows no
run. The held-run listing and the fallback's successor check now ignore
it.
- The older run's hold goes through the fallback's existing machinery. A
saved-work recovery that is still in progress keeps its bounded wait.
Holds that a person owns stay. Past the bound, the hold is settled and
the task continues in a new isolated workspace. Held files are kept,
recorded as `kept_not_applied`, and never applied.
- One recency rule, `interruptionEligibility` (the #21222 rule), judges
recency by the newest run that the hold refused at dispatch. It no
longer uses the end time of the hold's own run. If the last attempt is
older than 24 hours, no automatic continuation starts. A person's Retry
always continues the task.
- After the dispatch gate refuses a run for another run's hold, it hands
that hold to the fallback in the same transaction. The normal dispatch
path is not changed.
- A person's Retry and the isolated continuation (`heartbeat/queue.ts`,
`execution-recovery-resolution.ts`, new
`fresh-continuation-retry-fold.ts`):
- The wake-admission transaction that inserts an isolated continuation
run (context `freshContinuationIsolation`) also coalesces the person's
waiting `task_retry` wake for that task, under the task row lock. This
applies only to a Retry with no message attached. The wake gets `status:
coalesced` and `runId` set to the continuation. A server stop after the
commit cannot leave the Retry unfolded.
- The delivery pass is idempotent about this. An action whose own
continuation is already queued is no longer hidden from the pass. When
the pass finds that continuation, it marks the action delivered and
folds any Retry that is still waiting.
- Before this change, the fold covered only a Retry that was already
waiting when the continuation was recorded. A Retry that carries a
person's message is not folded. Only a Retry whose source run is the
continuation's own source run is folded, and only while the continuation
is queued or running. A Retry for the continuation's own failure gets
its own run.
- Plain wording (server and UI together):
- "No live execution path" is now "Task could not continue". One seed
(`buildNoRunningExecutionNoticeSeed`) builds it for every producer.
- Exhausted saved-work recovery: "Paperclip could not recover the
agent's saved files … Press Retry to try again".
- The non-retryable continuation notice no longer shows the error code.
Recovery-task comments no longer show `stranded_issue_recovery`, raw IDs
or retry reason codes.
- `legacy-execution-recovery.ts` uses plain text for remote files that
could not be restored.
- `IssueRecoveryActionCard`: "Board decision required" is now "Waiting
for you".
- The UI still recognizes notices saved with the old text
(`adjacent-errors.ts`, `system-notice-humanizer.ts`).
  - The precise codes stay in the evidence and in the notice details.
- Merge with current master: `recovery/service.ts` keeps master's
three-way choice (retry exhausted / continuation refused /
non-retryable). The non-retryable branch now uses the plain notice
without the code.
- Code comments that the original commits added are removed
(CONTRIBUTING).

## Verification

Red-to-green tests:
- `server/src/services/fresh-continuation-fallback.test.ts`:
- "continues a task held by an older run's saved-work hold in a new
isolated workspace…"
  - "keeps the wait of a saved-work recovery that is still advancing…"
  - "keeps a hold a person owns blocking…"
- "leaves an old hold for a person's Retry, and that Retry runs
isolated" (refused run and waiting Retry)
- "continues a recent restart-stopped run whose prepared remote
workspace could not be restored…" (red on the wording)
- "leaves an older restart-stopped remote restore failure for a person's
Retry, which then runs isolated"
- Same file, "consumes a person's Retry exactly once when the server
stops right after admitting the isolated continuation": the test runs
the admission and nothing after it. The Retry is already coalesced. The
test then resets it to unfolded, as an older release would leave it, and
runs the next delivery pass. The result is exactly one run and the Retry
consumed. Red without the in-transaction fold, and red without the
delivery-pass fold.
- Same file, "never folds a Retry for the admitted continuation's own
failure into that ended continuation": the continuation ends (real
`cancelRun`), then the person presses Retry on it. The Retry gets its
own run. Red without the source-run and liveness guard.
- Same file, "runs a person's Retry pressed before delivery as the
recorded isolated continuation, never in the shared directory": a
projectless task, an old legacy writer that may still be alive, and an
isolated continuation pending. Delivery starts exactly one run, which
carries `freshContinuationIsolation` in a `fresh_isolated_workspace`.
Extended here: after the continuation run ends (real `cancelRun` release
path, then `resumeQueuedRuns` and delivery), no run comes from the
person's Retry. The Retry wake stays `coalesced` into the continuation.
Red without the delivery fold (the Retry became a second run), green
with it.

Local runs on this branch (based on master 566843820c, independent of
#21347), one file at a time:
- `fresh-continuation-fallback.test.ts`: 72 passed
- `__tests__/recovery-wake-herd.test.ts` 18,
`__tests__/issue-recovery-actions.test.ts` 112,
`workspace-recovery-fallback.test.ts` 19,
`__tests__/execution-control-work.test.ts` 27, `heartbeat/queue.test.ts`
101: all passed
- `recovery/stranded-notice.test.ts`: 32 passed
- `system-recovery-state.test.ts`: 11 passed
- `workspace-recovery-warm-handoff.test.ts`: 44 passed
- `modules/run-dispatch/adapters/postgres.test.ts`: 88 passed
- `__tests__/legacy-workspace-restore-recovery.test.ts`: 73 passed
- `__tests__/heartbeat-process-recovery.test.ts`: 457 passed
- UI: `IssueRecoveryActionCard.test.tsx` 70,
`task-chat/adjacent-errors.test.ts` 26,
`lib/system-notice-humanizer.test.ts` 10,
`task-chat/TaskChatSystemNotice.test.tsx` 14, all passed
- `server` `pnpm run typecheck` (CI's layered typecheck, including
tests) and `ui` `tsc --noEmit`: clean

Test expectations changed on purpose, because they asserted the old
wording or behavior:
- `stranded-notice.test.ts`: seed title "No live execution path" is now
"Task could not continue". "retried dispatch/continuation" is now "tried
to start/continue this task again". A new test covers the plain wording.
- `system-recovery-state.test.ts`: the exact exhausted text is now
"Paperclip could not recover the agent's saved files. Press Retry to try
again; …". "stopped after 6 attempts" is now "after 6 tries". The
refusal text no longer contains "(allocation_ownership_ambiguous)".
"names the last failure and its cause" is replaced by "keeps the failure
code and cause in the evidence".
- `workspace-recovery-warm-handoff.test.ts`: `/stopped after 6
attempts/` is now `/after 6 tries/`.
- `heartbeat-process-recovery.test.ts`: new wording replaces "bounded
retry budget" / "no live execution path" (x2), "retried continuation"
(x2), "retried dispatch" and "3x attempts". "non-retryable failure" +
"`budget_blocked`" is now "retrying alone would not fix", with no code
in the body. The recovery-task comment lines are now plain.
- UI: humanizer "Task paused - waiting on a recovery owner" is now "Task
could not continue". `IssueRecoveryActionCard` "Board decision required"
is now "Waiting for you".
- `fresh-continuation-fallback.test.ts` "runs a person's Retry pressed
before delivery…": it asserted that the Retry's wake stays
`deferred_issue_execution` after delivery. It now asserts that the wake
is `coalesced` into the continuation run.

## Risks

- An older run's hold is now settled automatically while the
interruption is recent. The continuation always runs in a new isolated
workspace, and the held files are never applied, so the old run's files
cannot be overwritten or reused.
- A person's Retry without a message, pressed while an isolated
continuation waits for delivery, does not start a run of its own. The
continuation is the answer to that Retry. A Retry that carries a message
still runs. Concurrent writers were already impossible on this path (the
Retry waited behind the continuation); this change removes the redundant
second run.
- Wording only changes what a person reads. The UI still recognizes
notices that older releases saved.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 05:59:52 -06:00
DottaandClaude Opus 5.5 0ec155ac31 fix(recovery): enqueue a fresh interruption's continuation at once and record why a folded wake did not start (#21354)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops before it finishes, for example when a server swap
reaches a sandbox late, the fresh-continuation fallback records one
continuation of the task from its saved work
> - The execution-continuation worker turns that record into a queued
run. It took one record per agent per pass, and it took the oldest held
record first
> - After a large swap, the backlog of old records is hours deep. A
record for a run that ended a few minutes ago waited behind the whole
backlog. The record also stayed outside the queue while the agent's
queue was deeper than its concurrency limit
> - Meanwhile the task showed "todo" with nothing running, and the
assigned-work sweep's wakes for it were folded into the pending record
without any trace
> - This pull request queues a fresh interruption's continuation on the
next pass, keeps a share of every pass for the old backlog, and records
a skipped wake row when a system wake is folded into a pending
continuation
> - The benefit is that an interrupted task gets its next run promptly,
it waits in the agent's queue where the dispatcher ranks it, and nothing
is lost or invisible

## Linked Issues or Issue Description

Refs #21324, #21257, #21234 (earlier pacing and dispatch ranking that
this builds on).

**What happened?**
On a hosted staging instance, five tasks had runs that ended after a
server swap ("The runner restored the conversation after process loss,
but the previous turn is no longer active."). The fallback recorded a
continuation for each task 3 to 6 minutes after the run ended. Each
record stayed `continuationDelivery: pending` for 30 to 110 minutes with
no delivery attempt. During that time the worker delivered continuations
whose source runs ended 4 to 7 hours earlier, one per agent per pass.
The tasks read as failed and needing Retry. The sweep tried 3 to 4 times
per task, and each wake returned `null` without a wake row.

**Expected behavior**
A recorded continuation for a recent interruption becomes a queued run
within about one delivery pass. Old backlog continues to drain and is
not starved. A wake that is not started records why.

**Steps to reproduce**
1. Give an agent a queue that is deeper than its concurrency limit.
2. Record pending continuations for older interruptions that are held
past the hold bound.
3. Record a continuation for a run that ended a few minutes ago.
4. Run `deliverReconciledExecutions`. Before this change the fresh
record is not delivered. The oldest held record is delivered first, one
per agent per pass.

**Agent adapter(s) involved**
Not adapter-specific (core bug).

## What Changed

- `deliverReconciledExecutions` now sorts pending continuations into two
classes by when the source run ended. "Fresh" means that the source run
ended within `RECOVERY_BACKLOG_AGE_MS` (30 minutes), which is the same
value the dispatcher uses to tell fresh recovery from backlog.
- A fresh continuation is enqueued without the queue-depth room check.
It is limited only by the per-agent limit (one fresh continuation per
agent per pass) and the per-company cap per pass. Among fresh records,
the newest interruption goes first. A record whose wake was refused
moves behind records that were not attempted yet.
- Backlog continuations keep their earlier rules. These are the
queue-depth room check, the bypass after the 5-minute hold bound, oldest
held first, and one per agent per pass.
- Each pass runs three phases: fresh up to the company cap minus
`RECOVERY_CONTINUATION_BACKLOG_SHARE_PER_PASS` (1), then backlog up to
the cap, then fresh again up to the cap. When both classes wait, the
backlog gets at least 1 of the 5 slots per company per pass. The
per-company cap (5) does not change, so a swap that interrupts many runs
does not enqueue them all at once. All admissions still go through the
one `wake(...)` call.
- In `enqueueWakeup`, a system wake that is folded into the task's
pending fresh continuation now writes a `skipped` wake row with reason
`issue_continuation_pending`. Before this change it returned `null` with
no trace.
- Tests in `recovery-wake-herd.test.ts`:
- New: a fresh continuation is queued on the first pass while the
agent's queue is deeper than its limit.
- New: a fresh continuation is delivered ahead of an hours-old backlog
for the same agent, and the backlog keeps draining.
- New: many fresh continuations for one agent are enqueued one per pass,
newest first, and none is left behind.
- New: the backlog keeps a share of each pass while fresh continuations
fill the company cap.
- New: a refused fresh wake stays pending and is enqueued on the next
pass.
  - New: a folded system wake leaves a skipped row.
- Updated: existing tests that used a recent source run to mean "held
because the queue is full" now use backlog-age sources, and assertions
that depended on exactly one admission per agent per pass now check the
record under test.

## Verification

- Red before the fix: the 6 new tests failed. The 18 earlier tests still
passed.
- Green after the fix: `cd server && npx vitest run
src/__tests__/recovery-wake-herd.test.ts` gives 24 passed.
- `npx vitest run src/__tests__/execution-control-work.test.ts` gives 27
passed. `src/__tests__/issue-recovery-actions.test.ts` gives 112 passed.
`src/services/fresh-continuation-fallback.test.ts` gives 62 passed.
`src/services/workspace-recovery-fallback.test.ts` gives 19 passed.
`src/services/native-runtime/native-safe-replacement.test.ts` gives 62
passed.
- `cd server && npx tsc --noEmit -p tsconfig.json` passes.

## Risks

- Behavior shift: an agent with a full queue now gets fresh recovery
continuations queued at once. They wait in its queue, where the
dispatcher ranking (person-initiated first, recovery share) decides when
they start. They are no longer held outside the queue. The per-company
cap per pass still limits the burst.
- Old backlog drains at a similar pace as before. It always gets at
least one slot per company per pass when it waits.
- The skipped wake row adds one small row each time the sweep retries a
task that has a pending continuation. The sweep spaces its retries, so
the volume is low.
- No migration. No API or UI contract change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, read-only staging API calls).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 05:46:12 -06:00
DottaandClaude Opus 5.5 e23549cd9c fix(presentation): a task with a continuation waiting to start shows as continuing, not as needing Retry (#21353)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run dies (for example during a server swap), the
fresh-continuation fallback records that Paperclip will continue the
task in a fresh run from saved work
> - The record says `continuationDelivery: "pending"` until a delivery
pass starts the new run. Delivery is paced, so a record can wait a long
time behind an older backlog
> - The task view showed such a task as "working" only for 15 minutes
after the record was made. After that it showed a failed attempt with a
Retry button, although the record still said Paperclip would continue
the task
> - Product rule: a task that the system has recorded it will continue
by itself never shows as failed with a Retry button
> - This pull request ties the "continuing" state to the record's own
delivery state instead of a timer
> - The benefit is that the task view stays truthful however long
delivery takes, and Retry comes back as soon as the system gives up

## Linked Issues or Issue Description

Refs #21264, #21324, #21297, #21320, #21200

**What happened?**

On hosted staging (build d1d71a7755, which contains #21264 and #21324),
five tasks had a run that died after a server swap. Each task had a
recorded fresh continuation that was not delivered yet. The primary
recovery record had cause `native_continuation_requires_reconciliation`,
status `resolved`, `continuationDelivery: "pending"`, isolation
`remote_epoch`, and no `continuationDeliveryAttemptedAt`. The other
holds of the task were coalesced into it. The records were 30 to 110
minutes old. The task view showed responsibility `system`, state
`failed`, and `canRetry: true`.

**Expected behavior**

While a continuation is recorded and its delivery is pending, the task
shows as continuing (`working`, `canRetry: false`, responsibility
`system`). This lasts until delivery starts the new run (then the new
run decides the state) or the system gives up (`invalidated`; then Retry
is offered).

**Steps to reproduce**

1. Record a fresh continuation for a failed run (`continuationDelivery:
"pending"`) and coalesce the task's other holds into it.
2. Let more than 15 minutes pass without delivery.
3. Read the task. It shows `failed` with Retry.

## What Changed

- `taskInterruptionPresentations` in
`server/src/services/task-interruption-presentation.ts`: a primary
fresh-continuation record with `continuationDelivery: "pending"` reads
as "continuing" with no time limit. Before, this applied only for 15
minutes (`RECOVERY_CONTINUATION_DELIVERY_BOUND_MS`). The state is the
existing `working` state that the code already uses for a recorded
continuation; no new state is added. The UI shows the agent's turn as
queued and shows no Retry button.
- A refused delivery attempt (`continuationDeliveryAttemptedAt`) does
not end the state. The record stays `pending` and the delivery pass
tries again on the next pass, so the system still owns the next step.
- The select no longer reads the hold time columns, because the
projection does not use them now.
- `recovery-wake-herd.test.ts`: the #21324 test that expected "not
working" past the bound now expects `working` with `canRetry: false`,
and then proves that the delivery pass still delivers that overdue
continuation.
- No change to the delivery pass, the fallback, or the Retry path.

## What a Retry click does in this state today

`requestTaskRetry` admits a person's Retry as a user wake. The queue's
"a pending fresh continuation absorbs this wake" check
(`hasPendingFreshContinuation` in `heartbeat/queue.ts`) applies only to
system wakes, so the Retry starts its own run from saved work. The
delivery pass then sees a run created after the record and marks the
continuation `delivered` (superseded). In the normal order this does not
start a second run. But the click only asks for what the system already
recorded it would do, and it races the delivery pass: if the pass has
already checked for a newer run and is waking the agent when the click
is admitted, the task can get two continuation runs, one after the
other. That is why the button must not show while the continuation is
pending.

## Verification

- Red (before the fix), `pnpm exec vitest run
src/__tests__/task-interruption-automatic-path.test.ts` in `server/`: 2
failed. The exact staging record shape at 30 minutes read `state:
"failed"`, `canRetry: true`, and a pending record after a refused
attempt read the same.
- Green (after the fix), in `server/`: `pnpm exec vitest run
src/__tests__/task-interruption-automatic-path.test.ts
src/__tests__/recovery-wake-herd.test.ts
src/__tests__/task-interruption-query.test.ts
src/services/task-interruption-presentation.test.ts
src/services/fresh-continuation-fallback.test.ts`: 5 files, 202 tests
pass.
- New tests in `task-interruption-automatic-path.test.ts`:
- The staging record shape at 1, 30 and 110 minutes shows `working`,
`canRetry: false`, responsibility `system`.
  - A refused delivery attempt keeps `working`.
- Delivery given up (`invalidated`, with the coalesced hold) offers
Retry.
- Delivered: the task follows the new run (queued: `working` on the new
run; the new run fails: Retry on the new run).
- A resolved recovery record with no recorded continuation offers Retry
as before.
- A board-owned hold with no recorded continuation offers Retry as
before.
- The #21200, #21264 and #21320 guard tests in the same file pass
unchanged.
- `node scripts/typecheck-layers.mjs` in `server/` passes.

## Risks

- #21324 limited "working" to 15 minutes so that a continuation that
never gets delivered does not hide behind "working". With this change, a
continuation that stays pending forever shows "working" forever. #21324
also made delivery bounded (an overdue record is read on every pass and
gets one wake per agent per pass), and a separate change is making
delivery prompt. A record that cannot be delivered (task done,
cancelled, or reassigned) is marked `invalidated`, and then Retry shows.
- Person-owned waits still win: approval, reconnect, budget, pause and
project access are checked before this state.
- Presentation only. No schema, migration, or execution change.
- This change supersedes the expectation of the #21324 test "does not
show a task as working when its continuation is pending past the
delivery bound". That test hid the continuing state after 15 minutes.
The state must stay truthful: the system has recorded that it will
continue the task, so the task must not show a failed attempt or a Retry
that can start a second run.
- Follow-up: while a continuation waits, a person has no control. On
staging a continuation can wait a long time behind a deep queue. After
#21348 is on master (a person's Retry folds into the pending
continuation in the admitting transaction, and runs that a person starts
rank ahead of system recovery), the planned follow-up keeps the
continuing state and, after 15 minutes, also offers a Retry worded as
"move it forward now" that folds into the pending continuation. This
pull request does not add that Retry.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 05:24:53 -06:00
DottaandClaude Opus 5.5 f0679ee0c7 fix(recovery): bound automatic attempts per task, one deadline per interruption, and batch the invariant report's reads (#21347)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a server swap or a failure interrupts an agent's task, recovery
continues the task automatically: the assigned-work sweep, the
scheduler's bounded retries and native session restart recovery.
> - Release-swap testing on staging found four gaps in that recovery.
The assigned-work report took 60–81 s. One lasting outage was retried by
two owners under two separate caps. A server with a fast clock could
take a session lease that another server still held. Each server
generation started its own 15-minute recovery wait, so the waits chained
for hours.
> - Each gap either slows recovery down or lets a task loop or stall
without any limit.
> - This pull request batches the report's reads, gives each task one
shared hourly cap for automatic attempts, uses the database clock to
judge a lease, and gives each interruption one deadline across server
generations.
> - The benefit is that recovery stays fast and bounded. A lasting
outage gets at most six automatic attempts an hour. A run whose provider
makes no progress ends after 15 minutes, and the fallback then continues
it in isolation.

## Linked Issues or Issue Description

**What happened?**
1. On staging, the assigned-work invariant report took 60–81 s with more
than 1,000 open agent tasks. For each task with nothing live or queued,
the report read the tree pause gate in sequence: one round trip, plus
one per ancestor while a pause hold existed. It also read the company's
agent list once per assignee.
2. A lasting repository catalog outage was retried by two owners. Each
sweep continuation started a fresh run. Each failure of that run got two
more bounded transient retries from the scheduler. One failure could
start up to 18 runs an hour.
3. `claimNativeSessionResumptions` compared the lease expiry with the
claiming server's own clock. A server whose clock ran fast could take a
lease that another server still held.
4. Each server generation's restart-recovery claim started its own
15-minute wait for the provider to recover. Swaps came every 10–14
minutes, so the waits chained for up to 12 generations and none of them
expired.

**Expected behavior**
The report reads pause holds in a fixed number of statements. One task
gets at most one hourly cap of automatic attempts across all owners. The
database clock judges a lease. An interruption with no provider progress
ends after one 15-minute deadline, whatever the number of server
generations.

**Steps to reproduce**
1. Seed 1,500 open tasks with 300 held roots and 400 queued wakes, then
run the invariant report. Before this change it used 2,279 statements.
2. Make the repository catalog unavailable for longer than an hour on an
assigned task. Count the runs.
3. Let a server whose clock is ten minutes fast claim a native session
resumption whose lease is still live.
4. Replace the server every 7 minutes while a native run's provider
makes no progress.

## What Changed

- `issue-tree-control.ts`: a new batched pause-hold read. For a set of
tasks it uses at most two statements: one to find any active hold, then
one recursive ancestor read. It has the same depth bound and cycle guard
as the per-issue gate. `assigned-work-sweep.ts` uses it, and reads the
org chart once per batch to judge each assignee.
- New `recovery/task-automatic-attempts.ts`: one per-task ledger of
automatic attempts in the last hour (sweep continuations plus the
scheduler's `transient_failure` retries). The sweep counts both for its
cap and its spacing (`assigned-work-sweep.ts`). The scheduler's
transient retry lane (`heartbeat/retries.ts`) reads the same ledger.
Once the sweep has continued a task in the hour, a transient retry that
would exceed the cap of six is reported as exhausted. If the sweep never
continued a task, the scheduler keeps its own chain bound.
- `native-finalization-reconciler.ts` (`claimNativeSessionResumptions`):
the compare-and-set also requires the other owner's lease to have lapsed
by `clock_timestamp()`.
- `native-restart-recovery.ts`: each restart-recovery claim records
`interruptionStartedAt` in its recovery-history entry. A later claim
keeps that value while the interrupted attempt shows no provider
progress, and resets it when there is progress. A claim made 15 minutes
or more after the interruption began, with no progress, ends the run as
no longer resumable. This uses the existing terminal path (run failed,
slots and task lock freed), so the fresh-continuation fallback continues
the task in isolation. The bound is
`NATIVE_INTERRUPTION_NO_PROGRESS_DEADLINE_MS` in
`fresh-continuation-fallback.ts`, set equal to the fallback's
exact-recovery cap. Coordinators saved by older releases have no start
time, so the first claim by this code starts the clock. The normal run
path is not changed.
- Review fixes:
- A graceful drain no longer counts as "no progress". The claim always
reads provider progress. The drain only decides whether the resume
spends an attempt, so a run that progressed before a deploy is never
ended by the deadline.
- The transient lane checks the hourly cap again inside the task-locked
transaction, after it reuses an existing successor and before it inserts
a new one. The sweep already reads the ledger under the same lock.
- A same-run resume claim that the database clock rejects (another
owner's lease is still live) no longer pushes that owner's
`nextAttemptAt` out.
- Code comments that the original commits added are removed
(CONTRIBUTING). Merged with current master (includes #21336):
`assigned-work-sweep.ts` keeps master's `BLOCK_ATTRIBUTION_WINDOW_MS`
import and adds the ledger import.

## Verification

Red-to-green tests, each red on the parent commit and green with the
fix:
- `server/src/__tests__/assigned-work-sweep-round-trips.test.ts` "reads
pause holds and assignees once per batch on 1,500 open tasks…": 1,500
tasks, 300 held roots plus 300 children, 400 queued wakes, 20 agents.
Red: 1,100 per-task gate reads and 2,279 statements. Green: 0 and 239.
Report verdict counts are asserted to be the same before and after.
- Same file, "the batched pause read gives the per-issue gate's answer:
deep chains, cycles, released holds, other companies": checks that the
batched read gives the same result as the per-issue gate.
- `server/src/__tests__/assigned-work-invariant.test.ts` "counts the
scheduler's transient retries of the same task against the one
continuation cap": red (repaired), green (deferred_cap).
- `server/src/services/heartbeat/retries.test.ts` "a lasting outage gets
at most the task's hourly cap of automatic attempts across the sweep and
the transient lane".
- `server/src/__tests__/native-session-resumption.test.ts` "judges
another owner's lease by the database clock, not a server clock that
runs fast".
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
"resumes a run that progressed for 20 minutes before a graceful drain
suspended it" (red before: the deadline ended it) and "still ends a run
with no provider progress at the deadline when a graceful drain
suspended it".
- `server/src/services/heartbeat/retries.test.ts` "re-checks the task's
hourly cap under the task lock when the sweep records an attempt while
the retry is being prepared" (red before: attempt seven was scheduled).
- `server/src/__tests__/native-session-resumption.test.ts`: the
clock-skew test also asserts that `nextAttemptAt` is not pushed out (red
before).
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
"ends a run whose generations show no progress at the deadline from the
first interruption, not per generation", "restarts the clock when a
generation's resume made provider progress", "starts the clock at the
first claim for a coordinator saved before the deadline existed".

Local runs on this branch (based on master 566843820c):
- `cd server && pnpm exec vitest run <file>`, one file at a time:
  - `src/__tests__/assigned-work-sweep-round-trips.test.ts`: 2 passed
  - `src/__tests__/assigned-work-invariant.test.ts`: 31 passed
  - `src/services/heartbeat/retries.test.ts`: 30 passed
  - `src/__tests__/native-session-resumption.test.ts`: 20 passed
-
`src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
217 passed, 14 skipped
- `src/__tests__/heartbeat-native-restart-suspension.test.ts`: 115
passed
- `cd server && pnpm run typecheck` (CI's layered typecheck, including
test files): clean.

- `cd server && pnpm exec tsc --noEmit`: clean.
- Timing note: local round trips take about 0.2 ms, so local
milliseconds do not show the gain (736 ms became 303–583 ms on a loaded
machine). At staging's ~50 ms per round trip, the ~2,040 sequential
statements removed are about 100 s.
- No existing test expectation was changed.

## Risks

- After this change, a task the sweep continued in the last hour can get
fewer transient retries. This is intended: one task gets at most six
automatic attempts an hour. A person's Retry is not counted in this
ledger.
- The interruption deadline ends a native run that shows no provider
progress for 15 minutes across server swaps. The fallback then continues
it in a fresh isolated workspace, so the task is not left for a person
to fix. A run that shows progress resets the clock.
- Coordinators saved by older releases start the clock at their first
claim by this code, so an upgrade does not expire any run immediately.
- No migration. No schema change. The new history field is inside
existing JSON.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 05:10:28 -06:00
DottaandClaude Opus 5.5 ff971149c2 fix(dispatch): bound how long a queued run waits and keep a person's interrupted Retry person-first (#21349)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue decides which queued run an agent starts when a
slot frees up
> - Recent changes start person-initiated work first and give recovery
runs one start in four
> - A system continuation of a person's interrupted Retry ranked as
old-backlog recovery, and the recovery share always went to fresher
recovery runs
> - With a steady stream of new work, that continuation never started,
so the person's task sat queued for hours
> - This pull request adds aging and a longest-waiting share, keeps a
person's interrupted request person-first, and passes the turn on when a
long-queued run cannot be claimed
> - The benefit is that every ready queued run starts within a bounded
number of starts, and a person's Retry is not lost behind system work

## Linked Issues or Issue Description

**What happened?**

On a hosted staging instance, one agent had 28 runs running and kept
starting newer runs. The company had 67 queued runs, and 54 of them were
older than one hour. A person pressed Retry on a task. The Retry run was
interrupted by a server restart two minutes later. The system queued an
automatic continuation (`issue_continuation_needed`) with
`previousRunId` set to the interrupted Retry run. That continuation
stayed queued for more than 2.5 hours. The task showed as "working".
Other queued runs of the same person (some queued for more than 10
hours) also did not start.

Root cause:
- The dispatcher ranked the continuation as `recovery_backlog`, the last
tier, because it was a system wake and its source run ended more than 30
minutes ago. The person-first check only looked at wake requests
attached to the queued run itself. It did not look at the source run,
which was the person's `task_retry`.
- The 1-in-4 recovery share picked the first recovery run in sort order.
Fresh recovery runs sort before backlog runs, and fresh recovery runs
kept arriving, so the backlog run never got the share.
- Nothing aged a queued run. A run in a lower tier or with a lower
priority could wait forever.

**Expected behavior**

A system continuation of a person's Retry ranks as person-initiated.
Every ready queued run starts within a bounded number of starts, even
when fresh person and system work keeps arriving. A run that keeps
failing to claim is reported and does not block the next-oldest run.

**Steps to reproduce**

1. Give an agent one free slot and several queued runs.
2. Queue a person's `task_retry` run, start it, and interrupt it.
3. Queue a system continuation with `previousRunId` set to the
interrupted run, and let its source age past 30 minutes.
4. Keep adding fresh system and person work.
5. See that the continuation never starts.

**Paperclip version or commit**

Staging Core 7e7452e88f (contains #21234 and #21277). Fixed on top of
origin/master cb94108484.

**Deployment mode**

Hosted (authenticated, public exposure, managed cloud).

**Agent adapter(s) involved**

Any adapter. The bug is in queued-run dispatch order, before the adapter
starts.

Refs #21234, #21277

## What Changed

- `queued-run-dispatch-order.ts`: a ready run that has waited for one
hour (`QUEUED_RUN_AGING_MS`) moves up to at least the work tier.
- `queued-run-dispatch-order.ts`: every fourth start
(`OLDEST_DISPATCH_SHARE`) goes to the longest-waiting ready run that has
aged, whatever its class or priority. The dispatcher tracks this share
inside one pass.
- `queue.ts`: the dispatcher reads the agent's recent starts to know how
many starts passed over the oldest run. One query now serves both the
recovery share and the oldest-run share.
- `queue.ts`: a queued run is person-initiated when any run in its
source chain (`previousRunId`, then `retryOfRunId`, up to 8 runs, in one
recursive query) was a person's request: a `task_retry` or chat wake, a
run with a `taskRetryRequestId`, or a run whose wake request a user
made. A person's Retry stays first after repeated interruptions.
- `queue.ts`: when a run that has waited past the aging bound is offered
a slot and cannot be claimed, the server logs a warning with the reason
and the wait time. That run skips the oldest-run share for one aging
period, so the next-oldest run takes the turn. It is still offered a
slot in normal order on every pass.

## Verification

- `npx vitest run
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/recovery-wake-herd.test.ts` (38 passed).
- Red/green: with the old dispatcher and queue, these new tests fail and
the fix makes them pass:
- "starts an old queued continuation within a bounded number of starts
while fresh person and system work keeps arriving" (unit, continuous
stream of fresh person, work and recovery runs)
- "serves the longest-waiting run once every share even when fresher
work outranks it" (unit)
  - "lets an old-backlog recovery run age into the work tier" (unit)
- "starts the system continuation of a person's interrupted Retry ahead
of fresher system work" (embedded Postgres, the staging case)
- "gives the longest-waiting queued run a free slot after three fresher
starts passed it over" (embedded Postgres)
- "keeps a person's Retry first after its system continuation is
interrupted again" (embedded Postgres, two successive interruptions)
- "starts the oldest continuations of a long backlog within a bound
while later assignments and fresh recovery keep arriving" (unit, the
staging shape: 3 continuations 4-5 hours old behind 80 later
assignments)
- Added coverage that passes before and after: "starts a system
continuation that a person's Retry coalesced into ahead of fresher
system work", "passes the oldest-run share to the next oldest run while
the oldest one keeps failing to claim", and "does not use the oldest-run
share for a run that has not waited past the aging bound".
- Related suites pass: heartbeat-run-dispatch-work,
heartbeat-run-dispatch-lifecycle, heartbeat-queued-run-claim-isolation,
issue-reassignment-queued-run, heartbeat-startup-fresh-dispatch,
heartbeat-stale-queue-invalidation,
heartbeat-ephemeral-queued-migration, queued-comment-consumption (116
tests).
- `tsc --noEmit -p server` passes.
- Not related to this change: `heartbeat-stale-queue-invalidation`
"resumes a rejected connection_intent outcome and promotes the claimed
review task" also fails locally on the base commit without this change
(one queued run, so the new code does not run).

## Risks

- Behavior change: when runs have waited more than one hour, one start
in four goes to the oldest of them, even when fresher person work is
queued. A person's fresh work still gets the other three starts.
- More runs can rank as person-initiated: system continuations and
retries of a person's run. This is the intended outcome.
- One more small recursive query per dispatch pass (the source chain, at
most 8 hops per run, by primary key and the wake-request run index). It
only runs when more than one run is queued and some have a source run.
- The record of runs that failed to claim is in memory. After a restart,
a stuck run gets the oldest-run share again once, then is skipped again.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, browser for read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 04:59:56 -06:00
DottaandClaude Opus 5.5 566843820c fix(recovery): a queued continuation is not shown blocked, recent temporary failures continue, and notices say what a person can do (#21336)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, recovery continues the task, shows the
person a notice, and sweeps for tasks that nothing will continue.
> - On staging, a task stayed "Blocked / needs attention" for 24 minutes
while its Retry's run was queued.
> - Two tasks escalated to the board after a sandbox resume timeout, a
temporary failure, and nobody continued them.
> - Recovery notices asked a "Board operator" to inspect evidence and
repair sandboxes. Recovery must work without that.
> - A children-completed system wake could start on a parent that was
already closed.
> - This pull request fixes those four gaps.
> - The benefit is that interrupted tasks continue on their own, and
notices tell a person only what happened and the one thing they can do.

## Linked Issues or Issue Description

Refs #21250, #21257 (merged), #21258, #21234

**What happened?**
1. A task whose continuation was queued was shown as blocked (one
staging task). Execution recovery blocked the task at 23:51:28, and
restart recovery's Retry queued the successor at 23:51:50 through
`admitExplicitNativeContinuation`. That path resolves the interruption's
holds, but it never lifted the block those holds owned. The run started
at 00:15, so the task was queued for 24 minutes and shown blocked the
whole time. This was not a race: the issue's `updatedAt` equals
`blockedTransitionAt`, so nothing wrote to the issue after the Retry.
2. Board-owned stranded holds after a temporary failure had no continuer
(two staging tasks). A person's Retry failed while resuming the saved
sandbox (`reusable_sandbox_resume_failed`, cause `transport_timeout`,
`providerWorkStarted=false`), and stranded recovery escalated the task
to the board. The sweep leaves a blocked task that has any active hold
to the hold's owner. The fresh-continuation fallback takes only
execution-reconciliation holds. So nobody continued these tasks.
3. Recovery notices addressed a "Board operator" and asked people to
inspect evidence and repair runtimes, sandboxes or workspaces.
4. An `issue_children_completed` system wake could start on a parent
that was already done or cancelled.

**Expected behavior**
A task with a queued continuation is not shown blocked. A recent
temporary failure is continued automatically. A notice says what
happened and the one thing a person can do. A system wake does not start
on a closed task.

**Steps to reproduce**
1. Fail a run so that execution recovery blocks the task, then press
Retry while the agent's queue is full. The task stays Blocked until the
run starts.
2. Make a saved-sandbox resume time out on a person's Retry. The task
escalates to the board and stays blocked.

## What Changed

- **Queued continuation is not shown blocked**
(`explicit-native-continuation.ts`,
`recovery/assigned-work-invariant.ts`,
`recovery/assigned-work-sweep.ts`):
- `admitExplicitNativeContinuation` (a person's Retry, restart
recovery's Retry, or a saved message) moves the task from Blocked back
to To do when the replaced interruption set the block. That means the
hold's own block receipt, or a block written at or after the source run
ended with no later decision. A declared unblock action or an open
blocker keeps the block.
- New invariant violation `blocked_with_live_run`: the task is blocked,
has a queued or running run, and is not waiting on anyone else. Under
the row lock, the sweep lifts that block without starting a run or
spending the cap. Its suspect pre-filter now lets blocked tasks with
live runs through.
- **Recent temporary failures are continued** (new
`recovery/stranded-hold-continuation.ts`,
`recovery/assigned-work-sweep.ts`, `fresh-continuation-fallback.ts`,
`recovery/stranded-notice.ts`):
- The assigned-work sweep is the single owner of board-owned
`stranded_assigned_issue` holds whose failed run ended within the last
24 hours with a restart-class code, or with a temporary setup failure
before provider work (for example a sandbox resume refused for a
timeout).
- Under the row lock it resolves the hold (outcome `restored`, owner
system), restores the task's status from before the block, and queues
one continuation under the usual per-task cap and spacing.
- Older escalations and lasting causes (access denied, lost sandbox,
unknown) stay with their owner. The fallback explicitly never takes a
`stranded_assigned_issue` hold.
- **Plain recovery notices** (`recovery/service.ts`,
`recovery/stranded-notice.ts`, `recovery/successful-run-handoff.ts`):
see the wording table below.
- **Closed parent** (`heartbeat/queue.ts`): `issue_children_completed`
is added to the claim-time cancel for system continuations on a done or
cancelled task. A person's saved message still runs, and so does
onboarding's completion report (`onboardingCompletion: true`), which is
meant to run on a done parent.
- **Review fixes** (Greptile, first round):
- A person's block is never lifted by a Retry or saved message. The
block-time inference applies only when no hold carries a block receipt,
and a block whose activity entry was written by a person is kept. This
is the same signal the sweep uses (`attributeBlock`, so `blockedBy:
"user"` is a wait).
- The sweep's suspect query lets a blocked task with a live run through
the queued/claimed-wake filter, so a continuation queued by
`enqueueWakeup` (a queued wake bound to its run) reaches the
`blocked_with_live_run` repair.
- The saved-sandbox and workspace-setup timeout notices promise an
automatic retry only when the sweep will continue that hold. Both use
one decision (`strandedRunCauseIsTemporary`, inside
`strandedHoldContinuesAutomatically`). Otherwise the notice says to
press Retry.

### Old → new wording

| Where | Old | New |
|---|---|---|
| service.ts hold nextAction, successful run with missing state | Board
operator: inspect the run evidence, then explicitly choose a valid issue
disposition... | The agent finished without saying whether the task is
done. Mark the task done if it is, or press Retry to let the agent
continue from the saved work. |
| service.ts, process_lost / default | Board operator: inspect the retry
history / inspect the evidence, repair the runtime if appropriate... |
This attempt could not be finished. Press Retry to start it again from
the saved work. (`RETRY_FROM_SAVED_WORK_NEXT_ACTION`) |
| service.ts, provider_quota | Wait for provider quota recovery, then
retry the original assignee; do not wake a takeover owner. | The AI
provider's usage limit was reached. The same agent continues the task
once the limit resets. |
| service.ts, environment provider | No agent work started. Retry the
task once the environment provider is available. | The sandbox service
was unavailable, so no agent work started. Press Retry once it is
available again. |
| service.ts, inactivity | Board operator: inspect the inactivity
evidence... | The agent stopped responding, so this attempt was ended.
Press Retry to start it again from the saved work. |
| service.ts, worktree branch incoherence | Board operator: repair the
source task git worktree branch incoherence... | The task's workspace no
longer matches its branch, so the agent did not start. Press Retry to
start it again. |
| service.ts, base materialization | Board operator: repair the project
workspace repository URL or clone access... | Paperclip could not copy
the project's repository. Restore access to the repository in the
project settings, then press Retry. |
| service.ts, workspace link | Board operator: repair the source task
workspace link, project workspace cwd, or git checkout... | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| service.ts, base ref | Check the starting branch and repository
access, then repair the task's branch... | The starting branch for this
task's workspace was not found in the repository. Set a starting branch
that exists, then press Retry. |
| service.ts, sandbox plugin | Board operator: the sandbox provider
plugin named in the run failure is not ready; ... explicitly retry... |
The plugin that starts this agent's sandbox is not running. To fix it,
&lt;remedy&gt;, then press Retry. |
| service.ts, missing secret | Board operator: bind the missing
secret(s) named in the run failure... | A secret this agent needs is not
set. Add it in the agent's settings, then press Retry. |
| service.ts, review participant | Board operator: repair the failed
review participant path... | The reviewer did not finish the review.
Choose a reviewer, or press Retry to ask the same reviewer again. |
| service.ts failure summary | Latest retry failure details were
withheld from the issue thread; inspect the linked run for evidence. |
The failure details are kept with the run. |
| service.ts nested recovery comment | the assigned recovery owner or
board operator should fix the runtime/adapter problem... | press Retry
on the original task to start it again from the saved work, then close
this task. |
| service.ts recovery-issue stop comment | the current recovery owner
should inspect the failed run evidence, restore a live execution path...
| press Retry to start this task again from the saved work. |
| service.ts deliberate-wait exhaustion hold and comment | Inspect the
evidence and choose whether to repair, retry... / Next action: repair
the liveness disposition or request an explicit source-owner decision. |
The agent stopped without saying what happens next with this task. Mark
the task done if it is, or press Retry... / Next action: mark the task
done if it is, or press Retry... |
| stranded-notice REMOTE_WORKSPACE_RESTORE_NEXT_ACTION | Open the failed
run... inspect and move it aside, then retry | The saved work could not
be put back into the sandbox, so no agent work started. The saved
version was kept. Press Retry to start again from it. |
| stranded-notice scan notices (timeout, plugin worker, saturated,
output limit, failed, cancelled) | Check repository access and server
load / Check the worker status / Inspect the failed run... | "&lt;what
happened&gt;, so no agent work started. Press Retry ..." |
| stranded-notice workspace validation seed | ...Moving it to `blocked`
so the workspace link, cwd, or git checkout can be repaired | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| stranded-notice base ref seed | ...blocked until its workspace
configuration or repository access is repaired / Check that the
configured base ref exists... | starting branch not found / Set a
starting branch that exists in the repository, then press Retry. |
| stranded-notice stale shared holder | Check whether the other run is
still working and stop it if it is stuck... | Stop the other run on this
task, or choose a separate workspace for this task, then press Retry. |
| stranded-notice secret default | required secret/env bindings are
missing. Moving it to `blocked` so an operator can bind... | The agent
did not start because a secret it needs is not set. Add the secret in
the agent's settings, then press Retry. |
| stranded-notice review seeds (2) | ...so the board can inspect the
evidence, repair the reviewer runtime... | The reviewer ... did not
finish / cannot take work. Choose a reviewer, or press Retry ... |
| stranded-notice default body | could not restore a live execution
path... visible for intervention | This attempt could not be finished
automatically. Press Retry to start it again from the saved work. |
| stranded-notice metadata | Board decision required / "Inspect the
evidence, then retry ... repair the execution path ..." / "The recovery
owner should either restore a live execution path..." | Waiting for you
/ `RETRY_FROM_SAVED_WORK_NEXT_ACTION` / The assigned agent continues the
task from the saved work |
| successful-run-handoff.ts | Board decision required / inspect the
evidence, then retry the original owner... | Waiting for you / Mark the
task done if it is, or press Retry to let the agent continue from the
saved work |
| stranded-notice, saved sandbox resume timeout and workspace setup
timeout | asked to inspect the run or the provider | No agent work
started; Paperclip tries again automatically. |

Kept on purpose:
- `buildImmediateExecutionPathRecoveryNoticeSeed` ("No live execution
path" title, "retried continuation ... visible for intervention" body).
The UI recognizes this exact text
(`ui/src/components/task-chat/adjacent-errors.ts` regex and title,
`ui/src/lib/system-notice-humanizer.ts`) and humanizes it, so changing
it needs a UI change in the same PR.
- The UI's own "Board decision required"
(`IssueRecoveryActionCard.tsx`).
- Recovery-issue comments that still contain internal codes
(`stranded_issue_recovery`).
- The agent-facing disposition instruction in
`successful-run-handoff.ts`.

### Deliberately changed test expectations

Each of these asserted the old wording or the old behavior:
- `server/src/services/recovery/stranded-notice.test.ts`: stale shared
holder; base ref body and next action; scan timeout metadata; plugin
worker metadata; secret default body; owner row is now "Waiting for
you".
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: owner row
is now "Waiting for you" (3 places); failure-summary line;
board-escalation nextAction expectations (review participant,
process_lost, missing disposition, deliberate wait, default); sandbox
plugin; workspace link and validation comments; missing secret action
and comment; review participant comment.
- `server/src/__tests__/assigned-work-invariant.test.ts`: the old "left
to the hold owner" test now uses a 30-hour-old run. A recent temporary
failure is now continued, which is the new behavior.
- `server/src/services/heartbeat/queue.test.ts`:
`issue_children_completed` is now in the claim-time cancel list.
- `server/src/services/heartbeat/run-completion.test.ts`: the
restore-failure next action now says "The saved version was kept".
- `server/src/__tests__/issue-recovery-actions.test.ts` and
`server/src/__tests__/heartbeat-workspace-branch-containment.test.ts`:
the branch-incoherence next action now says the "workspace no longer
matches its branch".
- `server/src/services/recovery/stranded-notice.test.ts`: the timeout
notice is checked for both an automatic and a manual retry.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `task-retry.test.ts` "lifts the failure's block once the Retry's run
is queued..."; invariant unit test and sweep integration test for
`blocked_with_live_run`.
- sweep integration test "continues a recent board escalation of a
temporary setup failure...". A recent one is continued; a 30-hour-old
one and `provider_access_denied` are left with their owner. Plus a
`stranded-notice.test.ts` case.
- the wording expectations above.
- `heartbeat/queue.test.ts`.

Review-fix tests, each red before its fix:
- `task-retry.test.ts` "lifts the failure's block...": new `newer` case
(the hold's receipt names an older status version) and `person` case (a
person's activity entry set the block). Both stay blocked.
- `__tests__/assigned-work-invariant.test.ts` "a recovery block whose
task already has a queued run...": now seeds a queued wake bound to the
queued run, as `enqueueWakeup` does, and asserts `listSuspects` returns
the task.
- `heartbeat-dependency-scheduling.test.ts` "dispatches only a verified
completed onboarding report" (it failed in CI on the first push) passes
again.

Evidence:
- Item 1: the staging task's activity log. The issue's `updatedAt`
equals `blockedTransitionAt`; nothing wrote after the Retry.
- Item 2: the two staging runs ended at 12:28Z and 14:23Z on Nov 12,
inside the 24-hour window.
- Ruled out, so there is no change for it: a third staging task is
waiting on a real person's gate. A pending `request_confirmation` was
created before the fallback run failed.

Local runs, on master after #21257 (e063587edd) plus these commits
(server `tsc --noEmit` also passes):
- `stranded-notice.test.ts`, `recovery/assigned-work-invariant.test.ts`,
`task-retry.test.ts` and `__tests__/assigned-work-invariant.test.ts`: 4
files plus `heartbeat/queue.test.ts`, 329 tests passed.
- `heartbeat-process-recovery.test.ts` and `recovery-wake-herd.test.ts`:
475 tests passed.

Code comments that the original commits added were removed to follow
CONTRIBUTING; the explanations are in this description.

## Risks

- The sweep now resolves board-owned stranded holds for recent temporary
failures. It is bounded to failures within the last 24 hours with a
restart-class or temporary-setup cause, one continuation per task under
the usual cap and spacing. Lasting causes stay with their owner.
- Lifting a block on explicit continuation is limited to blocks the
replaced interruption set. A declared unblock action or an open blocker
keeps the block.
- Wording only changes server text. The UI-recognized "No live execution
path" notice is unchanged.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 04:04:20 -06:00
DottaandClaude Opus 5.5 43aabb7080 fix(runner): keep configured secret values out of OpenCode output, storage and model requests (#21330)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can configure a secret that Paperclip delivers to an
agent's commands as an environment variable
> - The value must stay out of what the agent runtime stores and what it
sends to the model
> - With the OpenCode runtime, a command that printed the variable put
the value in OpenCode's tool output. OpenCode stored that output in its
session database and sent it to the model. Paperclip's own run log
already redacted it
> - Paperclip did not tell the runner which configured values are
secrets, so the runner could not protect them without also hiding
ordinary settings
> - This pull request sends secret names (never values) to the runner,
filters exact secret values before OpenCode captures any tool output,
redacts them from model requests, and tells the agent how to check a
secret without printing it
> - The benefit is that a configured secret stays out of model context
and out of runtime storage, even when an agent prints it

## Linked Issues or Issue Description

**What happened?**
The `chat-features.runner-opencode.local.secrets.personal-secret` eval
failed because the persisted-state scan found a configured secret value
in OpenCode's session database. The person entered the value through the
normal secret entry form. Nobody pasted it in chat. The agent ran a
shell command that printed the environment variable it was given.
Paperclip's run log redacted the value. OpenCode stored the raw output
in its database and included it in the next model request.

**Expected behavior**
A configured secret must never reach model context or runtime
persistence. Ordinary configured values must not be redacted.

**Steps to reproduce**
Configure a secret environment variable for an OpenCode agent. Have the
agent run a command that prints the variable. Then search the OpenCode
session data and the model requests for the value.

## What Changed

- **Secret names cross to the runner.**
`configuredEnvironmentProjection` takes the controller's secret key
names and adds `PAPERCLIP_SECRET_ENV_KEYS`. This marker holds only the
names of configured entries that came from secrets. It travels with the
configured projection across the TypeScript and Rust runner boundaries
(`configured-environment.ts`, `configured_environment.rs`). The server
passes `secretKeys` in `runtime-execution.ts`.
- **Command output is filtered before OpenCode reads it.** For a launch
that has configured secrets, the runner sets OpenCode's `shell` to a
small POSIX wrapper (`drivers/opencode/output-custody.ts`). The wrapper
runs the shell OpenCode would use anyway. It sends stdout and stderr
through an `awk` filter that replaces exact secret values. OpenCode
never receives the raw bytes, so its streamed previews, its database and
its model requests stay clean. The wrapper keeps the exit status. It
does not wait for background processes. The values come from the
environment that OpenCode already holds and are never written to disk.
- **Other tool results are rewritten before they are saved.** A local
OpenCode plugin rewrites completed non-shell results (file reads,
searches, MCP results) in `tool.execute.after`. OpenCode runs that hook
before it stores or returns the result. The wrapper refuses to run any
command until the plugin has started for the current launch (per-launch
nonce). So no command can write a secret to a file that a later read
returns without filtering.
- **Model requests are redacted as a second layer.** The OpenCode
provider proxy removes exact secret values from the string values of
each parsed model request. Field names and other literals are never
changed. A quick byte scan for each value, raw and JSON-escaped, sends
the original bytes unchanged when no value is present.
- **The agent gets guidance.** Native task prompts list the configured
secret names. They say to check a secret with `[ -n "$VAR" ]` and never
to print, echo, log or write a secret value.
- **No added startup delay.** The wrapper is content-addressed under the
runtime directory and reused across launches, so it is not rewritten on
every launch. OpenCode's plugin-dependency install is marked as already
done, so loading the plugin does not add wait time to the first tool
call.
- **Only exact values are redacted.** Command output is filtered line by
line, so each line of a multi-line value that has at least 4 characters
is redacted too. If a configured secret has no line of 4 or more
characters, the launch is refused with
`opencode_output_custody_unavailable` instead of running unprotected.

## Verification

-
`packages/paperclip-runner/src/drivers/opencode/output-custody.test.ts`
(10 tests): the wrapper filters stdout, stderr and a partial last line;
keeps exit codes (7, and 143 for SIGTERM); filters each line of a
multi-line value; refuses to run before plugin readiness or with an old
nonce; returns at once when a background process keeps running; leaves
ordinary values unchanged. The plugin rewrites nested results.
Model-request redaction covers raw and JSON-escaped values and never
changes field names or literals (a secret equal to `true` leaves
`"stream": true` intact). Secrets the filter cannot match are refused.
No value is written to disk.
- `opencode-server-driver.test.ts`: one saved session launches three
times. First with no secret marker (older controller), then with the
marker, then with no marker again (rollback). The first launch is
unchanged, the second applies custody, and the third launch still
starts. This test fails without the driver change.
- `configured-environment.test.ts`: the marker names only secret
entries, crosses the runner spawn and OpenCode boundaries, reads as "no
secrets" when absent, and is refused when malformed. Rust:
`secret_name_marker_crosses_only_with_its_projection`.
- `server/src/__tests__/configured-secret-guidance.test.ts`: guidance
content, and no guidance when there are no secrets.
- Real OpenCode 1.18.34 qualification (`PAPERCLIP_OPENCODE_QUALIFY=1
node --test scripts/qualify-opencode-runtime.test.mjs`). A fake model
asks for a command that prints the secret and writes it to a file, then
a read of that file. Unprotected baseline: the value is in the session
database and in model requests (red). With custody: it is in neither,
and the exit status is kept (green).
- **Latency** (local, macOS, real OpenCode, 8 runs each): boot 629 ms vs
697 ms baseline; first model request after the prompt 678 ms vs 698 ms
(no regression); first command 139 ms vs 191 ms; each later command +9
ms (36 vs 27 ms). Proxy cost per request when no secret is present (the
normal case): 33 µs at 50 KiB, 164 µs at 300 KiB, 0.55 ms at 1 MiB, 2.3
ms at 4 MiB. When a secret is present: 91 µs, 0.48 ms, 1.6 ms and 6.8
ms. The scan runs only for launches that have configured secrets, and it
adds nothing to OpenCode's own startup.
- A narrow paid run of
`chat-features.runner-opencode.local.secrets.personal-secret` on this
branch is linked in a comment.

## Risks

- **Previous-release story.** The marker is new and optional. A runner
that receives no marker (older controller) launches exactly as before.
An older runner drops the marker at its environment boundary. The change
saves no new run or session state. OpenCode configuration is regenerated
on every launch, so sessions saved before this change resume under
custody (covered by the cutover test). Warm-process reuse keys include
the marker, so a warm process from before the deploy is not reused for a
run that has secrets.
- **Commands go through the wrapper.** Output arrives line by line.
Output written after the shell exits is dropped, which matches how
OpenCode stops reading at exit. If `awk`, `mkfifo` or `rm` is missing,
the launch is refused with `opencode_output_custody_unavailable` and
does not run unprotected.
- **Scope.** Only exact values are redacted. An agent that encodes a
value (for example base64) is out of scope. The model-request layer
applies when Paperclip routes the model through its proxy (OpenRouter
and Paperclip connections).

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:23:56 -06:00
DottaandClaude Opus 5.5 0986c3eb3a fix(native): let a same-run reattachment take over its own retained remote Runner cleanup (#21328)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 keeps an unreachable remote Runner alive when the failed
attempt closes, so that the same-run reattachment can adopt it
> - The same close also leaves the session in the in-process cleanup
quarantine. The quarantine refuses every new session of the task, and
that includes the reattachment that the Runner was kept for
> - Close retries cannot settle it. After the first close fails, the
Runner transport returns the same rejected close for every retry
> - On staging, this was the largest failure class: 14 Codex runs in one
hour failed with `native_session_cleanup_quarantined`, while only 5 runs
succeeded. Each task then needed Retry
> - This pull request lets the same run's reattachment to that Runner,
in the same remote sandbox, take over its own retained cleanup
> - The benefit is that the task continues on its own, with no spent
attempt, no failed run and no Retry

## Linked Issues or Issue Description

Refs #20923, #21031

**What happened?**
A remote native run (Codex on Daytona) failed a turn with a recoverable
error, for example `runnerd did not report the provider turn identity`
or `native_finalization_missing: session returned no semantic result`.
The close then logged `runner did not prove durable suspension before
checkpoint` and `remote runner left alive unsettled for same-run
reattachment`. Each later same-run reattachment was refused at
admission:
- When the close error was retryable, admission waited out the grace and
failed with `native_session_cleanup_quarantined: prior session cleanup
remains incomplete`. That spent a same-run attempt. After the attempt
budget, the run failed. All close retries replayed the same failed close
within milliseconds (`runner transport close requested: native session
quarantined cleanup recovery`), so they could never settle the
quarantine.
- When the close error was `native_session_close_unrecoverable`
(operator-required), the next attempt failed at once with
`native_session_cleanup_quarantined: the previous session is still
shutting down; Paperclip finishes the cleanup automatically and the
saved work is kept.` That code is a permanent failure.

"Paperclip finishes the cleanup automatically" did not happen in the
process. Only a server restart (a deploy) removed the in-memory
quarantine, and by then the run had failed and the task waited for
Retry. Staging restarts every 20 to 45 minutes, so attempts often end
mid-cleanup and meet this gate.

**Expected behavior**
The same run's reattachment adopts the Runner that its earlier attempt
kept for it. It does not wait for a close that cannot settle, and it
does not spend an attempt. The task continues without Retry. Other runs
of the task and other sandboxes stay gated.

**Steps to reproduce**
1. Start a remote native run, and let its turn fail with a recoverable
transport error while the Runner cannot prove suspension.
2. The close keeps the Runner alive and quarantines the session.
3. The server schedules the same-run reattachment 30 s later.
4. The reattachment fails with `native_session_cleanup_quarantined`, and
the task needs Retry.

**Paperclip version or commit**
Deployed Core `c730872974` on staging. Reproduced on `master` at
`49f0030f06`.

**Deployment mode**
Hosted cloud (staging), Daytona remote sandboxes.

**Agent adapter(s) involved**
`paperclip_runner` native runtime with the Codex app server backend.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
`executeNativeSession` takes a new option `retainedRunnerReattachment`.
With a remote cleanup scope, admission retires quarantine entries that
belong to the same run in the same remote scope. Removal stops later
close retries. A close attempt that is still in flight stays an
admission owner until it settles, so the old and new controllers do not
overlap. Other runs, other sandboxes and local backends keep the gate.
- `server/src/services/native-runtime/native-session-executor.ts`: sets
`retainedRunnerReattachment` only for a `reattach_remote_runner`
recovery of the same run.
- `doc/execution-semantics.md`: records the takeover.
- Tests:
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: four
new cases. An operator-required close with a retained Runner. A
retryable close whose retries replay one failure. A close that a deploy
interrupts while it is in flight (the reattachment waits for it, then
takes over). Controls: another run, a run without the takeover option
and a local backend stay refused.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
asserts that the option is set for a same-run remote reattachment and is
not set for a fresh sandbox run.

## Verification

- Red on `master` (runtime source restored): 3 of the 4 new runtime
cases fail with exactly the two staging errors (`the previous session is
still shutting down…` and `prior session cleanup remains incomplete`).
The control case passes.
- Red on `master` (executor source restored): the 3 executor assertions
fail.
- Green with this change:
- `cd packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts` (294 passed)
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "exact
Stop cleanup|fresh sandbox|retirement"` (46 passed)
- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/completed-turn-cleanup-hold-settle.test.ts` (61 passed)
- `npx tsc --noEmit` in `packages/paperclip-runner` and `server` passes.
- Staging evidence: runs `7b7ded99` and `27c487ef` show the sequence
above in their events and logs.

## Risks

- The takeover applies only when the server recovers the same run with
`reattach_remote_runner` in the same remote cleanup scope. This is the
case that #20923 keeps the Runner alive for. After a deploy the
quarantine is already gone and the same reattachment runs today, so the
takeover gives the in-process retry the same behavior as a restart.
- A close attempt that is in flight is never overlapped. The
reattachment waits for it within the existing admission grace.
- The reattachment still verifies Runner ownership and its controller
generation before it adopts the Runner. If the Runner is gone, the
existing relaunch or reconciliation path handles it.
- No persisted shape changes. Saved state from the previous release is
not affected because the quarantine lives only in process memory.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:15:40 -06:00
DottaandClaude Opus 5.5 22c9300dac fix(recovery): a run that fails before the agent is asked anything continues by itself on a fresh computer (#21297)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox opens a provider session and then
waits at a launch gate before the first prompt goes to the provider
> - On remote OpenCode sandboxes the gate saves the provider launch
files; when that save fails, no prompt was ever sent
> - Today that failure ends as `native_session_interrupted` with
ambiguous recovery evidence, so the task goes blocked and the user must
press Retry
> - Nothing was sent to the provider, so nothing can be duplicated: the
system can safely continue by itself
> - This pull request types that failure, proves it from the Runner turn
ledger at the failure point, and continues the task on a fresh sandbox
in the existing interrupted-preparation lane
> - The benefit is that a setup failure before the first prompt no
longer needs a human, and the user sees plain words instead of internal
codes

## Linked Issues or Issue Description

Refs #21287
Refs #21304

**What happened?**
On a remote OpenCode sandbox, the provider launch-files gate failed
before the first prompt (for example
`native_provider_artifacts_transfer_timeout`). The run ended blocked
with a Retry banner. The run error showed internal codes and raw byte
counts to the user.

**Expected behavior**
A failure that happens before any prompt reaches the provider continues
automatically on a fresh sandbox. The task shows as working while the
continuation is pending. The visible text is plain and has no internal
codes or raw numbers.

**Steps to reproduce**
1. Use an OpenCode agent on a Daytona sandbox environment with
controller persistence.
2. Arm the crash point `native.provider_artifacts.before_capture` in
fail mode (or let the launch-file upload time out).
3. Start a task. The first run fails at the gate before any prompt.
4. On master the task goes blocked with Retry. With this change a
successor starts on a new sandbox and the task completes.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- New typed cause `native_failed_before_provider_turn`
(`server/src/services/native-runtime/failed-before-provider-turn.ts`).
The executor raises it only when the launch gate fails, the run was not
cancelled, and the Runner turn ledger proves that no turn command was
issued. The proof is taken at the failure point, before the session
closes. In all other cases the original error is rethrown and today's
path stays.
- The executor quarantines the failed run's session root, so the
successor starts in a new root.
- `heartbeat/predecessor-retirement-fallback.ts` accepts the new cause:
it records the proof on `runnerProfileJson.failedBeforeProviderTurn`,
marks the run's reusable sandbox leases for replacement, and schedules
the successor in the existing `interrupted_preparation` lane (same cap
of 12 and the same backoff).
- `remote-workspace-unchanged-proof.ts` accepts the runner process,
runner command and pre-turn events only when that proof record is
present. It still refuses any turn evidence, an unsettled save by
another run on the same workspace, and setup writes. When the proof
refuses, the run's epoch is abandoned unapplied: the files are kept and
recorded, and the next run restores the last committed save into a fresh
root.
- When the proof refuses and the epoch is abandoned, the replacement
mark stays. The next acquisition replaces the sandbox. #21304's
retention hold makes `retireReplacedReusableSandboxLease` park the old
lease as `retained` instead of `pending_cleanup`, so the task continues
on a fresh sandbox and the sandbox that holds the abandoned files is
kept. When the proof holds (only the Runner's own session folder was
written), the old sandbox is replaced and cleaned up as usual.
- `task-interruption-presentation.ts` shows a pending
`interrupted_preparation` retry as `working` with no Retry until it is
due (plus the dispatch grace).
- The visible run error is now plain: "Setting up the computer took too
long, so the task is continuing on a fresh one." (for a setup timeout)
or "The computer could not be set up before the agent started, so the
task is continuing on a fresh one." The technical cause code stays in
`runnerProfileJson.failedBeforeProviderTurn.technicalCause`, the server
log and the run log.
- New crash point `native.provider_artifacts.before_capture` (fail mode
throws the typed transfer timeout at the gate).
`tests/runner-e2e/source-sdk-public-inputs.json` is refreshed for the
changed registry file.
- `tests/restart-chaos/main.ts`: `RESTART_CHAOS_EXTERNAL_CRASH_CONTROL`
lets an outside driver arm fail-mode crash points without a restart
fault.

## Verification

Local (targeted):
- `server/src/__tests__/heartbeat-native-predispatch-retry.test.ts`: 8/8
pass. New cases: fresh remote sandbox and fresh plugin-task sandbox
continuation (now also asserts the plain visible text with no
identifier-looking tokens, and the technical cause in the proof record),
the ambiguous hold when the ledger cannot prove no turn, and backoff
plus the lane cap.
- `server/src/services/remote-workspace-unchanged-release.test.ts`:
27/27 pass. New cases: hand-on when the ledger proves no turn; refusal
on a recorded turn, an unsettled predecessor save, and setup writes. Two
more cases follow the marked lease through the next acquisition's
replacement decision (`readReplaceBeforeReuseMarker` +
`reusableSandboxReplacementPlan` + `retireReplacedReusableSandboxLease`)
and then through `sweepPendingCleanupLeases`. "replaces the marked
sandbox when the failed run provably left its files unchanged": the old
lease goes to `pending_cleanup` and the sweep destroys it. "continues on
a fresh sandbox and parks the one that holds abandoned files, which
cleanup leaves alone": the old lease is `retained` with the
abandoned-files reason and is held by the receipt, the sweep does not
destroy it, and the file is still there.
- Red without the product change (product files reset to the base, tests
kept): 8 failed, 25 passed of 33. All 8 failures are the new cases. The
visible-text assertion alone was red on the previous commit: the error
read "The computer could not be prepared before the agent started
(native_provider_artifacts_transfer_timeout: the upload of 536870912
bytes did not finish within 120 s). The task continues on a fresh
computer."
- `task-interruption-presentation.test.ts`,
`native-provider-artifact-gate.test.ts`,
`crash-point-call-sites.test.ts`: 79/79 pass.
`packages/shared/src/crash-points.test.ts`: 16/16.
`tests/restart-chaos/crash-points.test.ts`: 12/12.
- `tsc --noEmit` clean for `server`, `packages/shared` and
`tests/restart-chaos`.

Daytona (local servers, Daytona sandboxes, image `sha256:787c3e9c…`,
these commits rebased onto a298ca7fbc, gate failures forced with the
crash point):
- F1, wave `w-20261113T042929-s2301`: 4 OpenCode tasks, one forced
failure on each first run. 4/4 failed runs recorded cause
`native_failed_before_provider_turn`. Each successor (retry reason
`interrupted_preparation`) was created 0.0–0.1 s after the failure and
started 1–3 s later, each on a NEW sandbox. Successor first agent event
at 32.8–56.6 s. 4/4 tasks done hands-off. 0 Retry banners. 0 blocked. No
operator wording. No reconciliation codes.
- F3, wave `w-20261113T043934-s2303`: 1 task, 3 consecutive forced
failures. Retries started 7 s, 35 s and 57 s after creation (backoff).
The 4th run succeeded. 0 Retry banners while pending. Not proven: that
retries 2 and 3 in F3 got fresh sandboxes (proven in F1 only).
- F2 (what bootstrap writes before the first turn): the whole sandbox
workspace was sampled every 5 s on all 4 failed runs, right after the
workspace was ready and at the gate failure. Bootstrap wrote nothing to
project files. All 14 new paths are under
`<workspace>/.paperclip-runtime/paperclip-runner/sessions/<session-sha>/`
(runner state, identity and lineage, `runtime-context.json`, the
instructions copy `AGENTS.md`, OpenCode config, data and log). The only
deletion was the staged agent-files `AGENTS.md` (or a
`.paperclip-upload-<uuid>` temp file). The successor never reuses them:
new session root, new workspace generation, restored from the snapshot.
3 of the files are byte-identical when recreated; the rest are
run-specific and recreated by the successor's own bootstrap. The project
workspace is not a git checkout there. Raw manifests are kept with the
PR author.
- The Daytona runs used the error text before the plain-wording commit.
The wording change is covered by the unit test above.

## Risks

- Scope: this applies only where the gate exists: remote OpenCode
sandboxes with controller persistence. Local runs cannot hit it.
- The proof is the Runner's in-memory turn ledger, read at the failure
point. In unit tests it is mocked. If a turn may have been sent, today's
ambiguous hold stays (tested).
- At the lane cap (12) the run stops with a working Retry instead of
waiting longer. A later change can move this onto the shared hourly
limit.
- A direct "is the workspace dirty" check by the Runner is not added,
because it needs a new Runner command. The unchanged proof instead
relies on the recorded evidence and refuses on any turn evidence,
unsettled saves or setup writes.
- Whether a replacement mark gives a fresh sandbox depends on
`reusableSandboxReplacementPlan`. When it refuses, the sandbox is
resumed, but the session root and workspace root are still fresh.
- Refused case (abandoned files): the task continues on a fresh sandbox,
in a fresh session root and a fresh workspace root. The old files stay
on the parked sandbox, recorded in the abandoned-epoch receipt and
unapplied. That sandbox is kept under #21304's hold until the files are
recovered, discarded, or the retention period ends; this PR does not
change that hold.
- The changes in `heartbeat/` and `remote-workspace-unchanged-proof.ts`
were cleared with their owner under these conditions: evidence-scoped,
latest committed epoch only, files kept when unproven, negatives tested.
- `server/src/services/fresh-continuation-fallback.ts` is deliberately
untouched.
- Upgrade safety: runs saved by an older build have no proof record, so
the unchanged proof keeps its previous behavior for them.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use (shell, file edits, test runs). Earlier commits in this branch
came from Claude sessions in the same fix loop.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:48:51 -06:00
Nicky LeachandClaude Opus 5.5 41998d551c feat(plugins): send the claimant's credential on plugin task resume (#21335)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugin environment drivers run agent tasks on remote computers, and
a warm follow-up run resumes a parked Runner through the `resume` task
operation
> - `resume` carries only the durable identity and a bootstrap ticket,
which is enough while the parked Runner process is still alive
> - A provider sometimes has to start the Runner again to resume the
session: after it moved the parked session to a new sandbox, or after
the computer rebooted
> - A Claude Runner cannot start without its credential, and today only
the portable handoff path (`handoffCredential`) delivers one
> - This pull request lets a provider opt in to receiving the claimant
run's harness credential on `resume`
> - The benefit is that a provider can continue a parked session on a
new computer without Core orchestrating a handoff

## Linked Issues or Issue Description

Refs #21289
Related: #21305

**What existing behavior does this improve?**
Warm resume for plugin task providers that restart the Runner to resume
a session.

**Subsystem affected**
Plugin SDK environment task schema, the server's plugin Runner task
owner, and the plugin environment task docs.

**Current behavior**
`submit` can carry the harness credential (`credential`), but `resume`
cannot. A provider that must start the Runner again on resume has no
credential for a Claude task.

**Proposed behavior**
A provider that returns `resumeCredentialVersion: 1` from `describe`
receives `credential` on `resume` whenever Core would send it on
`submit`, resolved for the claimant run. Other providers see no change.

**Reason and benefit**
Providers can restart a Claude Runner on resume without the Core-run
handoff protocol.

**Breaking changes**
None. The field is opt-in: the SDK worker parses operations strictly, so
Core sends it only to providers that advertise the capability.

## What Changed

- `resume` gains an optional `credential`, with the same schema and
validation as `submit`'s.
- `describe` gains an optional `resumeCredentialVersion: 1` capability.
- The plugin Runner task owner adds the credential on `resume` when the
provider advertised the capability and Core resolved a credential, using
the same resolution as `submit` (`preparePluginRunnerCredentials`).
After a warm rebind, the claimant run's credential is sent.
- `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` documents the field and its
handling rules.
- The runner-e2e SDK public-input pin is updated.

`handoffCredential` on the session handoff path is unchanged.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts` (68 passed): `resume` accepts and
rejects the same 11 credential cases as `submit`; the capability is
optional and rejects other versions; a resume credential reaches the
plugin through the serialized worker RPC.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts` (151 passed): credential sent with
the capability and a resolved credential; omitted without a credential
or without the capability; the rebound claimant's credential is sent;
the credential appears in no metadata write and not in the sanitized
error, even when the worker error echoes it.
- `native-session-executor.test.ts -t "uses a task-only"` (5 passed):
the executor forwards the capability to the owner.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts` (46 passed).
- `tsc --noEmit` for `packages/plugins/sdk` and `server`.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- No server-side provider check on resume, unlike `submit`: the lease
has no reliable harness at resume time. Protection relies on the owner,
whose rebind refuses a changed harness, account or credential identity.
- The credential is the claimant run's current one, which can be newer
than the one that started the parked Runner. That is intended.
- A provider must advertise the capability only after its SDK pin
includes this schema; roll Core out first.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:40:46 +00:00
DottaandClaude Opus 5.5 e063587edd fix(recovery): repair stale pending recovery states and tell the fresh run what the cut-off run already did (#21257)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The assigned-work invariant and its periodic sweep find open tasks
that nothing will continue, and continue them once.
> - The fresh-continuation fallback continues a cut-off run in a new
run, with an opening note about what the cut-off run did.
> - In release-swap waves, the invariant believed some "pending" states
forever, so stalled tasks were reported OK and never repaired.
> - The opening note did not say what the cut-off run had already
created or read, so fresh runs created subtasks again and posted DONE
twice.
> - This pull request bounds the stale "pending" states and adds the
cut-off run's work to the opening note.
> - The benefit is that a stalled task gets repaired within minutes, and
a fresh run does not repeat completed work.

## Linked Issues or Issue Description

Refs #21135, #21203, #21234

**What happened?**
1. The invariant counted a task as pending without limit when its
projection said "system recovery working", or when a wake was deferred
behind a run that had already ended, or when a wake was claimed but no
run was ever created. In release-swap waves, an undelivered continuation
and follow-ups that were never promoted were reported OK for 17 minutes
or more and not repaired. The suspect listing also excluded every task
with a deferred wake, so the periodic sweep did not find them.
2. A fresh continuation's opening note listed only actions with unknown
outcomes. Subtasks, comments and documents that the cut-off run had
completed were not mentioned, and follow-ups it had already read were
not mentioned. Continuations created the same subtasks again and posted
the DONE comment twice.

**Expected behavior**
A stale pending state stops counting after a bounded time, and the sweep
continues the task once. The fresh run knows what the cut-off run
already created and read.

**Steps to reproduce**
1. Run a release-swap wave where a run is cut off after it created
subtasks and posted a comment, and where a follow-up waits behind it.
2. See the continuation create the subtasks again, or see the task stay
"working" with no run while the invariant reports it OK.

## What Changed

- Invariant bounds
(`server/src/services/recovery/assigned-work-invariant.ts`,
`server/src/services/recovery/assigned-work-sweep.ts`):
- "System recovery working" counts as pending only for
`SYSTEM_RECOVERY_WORKING_BOUND_MS` (17 min) after the task's last run
ended: the fallback's progress bound (15 min) plus one disposition
re-check.
- A deferred wake with no live run counts as pending only for 5 minutes
after it was created. A claimed wake with no run counts only for 5
minutes after it was claimed.
- The suspect listing uses the same bounds, so the periodic sweep finds
these tasks.
- A held continuation (paced delivery from #21234) counts as the task's
pending wake only for `PENDING_CONTINUATION_BOUND_MS` (7 min) after it
was recorded: the pacing admits a held continuation within 5 minutes
even when the queue is full (`RECOVERY_CONTINUATION_MAX_HOLD_MS`), plus
2 minutes of 15-second delivery passes. Inside the bound the task still
shows "working" with no Retry, as #21234 intends.
- The sweep's wake still goes through admission, which hands a held task
to the fallback. It never starts a second writer.
- A continuation whose delivery is stuck is repaired, not just no longer
counted (`server/src/services/recovery-continuation-bounds.ts`,
`execution-recovery-resolution.ts`, `fresh-continuation-fallback.ts`):
- One module owns `RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 min) and
`PENDING_CONTINUATION_BOUND_MS` (that plus 2 min). The invariant, wake
admission and delivery all read them from there.
- Delivery re-drives first. A continuation held past 5 minutes was
already admitted when the agent's queue was full, but the delivery page
skipped every record of an agent that had another recovery run queued,
at any age. That exclusion now applies only within the hold bound.
- Admission never replaces the recorded continuation. The recorded
continuation alone carries the isolation and fresh-session decision. A
wake that `hasPendingFreshContinuation` coalesces into it now also
notifies the delivery queue. Delivery then re-drives the same record,
with the same isolation key, `forceFreshSession`, and one-successor
idempotency key.

### Where the isolation decision is read, and every path that starts the
successor

Run preparation reads the decision in one place:
`readFreshWorkspaceReadiness`
(`server/src/services/fresh-workspace-continuation.ts`), called from
`heartbeat/workspace-preparation.ts`. It reads
`contextSnapshot.freshContinuationIsolation`, which names the record,
and loads it through `readFreshContinuationReadiness`. Only a run that
carries that key gets a new isolated workspace.

| Path that can start the successor | How it keeps the isolation | Test
|
|---|---|---|
| Delivery worker (`deliverReconciledExecutions`) | Sends the record's
own wake with `freshContinuationIsolation` and `forceFreshSession` |
`fresh-continuation-fallback.test.ts` "gives the continuation a new
isolated workspace while the old local writer may be alive" |
| Re-drive after the bound | The same delivery wake, admitted past the
hold bound even when the queue is full or another recovery run is queued
| "re-drives an isolated continuation pending past the bound itself; a
repair wake never replaces it"; `recovery-wake-herd.test.ts` "re-drives
a continuation held past the hold bound..." |
| Sweep / monitor / timer wake | Never becomes the successor: coalesced
into the record, which it re-drives | "records a pending continuation
for an ordinary continuation wake and delivers it once"; the re-drive
test above |
| Saved input (a person's message) | Waits behind the isolated
continuation and is promoted when that run is released (#21250) |
`postgres.test.ts` "... (released, continuation pending)" (#21250) |
| A person's Retry started before delivery | Delivery counts the Retry's
run as the successor. The Retry run does not carry
`freshContinuationIsolation`. On a task with a workspace binding, the
fallback cleared the binding, so a new workspace is prepared. On a
projectless task this path is **not yet covered** | "never starts a
second run when a person's Retry already continued the task" (one
successor only). Isolation on this path is a known gap, reported for
follow-up |
- Opening note (`server/src/services/fresh-continuation-fallback.ts`):
- Names the subtasks, comments and document saves that the cut-off run
created, and asks the agent not to create them again. The record keeps
the list as `createdObjects`.
- Lists the person's messages that the cut-off run had read (through
`taskHistoryReadsForRun`, from #21203) before what that run then posted,
so the agent answers a re-delivered follow-up once. Only the revision
that was read counts as read. A message a person edited after the read
is listed as changed ("treat them as new").
- The note is written when the continuation is recorded. Nothing is
added before `turn.start`.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `server/src/services/recovery/assigned-work-invariant.test.ts` (unit)
and `server/src/__tests__/assigned-work-invariant.test.ts` (sweep
integration): stale "working", a stale deferred wake, and a claimed wake
with no run become violations past their bound and stay pending inside
it.
- `server/src/services/fresh-continuation-fallback.test.ts`: "names the
subtasks, comments and documents the cut-off run already created" and
"names the follow-up the cut-off run had read and its own final
comment".
- `fresh-continuation-fallback.test.ts` "re-drives an isolated
continuation pending past the bound itself; a repair wake never replaces
it" (projectless task, writer possibly alive). It was red on b76111e45f,
where the sweep wake started a run of its own without the isolation key.
- `recovery-wake-herd.test.ts` "re-drives a continuation held past the
hold bound even while another recovery run of its agent stays queued".
Red without the delivery change (`expected "vi.fn()" to be called 1
times, but got 0 times`).
- `fresh-continuation-fallback.test.ts` "does not list a follow-up
edited after the cut-off run read it as already read; it is marked as
changed". Red before the fix (the note said "had already read").
- `server/src/__tests__/recovery-wake-herd.test.ts` "reads a task whose
continuation is held back as waiting for its agent...": extended. A held
continuation recorded past the bound no longer counts as a pending wake.
Red without the bound (`expected 1 to be +0`), green with it.

Local command, on this branch based on current master (includes #21222
and #21234):
- `cd server && pnpm exec vitest run
src/__tests__/assigned-work-invariant.test.ts
src/services/recovery/assigned-work-invariant.test.ts
src/__tests__/recovery-wake-herd.test.ts
src/services/fresh-continuation-fallback.test.ts`: 4 files, 144 tests
passed.

CI on cfd42b0516: "General tests (server (1/12))" failed on two
timeouts, and I judged both unrelated to this diff:
- `deferred-execution-receipt-work.test.ts` "dispatches task Retry past
a repeatedly invalidated backlog into real process output": "Expectation
did not complete before its deadline".
- `voice-session-store.integration.test.ts` "retains both spoken sides
once on the public call task and preserves quarantine": it timed out in
15 s, and again on retry.

Why I treat them as unrelated:
- This PR does not touch the task-Retry dispatch, deferred receipt or
voice paths. Nothing in that path calls the sweep, the invariant or the
opening note.
- The same deferred-receipt test fails the same way locally on plain
`origin/master`. On this loaded shared Mac, the real-process wait never
completes.
- Both files passed in the same shard on #21250's CI an hour earlier
(run 43351247913).
- No existing test expectation was changed.

Local release-swap cells:
- R1 (6 tasks, 3 adapters, subtasks and follow-ups, 2 swaps): 4/4
hands-off, 0 banners, 0 duplicates, 0 invariant violations.
- Hour-long release-swap wave (w-20261112T233630-s7201) on the branch
that holds these fixes and the related recovery fixes: 11/11 tasks
finished hands-off (master: 9/11).

## Risks

- A task whose recovery is really working for more than 17 minutes after
its last run gets one sweep wake. Admission hands a held task to the
fallback, so this does not start a second writer.
- The opening note is longer when the cut-off run created many objects.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:35:36 -06:00
DottaandClaude Opus 5.5 7e7452e88f fix(native): keep a reattached Runner alive when its provider recovery deadline ends (#21321)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs on remote sandbox Runners. A same-run reattachment
adopts a live Runner after a failed attempt or a server deploy.
> - On staging, a remote Codex Runner is often gone when the next
attempt reattaches (`process_verification` / `process_not_running`). The
task then continues in a fresh session and loses its context.
> - The deploy does not stop the Runner. The logs show `controller
authority detached for restart; durable runner left alive`, and the new
controller adopts the same Runner after the deploy.
> - Our own cleanup stops it. The adopted Runner answers its recovery
reads slowly while its in-flight turn runs. The 900 s provider recovery
deadline then aborts the recovery, and the abort closes the transport
with `turn.stop` and `runner.suspend` and signals the process.
> - This pull request makes the recovery deadline detach from the
Runner, as a recovery read timeout already does. The server marks the
failure as an unanswered adopted Runner, so the next attempt replaces
the Runner from its saved journal and resumes the same provider thread.
> - The benefit is that a deadline no longer kills a live Runner and no
longer forces a fresh session. The task keeps its session context.

## Linked Issues or Issue Description

Refs #21314, #21272, #21161, #21023

**What happened?**
Codex runs on Daytona remote Runners failed often on staging. In three
sampled chains (attempt 2 of each), the same-run reattachment adopted
the live Runner, authenticated in under 1 s, and then waited the full
deadline: `native session provider recovery timed out after 900000ms`
(909 s, 930 s and 915 s). The deadline cleanup sent `turn.stop` and
`runner.suspend` (`provider turn stop timed out before runner
suspension`). The next attempt found the Runner gone 25–180 s later
(`process_not_running`, `runnerLifecycle: ready`, empty stderr) and
continued in a fresh session. During the 900 s wait, the in-flight turn
was still running Paperclip tools (`tool.execute` spans at 04:43 in a
recovery that started at 04:29).

**Expected behavior**
A remote Runner survives a recovery deadline and a control-plane deploy.
The next attempt continues the same provider session. A Stop still stops
the Runner.

**Steps to reproduce**
1. Start a Codex run on a remote Runner and let a turn start.
2. Fail the attempt so that its close keeps the Runner (for example, a
turn start that outlives its bound).
3. Let the same-run reattachment adopt the Runner, and make the Runner
stop answering recovery commands.
4. Let the provider recovery deadline expire.
5. See that the Runner process is gone and the next attempt reports
`process_not_running`.

**Paperclip version or commit**
master at b555087f0f (after #21314).

**Deployment mode**
Hosted staging (cloud), remote Daytona sandbox Runners, deploys every
20–45 minutes.

**Agent adapter(s) involved**
Native Codex (`paperclip_runner`, Codex app-server through
`paperclip-runnerd`).

## What Changed

- Add `NativeProviderRecoveryDeadlineError` (same message as before) and
use it for the provider recovery deadline in the native session runtime.
- In the Codex driver, a recovery aborted by this deadline detaches the
controller (no `turn.stop`, no suspend, no signal). A Stop and every
other abort still close and stop the Runner.
- On the server, `nativeRecoveryCommandUnanswered` recognizes the
deadline. The failure records `adoptedRunnerUnanswered`, so the next
claim carries `replaceUnansweredRunner`. That attempt fences the old
Runner lineage, launches a fresh Runner from the saved journal and
resumes the same provider thread (the #21272 path). It does not adopt
the slow Runner again for another 900 s.
- Update `doc/execution-semantics.md`.

## Verification

- Real `paperclip-runnerd` + fake Codex
(`runnerd-codex-transport-remote-retention.test.ts`): a retained Runner
is adopted, authenticates, then stops answering. A deadline abort leaves
it alive with no pending `turn.stop`/`runner.suspend`/`runner.shutdown`.
A Stop abort still stops it.
- Red/green: without the fix, the real-runnerd deadline test fails
(`expected 'gone' to be 'alive'`), the driver unit test fails (detach
called 0 times), and the server classification test fails. With the fix,
all pass.
- Measurements on real runnerd: the deadline cleanup took 10.0 s and
killed the Runner before. Now it returns in 0 ms and the Runner stays
alive. The replacement over the kept Runner's journal resumes the same
provider thread in about 2.2 s with one `thread/start` and one
`turn/start` (no replay).
- `npx vitest run
src/live/runnerd-codex-transport-remote-retention.test.ts
src/drivers/codex/codex-app-server-driver.recovery.test.ts
src/native-session-runtime.test.ts` (368 passed).
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "never
answered its recovery read"` (passed).
- `tsc --noEmit` for `packages/paperclip-runner` and `server` (clean).

## Risks

- A deadline now leaves a live Runner behind. The next attempt replaces
it through the launch fence, which fails closed when it cannot prove the
old lineage gone. This is the same risk as the existing recovery read
timeout detach.
- When the provider attempt budget is already spent, the kept Runner is
not replaced by this run. This also matches the existing timeout detach.
- Not in this PR: the upstream trigger. The first attempt abandons a
slow turn start (`runnerd did not report the provider turn identity` or
`turn.start timed out`) while Codex later runs that turn. Two sampled
chains then stopped the Runner on
`protocol_failure:tool_binding_mismatch` / `thread_binding_mismatch`
after a deploy reattachment. These need a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:30:22 -06:00
DottaandClaude Opus 5.5 4fe34df748 Do not show Retry on a parent with open subtasks while its queued message is settled (#21320)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The task view shows a "Retry" notice when a task needs a person to
start it again.
> - The product rule is that a task that continues by itself never shows
this notice, not even for a few seconds.
> - A parent task that waits for its subtasks showed the notice for 3.4
seconds after its run succeeded.
> - The cause is the queued-message grace in the task interruption
presentation: it does not apply to `blocked` tasks.
> - #21200 kept `blocked` out of that grace on purpose, so this pull
request adds only the exact observed shape: a parent with open subtasks
and a queued task message.
> - The benefit is that a parent that waits for its subtasks no longer
shows a false failure, and every other blocked task keeps its Retry.

## Linked Issues or Issue Description

Refs #21264, #21200 (added the queued-message grace and kept `blocked`
out of it)

**What happened?**
A Codex parent task ran across a two-server release swap. Its run waited
for two subtasks and then succeeded with the disposition `blocked`. The
task status became `blocked`. A user comment (a deferred wake, status
`deferred_issue_execution`) had been queued 13 minutes earlier, while
the run was active. For 3.4 seconds after the run finished, the task
presentation was `state: "failed"`, `canRetry: true`. Then the release
drain cancelled the wake as "Queued messages were already delivered to a
completed run", and the presentation changed to `resolved`. The subtasks
finished later. Their `issue_children_completed` wake continued the
parent, and it completed without help.

The cause is in `projectTaskInterruptionState` in
`server/src/services/task-interruption-presentation.ts`. The
`queuedDuringRun` check shows a succeeded run with user input queued
before it finished as `working` for up to 2 minutes. It applies only to
`todo`, `in_progress` and `in_review`. For a `blocked` task the
presentation fell through to the last line: `failed` with `canRetry:
true`.

The workspace finalization deferral
(`native_workspace_finalization_busy`) in the same window does not
change the presentation. The presentation does not read it.

**How this differs from what #21200 narrowed**
#21200 first applied the grace to every status. Its commit `a3fae9bb67`
removed `blocked` because input queued on a blocked task is not promoted
by itself, and because a recovery that gives up hands its hold to the
board and must end with Retry. This change does not add `blocked` back.
It adds one narrow case, and all of these must be true:
- The latest run **succeeded**. A failed, interrupted or given-up run is
not affected.
- The task is `blocked` **and has open subtasks of its own** (not `done`
or `cancelled`). Their completion wakes the parent again
(`issue_children_completed`).
- The pending item is an ordinary **task comment** wake (`commentId` or
`wakeCommentIds` in its payload), not a Retry request
(`taskRetry.sourceRunId`).
- The comment was queued before the run finished, the run finished less
than 2 minutes ago, and saved-work recovery is not exhausted. These are
the existing bounds.

A blocked task that waits for a person or a board decision has no open
subtasks, so it keeps Retry as before.

**Expected behavior**
After the run succeeds, the parent shows `working` with no Retry while
its queued message is settled. Then it shows `resolved` (message already
delivered) or `working` (message promoted to a new run). If nothing
settles the message in 2 minutes, Retry shows.

**Steps to reproduce**
1. Assign a parent task that creates subtasks and waits for them.
2. Post a comment on the parent while its run is active.
3. Let the run succeed with the disposition `blocked`.
4. Read the task presentation in the first second after the run
finishes. It shows `failed` with Retry.

**Paperclip version or commit**
master `bbafa78fe2`. The wave that found it ran on `62d673cc3f`.

## What Changed

- `server/src/services/task-interruption-presentation.ts`:
- The batched wait read also reads whether the wake carries a task
comment (`queuedComment`).
- One company-scoped read finds which `blocked` tasks with a succeeded
latest run and a queued comment have open subtasks
(`waitingOnOpenSubtasks`). It runs only for those tasks.
- The `queuedDuringRun` grace also applies to such a task. The `todo`,
`in_progress` and `in_review` behavior does not change.
- `server/src/__tests__/task-interruption-automatic-path.test.ts`, new
embedded-Postgres tests:
- The exact observed state: a `blocked` parent with two open subtasks, a
run that succeeded 1 second ago, a user comment wake queued 13 minutes
earlier. Shows `working`, no Retry.
- The same parent after the wake is cancelled as already delivered:
`resolved`.
- The same parent after the wake is promoted to a new run: `working` on
the new run.
- Guard: the same parent after the 2-minute grace: `failed` with Retry.
- Guard: a `blocked` task with a queued comment and no subtasks, or only
`done` or `cancelled` subtasks (waiting for a person or a board
decision): `failed` with Retry, unchanged.
- Guard: a parent with open subtasks whose queued wake carries no
comment: `failed` with Retry, unchanged.
- Guard: a parent with open subtasks and a queued comment whose run
failed and whose recovery hold went to the board: `failed` with Retry.
- Guard: a parent with open subtasks and a queued comment whose
saved-work recovery is exhausted: `failed` with Retry.
- A parent whose run succeeded with no queued message stays `resolved`
when its subtasks are open, blocked or cancelled.
- The existing #21200 assertion (a `blocked` task with queued input and
no subtasks shows Retry) is kept unchanged.

## Verification

- `cd server && npx vitest run
src/__tests__/task-interruption-automatic-path.test.ts`
- Red on master `bbafa78fe2`: "shows a parent whose run just succeeded
waiting on its subtasks as working while its queued message is settled"
returns `state: "failed"`, `canRetry: true`. This is the value that the
wave recorded. The other 15 tests pass on master.
  - Green with this change: 16 of 16.
- A plain `blocked` inclusion fails 3 of the guards: the #21200
assertion, "still offers Retry on a blocked task with a queued message
and no open subtasks" and "still offers Retry on a parent with open
subtasks when its queued wake carries no message". This shows that the
narrow condition is necessary.
- `task-interruption-presentation.test.ts`,
`task-interruption-query.test.ts`: green (116 tests with the file
above).
- `cd server && npx tsc --noEmit -p .`: 0 errors.
- Not verified: no new chaos wave. The flash was seen once in a loaded
release-swap wave. The server test reproduces its saved state.

## Risks

- Low risk. The change is presentation only. It does not change wake
promotion, dispatch or Retry.
- A parent with open subtasks and a stranded queued comment now shows
`working` for up to 2 minutes after its run succeeded, and then Retry.
Before, it showed Retry at once.
- One extra indexed read (`issues_company_parent_idx`) runs only for
blocked tasks with a succeeded latest run and a queued comment.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edits, test execution). Extended
reasoning was available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:27:22 -06:00
Nicky LeachandClaude Fable 5.2 314b893678 fix(native-runtime): deliver assigned gateway tools to every native harness through the PRP relay (#21274)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can have app connections. Paperclip gives the agent the
tools of those apps through an MCP gateway, with its approval and audit
rules.
> - Native harnesses reach the gateway in two ways today. In the relay,
the server keeps the gateway token and offers the tools over the
authenticated Runner channel. In the direct way, the server gives the
gateway URL and a bearer token to the harness.
> - Which way a harness uses comes from a list of special cases that
grew one harness at a time. Local Codex and all ACPX agents use the
direct way. The others use the relay.
> - The direct way puts a run-scoped token in a provider process. A
process that the agent starts can read it. A provider process that
serves more than one run can hold a revoked token.
> - This pull request replaces the list with one rule. Each native
harness gets its assigned gateway tools through the relay.
> - The benefit is one behavior for all native harnesses, and no gateway
token in a provider process.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #13942 (the relay), #17149 (ACPX accepts relayed tools) and #19912
(the design for turn-scoped ACPX MCP authority).

**What existing behavior does this improve?**

How a native harness reaches the tools of its assigned MCP gateway.

**Subsystem affected**

Native runtime in the server (`native-session-executor.ts`,
`native-session-resume.ts`). One Runner test file. One Rust test.

**Current behavior**

- `relayAssignedMcp` is true for plugin tasks, Cursor Cloud, OpenCode,
Codex on a remote target, and Dot.
- For local Codex and for each ACPX agent (Claude, Codex, Cursor, Pi,
Kimi, Grok, Copilot, Antigravity), Core puts
`PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` and
`PAPERCLIP_NATIVE_MCP_TOKEN` in the Runner environment. The ACPX host
gives them to the provider as an HTTP MCP server. The local Codex home
gets them in `config.toml`.
- The token is run-scoped. Core revokes it when the run ends.

**Proposed behavior**

- A native run that has an assigned gateway uses the relay. The
condition names no provider.
- Core builds the gateway tools on the control plane, offers them with
the other Runner tools, dispatches each call over PRP, and authorizes
each call again through the gateway.
- Core removes the three `PAPERCLIP_NATIVE_MCP_*` variables from each
Runner environment.
- The tool contract fingerprint has the relay marker for each target,
not only for remote targets.

**Reason and benefit**

- All native harnesses have the same behavior.
- No provider process holds a gateway token.
- A sandbox does not need to reach the gateway URL of the host.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
- The list of special cases is gone. Core validates the binding and
builds the relayed tools when a binding is present.
- Core does not list the gateway for a review run, Grok Bot or Muse. The
tool authority never offered app tools to those. A gateway error can
therefore not fail those runs.
- OpenCode keeps its `on_demand` catalog. That option depends on the
provider, not on the relay.
  - Core removes the three variables from each Runner environment.
- `server/src/services/native-runtime/native-session-resume.ts`: the
relay marker in the tool contract fingerprint is unconditional.
- `doc/execution-semantics.md`: the text about the tool contract
fingerprint now states the rule, the tool names and the cutover.
- App tool schemas. New module
`packages/paperclip-runner/src/drivers/tool-input-schema.ts` compiles
each schema by itself, in its declared dialect (draft-07, 2019-09 or
2020-12), with no shared `$id` registry. Core uses it when it builds the
relayed tools (`assigned-mcp-tools.ts`). Core does not offer a tool
whose schema has an unsupported dialect or does not compile, and writes
a log entry with the tool name and the reason. The run does not fail.
The ACPX tool bridge and the OpenCode MCP bridge use the same function.
The tools of Paperclip itself stay strict.
- Large arguments. The ACPX sidecar sends the arguments of a tool call
to runnerd with no display limit. The limit is the tool-call admission
limit of runnerd, 256 KiB. A larger call fails with a tool error that
gives the limit. The sidecar does not dispatch it.
- Tests in `native-session-executor.test.ts`,
`native-session-resume.test.ts`, `runtime-context-materializer.test.ts`
and `acpx_provider_backend.rs`.

Not changed in this pull request: the direct code path. It is still in
the Runner, but it receives no binding. #21311 deletes it.

## Behavior that each harness sees

| Harness | Before | After |
|---|---|---|
| Codex on a remote target, OpenCode, Cursor Cloud, Dot, plugin tasks |
Relay | No change |
| Local Codex | Direct: a `paperclip-assigned` server in `config.toml` |
Relay. App tools are dynamic tools named `app_<name>_<digest>` |
| ACPX agents | Direct: a second HTTP MCP server | Relay. App tools are
on the `paperclip` bridge as `mcp__paperclip__app_<name>_<digest>` |
| ACPX with a preconfigured home | No gateway | No change |
| Grok Bot, Muse | Got the variables and ignored them | Do not get the
variables |

- A relayed tool name has a maximum of 64 characters. With the
`mcp__paperclip__` prefix it has a maximum of 80.
- Permission behavior keeps the same intent. For Claude in the
`approve-reads` and `approve-paperclip` modes, app tools ask for
approval, as before. For the other ACPX agents the permission decision
does not use the server name.
- In a planning run or an ask run, the relay offers only tools with read
risk. The direct gateway offered all tools. This is stricter.
- When the catalog is too large, each harness gets the two compact tools
`paperclip_search_assigned_tools` and `paperclip_call_assigned_tool`.
- The four helper tools for resources and prompts
(`paperclip_list_resources`, `paperclip_read_resource`,
`paperclip_list_prompts`, `paperclip_get_prompt`) are not in the relay.
No harness loses them: a run token permits only `tools/list` and
`tools/call` (`run-preparation.ts`), so these four tools already fail
with `gateway_token_action_denied` on the direct path.

## State saved by the previous release

- **Local sessions of each provider start again one time.** The tool
contract fingerprint changes for local targets. The next run sees a
different fingerprint, starts a new provider session with the full task
context in the same workspace, and keeps the old saved state. This is
the existing rotation for a tool contract change. Codex needs it,
because Codex cannot add tools to a thread that it resumes. For the
local providers that already used the relay, the restart is not
necessary. It is accepted so that the rule names no provider.
- **Remote sessions continue.** Their fingerprint does not change. For
ACPX, the attach of the next run replaces the saved tool set.
- **A kept Codex home.** Paperclip writes `config.toml` again at each
start and recovery. The old gateway server and its token are removed.
- **A deploy lands during a run, and the Runner survives.** The
controller attaches to the same Runner for the same run. The old
provider keeps its direct gateway connection. The controller detach
keeps the gateway token of that run, and Paperclip revokes it when the
run ends.
- **A deploy lands during a run, and the Runner is dead.** The Runner
starts again for the same run with its saved tool set and no direct
gateway. The assigned app tools are not available for the remainder of
that one run. The next run gets them through the relay. This applies to
each harness that moves to the relay. It has no test.
- Paperclip refuses no saved state and discards none.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`: `1508 passed
| 7 skipped`, 1 failed. The failed test is "archives a damaged prior
epoch before completing guarded same-task replacement with real runnerd"
(`ENOENT … authority-epochs`). It fails in the same way on `master` at
53608bd1d6 and with the product change removed, on macOS.
- `cd server && npx tsc --noEmit -p tsconfig.json`: no errors.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/runtime-context-materializer.test.ts`: `18 passed`. `npx tsc
--noEmit -p tsconfig.test.json`: no errors.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes. `cargo test -p paperclip-runner-core --lib
acpx_provider_backend`: `50 passed`.

- `npx vitest run src/drivers/tool-input-schema.test.ts
src/drivers/runner-tool-bridge.schemas.test.ts
src/drivers/runner-tool-bridge.test.ts
src/cli/acpx-runtime-sidecar.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: `21 passed (21)` files, `377 passed` tests.
`pnpm run typecheck:typescript`: no errors.
- `cargo test -p paperclip-runner-core --test acpx_event_payload --test
tool_input_schema_fixtures`: 10 and 3 passed.
- Schema tests use one shared fixture
(`tool-input-schema-fixtures.json`) for the TypeScript compiler, the
tool bridge and the Rust `begin_call`: a draft-07 schema, a second
schema with the same `$id`, 2019-09, 2020-12, draft-04, an unknown
dialect, a `$ref` with no target, and an invalid keyword. Before the fix
the bridge failed with `no schema with key or ref
"http://json-schema.org/draft-07/schema#"`.
- A tool schema whose `$id` is the URI of a meta-schema cannot damage
the shared compiler. Before this fix, `removeSchema` deleted that
meta-schema, and later schemas failed to compile for each run in the
process. A test with 7 reserved ids failed before the fix and passes
now.
- Core limits the compile work of each run start. A schema over 512 KiB
is not compiled, and one listing compiles a maximum of 768 KiB of
schemas. These are the limits of the Runner. A tool over a limit is
served only through the search and call tools. Core keeps the outcome
for each schema digest (1,024 entries) and yields to the event loop
before each compile. Four tests failed before this change.
- A sidecar test sends a tool call with 200 KiB of arguments and
receives them byte for byte, refuses a call with 300 KiB with the named
error, and keeps the replacement of a lone surrogate.

Tests, each seen to fail without the product change:

- "keeps assigned MCP credentials on the control plane for %s" now runs
for remote Codex, local Codex, OpenCode and ACPX Claude, Pi, Kimi,
Copilot and Cursor. It checks that Core builds the relay, offers the
tool, dispatches a call, and gives no `PAPERCLIP_NATIVE_MCP_*` variable
to the Runner. Without the change, the five new harness cases fail.
- "relays the assigned gateway when a deploy lands mid-run…" runs for
Claude and Pi.
- The fingerprint tests keep the old local fingerprint as a retained
value and check that the new one is different. The remote fingerprints
for Codex and OpenCode do not change.
- "drops a retained direct gateway server when the relayed launch has no
binding": a Codex home with the old `config.toml` entry loses that entry
at the next start and at a recovery of the same run.
-
`next_run_attach_offers_relayed_gateway_tools_to_a_direct_gateway_checkpoint`
runs for Claude and Pi.

Not run: a live run with a real provider and a real app connection for
each harness. Not run: `pnpm verify` and the full suite. CI supplies
that coverage.

## Risks

- **An ACPX app call has a limit of 256 KiB for its arguments.** The
direct gateway accepted a request body of 10 MB. Codex and OpenCode
permit 480 KiB. To get nearer to 10 MB, runnerd needs a larger admission
limit for tool calls or a protocol that sends arguments in parts. That
is not in this pull request.
- **A tool with a draft-04 or draft-06 schema is not offered.** Ajv 8
does not support draft-04, and the TypeScript and Rust validators must
agree. On the direct path the provider got such a tool.
- **Core compiles app schemas on its event loop.** Before this pull
request Core did not compile them. With the limits above, one valid
schema near 512 KiB can hold the loop for about 0.65 s, and one listing
for about 1.2 s in total (measured on a developer machine). An app that
changes its schemas at each listing gets no help from the cache. A
worker thread with a timeout would remove this; it is not in this pull
request.
- **Tool names change for local Codex and all ACPX agents.** A saved
agent instruction that names a gateway tool directly does not match the
new name.
- **Each local native session starts again one time after the deploy.**
The provider conversation of that session does not continue. The task
context and the workspace stay.
- **A run that is in flight at deploy, with a dead Runner, has no app
tools until it ends.**
- **Names of 80 characters are not tested live** with Kimi, Copilot,
Grok, Antigravity and Codex through ACPX. Pi already shortens a name
longer than 64 characters.
- **Managed Claude and AWS AgentCore now receive relayed app tools.**
This is not tested live.
- **A gateway token expires after 1 hour.** The relay keeps the token on
the controller for the whole run, as OpenCode and remote Codex do today.
A longer run loses its app tools on both paths. This pull request does
not change that.
- **The gateway URL must have the path `/mcp/gateways/<id>`.** With a
path prefix in the API base URL, a run with an assigned gateway fails
with `assigned native MCP gateway path is invalid`. This was true for
the relayed harnesses before. It is now true for all.
- No test covers the rule that Core does not list the gateway for Grok
Bot and Muse.

## Follow-up

The direct path is now unreachable. #21311 deletes it. That pull request
is in the same stack, above the pull requests that change the same ACPX
host files.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff and ran the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated each harness
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 00:16:58 -08:00
DottaandClaude Opus 5.5 fde0328ff5 fix(recovery): deliver every pending continuation within a bounded time (#21324)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Execution recovery continues a task in a fresh run after a restart
or a lost run left it unfinished
> - #21234 paced these continuations: at most 1 per agent and 5 per
company per pass, and a continuation waits while its agent already has a
queued recovery run or a full queue
> - A continuation that waited 5 minutes must enter the queue anyway,
but that bound applied only to the full-queue check. The "agent already
has a queued recovery run" check was in the page query, and it had no
bound.
> - On staging, an agent had queued recovery runs that did not start for
hours. Its pending continuation was never read again, and its task
showed "working" with nothing running or queued
> - This pull request applies the 5-minute bound to both checks, stops
one undeliverable continuation from using its agent's or company's slot,
and stops the task view from showing "working" for a continuation that
is overdue
> - The benefit is that every pending continuation enters the queue
within a bounded time, also after a restart, and the pacing still
protects people from recovery herds

## Linked Issues or Issue Description

Refs #21234, #21135

**What happened?**

On staging, a task had a fresh-continuation hold with
`continuationDelivery: "pending"` since about 03:30Z. More than 3.5
hours later, no run had started for the task. The task view showed the
interruption state `working` with `canRetry: false`, so the user saw
nothing wrong. The task's agent had 20 concurrency slots and 11 running
runs. It also had queued `issue_continuation_needed` runs from 22:54Z
that had not started. The delivery pass reads pending continuations with
a query that leaves out every agent that has a queued recovery run
(`issue_recovery_action_restored` or `issue_continuation_needed`). That
condition had no time bound, so the pass never read this continuation
again. The 5-minute admission bound was never reached. Across the
company, 203 open tasks showed `working` while the API showed no queued
or running run for them. 162 queued runs were more than 1 hour old.

**Expected behavior**

Every pending continuation enters its agent's queue within a bounded
time, also when the agent's queue holds recovery runs that do not start,
and also after a server restart. Pacing still limits how many
continuations enter a queue at the same time. A task does not show
"working" when its continuation is overdue and nothing runs or is
queued.

**Steps to reproduce**

1. Give an agent a concurrency limit of 1.
2. Put one queued `issue_recovery_action_restored` run on that agent
that does not start.
3. Record a pending fresh continuation for another task of the same
agent.
4. Wait more than 5 minutes, and let the 15-second continuation pass
run.
5. See that the continuation stays `pending`, the task shows `working`,
and no run starts.

**Paperclip version or commit**

master at bbafa78fe2 (includes #21234, ade7e0643f)

**Deployment mode**

Hosted cloud (staging)

**Agent adapter(s) involved**

paperclip_runner (native runtime). The problem is in the shared
continuation delivery pass, so it applies to all adapters.

## What Changed

- `deliverReconciledExecutions` in
`server/src/services/execution-recovery-resolution.ts`: a continuation
held for 5 minutes or more is read even when its agent has a queued
recovery run. Before, only the full-queue check let it through.
- Each page of the pass now reads at most one candidate per agent
(`row_number() over (partition by agent)`), so one agent's backlog
cannot fill the page budget and hide other agents' continuations.
- A continuation counts against the company's limit only when its wake
puts a run in the queue. A continuation that was invalidated or replaced
by a newer run does not use its agent's turn.
- When an agent refuses a wake (no run, or an error), the pass skips
that agent for the rest of the pass and stamps the row with
`continuationDeliveryAttemptedAt`. Overdue rows always rank before rows
that are not overdue. Among overdue rows, the turn is the later of the
hold time and the last attempt, so a refused row goes behind rows held
before that attempt and ahead of rows held after it. One refused row
cannot pin its agent, and new arrivals cannot keep an older refused row
waiting.
- New module `server/src/services/recovery-continuation-pacing.ts` holds
the pacing constants and the "held since" rule. It adds
`RECOVERY_CONTINUATION_DELIVERY_BOUND_MS` (15 minutes). The old
constants are still exported from `execution-recovery-resolution.ts`.
- `taskInterruptionPresentations` in
`server/src/services/task-interruption-presentation.ts` shows a pending
continuation as "continuing" only within the delivery bound. After that,
the task shows its real state instead of `working`.
- No change to the schedule: the continuation duty runs only while
pending continuations exist, and it checks the database on start, so a
restart does not lose pending rows.

## Verification

- `pnpm exec vitest run src/__tests__/recovery-wake-herd.test.ts` (in
`server/`): 15 tests pass. New tests:
- A continuation held past the bound is delivered while the agent's
queue holds recovery runs that never start.
- After a restart with a continuation pending past the bound and a full
queue, a new schedule is due within one pass, the pass delivers the
continuation, and the schedule then goes idle.
- Continuations whose wake returns no run do not use the per-company
count. Another continuation of the same agent is delivered on the next
pass.
- An overdue continuation is refused once while the agent's queue is
full. On the next pass, it is delivered even though new continuations
keep arriving for the same agent.
- One agent refuses a backlog of 210 overdue continuations. Another
agent's continuation is delivered in the same pass, and the refusing
agent gets one wake attempt in that pass.
- A task whose continuation is pending past the delivery bound with
nothing running does not show `working`. A fresh pending continuation
still shows `working` with `canRetry: false`.
- Red/green: with the two source files reverted to master (and only the
new constant exported), all 4 first new tests fail. The backlog test
fails on the first PR head (200 refused wake attempts in one pass). The
temporary-refusal test fails on the second PR head (the overdue row
stays pending). With this change, all 15 pass.
- Related suites pass: fresh-continuation fallback, workspace recovery
fallback, native safe replacement, native workspace reconstruction, task
interruption presentation, queued-run dispatch order, execution recovery
resolution, assigned-work invariant, execution control work, task
interruption automatic path and query.
- `node scripts/typecheck-layers.mjs` (in `server/`) passes.

## Risks

- A continuation that waited 5 minutes now enters a queue that already
has recovery runs. The per-agent (1) and per-company (5) limits per pass
still apply, and the queue still starts person-initiated work first.
- An agent that refuses wakes gets one wake attempt per pass. Its
refused rows rotate through `continuationDeliveryAttemptedAt`, and they
do not block other agents.
- After 15 minutes without delivery, the task no longer shows `working`.
With this fix, delivery happens well within that bound, so this is a
safety net.
- No schema or migration changes. No new polling.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, and read-only staging
API requests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:12:13 -06:00
Nicky LeachandClaude Opus 5.5 2483cb46c2 feat(plugins): deliver configured environment to plugin Runner tasks (#21280)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent gets environment variables from its execution environment,
its own configuration, its project, and its routine. Secrets are
resolved for each run.
> - A plugin environment can run the agent as a native Runner task on
its own host, through the environment task contract.
> - That contract has no field for the agent's variables. Runs on a
plugin environment get only the variables that the plugin sets itself.
> - This pull request adds a versioned contract field for the values,
sends them on a session handoff, lets Codex commands inherit the
variables in an external sandbox, and lets a plugin protect the
variables that it sets itself.
> - The benefit is that plain and secret variables work on plugin
environments in the same way as on local environments.

## Linked Issues or Issue Description

Refs #21233. That pull request by @lucktastic finds the same gap and
proposes the same capability and field names
(`configuredEnvironmentVersion: 1`, `configuredEnvironment`). This pull
request uses the same names, so providers implement one contract. It
differs in these places:

- Codex commands get the variables through an external-sandbox denylist,
not through key names sent at `prepare`. A retained Runner does not need
a new preparation when a key is added or removed.
- A session handoff `resume` gets the values, because it starts a new
Runner on another computer. Value changes on the same computer need no
new field: Core's process reuse key includes the values, so Core starts
a new task.
- A driver can declare `protectedEnvironmentKeys`. Paperclip rejects or
warns when one is saved.
- The run log names skipped variables only for an older provider.

Thank you to @lucktastic for the diagnosis and the contract shape.

**What happened?**
An agent has environment variables, plain and secret. Its runs on a
plugin task environment do not get them. `PAPERCLIP_CONFIGURED_ENV_KEYS`
in the run lists only the variables that the plugin sets.

**Expected behavior**
Each run gets the agent's configured variables.

**Steps to reproduce**
1. Set a plain and a secret environment variable on an agent.
2. Assign the agent to a plugin environment that provides Runner tasks.
3. Start a run. Read the names in `PAPERCLIP_CONFIGURED_ENV_KEYS` in a
command that the agent runs.
4. The agent's names are not in the list.

## What Changed

- **SDK contract** (`packages/plugins/sdk/src/environment-tasks.ts`):
- A provider advertises `configuredEnvironmentVersion: 1` on `describe`.
- `submit` gets `configuredEnvironment`. `resume` gets it only for a
session handoff.
- Names, count, and sizes have the same limits as the Runner projection.
- **Server**:
- `createRunnerdBackendWithinSessionClaim` gives the run's configured
projection to the plugin task owner. The projection has no provider
credentials and no generated runtime keys.
- `createPluginRunnerTaskOwner` sends the fields only to a provider with
the capability. An older provider gets no new field, and the run log
names the variables that were not set. The log never prints values.
- **Codex** (`codex-security-config.ts`):
- An externally sandboxed launch uses `inherit="all"`,
`ignore_default_excludes=true`, and an `exclude` list of Codex's own
credentials, instead of an `include_only` list fixed at preparation
time.
- A test reads the Runner's Codex environment keys, and fails if a new
key is not classified.
  - Host launches keep the bounded allowlist.
- **runnerd** (`codex_provider.rs`): `runtime_notification_codex_args`
still grants the notification files to read-only profiles when no
allowlist is present. Plan mode in an external sandbox needs these
grants.
- **Protected keys**:
- An environment driver can declare `protectedEnvironmentKeys` in its
manifest.
- Paperclip rejects a protected key on an agent or environment that uses
the driver. The 422 has code `protected_environment_variable`.
- On a project, a routine, or another agent, Paperclip saves the key.
The response has a `configurationWarnings` notice that names the
environments. The project and routine pages show the notice as a toast.
The agent pages already show configuration warnings.
- **Docs**: `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` describes the
provider obligations.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the reviewed
SDK source pin.

## Verification

- `packages/plugins/sdk`: `vitest run tests/environment-tasks.test.ts`
(38 pass). Accepts bounded values. Rejects bad names, NUL bytes, too
many variables, and oversized values.
- `server`: `vitest run
src/services/native-runtime/plugin-runner-task.test.ts` (95 pass).
Values go only to a provider with the capability, never in `prepare`.
`resume` sends values only for a handoff.
- `server`: `vitest run src/services/protected-environment-keys.test.ts
src/__tests__/project-routes-env.test.ts` (10 pass). Rejection, notices,
and the project route response.
- `server`: all route suites that import the project, routine,
environment, or agent routes (45 files, 1689 pass).
- `packages/shared`: `vitest run src/validators/plugin.test.ts` (64
pass).
- `packages/paperclip-runner`: `vitest run src/drivers/codex
src/configured-environment.test.ts
src/live/runner-provider-launch.test.ts` (444 pass).
- `packages/paperclip-runner/runner`: `cargo test -p
paperclip-runner-core --lib runtime_notification` (9 pass, 2 ignored as
before).
- `pnpm typecheck` in `packages/shared`, `packages/plugins/sdk`,
`packages/paperclip-runner`, `server`, and `ui`.
- Not run locally: `pnpm verify --full` and browser suites. Required CI
covers the final revision.

## Risks

- **Existing providers:** low risk. Core sends no new field to a
provider without the capability. Their runs behave as before, with one
added log line when variables were skipped.
- **Codex in an external sandbox:** commands now see every variable in
the Runner environment except Codex's own credentials. The execution
environment builds that environment from an allowlist. If the Runner
gives Codex a new credential, the drift test fails until the key is
classified.
- **Saving env:** new rejections happen only when an installed driver
declares `protectedEnvironmentKeys`. No driver declares them today.
- **Providers:** they must keep the values transient and never save or
log them. The doc lists these obligations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:11:23 +00:00
DottaandClaude Opus 5.5 e73fc71331 fix(environments): continue on a new sandbox when an idle runner sandbox never starts (#21323)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on the next wake
> - On staging, Daytona left two idle sandboxes in the `archiving` state
for more than three hours, so no resume call could succeed
> - The native runner leases had no verified harness backup, so the run
never replaced the sandbox, even though the task files were saved
> - Each retry spent six minutes on the same stuck sandbox, the bounded
retries ran out, and the task waited for a person to press Retry
> - This pull request continues on a new sandbox when the provider has
proven it cannot start the old one and the workspace is saved, and it
keeps the old sandbox
> - The benefit is that the task recovers by itself and a sandbox that
may hold unsaved runner state is never destroyed

## Linked Issues or Issue Description

Refs #20626, #21113

**What happened?**
Eight Codie runs on staging failed in one hour with
`reusable_sandbox_resume_failed`: `Reusable sandbox lease "<id>" could
not be resumed (cause=transport_timeout); the lease was preserved and no
replacement was created.` They came in two clusters (05:59 to 06:00Z and
06:32 to 06:38Z). Each run took about six minutes, so the 330-second
idle resume budget from #21113 was applied. All runs were already
`transient_failure_retry` wakes.

Investigation of one instance (lease `c23d6120…`, run `d16c272e…`):
- The lease was `released` at 02:37Z with a confirmed stop-and-retain.
No live run held it, so the lease was idle and the extended budget
applied.
- A read-only Daytona API read at 06:5xZ showed the sandbox in state
`archiving` (desired state `archived`), last updated 03:38Z, one hour
after the stop (the auto-archive interval is 60 minutes). The second
sandbox (`e0e6cfe5…`) showed the same state since 03:35Z. Daytona does
not start a sandbox while it archives, so every resume call timed out.
More time or a new HTTP connection could not help.
- The task workspace was clean and had a committed saved snapshot.
- The agent uses `paperclip_runner`. The last run on the sandbox failed
while it saved the runner session, so the lease had no
`nativeHarnessBackup` stamp. `reusableSandboxReplacementIsSafe` requires
a verified harness backup for a runner lease, so the run never replaced
the sandbox. It failed strictly, the bounded retries (2) ran out, and
the task waited for a manual Retry.

**Expected behavior**
When an idle sandbox never starts through the whole provider start
deadline and the task files are saved, the run continues on a new
sandbox automatically. A sandbox that may hold unsaved runner state is
not destroyed.

**Steps to reproduce**
1. Run a `paperclip_runner` task on a reusable Daytona sandbox. Let the
last run end without a harness backup (for example, the runner exits
while it saves the session). The workspace is clean with a committed
snapshot.
2. Let Daytona start to archive the stopped sandbox and get stuck in
`archiving`.
3. Wake the task. Every `environmentResumeLease` call times out, the run
fails after about 330 seconds with `reusable_sandbox_resume_failed`, and
the retries fail the same way.

**Paperclip version or commit**
Staging Core `0c04ff76eb` (current `master` has the same code path).

**Deployment mode**
Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases
on.

**Agent adapter(s) involved**
`paperclip_runner` (native runner) on a Daytona sandbox environment.

**Root cause**
`server/src/services/reusable-sandbox-resume-replacement.ts`: a runner
lease without a verified harness backup was never replaceable, for any
failure. The only other path was to retry the same sandbox, which cannot
work while the provider is stuck. Replacement also always retired the
old lease to `pending_cleanup`, so allowing it without a backup would
have destroyed the only copy of the runner session state.

## What Changed

- `reusable-sandbox-resume-replacement.ts`:
`reusableSandboxReplacementIsSafe` becomes
`reusableSandboxReplacementPlan`. It returns `retire` (same rules as
before), `keep`, or `null`. It returns `keep` for a runner lease without
a verified harness backup only when the caller passes
`providerCannotStart`. All other conditions still apply: idle lease,
same workspace, clean workspace, committed snapshot, this run is the
writer, and no access refusal.
- Add `keepReplacedReusableSandboxLease`. After the replacement lease is
recorded, one guarded write marks the old lease with
`reusableSandboxKeptAfterReplacement`. The lease keeps its status and
its sandbox. Nothing destroys it.
- `environment-runtime.ts`: set `providerCannotStart` only when the idle
start deadline was extended and the run then used all of it on timeouts.
Lease selection skips a lease with the keep marker, so later runs do not
spend time on it and obsolete-lease cleanup does not destroy it.
- The replacement records `replacement` / `resume_failed` with no
harness backup. The native executor already starts a fresh session in a
new sandbox in this case (`freshReplacement`), and workspace preparation
restores the committed snapshot.
- The keep plan needs every resume call to time out. A rate limit, a
reset, or any other failure in between keeps the strict behavior.
- `reusable-sandbox-kept-marker.ts`: the marker key and a SQL guard.
Every automatic teardown path skips a marked lease: retained-orphan
listing and settle, `destroyReusableSandboxLease` (obsolete cleanup,
expiry, run destroy), `destroyClaimedSandboxLease` (issue and workspace
teardown), recovered-workspace cleanup, and abandoned-file release. Only
an operator-consented environment delete still destroys it.
- `doc/execution-semantics.md`: document the rule.

## Verification

- Red before the fix: the new staging-shape test in
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`
(released runner lease, no harness backup, clean committed snapshot,
every resume times out) failed with the staging error `could not be
resumed (cause=transport_timeout); the lease was preserved and no
replacement was created.`
- Review round: "keeps an idle sandbox with no harness backup when some
resume calls failed for another reason", "never destroys a sandbox kept
after replacement through the run or workspace teardown paths", and the
retained-orphan case "keeps a warm sandbox while a replacement kept it"
each failed before the fix and pass after it.
`environment-retained-orphan-sandbox.test.ts`,
`abandoned-workspace-files-hold.test.ts`,
`workspace-recovery-preparation-gates.test.ts`,
`workspace-recovery-admission.test.ts`: all pass.
- Green after the fix: 22/22 pass in the replacement suite. The run gets
a new sandbox within 330 seconds, makes no destroy call, the old lease
stays `released` with the keep marker, and the next run resumes the new
sandbox only. With unsaved work or no saved snapshot, the run still
fails strictly and does not mark the old lease. Earlier tests (a refusal
without a backup, a live owner, a dirty workspace with a timeout) are
unchanged.
- `pnpm exec vitest run src/__tests__/environment-runtime.test.ts
src/services/environment-runtime-reusable-lease-probe.test.ts
src/__tests__/reusable-leases-default.test.ts`: 215 pass.
- `tsc -p server/tsconfig.json --noEmit` passes.
- Not run: a live replacement on staging. It is verified after deploy on
the next wake of the affected tasks.

## Risks

- A runner task that hits this path starts a fresh runner session. The
files come from the saved snapshot, and the task, comments, and run
history are kept. Before this change, the task stopped and waited for a
person.
- The kept sandbox stays with the provider (archived storage) until
someone removes it. This is the cost of never destroying unsaved runner
state.
- The rule needs the full extended start deadline (about 330 seconds) of
timeouts, so a slow but healthy start still resumes the same sandbox.
- No schema change. Old leases follow the same rule.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:06:25 -06:00
Nicky LeachandClaude Fable 5.2 503a736195 feat(ci): read a registry build cache in the Runner image build (#21310)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - `.github/workflows/runner-image.yml` builds the Runner alone and
publishes it from `master` as `ghcr.io/paperclipai/paperclip-runner`,
one time for each set of inputs
> - The build used no cache, so every run compiled the daemon again. The
first publication took 25m46s from push to tag
> - The toolchain and the daemon change only with the Rust sources. Most
commits that change the Runner's inputs do not touch them
> - This pull request lets the build legs read a registry build cache
for those stages. Only a new job can write the cache, and only for a
`master` push
> - A weekly run builds with no cache, so the cache cannot hide a broken
build for long
> - The benefit is about 3 minutes less on each build leg when the Rust
sources did not change, with no change to the published image

## Linked Issues or Issue Description

Refs #21230 (publication of the Runner image), #21229 and #21204 (the
build this change makes faster). No issue exists for this change.

**What existing behavior does this improve?**

The Runner image build in `.github/workflows/runner-image.yml`. It
builds the same image in less time.

**Current behavior**

Each build leg starts from an empty builder. It installs the Rust
toolchain and compiles the daemon in every run. In the first `master`
publication (run 43365670428) the amd64 leg spent 7m22s in the build
step. The toolchain and the cargo build took 3m26s of that.

**Proposed behavior**

The build legs read
`ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>` for the stages
up to the compiled daemon. A new `cache` job writes that cache after a
publication. A scheduled run builds with no cache one time each week.

**Reason and benefit**

The stages before the last one read the Rust sources, the protocol files
and the toolchain pin. In the last 30 days, 438 `master` commits changed
an input of the image, and at most 181 of them changed a file those
stages read. For the other commits, and for each pull request that does
not touch those files, the build step is about 3 minutes shorter on each
leg.

**Breaking changes**

None. The `Dockerfile` and each file in the image identity are
unchanged. The published digest for a set of inputs does not change.

This is part 4 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build (this pull
request)

This pull request uses no code from parts 1 to 3. It is stacked on them
because all four parts edit `doc/runner-image-contract.md`.

## What Changed

- The build legs pass `--cache-from
type=registry,ref=ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>`
to the first build command. They do not log in. Their permissions stay
`contents: read` and `packages: read`.
- A new `cache` job writes the cache with `--cache-to
type=registry,...,mode=max` and `--output type=cacheonly`. It runs only
for a `master` push in the canonical repository, for inputs that had no
image. Its permissions are `contents: read` and `packages: write`. It
has no `id-token` and no `attestations` permission.
- The `cache` job needs the `publish` job, and no job needs the `cache`
job. A cache write cannot delay or fail a publication.
- The `cache` job builds the target `runner-artifacts-runnerd` and no
later stage. The last stage takes the commit as a build argument, so no
other commit can reuse it. The install and its dependency lifecycle
scripts are in that stage, so the job with the write token does not run
them.
- The `cache` job has one concurrency group for each architecture, a job
time limit and a step time limit.
- A `schedule` trigger starts a weekly build. It passes `--no-cache` to
the first command and reads no cache. It runs all checks and publishes
nothing. A fork's schedule runs nothing.
- `server/src/__tests__/runner-image-workflow.test.ts` pins the new
security properties: which jobs hold `packages: write`, that the build
legs have no credential and cannot write the cache, the conditions of
the `cache` job, that the `publish` job does not depend on it, and that
a scheduled run cannot publish or write the cache.
- `doc/runner-image-contract.md` has a "Build cache" section.

## Verification

Tests, on macOS (Node 26.12.0) and in a `node:24.22.3-bookworm`
container (Linux arm64, Node 24.22.3):

- `pnpm exec vitest run src/__tests__/runner-image-workflow.test.ts
src/__tests__/runner-image-identity.test.ts` in `server/`: 55 of 55
tests pass on both.
- `node --test .github/scripts/tests/*.test.mjs`: 807 of 807 tests pass
on both. The container needs `jq`.
- `pnpm verify` exits 2 for this change, because a workflow file is a
global CI input. Required CI supplies the broad coverage.
- Each new assertion fails when the workflow breaks its rule. I checked
seven such edits, for example: the `publish` job needs the `cache` job,
the `cache` job gets `id-token: write`, a build leg gets `--cache-to`.

Cache mechanics, on linux/arm64 with a local registry, BuildKit v0.34.2
(the version the hosted runners use), and an empty builder for each
build:

| Build of both variants | Time | Result |
| --- | --- | --- |
| No cache, cache ref does not exist | 260 s | An importer error in the
log, exit 0 |
| One changed TypeScript file, with the cache | 149 s | Toolchain and
cargo steps cached |
| Unchanged inputs and a new commit, with the cache | 129 s | Toolchain
and cargo steps cached |
| Each stage in the cache (a whole-build cache, which this change does
not write) | 16 s | Both OCI archives complete |
| `--no-cache` on the first command | 272 s | No cached step in the
first command. The second command reuses all of it |

- The images built from the cache pass `runner-image-layout.mjs check`
and `runner-image-oci.mjs check-layers`. The daemon has the same SHA-256
as the daemon of a cold build.
- A cache ref that does not exist, a host that refuses the connection, a
name that does not resolve, and the real
`ghcr.io/paperclipai/paperclip-runner-buildcache:arm64` (which does not
exist yet) each give an importer error in the log and a complete build
with exit 0.
- The `cache` job command on an empty builder with the cache present
takes 6 s. It downloads no layer.

Not tested before merge:

- The `cache` job. It runs only on `master`.
- The scheduled run. It starts only from the default branch.
- The cache package. It does not exist. I did not test a private
package, which answers an anonymous read with an authorization error. I
expect the same importer error and a cold build.
- A fully cached build on the hosted runners. The local test used the
same BuildKit version and a different buildx version (0.37.0, and the
runners have 0.38.1).

## Risks

- **A second job can write packages.** Before this change, only the
`publish` job held `packages: write`, and it runs no build. Now the
`cache` job holds it too, and it runs BuildKit on the repository's
`Dockerfile`. GitHub cannot limit that token to one package, so it can
also write to `ghcr.io/paperclipai/paperclip-runner`. The limits are:
the job runs only for a `master` push in the canonical repository, the
token goes only to the registry login, the build steps run inside
BuildKit without it, the job does not run the install, and the job
cannot sign. An image that it wrote has no attestation.
- **Part of a published image can come from an earlier run.** The daemon
in a published image can come from a step that an earlier `master` run
compiled and cached. The attestation names the run that published, not
the run that compiled. BuildKit reuses a step only for the same
instruction and the same file content, but a party that can write the
cache package can replace what a later build reuses. Write access to the
cache package must stay as narrow as write access to the image package.
- **A cached download is not repeated.** The toolchain installer, the
toolchain and the crates are not downloaded again until the inputs of
their step change. The weekly cold build finds a download that is gone.
It does not compare its output with the cache.
- **A registry failure after the cache import fails the leg.** A cache
that cannot be read at the start is not a failure. A layer download that
fails later is a failure. Run the job again.
- **The cache package starts private.** GitHub creates it on the first
write. A maintainer must make it public one time. Until then each build
runs cold and nothing fails.
- **The first cache write waits for an input change.** The merge of this
pull request changes no input of the image, so its `master` run builds
nothing and writes no cache.
- **A cancelled cache job marks its run as cancelled.** When three cache
writes for one architecture wait at the same time, GitHub cancels the
middle one. The image of that run is published.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, through Claude Code.
Tool use and shell execution. It wrote the change, ran the local builds
and tests, and wrote this description.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:46:15 -08:00
Nicky LeachandClaude Opus 5.5 4f4d823cfc feat(environments): forward the warm idle window to plugin lease acquisition (#21305)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugin environment drivers run agent tasks on remote computers, and
warm leases keep a parked Runner between runs for an idle window that
Core records
> - Some providers stop an idle computer on their own schedule (for
example a sandbox auto-stop interval)
> - If that provider timer is longer than the Runner idle window, the
provider keeps paying for a computer whose Runner is already gone; if it
is shorter, a warm resume can find the computer stopped
> - The plugin cannot align the two because it never sees the idle
window at acquire
> - This pull request forwards the warm idle window to
`environmentAcquireLease` as `runnerLifecyclePolicy`
> - The benefit is that a provider can stop its computer just after the
Runner idle window, without Core knowing anything about the provider's
placement

## Linked Issues or Issue Description

Refs #21289
Refs #21066

**What existing behavior does this improve?**
Warm lease setup for plugin environment drivers.

**Subsystem affected**
Server environment runtime (`createPluginEnvironmentDriver`) and the
plugin SDK acquire parameters.

**Current behavior**
Core records the warm lifecycle policy (`runnerTaskLifecyclePolicy`,
including `idleTimeoutMs`) on the lease, but `environmentAcquireLease`
does not receive it, so a provider with its own idle shutdown has to
guess.

**Proposed behavior**
For a warm lease, `environmentAcquireLease` receives
`runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs }`, the same value
Core records. Other lifecycle modes do not receive the field.

**Reason and benefit**
A provider can align its idle shutdown with the Runner idle window.

**Breaking changes**
None. The field is optional and additive.

## What Changed

- `PluginEnvironmentAcquireLeaseParams` has a new optional
`runnerLifecyclePolicy`.
- Core sends it only for warm leases.
- `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` documents the field.
- The runner-e2e SDK public-input pin is updated for the SDK change.

An earlier revision of this PR also made Core try
`environmentResumeLease` before a session handoff. That part is removed:
placement is the plugin's decision, so Core should not decide to retry
the same computer before moving the session. The plugin will own that
inside its own resume instead.

## Verification

- `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed.
- `vitest run src/__tests__/environment-runtime.test.ts -t "portable
task placement|delegates plugin environment leases with acquisition
budget"` passed (5 cases). The portable test asserts the first warm
acquire receives `runnerLifecyclePolicy` and the lease records
`runnerTaskLifecyclePolicy`; the non-warm delegation test asserts the
field is absent. Making the forwarding unconditional fails both
delegation cases.
- The full suite was not run locally; required CI covers it.

## Risks

- A plugin that does not know the field ignores it.
- Rollback: revert this pull request. No migration.

Note: the branch name still reflects the earlier scope (resume before
handoff).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:45:31 -08:00
DottaandClaude Opus 5.5 86f9555695 fix(native): give a Runner whose first unanswered read came on the last attempt one replacement attempt (#21322)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner, and a same-run retry
reattaches that Runner after an interruption
> - #21272 made the next attempt replace a Runner that left its recovery
read unanswered, instead of adopting it again
> - On staging, three runs after #21272 went live still failed: their
first unanswered read came on the third and last attempt, so no next
attempt existed
> - This pull request gives such a run one more attempt, used only for
that replacement
> - The benefit is that the replacement from #21272 can run when a
restart used up the middle attempt

## Linked Issues or Issue Description

Refs #21272, Refs #21161

**What happened?**

After #21272 (497ccba8f1) went live on staging at 05:34Z, three Codex
runs still failed with `native_session_recovery_failed:
HarnessCapabilityUnavailableError: read is unavailable: Error:
provider_transport_failed: PRP command session.snapshot timed out
(runner connected; pending ahead 0; last runner event seq … 55–84s
ago)`. The run events of all three show the same sequence:

1. Attempt 1 fails at 06:24 with `runnerd did not report the provider
turn identity`.
2. Attempt 2 starts at about 06:25. The 06:24–06:25 server restart ends
it with `local_runtime_api_listener_unavailable`. This attempt sends no
recovery read.
3. Attempt 3 adopts the same Runner. Its first recovery read gets no
answer. Attempt 3 is the last of three, so the run ends.

`adoptedRunnerUnanswered` was recorded on attempt 3, but no next attempt
read it. The replacement from #21272 never ran.

**Expected behavior**

A Runner whose first unanswered read comes on the last attempt is still
replaced once by a fresh Runner from the saved journal. The provider
turn is not sent again. The run does not end only because a restart used
the middle attempt.

**Steps to reproduce**

1. Start a Codex run in a remote Daytona Runner. Let attempt 1 fail at
turn start, so the Runner stays alive.
2. Restart the server during attempt 2, so that attempt fails without a
recovery read.
3. Let the Runner leave its recovery read unanswered on attempt 3.
4. See the run end with `session.snapshot timed out` and no replacement.

**Paperclip version or commit**

Staging with #21272 (497ccba8f1). Fixed on `master` after `bbafa78fe2`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`).

## What Changed

- `nativeUnansweredRunnerReplacementDisposition` (executor): when
attempt 3 fails because its adopted live Runner left a recovery read
unanswered, and that attempt was not itself a replacement, the failure
stays `retryable_failure` (`native_session_interrupted`, retry in 30 s)
instead of `native_session_retry_exhausted`.
- `claimNativeRestartRecoveries`: a failed remote retry at attempt 3
with `adoptedRunnerUnanswered` is not treated as an exhausted budget.
The claim is `reattach_remote_runner` at provider attempt 4 with
`replaceUnansweredRunner`.
- `nativeAdmittedProviderAttemptLimit` (executor admission gate): a
`reattach_remote_runner` claim with `replaceUnansweredRunner` is
admitted up to provider attempt 4. Every other claim keeps the limit of
3.
- `verifyRemoteRunnerOrSettledExit`: the replacement of a live Runner is
allowed up to attempt 4. An exited Runner keeps the fail-closed path.
- Bound: the extra attempt is a replacement (`replaceUnansweredRunner`),
so it never gets another one. Attempt 4 is exhausted by the existing
rule.
- `doc/execution-semantics.md` describes the extra attempt.

## Verification

- End to end (embedded Postgres):
`native-remote-retry-admission.test.ts` › "executes the one replacement
attempt after the last attempt's first unanswered recovery read, and no
further one". It seeds attempt 3 with the staging failure, claims it,
and executes the claim through `executePaperclipNativeSession`. The
provider is reached once, the run then ends at attempt 4, and no further
claim is made. With the old `nextAttempt > 3` gate this test fails
before the provider is reached.
- Red on `master`, green with the fix (7 tests, including "admits
provider attempt 4 at execution only for the replacement claim"):
- `native-runner-restart-recovery.integration.test.ts` › "gives a Runner
whose first unanswered read came on the last attempt one replacement
attempt: true" (embedded Postgres; the coordinator is seeded at attempt
3 with the staging message). On `master` no reattach claim was made.
- `native-session-executor.test.ts` › "bounds the replacement after …"
(4 cases: the last attempt's first unanswered read is retryable; a
replacement that also went unanswered, another failure, and attempt 4
stay terminal) and "replaces the live Runner on the extra replacement
attempt".
- Focused suites: `native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation`: 1737 passed, 21 skipped.
- `pnpm typecheck` in `server` passes.
- Not run locally: the container-gated remote Runner suites and the full
suite (CI covers it). Why the remote Runner stops answering is a
separate investigation and is not changed here.

## Risks

- Low to medium. This adds at most one provider attempt per run, and
only after an unanswered recovery read on a live adopted Runner. The
attempt replaces the Runner through the launch fence, which proves the
old Runner and its lineage gone or fails closed. No provider turn is
sent again.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events),
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:27:35 -06:00
DottaandClaude Opus 5.5 49f0030f06 fix(heartbeat): rank a person's joined comment first and refill free run slots (#21277)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts each agent's queued runs, up to the
agent's `maxConcurrentRuns`
> - #21234 starts person-initiated runs before system recovery runs, and
it reads "person-initiated" from the run's own wake request
> - When a person comments on a task that already has a queued system
run, the comment joins that run. The run keeps its system wake, so
dispatch ranks the person's message as system work
> - Dispatch is event driven. The queued-run re-drive backstop covers
only agents with no live run, so an agent with several slots keeps a
free slot empty until one of its runs finishes, and runs can take hours
> - This pull request makes dispatch read the person wakes that joined
each queued run, and makes the re-drive backstop cover an agent that
still has a free slot
> - The benefit is that a person's comment starts first, and free slots
fill within about a minute instead of waiting for an unrelated run to
finish

## Linked Issues or Issue Description

Refs #21234, #20986

**What happened?**

On staging, one agent had `maxConcurrentRuns` 20, 18 running runs and 76
queued runs at 03:12Z. No run started on that agent from 03:01:33Z to
03:16:57Z (15.4 minutes). From 03:11:10Z, after a run finished, at least
2 slots were free. Other agents in the same company started runs during
that time. Runs started again only after the next deploy restarted the
server.

Five queued runs came from a person: comments at 02:43Z, 02:44Z and
02:45Z, and Retries at 02:46Z, 02:49Z and 03:01Z. The comment runs had
the wake reasons `issue_reopened_via_comment` and `issue_commented`, the
invocation source `automation`, and no person on the run's own wake
request.

**Expected behavior**

A run that carries a person's comment ranks as person-initiated, also
when the comment joined an existing queued run. When an agent has a free
slot and a queued run is ready, the run starts promptly, without waiting
for another run on the agent to finish.

**Steps to reproduce**

1. Give an agent `maxConcurrentRuns` 3. Start 2 runs on it.
2. Queue a third run while no dispatch signal reaches the queue (for
example, a claim declines it while a gate is still closed, and the gate
then clears).
3. See that `listUndispatchedQueuedRuns` returns nothing, because the
agent has a live run. The run stays queued until one of the 2 running
runs finishes.
4. Separately, queue a system run on a task, then post a person's
comment on the task so that the comment wake coalesces into that queued
run. Queue older system runs on other tasks of the same single-slot
agent.
5. See that dispatch starts the older system runs first.

**Root cause**

1. `coalesceIntoActiveExecutionRun`
(`server/src/modules/wake-queue/adapters/postgres.ts`) merges the
person's comment context into the queued run and records the person's
wake as a separate `coalesced` row with `runId` set to that run. The
run's own `wakeupRequestId` still points to the system wake.
`queuedRunDispatcherForAgent` (`server/src/services/heartbeat/queue.ts`)
only read the wake behind `wakeupRequestId`. The task routes do record
the comment author as the requester (`requestedByActorType: "user"`).
The person was lost only at dispatch time.
2. `listUndispatchedQueuedRunPage`
(`server/src/services/queued-run-dispatch-liveness.ts`) skipped every
agent with any `running` or `scheduled_retry` run. That rule assumes one
slot per agent. For an agent with several slots, a lost or declined
dispatch is not re-driven until a run on that agent finishes.

This change is not in `claimQueuedRun`, the fresh-isolated preflight,
the wake-queue promoter or `fresh-continuation-fallback.ts`.

**Paperclip version or commit**

Staging Core `ddc0aacad8` (contains #21234 `ade7e0643f`). This branch is
based on master `6cb0dd0c6b`.

**Deployment mode**

Authenticated, public, Paperclip Cloud managed staging.

**Agent adapter(s) involved**

`paperclip_runner` (Codex native runtime). The fix is
adapter-independent.

## What Changed

- `queuedRunDispatcherForAgent` also reads person wakes whose `runId` is
a queued run of the agent (by `run_id IN (queued run ids)`, so an
earlier saved comment that a run adopted is included). A run with such a
wake ranks as person-initiated and is never counted as system recovery.
- `listUndispatchedQueuedRuns` now returns a queued run when its agent
has fewer `running` plus `scheduled_retry` runs than its
`maxConcurrentRuns`, and its task has no `running` or `scheduled_retry`
run. The re-drive backstop and the ownership I7 check use this query.
- The query counts each agent's live runs once, in one grouped subquery
that is joined to the queued runs.
- Migration 0476 adds `agent_wakeup_requests_company_run_idx`
(`company_id`, `run_id`, partial on a non-null run) with `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`. It is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` and classifies as
`online_compatible`. `doc/DATABASE.md` describes it.
- Added `maxConcurrentRunsSql()` to `timer-policy.ts`. It reads the same
limit as `parseHeartbeatPolicy`, including the single-run runner
providers, which now share one constant.

## Verification

- Red/green: `server/src/__tests__/recovery-wake-herd.test.ts` "starts a
queued run that a person's comment joined ahead of older system work".
It fails on master (`expected null not to be null`: the older system run
started first) and passes with this change.
- Red/green:
`server/src/__tests__/ownership-reconciliation-report.test.ts`
"re-drives a queued run while its agent still has a free slot, and not
once every slot is running". It fails on master (`expected [] to deeply
equal [ Array(1) ]`) and passes with this change.
- Red/green: "starts a queued run that adopted a person's comment saved
before every queued run". It fails with a `requestedAt` bound and passes
with the `run_id` lookup.
- Red/green: "counts an agent's live runs in one grouped aggregate, not
once per queued run" runs EXPLAIN on the page query. It fails with a
per-row correlated count and passes with the grouped subquery.
- New: "uses the agent's single-run limit for a runner provider that
runs one turn at a time".
- Two existing tests now set `maxConcurrentRuns: 1` so that they keep
testing "the agent's only slot is busy".
- `npx vitest run src/__tests__/recovery-wake-herd.test.ts
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/ownership-reconciliation-report.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/services/heartbeat/dispatch-work.test.ts`: 71 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: no errors.

## Risks

- The re-drive backstop now also covers multi-slot agents with live
runs. Each queued run is re-driven at most on its existing backoff (1
minute, then doubling to 15 minutes). A queued run whose task already
has a live run is not re-driven.
- The ownership I7 report can now list a queued run on a multi-slot
agent that has a free slot. This is the stall that this change targets.
- One more indexed read per dispatch pass (`agent_wakeup_requests` by
company and run id). The new index is built concurrently, so the
previous release keeps writing during the build.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, read-only staging API reads in a
browser tab).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:17:57 -06:00
Nicky LeachandClaude Fable 5.2 23c2343015 feat(server): read a retained provider pack from its Runner image (#21308)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner executes agent work in a sandbox. It needs a provider
pack of agent tools, and the server uploads that pack when the sandbox
does not have it
> - A retained session keeps the pack it started with. When that pack is
not the server's own, the server downloads the pack's transport archives
from a registry (#19779)
> - Today the server looks for them in the application image tagged
`sha-<commit that built the pack>`
> - `master` now publishes the Runner one time for each set of inputs,
as `ghcr.io/paperclipai/paperclip-runner`. A pack from that image names
the commit that published the image, and few commits get an application
image, so that tag often does not exist
> - This pull request reads such a pack from its Runner image, and keeps
the old source for every other pack and for a Runner image that cannot
be read
> - The benefit is that a retained session can still get its pack when
the pack came out of the published Runner image

## Linked Issues or Issue Description

Refs #19779 (the retained pack source), #21228 (the pack field
`runnerInputsDigest`), and #21204, #21229, #21230 (the Runner image). No
issue exists for this change.

**Problem or motivation**

`retained-provider-pack-oci.ts` reads
`ghcr.io/paperclipai/paperclip:sha-<runnerSourceRevision>`. It needs
that image to exist, to carry the pack, and to have one layer with a
known build-history line. For a pack from a published Runner image,
`runnerSourceRevision` is the commit that published the image.
`docker.yml` writes `sha-<commit>` only when its `master` run for that
commit completes, and a newer push can replace a waiting run. The
restart-chaos harness has the same assumption in another place: it
requires the pack of a sandbox image to be built by the commit in the
image's `org.opencontainers.image.revision` label.

**Proposed solution**

A pack that names its Runner image in `payload.runnerInputsDigest` is
read from `ghcr.io/paperclipai/paperclip-runner:inputs-<digest>`. A
sandbox image can name the commit that built its Runner in a label of
its own. No application image carries a pack with that field yet, and no
image carries that label, so behaviour today does not change.

**Alternatives considered**

- Keep the application image as the only source and require an
application image for every Runner publication. Publication would then
depend on `docker.yml`, which serves one ref at a time.
- Take a fixed number of layers ("the image has four, take the last
two"). A server in the field cannot be changed when the image gets
another layer. The server now uses the rule that holds for any number of
layers: the full image is the slim image plus transport layers.
- Fall back to the application image for every failure of the Runner
image. A Runner image that exists and does not match the pack is a
fault, and a second source would hide it.

**Roadmap alignment**

This is build and runtime infrastructure. It adds no product feature and
does not overlap an item in `ROADMAP.md`.

This is part 3 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image (this
pull request)
4. #21310: a registry build cache for the Runner image build

This pull request uses no code from parts 1 and 2. It is stacked on them
because all four parts edit `doc/runner-image-contract.md`.

## What Changed

- `server/src/services/native-runtime/retained-provider-pack-oci.ts`,
for a pack whose `payload.runnerInputsDigest` is 64 hexadecimal
characters:
- The server reads the full index `inputs-<digest>` of the Runner image.
The index must carry `io.paperclip.runner.inputs-digest` equal to the
pack's value, `org.opencontainers.image.revision` equal to the pack's
`runnerSourceRevision`, and a valid digest in
`io.paperclip.runner.slim-digest`.
- It reads the slim index by that digest, never by the slim tag, and
checks the bytes it gets against the digest.
- For the pack's platform, the layer digests of the slim image must be
the first layer digests of the full image, in the same order. The layers
after them are the transport layers. There must be at least one and at
most eight, and their total size has the existing limit.
- The server downloads only those layers. It checks each against its
descriptor and refuses every entry that is not a transport archive, the
receipt or the `provider-pack.archives` directory. A file may not be in
two layers.
- When the Runner image cannot be read, the server reads the application
image as before. That covers a registry error status for the token, for
either index or for either manifest: for example 404 for a tag or a slim
index that does not exist. Nothing is written before that point.
- A Runner image that can be read and does not match fails with
`runner_remote_provider_source_invalid`. The server does not try the
application image then.
- The same file, for every other pack: the old path, unchanged in what
it checks. Its code moved into functions that both paths use (`request`,
`pullToken`, `platformImage`, `receiveTransportLayer`).
- The downloaded layer file is removed as soon as it is unpacked.
Before, the importer removed it after its checks, so
`retained-provider-pack-source.ts` loses that one line.
- The cache record gets an optional `runnerImage` field with the inputs
digest and the digests of the transport layers. `transportLayerDigest`
is the last of them and `transportLayerBytes` their total. An older
server ignores the new field.
- `tests/restart-chaos/daytona-image.ts`: when the image has a non-empty
label `io.paperclip.runner.source-revision`, the pack must be built by
that commit. Otherwise the pack must be built by the commit in
`org.opencontainers.image.revision`, as before. The receipt field
`providerPackRunnerSourceRevision` records the pack's commit.
- `doc/runner-image-contract.md` has a new section, "Reading the
transport archives alone". `tests/restart-chaos/README.md` describes the
label.

One choice needs a reviewer's eye. When the full index can be read and
the slim index cannot (a registry error status), the server falls back
to the application image; it does not fail. Without the slim manifest
the server cannot tell which layers are transport layers, so it has not
accepted any byte of the Runner image, and the old source has its own
complete checks. A slim index that is read and has other bytes than its
digest still fails.

## Verification

Linux (`node:24.22.3-bookworm`, GNU tar 1.34, zstd 1.5.4, the worktree
mounted read-only):

- In `server/`: `retained-provider-pack-source.test.ts` (34 tests, 15 of
them new), `provider-pack-archive.test.ts`,
`remote-provider-pack-verification.test.ts`,
`remote-provider-pack.test.ts`: 127 tests pass in 4 files.
- `node node_modules/vitest/vitest.mjs run --config
tests/restart-chaos/vitest.config.ts
tests/restart-chaos/daytona-image.test.ts`: 25 tests pass, 5 of them
new.

macOS (Node 26.12.0):

- `daytona-image.test.ts`: 25 tests pass.
- `retained-provider-pack-source.test.ts`: the 15 new tests pass. Six
older tests fail, with and without this change. They run the real
uploader, which needs GNU tar, and this host has bsdtar.
- `tsc` on the two changed server files and the test, and on the two
restart-chaos files, with the settings of their test projects: no error.

The new cases:

- Read from the Runner image with 6 registry requests (two indexes, two
manifests, two layers), none to the application image and none for the
slim tag.
- Take every layer after the slim image's: one transport layer, and
three.
- Fall back when the tag does not exist, when the token is refused, and
when the slim index cannot be reached.
- Refuse, with no request to the application image: an index for other
inputs, an index for another revision, no slim annotation, a slim
annotation that is a tag, a slim index with other bytes than its digest,
a slim image that is not a prefix of the full image, a slim image equal
to the full image, nine transport layers, and a transport layer that
holds a pack file.
- Four deliberate faults in the reader each make a test fail: the slim
index bytes are not checked, no prefix check, the slim annotation is not
validated, and no bounds on the number of transport layers.

A real run, in the same Linux container (`linux/arm64`), with no
credential:

- The script took `provider-pack/provider-pack.json` of `linux/arm64`
from the published image
`ghcr.io/paperclipai/paperclip-runner:inputs-0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`.
Pack digest
`sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f`,
`runnerSourceRevision` `25e9a9c2e9277f3be8649be8c809cc81bde6f0c5`.
- `resolveProviderPackUploadSource`, with another pack as the server's
own, returned `{ archiveOnly: true }` after 23 seconds. That function
calls `receiveRetainedProviderPack` and then the importer's checks: the
manifest inside the overlay archive is the selected manifest, every
archive matches the receipt, and the content of every file archive
matches the digest of its file.
- The server code made 9 requests: one anonymous token request, the full
index, the `arm64` manifest, the slim index `sha256:f63ea436…4f3c` by
digest, its `arm64` manifest, and two layers, each with one redirect. No
request went to the application image.
- The cache record:

  ```json
  {
    "schema": "paperclip.retained-provider-pack-source.v1",
    "sourceRevision": "25e9a9c2e9277f3be8649be8c809cc81bde6f0c5",
"providerPackDigest":
"sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f",
"imageIndexDigest":
"sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162",
"imageManifestDigest":
"sha256:73dbe864fbab6208616f0aa6f1610a7ece4ef22f68a73d30901cbaea60139589",
"imageConfigDigest":
"sha256:740e48463d5caf0bfead4b923cdbd8b220acb93d058419c3db9d00c7f1831d65",
"transportLayerDigest":
"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf",
    "transportLayerBytes": 426602696,
    "runnerImage": {
"inputsDigest":
"0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca",
      "transportLayerDigests": [

"sha256:6a340088ca2cfa85f10f0385d9c8f5cdccb22c3a2bbb2062620b1d90616ba5b5",

"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf"
      ]
    },
"receiptSha256":
"2afdd26645845369d9e851b1e6e2884eb6c3884ea7c4866a1eeb2efb9fb230d7",
    "verifiedTarPaths": true
  }
  ```

- `uploadVerifiedProviderPack` then installed the pack from that source
through a local stand-in for the sandbox, with no further registry
request, in 4 seconds. The installed tree is 1,249,060,841 bytes. Its
`provider-pack.json` is the selected manifest, all 7 artifacts of the
manifest have the digest the manifest names, and the pack's own Node
prints `v24.22.3`.

Never run:

- The same real run for `linux/amd64`, whose transport layers are about
1 GB.
- A session in a real sandbox. The real run used a local stand-in for
the sandbox.
- The fallback against the real registry. Fixtures cover it.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The old path was moved into shared functions. Its checks are the same,
and its 19 existing tests pass on Linux, but a review must read that
code as changed.
- The layer file of the old path is now removed before the importer's
checks, not after them. The importer does not read it.
- The reader depends on two rules of the image contract: the full index
names its slim index, and the full image starts with the slim image's
layers. A Runner image that breaks either rule is refused, with no
fallback.
- The fallback covers a registry error status only. A network error ends
the import, as it does today.
- The label `io.paperclip.runner.source-revision` is new and no image
carries it yet. The change to the sandbox image that sets it must use
this name.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:02:19 -08:00
DottaandClaude Opus 5.5 0477f622d6 fix(recovery): keep the sandbox that holds abandoned unsaved files out of replacement and cleanup (#21304)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in remote sandboxes, and the workspace files live in
those sandboxes until a save
> - When a run cannot resume exactly, the fresh continuation fallback
(#21135) abandons the old workspace epoch and keeps its unsaved files
"recorded and unapplied" on the old sandbox
> - The abandonment clears the dirty flag, the writer and the location,
so the workspace looks clean. No code reads the abandonment receipt
> - A later failed resume can then replace the sandbox, and cleanup can
destroy it. That sandbox holds the only copy of the unsaved files
> - This pull request adds one shared hold. The task continues on a new
sandbox, and the old sandbox is parked until its files are saved,
discarded, or kept for 7 days
> - The benefit is that the fallback keeps its promise. Abandoned files
are not lost

## Linked Issues or Issue Description

Refs #21135
Refs #21297

**What happened?**
After the fresh continuation fallback abandoned a workspace epoch, the
receipt in `remoteWorkspaceAbandonedEpochs` was write-only.
`reusableSandboxReplacementIsSafe` saw a clean workspace with a
committed save and allowed replacement.
`retireReplacedReusableSandboxLease` then moved the old lease to
`pending_cleanup`, and the cleanup sweep destroyed the sandbox with the
unsaved files. `nativeWorkspaceExportPendingForLease` (the check that
the sweep, the retained orphan settlement and the scoped destroy paths
use) also answered "nothing pending".

**What did you expect?**
A sandbox that holds abandoned, unsaved files stays until the files are
saved, a person discards them, or a stated retention period ends.

**Steps to reproduce**
1. Abandon a remote workspace epoch whose location names sandbox L.
2. Let the next run fail to resume L.
3. The replacement decision says safe, L goes to `pending_cleanup`, and
the sweep destroys it.

## What Changed

- New `server/src/services/remote-workspace-abandoned-files-hold.ts`
holds the single predicate for "this lease holds abandoned files". It is
one SQL fragment (`abandonedWorkspaceFilesHeldSql`) plus a TypeScript
wrapper (`abandonedWorkspaceFilesHeldOnLease`). A lease is held while a
workspace in its company has an abandoned-epoch receipt whose
`location.providerLeaseId` equals the lease's provider lease id, and
that receipt has no `released` object. A receipt written by the current
release (no new fields) counts as unreleased.
- Replace and park. A failed resume of a held sandbox still lets the run
replace it, so the task continues at once on a new sandbox restored from
the last committed save. `retireReplacedReusableSandboxLease` parks the
old lease as `retained` with the `abandonedWorkspaceFilesParked` marker
(failure reason `abandoned_workspace_files_held`) instead of
`pending_cleanup`. `reusableLeaseCanBeResumed` and
`reusableLeaseCanBeCleanedUp` skip a parked lease, so later runs never
select it again. The check is a metadata read with no query, so the run
path does no extra work.
- Every path that can destroy a sandbox now honours the hold:
- `nativeWorkspaceExportPendingForLease` answers pending for a held
lease. This covers the pending cleanup sweep (defer), the orphaned
active lease sweep, the retained orphan settlement `heldBy`,
`destroyClaimedSandboxLease` (terminal workspace reaper
`destroyRemoteWorkspaceLease`, `destroyReusableSandboxLeases` and the
environment delete path, and run lease release with `destroy`), the
`destroyRunLease` dispatcher, and the plugin sandbox teardown retry.
- The run lease settlement (`environment_run_lease_active`) is the one
caller that ignores the hold (`{ destructive: false }`). It never
destroys a sandbox. It stops the sandbox and keeps it, or it moves a
lease without a sandbox to `pending_cleanup`, where the sweep parks a
held lease. If it waited for the hold, a dead run's lease would stay
`active` and block the retry for the whole retention period.
- `destroyReusableSandboxLease` parks the lease and returns without a
provider call. This covers obsolete reusable lease cleanup, the destroy
after an expired or mismatched resume, release with `expired`, and the
sandbox driver's `destroyRunLease`.
- The `retryPendingSandboxTeardown` dispatcher refuses a destructive
teardown of a held lease. Stop-only cleanup still runs, because a stop
keeps the filesystem.
- The retained orphan listing (`alwaysHeldSql`) uses the same SQL
fragment, so held orphans are not polled.
- Release. `releaseAbandonedWorkspaceFiles` marks a receipt `released`
with a reason (`recovered`, `discarded`, `retention_expired`) and an
actor. It moves the parked leases for that sandbox to `pending_cleanup`
when no other unreleased receipt holds them, and it writes a
`workspace.abandoned_files_released` activity entry on the task in plain
language. No export path for abandoned epochs exists today. The
`recovered` reason is the hook for one; this pull request adds no new
export mechanism.
- Retention expiry. New receipts carry `retainUntil` (7 days). An old
receipt uses `abandonedAt` plus 7 days. `nextEnvironmentCleanupAt`
includes the earliest retention end of any unreleased receipt. The
shared cleanup worker therefore wakes at that real deadline, and a
restart rebuilds it from the database. At the deadline the sweep runs
`expireAbandonedWorkspaceFileHolds`. This releases each due receipt as
an attributed system decision (actor `workspace-retention`) and logs the
held receipts and parked leases for each company. Each expiry always
writes a task entry: "Unsaved files from an interrupted run were kept
for N days, and their retention period has ended." The entry adds "They
are scheduled for removal." only when a parked lease was queued for
cleanup. A separate entry, "... kept for N days and have now been
removed.", is written only after the provider confirms the delete. Each
receipt is released in its own transaction. A failed receipt is logged
and skipped, and it is retried after a backoff (5 minutes, doubling, at
most 6 hours). The other receipts are still processed. The entries
reference a run or a task only if it still exists. If the task was
deleted, the decision goes to the server log only.
- Upgrade. A `pending_cleanup` lease (not in flight, not stop-only) that
an unreleased receipt holds goes back to `retained` and is parked by the
sweep. A sandbox that a release before this change already destroyed
cannot be recovered. Its receipt ends at its retention time like any
other receipt.
- Receipt trim (review P1). `abandonUnverifiedRemoteWorkspaceEpoch`
trims only released receipts and keeps the last 20 of them. It never
drops an unreleased receipt. A new abandonment of the same sandbox and
root replaces the unreleased receipt for that root. So a workspace holds
at most one unreleased receipt for each distinct sandbox and root
abandoned in the last 7 days.

## Verification

- `server/src/__tests__/abandoned-workspace-files-hold.test.ts` (19
tests). It follows the lease through the real acquisition path with a
failed resume, the real `abandonUnverifiedRemoteWorkspaceEpoch`, the
real `sweepPendingCleanupLeases`, and `settleRetainedOrphanSandbox`:
- `gives the next run a new sandbox at once and parks the held one,
which cleanup and later runs leave alone`
- `pulls a held lease already queued for cleanup back to retained, and
leaves it on the provider`
- `treats a receipt written before release fields existed as unreleased`
  - `records a retention expiry on the receipt`
- `hands the parked lease to normal cleanup once the files are
recovered` / `... discarded`
- `removes the files only after the retention period, and says removed
only after the provider confirms it`
  - `does not report removal when the provider delete fails`
- `records the end of retention without scheduling removal when another
receipt still holds the sandbox`
- `records the end of retention when it expires before the lease was
ever parked`
- `wakes the cleanup worker at the receipt's retention end, also after a
restart, and then goes quiet`
- `releases a receipt whose run was deleted and records it without the
run reference`
- `releases a receipt whose task was deleted without writing a task
entry`
- `keeps expiring later receipts when one fails, and retries the failed
one after a backoff`
- `keeps a held sandbox when the retained orphan settlement reviews it`
- `lists the sandbox for the retained orphan settlement again once the
receipt is released`
- `keeps holding the first sandbox after many later abandonments on
another sandbox`
- `replaces and cleans the sandbox as before when the receipt names a
different sandbox`
- `replaces and cleans the sandbox as before when no epoch was
abandoned`
- Red on the base revision (62d673cc3f): the park, upgrade, old-receipt,
expiry, many-abandonments and scheduling tests fail. The controls pass.
- Green: 19 of 19. Also
`environment-runtime-reusable-resume-replacement.test.ts` 17/17,
`environment-retained-orphan-sandbox.test.ts` 32/32,
`heartbeat-pending-cleanup-sweep.test.ts` 51/51,
`fresh-continuation-fallback.test.ts` 58/58.
- `tsc --noEmit -p server/tsconfig.json` and a tsc pass over the new
test file: clean. `environment-cleanup-work.test.ts` +
`heartbeat-pending-cleanup-sweep.test.ts`: 80/80.

- No existing test expectation was changed. Three CI failures on bb43f52
came from this change. Two came from the hold blocking the run lease
settlement: `ownership-reconciliation-report.test.ts > releases the
lease and frees the workspace for the retry once a preparation-only
epoch ends` and `remote-workspace-save-concurrency.test.ts > lets a
retry prepare the workspace again after its first writer was killed
during preparation`. The third, `native-session-executor.test.ts >
restores a warm backup into a created lease only when a settled restore
hold retained the stopped source`, came from a mocked database that has
no `execute`; the predicate now runs through `select`. All three pass
locally now. So do `heartbeat-orphaned-active-lease-sweep`,
`environment-run-lease-ownership`, `native-workspace-warm-reuse` and
`remote-workspaces.integration`.

## Risks

- Cost of a parked sandbox. Parking makes no provider call and changes
only the lease row. A sandbox that could not be resumed is not running.
A reusable sandbox that was released is already stopped and kept. In
both cases the cost is stopped storage for up to 7 days. One case is
different: if a destructive path finds a held warm sandbox that is still
running (`retained` after a keep-running release), the sandbox stays
running until the receipt is released. The expiry log line counts parked
leases for each company, so this cost can be seen.
- Run lease release for an `ephemeral` lease calls the provider's
release. Some providers delete the sandbox on that call. A held sandbox
normally has a reusable lease, so this pull request does not gate that
call.
- No schema change. The receipt is JSON in
`execution_workspaces.metadata`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:00:20 -06:00
Nicky LeachandClaude Fable 5.2 e363b0e52d fix(docker): keep local build output out of the Runner image's build context (#21312)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as the image `ghcr.io/paperclipai/paperclip-runner`, one time
for each set of Runner inputs (#21204, #21229, #21230)
> - The name of a published image is a digest of every file the Runner
build receives from the build context
> - `.dockerignore` keeps the Runner's own build output
(`packages/paperclip-runner/dist`) out of the context, but not the eval
kernel's (`packages/paperclip-eval-kernel/dist`), which the Runner's
build also writes
> - A tree where the Runner was built therefore has another inputs
digest than a clean checkout of the same commit
> - The Runner package's Vite builds write five more directories with
the same problem
> - This pull request adds the missing ignore rules, and a test that
finds the next such directory
> - The benefit is that a developer's tree and a clean checkout of the
same commit name the same Runner image

## Linked Issues or Issue Description

Refs #21204, which added the image identity. No issue exists for this
change.

**Problem or motivation**

The inputs digest covers every file the build receives under
`packages/paperclip-eval-kernel`. The Runner's build compiles the eval
kernel and writes `packages/paperclip-eval-kernel/dist`. After a local
Runner build, that directory is in the build context, so the digest of
the tree differs from the digest of a clean checkout. A lookup of the
published Runner image for such a tree finds nothing.

CI checkouts are clean, so no published image is affected.

**Proposed solution**

Add `packages/paperclip-eval-kernel/dist` and
`packages/paperclip-runner/dist-*` to `.dockerignore`, next to the rule
for the Runner's own `dist`. Add a test that derives every build-output
directory inside the image's inputs and requires a rule for each.

**Alternatives considered**

- Exclude the directory in the identity script only. Then the digest
would not describe what the build receives, which is the property the
identity is built on.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This pull request stands alone. It is based on `master` and needs no
other open pull request.

## What Changed

- `.dockerignore`: two rules.
- `packages/paperclip-eval-kernel/dist`: the Runner's own build writes
it.
- `packages/paperclip-runner/dist-*`: the output of the Runner package's
Vite builds (`dist-browser`, `dist-issue-thread`, `dist-scenarios`,
`dist-sdk`, `dist-standalone`). The Runner's `build` script does not
write them, but a tree where one of those builds ran had the same digest
drift. No tracked path matches the rule, and no later `!` rule
re-includes those directories.
- `server/src/__tests__/runner-image-identity.test.ts`: one new test.
For each package among the identity's inputs it collects the `outDir` of
every tsconfig a script compiles with `tsc -p`, the `target` directory
beside every Cargo manifest a script names, and the `outDir` of every
Vite configuration. It requires that the last `.dockerignore` rule
matching each directory is an exclusion. Today that is eight
directories. A new output directory inside the inputs fails the test
until it has a rule.
- No image build reads that directory from the host. The Runner stages,
the `build` stage of the application image and `docker/daytona-runner`
all compile the kernel themselves (`tsc -p`, not incremental), and the
eval kernel is a development dependency of the Runner only.

## Verification

The inputs digest, as the `runner-artifacts-identity` build target
reports it, on a `linux/arm64` Docker host:

| Tree | Without the rule | With the rule |
| --- | --- | --- |
| clean export at `62d673cc3f` | `0e6662ad…` | `0e6662ad…` |
| the same, with an eval-kernel `dist` | `e4cf3095…` | `0e6662ad…` |
| clean export at `bc71c23e0a` | `88fe2929…` | `88fe2929…` |
| the same, with an eval-kernel `dist` | `ac331275…` | `88fe2929…` |
| clean export at `731b52453c` | `88fe2929…` | `88fe2929…` |
| the same, with two files in `dist-browser` | `44fe867d…` | `88fe2929…`
|
| the same, with one file in each of the five Vite directories | not
measured | `88fe2929…` |

- `0e6662ad…` is the digest of the image that `master` published at
`25e9a9c2e9`, so the rule does not change the digest of a clean
checkout.
- A worktree after a real `pnpm --filter
@paperclipai/paperclip-eval-kernel build` also gave the clean digest
with the rule.
- `runner-image-identity.test.ts` (42 tests) and
`tests/runner-e2e/daytona-image.test.ts` (19 tests), which reads
`.dockerignore`, pass on macOS and in `node:24.22.3-bookworm`.
- The new test fails on the `.dockerignore` of `master` and names
`packages/paperclip-eval-kernel/dist`. It fails without the `dist-*`
rule and names `packages/paperclip-runner/dist-browser`. It fails
without either of the two older Runner rules, and with a trailing `!`
rule that re-includes one of the directories.

Never run:

- Any image build with the new rules beyond the identity target,
including `.github/docker-context-checks.Dockerfile`. On a CI checkout
the directory does not exist, so the rules remove nothing there.
- `linux/amd64`.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- **The sandbox image's content id changes one time.** `.dockerignore`
is an input of that id (`tests/runner-e2e/daytona-image-content.ts`). At
`62d673cc3f` the id goes from `be208823…` to `1e13a151…`. The next
end-to-end run builds a new sandbox image, and the restart-chaos harness
needs an image built from this tree. The image's contents do not change.
- Low otherwise: the rules remove directories that no image build reads.
The `Dockerfile`, `docker/daytona-runner/Dockerfile` and
`.github/docker-context-checks.Dockerfile` name none of them and run no
Vite build.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:55:23 -08:00
DottaandClaude Opus 5.5 bbafa78fe2 fix(retry): offer Retry only when admission can take it, and start held saved-work Retries (#21315)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run fails and its saved work is held for recovery, the task
shows a Retry button. Admission then decides if the Retry can start a
new turn.
> - The task view, the task's execution blocker, and admission each
computed their own answer. The task view said `canRetry: true`. The
blocker said `canRetry: false`, and its own `workspaceRecovery` part
said `canRetry: true`.
> - Admission refused these Retries with no reason. The receipt kept the
generic `execution_recovery` wait ("Retry is waiting for execution
checks"), and each later click joined the same held receipt. Nothing
ran.
> - The refusals had real causes. A preparation proof rejected
host-written workspace notices in the run log. A repair proof did not
accept a snapshot that the failed preparation only restored and never
changed.
> - This pull request fixes both proofs, names every admission refusal
of a held Retry, and makes the task view and the blocker read one shared
Retry answer.
> - The benefit is that Retry starts work in these cases, and Retry is
offered only when admission can take it.

## Linked Issues or Issue Description

**What happened?**

On hosted staging, several blocked tasks showed a Retry button. Pressing
Retry was accepted but nothing ran. The task said "Retry is waiting for
execution checks. Your message is saved." The task view had
`interruption.canRetry: true` and `retryWait.reason:
execution_recovery`. The task's `executionBlocker` had `canRetry:
false`.

- Saved-work recovery failed with `source_unavailable`. The legacy
preparation had failed before the provider started
(`executionRecovery.providerWorkStarted: false`), its sandbox epoch was
abandoned, and its sandbox was destroyed. The blocker contradicted
itself: `workspaceRecovery.canRetry: true`, top-level `canRetry: false`.
- Saved work showed "recovered", but `workspaceRepairRequired` stayed
`true`. The failed preparation had restored a snapshot that an earlier
run produced. The repair proof accepted only a snapshot that the run
itself saved.

**Expected behavior**

The task view, the execution blocker, and admission give the same Retry
answer. A Retry that admission can take starts work. When saved work
cannot be recovered, Retry continues in a fresh session from the
recorded work. Every admission refusal of a held Retry has a named
reason. No Retry is offered for a refusal that another click cannot
change.

**Steps to reproduce**

1. Let a legacy `paperclip_runner` preparation fail before the provider
starts, after the host writes its workspace notices ("Execution
workspace reuse freshness action ...", "Skipping saved session resume
...") to the run log.
2. Let saved-work recovery end with `source_unavailable` and
`use_alternative_strategy`.
3. Press Retry.
4. Observe that the Retry receipt is held with `executionWait.reason =
execution_recovery` and no run starts.

Or:

1. Let a legacy preparation restore the task's saved snapshot and fail
with `setup_failed` before the provider starts.
2. Let saved-work recovery complete.
3. Observe that the blocker keeps `workspaceRepairRequired: true`, and
that Retry is held with `execution_recovery`.

**Paperclip version or commit**

Staging on `origin/master` `62d673cc3f` (after #21267 and #21295).

**Deployment mode**

Hosted cloud (staging), remote Daytona sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (legacy runtime runs, native agent).

Refs #21267, #21295.

## What Changed

- `hasVerifiedPreparationLog` (`verified-preparation-log.ts`) now
accepts a log made only of host notices that the host writes before it
hands the turn to an adapter: the mentioned-skills line (first record
only, checked against the saved skill selection) and the workspace
freshness and session-reset notices. Every record must have the same
attempt, sequential `seq`, `stdout`, and the excerpt and watermark must
match. Any other line still keeps the run held.
- `verifiedUnchangedRestoredSnapshot`
(`remote-workspace-snapshot-reuse-receipt.ts`) proves that a failed
preparation's saved state is the snapshot it restored. It needs
`executionStage: preparing`, `providerWorkStarted: false`, exactly one
succeeded restore of that snapshot at that epoch and lease, and no
snapshot save by the run. `verifiedWorkspaceRepairReceipt` uses it next
to the existing reuse proof.
- `savedWorkOffersRetry` (`workspace-recovery-receipt.ts`) is the one
rule for Retry on a saved-work hold. Retry is offered when the saved
work is recovered and verified, or when recovery is exhausted in a state
that only a Retry restarts (a new recovery generation, or the
fresh-session fallback after `source_unavailable`). Retry is not offered
while recovery re-arms by itself (a stop recheck or a newer capture
policy), or while a copyback owner's stop is unconfirmed. The blocker
uses this rule for the top-level `canRetry` and for
`workspaceRecovery.canRetry`, so the two parts cannot contradict.
- `task-retry-eligibility.ts` lists the held-Retry refusals that another
click cannot change. The task view and the blocker both use it.
- Admission (`explicit-native-continuation.ts`) names each refusal of a
held request that was silent before: `previous_execution_unverified`,
`execution_settling`, `cleanup_pending`, and `remote_cleanup`. It names
a refusal only when an execution blocker holds the task, so a request
without a hold still takes the normal path.
- The task view now reads the blocker's answer for a task with a
saved-work hold. It shows no Retry when the blocker offers none, and it
names the recovery wait.

## Verification

- Red/green: `cd server && npx vitest run
src/services/saved-work-retry-eligibility.test.ts`. The tests fail on
`origin/master` sources and pass with this change. They are built from
the staging shapes:
- offers the same Retry in the task view and the blocker when saved work
is no longer available
- offers no Retry in either view while saved-work recovery waits for an
unconfirmed file copy to stop
- continues in a fresh session from the recorded work when saved work is
no longer available
- verifies the repair of a restored snapshot that the failed preparation
never changed
- keeps the repair unverified when the failed preparation may have
changed the restored snapshot
- names a refusal that another click cannot change and offers no Retry
for it in either view
- names the admission refusal of a held Retry and offers Retry only when
admission would take it
- `npx vitest run src/services/verified-preparation-log.test.ts`: new
cases accept the host notices and keep a provider line, a different
attempt, or a late skills line held.
- Related suites: deferred execution wait, execution blocker read,
explicit native continuation, governed wait runner loss, legacy
preparation retry hold, remote workspace snapshot reuse, task
interruption presentation, task retry, and all `workspace-recovery-*`
suites.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Medium-low. The preparation-log proof accepts more host lines. Each
accepted line has a fixed host format, and the run must still pass every
other preparation proof (events, leases, no provider evidence).
- The restored-snapshot proof applies only when the run recorded that
the provider never started and it never saved its own snapshot.
- Admission now records a named wait instead of the generic one for some
held requests. It changes no admission decision.
- The task view calls `getExecutionBlocker` for tasks that offer Retry
and have a saved-work hold. These tasks are few, so the extra reads are
small.
- Not covered here: a copyback owner on a replaced deployment still
needs its stop confirmed before saved work can be copied out of a full
sandbox. The task view now shows that wait instead of a Retry that
admission refuses.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:52:25 -06:00
DottaandClaude Opus 5.5 b555087f0f fix(native): continue again when a deploy interrupts an automatic continuation (#21314)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner (Daytona) that the
server controls; a deploy restarts the server while runs are in flight
> - When the restart's reattachment proves the Runner process gone
mid-turn, #21039 makes the server request one automatic Retry that
continues in a fresh session from the saved work, without replaying the
turn
> - Staging deploys every 20 to 45 minutes, and Codex runs often take
longer, so that automatic continuation is often interrupted by the next
deploy too
> - The hook treated that second loss as "automatic Retry already used"
and left the task blocked for a person
(`native_continuation_requires_reconciliation`)
> - This pull request gives an automatic continuation one more
fresh-session continuation each time a server restart or deploy took its
controller, and a deploy spends no automatic budget
> - The benefit is that a task interrupted by repeated deploys continues
by itself, and no provider turn that may have run is sent again

## Linked Issues or Issue Description

Refs #21039, Refs #21052, Refs #21272, Refs #21023, Refs #21120, Refs
#21161

**What happened?**

On staging, the largest failure class for a Codex agent on Daytona was
`runner_remote_process_identity_unavailable` with error code
`native_continuation_requires_reconciliation`: 18 runs in 3 hours. The
error name is misleading. It is the shared code for every refused remote
reattachment (`REMOTE_RUNNER_RECOVERY_ERROR_CODE`). In every sampled run
the refusal was `process_verification` / `process_not_running`: the
Runner process was gone, its stderr log was empty, and the last turn
event was `turn.submitted` or `turn.accepted` (unsettled). It was not a
missing identity file.

Sampled 12 such runs. In 9 the #21039 hook worked: it recorded
`native.interrupted_turn_outcome_unknown` and requested the
fresh-session Retry. The tasks that stayed blocked had one shape:

1. A run lost its Runner after a deploy. The hook requested the
automatic Retry (key `task-retry:restart-interrupted-turn:<run>`).
2. The Retry run started a fresh session. The next deploy came during
its turn (a graceful `SIGTERM` shutdown transition, then "Recovering the
existing native runner process after server restart").
3. The same-run reattachment found that Runner gone too
(`process_not_running`).
4. The hook returned `automatic_retry_used`, because the run's Retry
intent came from the hook. The task stayed blocked.

Examples: a `task_retry` run created 40 s after its predecessor failed
(the hook's Retry), interrupted by the 05:17 deploy, failed at 05:18,
task blocked. A second chain (assignment, then the hook's Retry) failed
the same way at 05:12.

**Expected behavior**

A deploy is not the task's failure. When a server restart or deploy took
the controller of an automatic continuation and its Runner is proven
gone, the server continues the task again in a fresh session from the
saved work. It never replays the turn. A loss without restart evidence
keeps the existing one-time rule.

**Steps to reproduce**

1. Start a long Codex run in a remote Daytona Runner.
2. Deploy the server during its provider turn so that the reattachment
finds the Runner gone. The server requests the automatic Retry.
3. Deploy again during the Retry run's turn so that its Runner is also
lost.
4. See the Retry run fail with
`runner_remote_process_identity_unavailable` and the task stay blocked
with no new run.

**Paperclip version or commit**

Staging Core `25e9a9c2e9`. Fixed on `master` after `731b52453c`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `restart-interrupted-turn.ts`: `lostControllerToRestart` finds restart
evidence on the run: a `native.recovery.transition` with `restartKind:
"graceful"` (shutdown), or a restart claim (`reattach_remote_runner`,
`reattach_existing_runner`, `resume_dead_runner`,
`bootstrap_incomplete`) without the `same_run_retry` trigger.
- Both `automatic_retry_used` refusals (a person's Retry that the hook
requested, and a system retry of owed work) now continue when the run
has restart evidence. A deploy interruption spends no budget, so a task
that only deploys interrupt never runs out. Each continuation needs a
new deploy during its run, so deploys bound it. A loss without restart
evidence keeps the one-time rule. All other gates are unchanged: proven
Runner exit, unknown-turn record, the issue still assigned and open,
idempotency per run, and the Retry's admission gates.
- `doc/execution-semantics.md` describes the rule.

## Verification

- Red before the fix, green after:
`server/src/services/runner-lost-mid-turn-continuation.test.ts` › "an
automatic continuation that the next deploy interrupted". The test seeds
the staging sequence: a lost run, the hook's Retry run with a
`turn.submitted` turn, the shutdown and restart transitions with staging
payloads, then a same-run reattachment that proves the Runner gone. On
the old code the two continuation tests return `automatic_retry_used`.
With the fix they request one fresh-session Retry, record the unknown
turn outcome once, recover the saved work and queue the successor with
`forceFreshSession: true`.
- A chain of 12 deploy continuations, each with a system retry that
inherits its Retry receipt, still continues (red on the first revision
of this PR, which had a budget of 8). Losses without deploy evidence
stay refused (with and without earlier deploy continuations).
- `npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts` 23/23 pass.
`governed-wait-runner-loss.test.ts` and
`workspace-recovery-retry-copyback.test.ts` 103/103 pass.
- `pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify` exits 2 (a global doc input changed), so required CI
supplies the broad coverage.

## Risks

- Low to medium. A task whose Runner is lost on every deploy now
continues after each deploy instead of once. Each continuation is a full
run, so it costs tokens. The deploy count bounds it, because each
continuation needs a new restart during its run.
- Not fixed here: why the reattached Runner is gone after a deploy. In
the sampled runs the reattachment after the restart hung in `native
session provider recovery timed out after 900000ms`, and the next
attempt found the process gone with empty stderr. That needs its own
investigation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, browser read-only API GETs on staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:41:32 -06:00
Nicky LeachandClaude Fable 5.2 03f9b83e52 feat(ci): report the Runner image on master and build the app image's Runner path on pull requests (#21307)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work, and `master` now
publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`
> - The application image will soon copy that Runner instead of
compiling its own. Part 1 of this stack adds the resolver that picks the
Runner for a build
> - Two gaps make that next change risky. The resolver has never run in
CI on `master`, so its lookup time and how often a commit has a
published Runner are unknown. And `docker.yml` has no pull-request
trigger, so a change to how the application image gets its Runner first
runs after it merged
> - This pull request adds a report-only job that runs the resolver on
every `master` push, and a pull-request check that builds the Runner
path of the application image
> - The benefit is real data from `master` before any build depends on
the resolver, and a failed check on the pull request instead of a failed
image build on `master`

## Linked Issues or Issue Description

Refs #21204, #21229 and #21230 (the Runner image). No issue exists for
this change.

**Problem or motivation**

The resolver decides whether a build copies the published Runner. On
`master`, a commit that changes the Runner's inputs is ahead of its
publication for some minutes, and nobody knows yet how often an
application build will meet that. The application image itself is built
only on `master` and on tags. The change that removes the Runner's
compile stages from that image will edit the `Dockerfile`, and today
nothing builds those stages on a pull request.

**Proposed solution**

Add one job to `docker.yml` that only reports, and one new workflow that
only checks. Neither changes what an image contains, and no existing job
reads either of them.

**Alternatives considered**

- Run the resolver inside the build legs. A failure there could fail the
image build. A job of its own, with `continue-on-error`, cannot.
- Give the lookup the job token. The Runner package is public, so an
anonymous lookup works and the job needs no credential.
- Build `production` in the pull-request check. It adds three `COPY`
instructions and a readability check to the two stages that this check
builds, at the cost of the tool layer and two more copies of several
gigabytes.
- Trigger the check on every Runner source change. `runner-image.yml`
already builds the Runner on those pull requests, and CI builds the
server and the UI.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 2 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path (this pull request)
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build

## What Changed

- `.github/workflows/docker.yml`: new job `runner_image_report`.
  - It runs only for a `master` push in this repository.
- It runs `runner-image-resolve.mjs inputs`, then `resolve`, and the
result goes to the job summary with the line "Report only: no build read
this result."
- It has `continue-on-error: true`, declares no `outputs`, and no job
needs it. Its failure does not fail the run.
- It holds no credential. Its only permission is `contents: read`, the
checkout does not keep its token, and the lookup is anonymous. It does
not log in to a registry and writes to none.
- Time limits: 8 minutes for the job, 5 for the step that builds, 2 for
the lookup.
- It is the first job in the file. Existing tests cut the other jobs out
of the file by name, up to the next job or to the end of the file, so no
other place leaves those tests unchanged.
- `.github/workflows/docker-app-check.yml` (new): on a pull request,
builds the `build` and `runner-provider-pack` targets with
`PAPERCLIP_BUILD_COMMIT` set, for `linux/amd64` only.
- `permissions: {}` at the top, `contents: read` for the job, no
credential, no `pull_request_target`.
- It exports nothing (`--output type=cacheonly`) and pushes nothing. It
reads the public build cache that `docker.yml` writes and never writes
it.
- It refreshes the lockfile in the build context first, as `docker.yml`
does.
  - Time limits: 60 minutes for the job, 45 for the two builds.
  - Trigger paths, and why:

    | Path | Reason |
    | --- | --- |
    | `.github/workflows/docker-app-check.yml` | the check itself |
| `Dockerfile`, `.dockerignore` | the recipe, and the files it receives
|
| `server/scripts/prepare-runner-vendor.mjs` | the step of `build` that
reuses or rebuilds the Runner |
| `packages/paperclip-runner/scripts/build-inputs.mjs` | the build stamp
that step compares with the files the image build received |

- `server/src/__tests__/docker-runner-workflows.test.ts` (new, 90
lines): only security properties. The report job has no credential and
one read permission, nothing can read it, and its two steps have time
limits. The check has no credential, cannot publish, and exports
nothing. In both, every action is pinned by commit and no shell script
holds an expression.
- `doc/runner-image-contract.md` and `doc/DOCKER.md` name the job and
the check.

## Verification

macOS (Node 26.12.0):

- In `server/`: `docker-runner-workflows`, `runner-image-workflow`,
`docker-build-stamp`, `container-init-reaping`,
`cloud-image-bundled-plugins`, `docker-image-layer-reuse` and
`cloud-image-sentry` tests: 52 tests pass in 7 files.
- `tests/runner-e2e` workflow tests (`workflow-security`,
`private-trust-workflow-security`, `workflow-image-only`,
`paid-runner-routing`): 27 tests pass in 4 files.
- `node --test .github/scripts/tests/*.test.mjs
scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: 962 of 962 pass.
- Both workflow files parse as YAML.
- `docker buildx build --target runner-artifacts-context --cache-from
type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-amd64
--output type=cacheonly .` succeeds, so both flags are accepted and the
cache is readable.

Linux (`node:24.22.3-bookworm`, with `zstd` and `jq` added): the same
three sets pass: 52, 27 and 962 of 962.

The two commands the report job runs were run by hand, with no
credential, on clean exports of the tree. For the inputs `0e6662ad…d1ca`
the answer was `published`, with the two index digests that `master`
published. For a commit whose image was not yet published the answer was
`inline`, `absent`. Part 1 has the output.

With the job at the end of `docker.yml`, two existing tests failed. They
are the reason for its place: `docker-canary-promotion.test.mjs` reads
the last job to the end of the file, and
`release-verify-workflow.test.mjs` allows no BuildKit configuration
after `merge-and-push`.

Never run:

- Either workflow. Only GitHub can run them. This pull request starts
the check by its own path; the report job first runs on `master`.
- The two builds of the check (`--target build`, `--target
runner-provider-pack`) on a local host. Their time and disk use on a
hosted runner are not known.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The check builds the whole `build` stage, with the UI and server
builds. Its duration is not measured. It runs only when one of five
paths changes, and its first real run is on this pull request.
- `docker.yml` holds one concurrency slot for each ref until all its
jobs end. The report job normally ends long before the build legs. A
stalled job can hold the slot for at most 8 minutes when the legs end
sooner.
- `scripts/watch-release-source.mjs` warns about every job of the image
run that waited more than 60 seconds for a runner. The report job is one
more job that it can name.
- `scripts/merge-group-image.mjs` refuses to promote a merge-queue image
for a push that changes `docker.yml`. The push that lands this change is
such a push. The merge queue is not enabled.
- The lookup has 2 minutes. The registry client retries a failing
request for longer than that, so a registry outage ends the step by its
time limit, and the job then reports nothing. The run stays green.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:29:33 -08:00
DottaandClaude Opus 5.5 bcadbde449 feat(runner): record the OpenRouter host that served each OpenCode model call (#21298)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agents can use models through OpenRouter, and the Runner
proxies those model calls
> - OpenRouter routes one model across many hosts, and these hosts use
different quantizations (fp4, mxfp4, fp8)
> - Chat evaluations with identical prompts behave very differently
between runs, but the run log does not record which host served a call
> - The run usage shows `provider: "deepseek"` and `biller:
"openrouter"`, and `providerRequestId` is always `null`
> - This pull request records the serving host and the OpenRouter
generation ID for each model call in the run log
> - The benefit is that we can compare evaluation cohorts by host, and
the cost event names a real OpenRouter generation

## Linked Issues or Issue Description

**Subsystem affected**
Runner OpenCode provider proxy, runnerd usage normalization, and native
run usage accounting.

**Problem or motivation**
OpenRouter's default routing sends a request to one of many hosts. Each
host can use a different quantization. Chat evaluation results change a
lot between runs with identical prompts. Paperclip does not record which
host served a call, so we cannot compare results by host.
`usageJson.providerRequestId` is always `null` for OpenCode runs.

**Proposed solution**
Every OpenRouter completion response has a top-level `provider` field
and an `id` (the generation ID). Each streamed chunk has them too. The
Runner proxy reads them from the first chunk after it forwards the
bytes. The Runner attaches them to the usage of that call. The values go
to the run log, and the run's first generation ID goes to
`usageJson.providerRequestId`.

**Alternatives considered**
A new PRP event type such as `provider.upstream.served` would need a new
event enum value. An older control plane rejects an unknown event type.
A `harness.diagnostic` event shows in the task chat as a diagnostic. The
usage payload already flows to every consumer, and older consumers
ignore extra fields. Thus this change uses the usage payload.

**Additional context**
Run-log change only (AGENTS.md section 7). The data stays in
`heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch
`packages/shared/src/telemetry/` or OpenTelemetry.

## What Changed

- New `createUpstreamServedTap`
(`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`).
It is a pass-through stream. It pushes each chunk downstream first, then
it reads `provider` and `id` from the first SSE `data:` event (or from a
JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of
a JSON body. It skips encoded bodies and unknown content types. Any
parse problem or recorder error stops the scan and leaves the response
unchanged.
- `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The
OpenCode session keeps the most recent served call. Each
`item.completed` usage payload and `session.usage()` now carry
`upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint
does not include these fields, so the number of usage events does not
change.
- The OpenCode app-server proxy adds the two fields to
`thread/tokenUsage/updated`. Rust `usage.reported` now fills its
existing `providerRequestId` field and adds an optional
`upstreamProvider`. The TypeScript runnerd transport keeps both fields
when it rehydrates the usage notification. `usage.schema.json` documents
`upstreamProvider` (the protocol manifest and schema bundle are
regenerated).
- Server native turn accounting keeps the first generation ID the run
reports. The usage checkpoint and the native adapter result carry it as
`providerRequestId`, so `usageJson.providerRequestId` and the final cost
event name it.
- `doc/run-log-events.md` documents the fields in a new "OpenRouter
serving host" section.

## Verification

- Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's
stream through unchanged and records the host that served it on the
turn's usage". The test sends a streamed fixture with a comment line, a
host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks
that the bytes are identical and that the turn's usage names `Novita`
and the generation ID. The test fails without the proxy wiring.
- `upstream-served.test.ts`: streamed fixture, chunk split inside a
multibyte character, non-streamed JSON, malformed chunks, a truncated
JSON body, an encoded body, the 64 KiB limit, and a recorder that
throws.
- `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts`
(rehydration), Rust
`provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`,
and server `native-usage-accounting.test.ts` (first generation ID stays
stable across turns).
- Local: OpenCode driver, OpenCode proxy, and runnerd transport suites
(31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes
(one timing test failed once under load and passed alone). Runner and
server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks
pass.
- Latency: a local benchmark proxied a 400-chunk SSE stream 200 times
with the tap and 200 times without it. The median time to first byte was
292 µs with the tap and 259 µs without it (+33 µs, which is the extra
stream stage). The total stream time changed from 491.4 ms to 492.5 ms.
The tap spends about 5.5 µs of CPU to inspect the first chunk, after it
has forwarded that chunk. OpenRouter time to first token is hundreds of
milliseconds.
- Paid evidence: one single-case run of
`chat-features.runner-opencode.local.skills.fork-read-only` on this
branch. The result is in a comment on this pull request.

## Risks

- Low risk. All new fields are optional and additive. An older control
plane ignores the extra usage fields. `usage.schema.json` is not bound
by the event schema, and no validator enforces it on `usage.reported`.
- Saved state from the previous release: this change adds no persisted
run, session, checkpoint, or launch state. A saved session resumes
unchanged. A run that is in flight during a deploy reports usage without
the new fields until its Runner is new. `usageJson.providerRequestId`
then stays `null`, as before.
- `providerRequestId` on the run and its cost event is the run's first
generation ID, not each call's ID. The value is stable, so a late usage
receipt still matches the posted charge. Per-call IDs and hosts are in
the run log.
- If OpenCode starts a new model call before it reports the usage of the
previous call, a usage event can name the newer call. OpenCode reports a
message's usage when its step ends, so this is not expected.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code as an agent with tool use (shell, file edits, test runs). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:15:53 -06:00
DottaandClaude Opus 5.5 c730872974 fix(recovery): a chat turn waiting on the person is not stranded work (#21302)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In a chat, the agent's turn ends and the conversation waits for the
person's next message. The stranded-work sweep must leave that wait
alone
> - The sweep recognizes a finished chat turn by
`finalizationReasonCode: "conversation_turn_finished"` on the run record
> - Run completion re-reads the run through the 64 KiB read projection.
When the stored result is larger, that projection drops the finalization
fields, and completion wrote the reduced copy back
> - The chat turn then lost its finalization reason. Its reply was not
presented, and the sweep queued
`issue.productive_terminal_continuation_recovery` 30 s later. That is an
extra run while the agent waits on a person
> - This pull request keeps the full stored result through run
completion, and makes the sweep read the committed decision from its
durable rows
> - The benefit is that a turn waiting on a person is never treated as
stranded work

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `app-permissions.new-quarantined-action`, the
CEO's chat turn yielded with a `response_wake` continuation and waited
for a person to allow an app action. The stored run result was 113,937
bytes. Its saved record had the read projection's `truncated`,
`truncationReason: oversized_result_json` and `originalSizeBytes` fields
and no `finalizationReasonCode` or `decisionId`. The run's presentation
decision was `interaction_owns_conversation_response`, so no reply
comment was written. 30 s later the sweep queued
`issue.productive_terminal_continuation_recovery`, and the case failed
its run limit.

**Steps to reproduce**
1. Finish a native chat turn that yields `response_wake`, so
finalization records `conversation_turn_finished`.
2. Let the stored run result be larger than 64 KiB when run completion
reads it.
3. Run completion writes back the reduced copy without
`finalizationReasonCode`.
4. Run the stranded-work sweep. It queues a productive continuation run.

**Expected behavior**
The run record keeps its finalization fields, and the sweep leaves the
waiting chat turn alone. No new run starts until the person replies.

## What Changed

- `server/src/services/heartbeat/run-completion.ts`: run completion
reads the run with `unsafeFullResultJson` before it merges the native
result. It no longer writes the read projection's reduced copy back.
- `server/src/services/native-runtime/native-response-wake-wait.ts`:
`hasCommittedNativeConversationTurnWait` checks the run's committed
finalization decision (`conversation_turn_finished`) through
`native_run_finalizations` and `status_decisions` for the conversation's
own agent.
- `server/src/services/recovery/service.ts`: the sweep's
finished-chat-turn skip also uses that durable check, so runs already
saved with the reduced copy are recognized.
- Tests: `run-completion.test.ts` keeps the finalization record when the
stored result is larger than the read projection.
`heartbeat-process-recovery.test.ts` finalizes a real chat turn with the
eval's yielded result, saves the reduced run record, and expects two
sweeps to start no run.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts -t "larger than the read
projection"` — fails on master (the saved result is the truncated copy)
and passes here.
- `cd server && npx vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "chat turn that
yielded"` — fails on master (`continuationRequeued: 1`) and passes here.
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts
src/__tests__/heartbeat-process-recovery.test.ts` (614 passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- Run completion now reads the full stored result for one run. This is
one row; the read projection still serves every API read.
- The sweep check adds one indexed query for chat issues whose latest
run succeeded and whose run record lacks the reason.
- Previous-release saved state: no persisted shape changes. Runs already
saved with the reduced copy are covered by the durable decision check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:11:30 -06:00
Nicky LeachandClaude Fable 5.2 f79e2c5d84 feat(ci): add the Runner image resolver and a no-rebuild mode for the Runner vendor step (#21306)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - `master` now publishes the Runner alone as the image
`ghcr.io/paperclipai/paperclip-runner`, one time for each set of Runner
inputs (#21204, #21229, #21230)
> - The application image still compiles its own Runner on every build.
The next steps make it copy the published Runner instead
> - Before a build can do that, it must answer one question the same way
as every other build: "does a published Runner exist for exactly this
tree?" It must also fail clearly when the Runner it was given does not
match the tree
> - This pull request adds the script that answers the question, and a
mode of the Runner vendor step that fails instead of rebuilding
> - The benefit is that the later change to the image build is small,
and that a wrong or unreachable answer can only mean "build the Runner
yourself"

## Linked Issues or Issue Description

Refs #21204, #21229 and #21230 (the Runner image). No issue exists for
this change.

**Problem or motivation**

Three builders make images that contain the Runner: `docker.yml`, the
merge-queue build and the release preview build. Each one would have to
repeat three steps to find the published Runner of its tree: check the
lockfile, compute the inputs digest in the build, and look up
`inputs-<digest>`. Three copies can decide differently. A build that
copies a Runner built elsewhere also has no Rust toolchain. If the
copied Runner does not match the tree, `prepare-runner-vendor.mjs`
starts a rebuild, and the build fails late at `cargo` with a message
that hides the cause.

**Proposed solution**

Add `scripts/runner-image-resolve.mjs`, the one place that decides. Add
`PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` to
`prepare-runner-vendor.mjs`. Nothing calls the resolver and nothing sets
the variable in this pull request, so no workflow job changes.

**Alternatives considered**

- One command that builds and looks up. Two commands keep a registry
credential, when a caller has one, out of the step that builds.
- A wait for the publication of the commit. The decision whether the
application build waits is open, so the resolver has no wait mode. It
reports `absent` and stops.
- A build argument that names the expected inputs. No `Dockerfile`
declares one yet. The change that makes the application image copy the
Runner will add both ends together.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 1 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode (this pull request)
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build

## What Changed

- `scripts/runner-image-resolve.mjs` (new, 185 lines), with two commands
that share a directory:
- `inputs --dir <dir>` exports the `runner-artifacts-context` target,
runs the lockfile check on that export, and builds the
`runner-artifacts-identity` target when the lockfile is current. It
reads no credential. A lockfile that is behind stops there: no image is
published for such a tree.
- `resolve --dir <dir>` looks up `inputs-<digest>` and prints `mode`,
`reason`, `inputs_digest`, `image`, `slim_image`, `build_contexts` and
`labels`. It also writes them to `$GITHUB_OUTPUT` and a short report to
`$GITHUB_STEP_SUMMARY`.
- `mode` is `published` only for a complete publication whose full index
names these inputs and both platforms. Both images are pinned by index
digest. The slim digest is the one the full index names in
`io.paperclip.runner.slim-digest`; the slim tag is never read on its
own.
- Every other answer is `inline`, with the last four values empty:

  | `reason` | Meaning |
  | --- | --- |
  | `lock-behind` | `pnpm-lock.yaml` is behind the Runner's manifests |
  | `absent` | no tag exists |
| `incomplete`, `broken` | a publication stopped, or its two tags do not
belong together |
| `mismatch` | the full index names other inputs, or lacks a platform |
| `unknown` | the registry refused, failed or could not be reached, or
the lookup threw |

- The resolver reuses `runnerLockState`, `readRunnerImageManifest`,
`runnerPublicationState` and `runnerImageTags` from
`scripts/runner-image-publish.mjs`. That file now exports
`environmentCredentials`; it has no other change.
- `server/scripts/prepare-runner-vendor.mjs`: with
`PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1`, the step reuses a current
build as before. When the build is not current, it prints the reason,
starts no rebuild, leaves the stamp in place and exits with status 1. It
also fails when `PAPERCLIP_RUNNER_VENDOR_REBUILD=1` is set at the same
time. Without the variable the step is the same as before.
- `doc/runner-image-contract.md` has a new section, "Choosing the Runner
for an image build". `doc/DEVELOPING.md` describes the new variable.
- The resolver is not an input of the Runner image, so the inputs digest
does not change.

## Verification

macOS (Node 26.12.0), in `server/`:

- `node node_modules/vitest/vitest.mjs run
scripts/prepare-runner-vendor.test.mjs
src/__tests__/runner-image-resolve.test.ts
src/__tests__/runner-image-publish.test.ts
src/__tests__/runner-image-identity.test.ts
src/__tests__/runner-image-workflow.test.ts
src/__tests__/server-package-build-script.test.ts`: 179 tests pass in 6
files. 27 of them are new resolver tests and 3 are new tests of the
vendor step, whose end-to-end test also has a new case.
- `tsc` on the new test file, with the settings of
`server/src/__tests__/tsconfig.json`: no error.

Linux (`node:24.22.3-bookworm`, GNU tar 1.34, the worktree mounted
read-only): the same files, 179 tests pass.

Real runs, on clean exports of the tree (`git archive`), with no
credential, on a `linux/arm64` Docker host:

- On `master` at `62d673cc3f`: `inputs` gave `lock=current` and
`inputs_digest=0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`.
`resolve` gave `mode=published`,
`image=ghcr.io/paperclipai/paperclip-runner@sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162`
and
`slim_image=ghcr.io/paperclipai/paperclip-runner@sha256:f63ea43666ed708f1b0e5d2ade8bdc002885bf04252b9ed6cee21f7920e54f3c`.
These are the two indexes that `master` published for those inputs.
- The same tree with one line added to a Runner source file: another
digest, `mode=inline`, `reason=absent`.
- On `master` at `bc71c23e0a`, which changed the Runner's inputs, before
its image was published: digest
`88fe2929321b4706c77f11a098572ce990fec4449d505544c62ef3a84c28308a`,
`mode=inline`, `reason=absent`.
- Four deliberate faults in the resolver each make a test fail: no index
check, only `absent` is inline, the identity is built for a lockfile
that is behind, and the slim image is taken by tag.

Never run:

- A lookup with a registry credential. The package is public, so no
caller needs one now.
- Any workflow. No workflow calls the resolver in this pull request.
- `linux/amd64`.
- The full test suite, the full typecheck and `pnpm build`. `pnpm verify
--explain` reports exit 2, "broader verification required", so CI must
supply them.

## Risks

- Low for existing behaviour: nothing calls the resolver, and the vendor
step is the same unless the new variable is set.
- The resolver does not verify the attestation of the image. The
workflow that consumes the image must do that from the digest the
resolver reports.
- The resolver reports both build contexts, `runner-artifacts` and
`runner-artifacts-slim`. A build that reads only one stage gets one
context it does not use.
- A `broken` publication is `inline`, not a failure. The Runner image
workflow fails for that state.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:03:15 -08:00
Nicky LeachandPaperclip 731b52453c fix(environments): verify workspaces on run leases and release settled tasks (#21289)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core owns run leases and delegates remote execution to environment
plugins.
> - A retry must verify its old checkout on the placement that will run
the task.
> - A completed task must still release its provider resources.
> - This change uses the existing lease for workspace verification and
always calls the release hook after settlement.
> - Saved settlement proof lets cleanup retry without repeating
execution work.

## Linked Issues or Issue Description

Refs #21237. That PR records stop proof before lease release. This PR
keeps provider cleanup after settlement and changes workspace
verification sequencing. Both touch task completion; preserve both
changes if the merge order requires conflict resolution.

**What happened?**

Workspace verification ran before a portable task had a lease, so a
provider could not verify the checkout on its selected placement. Core
also skipped the provider release hook after successful task completion.
This could leave resources allocated after the run ended.

**Expected behavior**

Verify the old checkout through the selected run lease. A confirmed
refusal can use the authorized fresh-workspace fallback. A transport
failure must remain retryable. Call the provider release hook after
settlement and retry failed cleanup without changing the run result.

**Steps to reproduce**

1. Retry a task whose prior checkout uses plugin-managed storage and
portable task placement.
2. Verify that the plugin receives the same task lease for verification
and preparation.
3. Complete a task, fail the provider release once, and reconstruct the
cleanup controller.
4. Confirm that cleanup calls only the release hook and retains the
completed run outcome.

## What Changed

- Acquire the logical run lease before provider workspace verification
and reuse it for preparation and execution.
- Preserve the existing session and workspace choices. Recompute
workspace-dependent inputs only after a definite refusal; acquire
credentials and managed runtime inputs once.
- Save the exact refusal and lease before cleanup. Use a
workspace-scoped creation key only for the resulting fresh attempt.
- Replay interrupted verification on the same unlaunched lease. Keep
transport errors retryable and reject replay after execution starts.
- Always call the provider release hook after task settlement. Pass
refreshed settlement proof and route failed releases through the
existing cleanup reaper.
- Add database regressions for call order, restart, retained writers,
refusal, transport errors, and settlement-only cleanup replay.

`plugin-workspaces.ts` grows from 997 to 1,001 lines. The four added
lines extend the existing workspace transaction and lease callbacks.
Keeping the checks there preserves the ownership assertion, provider
result validation, and binding update in one operation. Extracting only
the verification branch would split that transaction without removing a
separate responsibility.

## Verification

- Server TypeScript check passed: `tsc -p server/tsconfig.json
--noEmit`. The complete `server/scripts/typecheck-layers.mjs` check also
passed, including all test-tree layers.
- The first CI typecheck caught a readonly array in a test fixture that
the narrower source check did not include. Commit 8d9ec19e7a corrects
that fixture; all layered checks pass afterward.
- Added a public heartbeat refusal regression in 7c598a4da3. All ten
session-preparation tests pass; independent execution of the refusal
case also passes. It checks successful completion, the saved fresh
workspace, a new conversation, and one credential acquisition.
- Nine database/service suites passed 335 tests: plugin warm tasks,
plugin runner tasks, startup cleanup recovery, credential writeback,
plugin workspaces, session preparation, plugin environments, retry
workspace continuity, and heartbeat workspace preparation.
- After the final restart change, both targeted refusal/restart tests
passed. The primary agent independently reran those cases and the final
server typecheck. After integrating current master, the primary agent's
independent rerun passed all 237 tests across the warm-task,
plugin-workspace, session-preparation, and heartbeat-workspace suites.
- `pnpm verify` returned exit 2 and requested broad CI coverage because
the plugin execution contract documentation changed. No full local
Core-suite pass is claimed.
- Previous-revision CI was green on 8d9ec19e7a: 59 checks passed and 3
were skipped. The first run failed the unchanged legacy-continuation
browser assertion and credential-cleanup server shard. Each passed on
one targeted retry with no code changes. The credential-cleanup
integration suite also passed all 21 tests locally on Node 22. The
browser failure concerned an existing continuation metadata marker; the
server failure cascaded after a cleanup error left a test spy installed.
The successful retries do not establish the cause of the first failures.
- Local database fixtures use simulated provider workers. No live
provider or deployed environment was changed.

- Additional verification for 59c7a1dae8: the layered server typecheck
passed. Independent execution passed all 265 tests across durable task
retries, workspace preparation, and public heartbeat preparation. The
new shutdown regression preserves the fresh Retry choice, waits for
parent cleanup, and prevents the successor from inheriting the parent
lease reference. Same-run recovery retains its receipt. Required CI is
green on 59c7a1dae8: 58 checks passed and 3 were skipped. Standard
review scored 5/5 with no unresolved comments. The separate Superagent
Security Scan is neutral/inconclusive because its workflow failed; a
direct rerun request returned HTTP 404. That scan is not claimed as
passed.

## Risks

- Providers must use the supplied task lease for verification and return
a definite `binding_mismatch` only when the checkout is unconfirmed.
Transport errors retain the retry decision.
- A refused checkout waits for confirmed release before a fresh
allocation. Failed deletion remains owned by the existing cleanup
reaper.
- Task settlement and physical resource release stay separate. Retained
handoff leases do not enter this release path.
- No migration or SDK schema change is required.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) implemented the changes and tests.
The primary Codex agent independently reviewed the connected Core and
plugin paths and ran checks. A separate GPT-6.1 Sol agent reviewed
lifecycle and restart boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue using the bug template labels
- [x] I have not referenced internal or private project details
- [x] My branch name describes the change
- [x] I have added relevant tests and documentation
- [x] I have considered and documented risks
- [x] All required CI gates are green
- [x] Greptile is 5/5 with no unresolved comments
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 21:56:02 -08:00
DottaandClaude Opus 5.5 61ab829d5d fix(queue): another agent's comment no longer hides a pending approval (#21303)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person approves a new plan while the agent is busy, the
approval waits in the task's message queue. Its Interrupt control stops
the current attempt and starts a fresh one with the approved plan
> - The task view shows one queue at a time. A saved card response (an
approval) gives way to any later queued message, so it never hides the
person's own later messages
> - Another agent's comment (here, the CEO's direction with `resume:
true`) is also a queued message. It took over the view, and the approval
and its Interrupt control disappeared
> - The person could then not interrupt for the plan they approved
> - This pull request lets only the person's own later messages hide a
pending approval. A queue that holds only agent comments no longer hides
it
> - The benefit is that the Interrupt control for an approved plan stays
available until the person uses it or writes a newer message

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `steering.interrupt-with-direction`, the person
approved a saved simpler plan while Morgan was busy. The approval queued
with `requiresFreshSession`, and its Interrupt control showed. The CEO
then posted its own direction comment on the task at 04:03:21. That
comment queued as a separate deferred wake. `GET /queued-comments` then
returned only the CEO comment's queue. The approval's Interrupt control
(`task-chat-queued-interrupt-<approval id>`) never rendered, the click
timed out, and no interrupt request was sent. The approval was not lost:
its deferred wake was still saved. Only the view and its control were
hidden.

**Steps to reproduce**
1. Approve a plan on a busy task, so the approval waits for a fresh
session.
2. Let another agent comment on the task, so its comment queues after
the approval.
3. Read `GET /api/issues/:id/queued-comments`. The approval is gone from
the queue.

**Expected behavior**
The approval stays the visible queue with its Interrupt control. A newer
message from the person still takes the view, as before.

## What Changed

- `server/src/routes/issues.ts`: `findQueuedCommentWake` returns a later
message queue only when it holds a message from a person. A queue with
only agent comments yields to a pending saved response. With no saved
response it is still returned, as before.
- `server/src/__tests__/issue-queued-comments-routes.test.ts`: the eval
ordering (a fresh-session approval, then another agent's comment) keeps
the approval as the queue, and a later person message in that queue
still takes the view.

## Verification

- `cd server && npx vitest run
src/__tests__/issue-queued-comments-routes.test.ts -t "another agent's
comment queues after it"` — fails on master (the CEO comment's queue is
returned) and passes here.
- `cd server && npx vitest run
src/__tests__/issue-queued-comments-routes.test.ts` (all tests passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- While a saved response waits, queued agent-only comments are not shown
in the composer queue. They stay saved and are delivered by the normal
deferred-wake path, as they were before the person approved anything.
The person's own messages are never hidden.
- One indexed comment lookup per queued message wake during the queue
read.
- No persisted shape or API contract change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:53:54 -06:00
DottaandClaude Opus 5.5 99d9e608f4 fix(heartbeat): hold the new owner's run while the reassigned owner's run settles (#21301)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person reassigns a task, the old owner's native run is
cancelled and the new owner's run is queued
> - The old run's executor keeps closing its provider session for
several seconds after the run is marked cancelled
> - #21031 holds a queued run while the task's previous session close is
pending. That hold reads the runner's cleanup gate, which exists only
after the close begins
> - The stale-lock sweep cleared the task lock as soon as the old run
was terminal. The new owner's run was then claimed before the close
began, met the close as it escalated, and failed with
`native_session_cleanup_quarantined`
> - This pull request keeps the new owner's run queued while the old
owner's stopped run still settles in this process, and wakes it when
that executor finishes
> - The benefit is that a reassignment never fails the new owner's run
and needs no Retry

## Linked Issues or Issue Description

Refs #21031

**What happened?**
In the paid chat eval `tasks.edit-properties`, the old owner's run
`6cf2ee38` was cancelled for reassignment at 04:03:47.611. Its process
stopped only at 04:03:57. The stale-lock sweep cleared the task lock at
04:03:47. The new owner's run was claimed at 04:03:48.626. The runner
admission waited for the old close, which escalated to operator-required
at 04:03:56, and the new run failed with
`native_session_cleanup_quarantined`.

Why #21031 did not apply: its hold reads the runner's in-memory cleanup
gate. At claim time the old executor had not started its session close,
so no gate existed. The existing `predecessor_settling_in_process` hold
covers only the same agent, and a reassignment changes the agent. #21260
does not touch this path: its new checkpoint runs only before a turn in
a recovered session, and both sessions here were new.

**Steps to reproduce**
1. Run a native task for agent A.
2. Reassign it to agent B. A's run is cancelled, but its executor is
still settling.
3. The stale-lock sweep clears the task lock.
4. B's queued run is claimed before A's session close begins.

**Expected behavior**
B's run stays queued until A's executor has finished. It starts by
itself after that, then #21031's gate covers any close that is still
settling.

## What Changed

- `server/src/services/heartbeat/queue.ts`: the
`predecessor_settling_in_process` hold in `claimQueuedRun` also holds
for another agent's native run on the same task when that run is
terminal but its executor is still active in this process. A live run of
another agent (for example a reviewer) does not hold it.
- `server/src/services/heartbeat/queue.ts`:
`notifyOtherAgentsQueuedOnSettledIssue` sends a queued-run dispatch
signal when a native executor settles and another agent has a queued run
on the same task. It does nothing when no run waits.
- `server/src/services/heartbeat.ts`: the executor calls it after its
cleanup, next to the existing same-agent `startNextQueuedRunForAgent`.
- `server/src/services/heartbeat/queue.test.ts`: the exact ordering
(cancelled old-owner run still in the executor set, lock cleared, new
owner's run queued), a control for a live run of another agent, and the
dispatch wake.

## Verification

- `cd server && npx vitest run src/services/heartbeat/queue.test.ts -t
"reassigned owner|another agent's live run|next owner once"` — the hold
and the wake tests fail on master and pass here.
- `cd server && npx vitest run src/services/heartbeat/queue.test.ts
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/__tests__/heartbeat-workspace-busy.test.ts` (all passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- A queued run on a task waits while another agent's terminal native run
on that task is still settling in this process. That wait ends when the
executor finishes, and the new wake re-drives dispatch. The queued-run
liveness redrive still covers a lost signal.
- The wake is event-driven: one indexed query when a native executor
settles, and one notification only when another agent waits on that
task. It adds no polling.
- Previous-release saved state: no persisted shape changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:53:49 -06:00
DottaandClaude Opus 5.5 f82ec65800 fix(runs): a person's Stop wins over lease-release teardown (#21300)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can stop an agent run with Stop (cancel). The run must then
end as cancelled
> - A run that a person stops just as it starts tears down its
environment lease at the same moment
> - The lease-release teardown wrote `interrupted`
(`lease_released_before_terminal`). It recognized only an acknowledged
Stop, and it decided from a run snapshot that it read before the Stop
request landed
> - An interrupted run looks like an infrastructure failure, and it is a
recovery candidate. That is against the person's decision
> - This pull request makes a recorded person Stop win over the teardown
in both orderings
> - The benefit is that a person's Stop always ends the run as cancelled

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `steering.stop-versus-pause`, the fixture
cancelled a resumed run as it reached native running. The Stop request
was recorded at 04:02:11.049. The teardown wrote `interrupted` with
`lease_released_before_terminal` at 04:02:11.083. The native
cancellation intent was still `pending`, so the teardown did not see it
as a Stop. The eval needs `cancelled`.

**Steps to reproduce**
1. Start a native run.
2. Record a person Stop request on the run (the cancel route does this
first).
3. Let the lease-release teardown run before the cancel finalizer writes
`cancelled`.
4. The run ends `interrupted`.

**Expected behavior**
The run ends `cancelled`. This is true when the Stop is recorded before
the teardown reads the run, and when it is recorded between that read
and the write.

## What Changed

- `server/src/services/acknowledged-native-stop.ts`:
`hasRecordedUserStopRequest` and `recordedUserStopRequestSql` recognize
a person Stop that the cancel route recorded (`cancelledByActorType:
"user"`, a user id, and a requested or acknowledged execution
cancellation or a startup cancellation fence).
- `server/src/services/heartbeat/run-lifecycle.ts`:
`terminalizeRunOnLeaseRelease` re-reads the run before it decides. A
recorded person Stop now gives `cancelled`. The `interrupted` write is
fenced on "no person Stop recorded", so a Stop that lands between the
read and the write makes the write fail. The function then reads again
and writes `cancelled`.
-
`server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts`:
two orderings (Stop recorded before the teardown read; Stop recorded
after the read, with the stale snapshot) and a control (an agent-only
request still reports `interrupted`).

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-lease-release-terminalization.test.ts` —
both new ordering tests fail on master (`interrupted`,
`lease_released_before_terminal`) and pass here.
- `cd server && npx vitest run
src/__tests__/heartbeat-run-lease-release-terminalization.test.ts
src/__tests__/heartbeat-run-terminalize-before-release.test.ts
src/services/heartbeat/run-lifecycle.test.ts` (67 passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- Low. Only the lease-release teardown changes, and only when a person
Stop is recorded on the run. The cancel path already wrote `cancelled`
for the same run when it won the race.
- The teardown now makes one more read of the run row. This is one
indexed primary-key read per teardown.
- Previous-release saved state: no persisted shape changes. Runs saved
by the previous release carry the same Stop fields.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:52:13 -06:00
Nicky LeachandClaude Opus 5.5 e65a3095a7 fix(runner): quiesce the Core-dialed route once the Runner commits its suspend (#21245)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Core runs agent sessions through the Paperclip Runner. With provider
ingress, Core dials the Runner over a PRP WebSocket.
> - When Core closes a session, it asks the Runner to suspend. The
Runner suspends, reports the result, and exits.
> - Core's ingress connection did not know that this exit was expected.
It treated the drop as a transport loss and redialed the Runner. Each
redial got a 404 until the close released the route.
> - The release (park) path already quiesces the connection before the
Runner goes away. The suspend path did not.
> - This pull request makes the suspend close quiesce the route in the
same way, once the Runner commits its suspend.
> - The benefit is that a clean suspend no longer looks like a lost
Runner. A drop before the suspend is committed still redials and
recovers as before.

## Linked Issues or Issue Description

**What happened?**
After a provider-ingress session closed cleanly, Core redialed the
Runner's ingress endpoint several times and got 404 each time. The
Runner state was `suspended`, `runner.drain` and `runner.suspend` had
both completed, and the Runner had exited with code 0. The redials
followed the outbound reconnect backoff (about 250, 500, 1000 and 2000
ms) and stopped when the registration release closed the outbound.

**Expected behavior**
After Core commits to `runner.suspend` and the Runner reports it
completed, the Runner's exit is expected. Core must not redial or
classify the drop as a transport loss. The release path already works
this way: it calls `outbound.quiesce()` before it releases the task.

**Steps to reproduce**
1. Open a session through a Core-dialed route (plugin provider ingress
or the `listen_ws` driver).
2. Close the session. Core drains, queues `runner.suspend`, and the
Runner suspends and exits.
3. See that the outbound reconnect loop redials the gone Runner until
the registration release closes it.

**Paperclip version or commit**
`master` at b4ca19f3a8.

**Agent adapter(s) involved**
Not adapter-specific (core bug). It affects any Core-dialed Runner
route: plugin provider ingress and the `listen_ws` driver.

**Additional context**
The redials do not fail the session. The plugin outbound uses a 7-minute
recovery grace, and `#failTransport` ignores failures after
`#closeAndSettle` sets `#closed`. The new test confirms this: without
the fix, the close still settles, but the route redials. This PR removes
the misleading redial. It does not change how a run's own failure is
classified.

## What Changed

- `RunnerExecutionDriver.controlPlaneRegistration` results can now
include an optional `quiesce()`. Core calls it when it has committed the
Runner's exit. The route then stops liveness and reconnect, and it keeps
the established receive/ACK path.
- `DurablePrpControlPlane.onRunnerExitCommitted(listener)` is new. The
authority calls its listeners after it saves a completed
`runner.suspend` or `runner.shutdown` result and before it sends the
acknowledgement. The Runner exits when it gets that acknowledgement, so
a route quiesced here never sees the exit as a loss.
- `awaitRunnerSuspensionBarrier` calls a new `onExitCommitted` callback
once, when its suspend command is `completed`. It also calls the
callback when it reuses a completed receipt and the durable state is
still `suspended`. A rejected, failed or unanswered suspend never calls
it.
- `DurablePrpCodexTransport` registers each control-plane route through
one `#registerControlPlane` helper. The helper records each route's
`quiesce` until that route is released. The suspend close subscribes to
`onRunnerExitCommitted` while its barrier runs and quiesces all of these
routes from that notification. The barrier's own `onExitCommitted`
covers a completed receipt that was acknowledged before this close.
Quiescing clears the tracked routes, so it runs once per close. The
helper copies property descriptors, so getters such as the `listen_ws`
driver's lazy `failure` stay live.
- The plugin task registration (`plugin-runner-task.ts`) and the
`listen_ws` driver registration (`native-runner-execution-driver.ts`)
implement `quiesce` with `outbound?.quiesce()`. This is the same call
that the release path makes.

## Verification

- New transport test with a real runnerd:
`packages/paperclip-runner/src/live/runnerd-codex-transport-suspend-route.test.ts`.
The test route acts like a Core-dialed provider ingress: on a socket
drop that it was not told to expect, it records a redial into a 404.
Before the fix the events were `connected, redial: 404, release`. After
the fix they are `connected, quiesce after runner.suspend completed,
release`. The close settles, and the durable Runner state is
`suspended`. A second case delays the durable-state read by 1.5 s, so
the Runner exits while the barrier still reads. That case fails without
the authority hook and passes with it.
- New barrier unit test in `runnerd-codex-transport.test.ts`. It shows
that the exit is committed only after the retried suspend completes,
never on a rejected or unanswered suspend, and on a reused receipt only
when the state is `suspended`.
- New server tests for the plugin task and `listen_ws` registrations.
`quiesce` reaches the active outbound and does not close it. Release
still closes it.
- Existing tests pass. These include the outbound release-path quiesce
test, the half-open and lease-expiry failure tests (a drop before the
commit still fails after its grace), and the plugin release/resume
facade tests.
- `pnpm verify` exited 2 (the change touches files outside the resolved
test graphs), so I ran focused suites. Locally: 8 runner transport files
and the 4 control-plane test files, 667 tests
(`runnerd-codex-transport*.test.ts` for suspension, release/resume,
remote retention, settlement, warm attachment and attachment recovery,
plus the new suspend-route test), and 5 server files, 148 tests (both
plugin Runner integration tests, the plugin task, the execution driver
and the outbound). Required CI supplies the broad coverage.
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Low risk. `quiesce` is optional. Registrations that do not supply it
behave as before.
- The quiesce happens only after the Runner reports a completed suspend.
A socket drop before that point still redials and recovers.
- The quiesced route keeps its receive/ACK path, so the Runner's last
frames before exit are still acknowledged.
- No persisted state or protocol change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:19:24 -08:00
DottaandClaude Opus 5.5 62d673cc3f fix(retry): start a held Retry once a stopped preparation is proven stopped (#21295)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails, a person can press Retry. Admission starts
a new turn from the recorded work only after it proves that the previous
execution stopped.
> - For a legacy runner run that failed while it prepared its sandbox,
the proof reads the run's events and its environment lease.
> - After the automatic retry budget is spent, execution recovery
settles a no-replay hold and appends its own lifecycle event to the run.
The proof accepted only the error and measured events, so this platform
event made the proof fail.
> - The proof also asked for a workspace repair receipt for every run
with a lease, even when the run never recorded a remote workspace and
its sandbox was destroyed. No repair could ever exist, so the proof
could never pass.
> - Admission then refused the Retry without a named reason. The receipt
kept the generic `execution_recovery` wait ("Retry is waiting for
execution checks"), and every later Retry joined the same held receipt.
> - This pull request accepts the recovery hold's own settlement event
when it is bound to a settled hold for the same run, and asks for a
workspace repair only when the run holds prepared files.
> - The benefit is that Retry starts work at once when the previous
preparation is proven stopped, and it continues from the recorded work.

## Linked Issues or Issue Description

**What happened?**

A board user pressed Retry on a blocked task. The Retry was accepted,
but no run started for more than 13 hours. The task said "Retry is
waiting for execution checks. Your message is saved." A second Retry the
next day attached to the same held request and also did nothing. The
execution blocker said `legacy_execution_requires_reconciliation` with
`canRetry: true` and "A person can use Retry on this task ... to
continue once the previous execution has stopped." The previous run was
a legacy `paperclip_runner` run that failed with `setup_failed` while
preparing (`executionRecovery.kind = bootstrap`, `providerWorkStarted =
false`). It had no process identity, no controller lease, and its
Daytona sandbox lease had a `destroyed` termination receipt. Execution
recovery had settled a no-replay hold on it and appended the "Automatic
recovery stopped" lifecycle event.

**Expected behavior**

When the previous execution is proven stopped, a person's Retry starts a
new turn promptly and continues from the recorded work. The task view,
the execution blocker and admission agree that Retry works.

**Steps to reproduce**

1. Let a legacy `paperclip_runner` run fail while it prepares a remote
sandbox (`setup_failed`, bootstrap evidence with `providerWorkStarted =
false`) on its third automatic attempt.
2. Let the sandbox lease be released with a `destroyed` termination
receipt.
3. Let execution recovery settle the
`legacy_execution_requires_reconciliation` hold. It appends the
"Automatic recovery stopped" lifecycle event to the run.
4. Press Retry on the task.
5. Observe that the Retry receipt keeps `executionWait.reason =
execution_recovery` and no run starts. Press Retry again and observe
that it attaches to the same held receipt.

**Paperclip version or commit**

Staging Core `53608bd1d6`. Fixed on top of `origin/master` `c4f639ed1b`.

**Deployment mode**

Hosted cloud (staging), Daytona remote sandbox.

**Agent adapter(s) involved**

`paperclip_runner` (legacy runtime).

Related: #21267 (a Retry held on a resolved decision,
`decision_pending`). This pull request does not change its files.

## What Changed

- `readFailedPreparationSettlement` (`failed-preparation-settlement.ts`)
accepts one trailing host lifecycle event: the no-replay settlement
event that execution recovery appends. The event must have the exact
payload shape and must name a resolved
`legacy_execution_requires_reconciliation` hold whose evidence binds the
same run. Any other event still fails the proof.
- `failedNativePreparation` (`cancelled-native-startup.ts`) asks for a
verified workspace repair only when the run holds prepared files: a
recorded remote workspace, a workspace restore failure, or required
workspace recovery. A run with no recorded remote workspace and a
stopped sandbox has nothing to repair. The lease termination receipt,
the bootstrap evidence and the settled-event proof are still required.
- New regression test `legacy-preparation-retry-hold.test.ts` uses the
exact shape of the staging run.

## Verification

- Red/green: `cd server && npx vitest run
src/services/legacy-preparation-retry-hold.test.ts`.
- "starts the person's Retry once the stopped preparation's sandbox is
destroyed" fails on `origin/master` (the receipt keeps
`executionWait.reason = execution_recovery`, no run) and passes with
this change (one queued successor with `previousRunId` set to the failed
run and the Retry request id).
- The Retry stays held when the proof does not hold: a settlement event
bound to another run's hold, a provider execution event, a lease without
a termination receipt, and a recorded remote workspace without a repair
receipt.
- Related suites pass (299 tests): `task-retry`,
`saved-bootstrap-continuation`, `interrupted-preparation-hold-settle`,
`workspace-recovery-preparation`,
`workspace-recovery-preparation-gates`, `heartbeat-session-preparation`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low. The change only widens two proofs for a run that never started
provider work: the platform's own settlement event is accepted only when
it binds a settled hold for the same run, and the repair receipt is
skipped only when the run recorded no files to repair.
- Retry still starts a new turn. It never replays the stopped run.
- Not in scope: two other tasks show the same generic wait with a
different cause on the saved-work side. Their blocker says `canRetry:
false` while the task view offers Retry. One has saved-work recovery
that failed with `source_unavailable`. The other shows recovered work
without a verified repair receipt. They need a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:07:17 -06:00
DottaandClaude Opus 5.5 0c63f0fdb3 test(skills): budget list read bytes for fixture rows, not the bundled catalog (#21285)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The skill library list reconciles the bundled skill catalog,
folders, and local sources on each read
> - #18023 added a test that limits the `company_skills` bytes one warm
`list()` call reads, so the folder pass and the list query keep their
narrow projections
> - The limit counts every `company_skills` read, so it also counts the
bundled catalog rows that every company gets
> - Each new bundled skill adds about 9.6 KB to that total, so pull
requests that only add skills under `skills/` fail a test about
projections
> - This pull request measures a catalog-only company next to the
fixture company and applies the same limit to the difference
> - The benefit is that the test fails only when the fixture rows are
read too widely, not when the catalog grows

## Linked Issues or Issue Description

Refs #18023 (added the test). Refs #20760, #20764, #20767, #20768,
#20769 (failed on this test).

**What happened?**

`companySkillService.list > keeps folder classification and stored
content while narrowing inventory refresh reads` failed on five open
pull requests that only add bundled skills under `skills/`. The error
was `expected 1163219 to be less than 950000`. Other runs had 983607,
1035956, 982321, and 985561. The test passes on master.

**Expected behavior**

The test must fail when `list()` reads the unused metadata or the
Markdown of company skill rows. It must not fail when the bundled
catalog gets larger.

**Steps to reproduce**

1. Check out master.
2. Run `git checkout <head of #20760> -- skills` to add its 34 bundled
skills.
3. Run `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/company-skills-service.test.ts -t "narrowing inventory
refresh"`.
4. The test fails with `expected 1163504 to be less than 950000`.

**Paperclip version or commit**

`9b613562b70b7bea9e7562ddc1087d4c1b1788b1` (master).

## What Changed

- The test creates a second company that has only the bundled catalog.
- The test lists both companies warm and captures the `company_skills`
response-value bytes of each call.
- The test applies the existing 950,000-byte limit to the difference.
The difference is about 663 KB and comes only from the ten fixture rows.
- The test also checks that the fixture company lists the catalog rows
plus the ten fixture rows.
- No service code changes.

## Verification

Where the bytes come from (master, one warm `list()` call):

| Read | Master catalog | Catalog of #20760 |
| --- | --- | --- |
| Main list query (`company-skills.ts:3324`) | 671,003 B, 20 rows |
691,694 B, 52 rows |
| Bundled reconciliation `getByKey` reads (one per bundled skill) |
165,423 B, 13 calls | 471,771 B, 45 calls |
| Total (old assertion) | 836,465 B, pass | 1,163,504 B, fail |
| Fixture company minus catalog-only company (new assertion) | 663,153
B, pass | 663,166 B, pass |

Red and green:

- Old test, master `skills/` plus the `skills/` tree of #20760: fails,
`expected 1163504 to be less than 950000`.
- New test, same enlarged catalog: passes, difference 663,166.
- New test, master: passes, difference 663,153.

The new test still catches the regression it exists for. I widened each
projection on purpose with the enlarged catalog, ran the test, and
reverted:

- Folder pass reads the full `metadata` column instead of `sourceKind`
only: fails, `expected 1323122 to be less than 950000`.
- Main list query also reads `markdown`: fails, `expected 1183162 to be
less than 950000`.

Commands:

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/company-skills-service.test.ts`: 81 passed.
- `pnpm verify`: not completed. It selected 2,615 of 3,435 test files
(conservative selection), and I stopped it to keep load low on a shared
machine. CI ran the full required checks on the final commit: 54 passed,
4 skipped.

## Risks

- Low risk. The change is in one test only.
- The new test runs one more warm `list()` call, about 0.5 s more.
- The bound is unchanged. A widened projection on the fixture rows still
adds at least 540 KB and fails.
- This test no longer limits the bytes the bundled catalog adds to each
list call. Each bundled skill adds about 9.6 KB per warm call, because
`ensureBundledSkills` calls `getByKey` with the full row for each
bundled skill. That cost is real but separate. A fix should narrow that
read, not let this test limit the catalog size.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (not
applicable: test-only change)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (Superagent Security Scan is
neutral: "Scan inconclusive")
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:03:23 -06:00
DottaandClaude Opus 5.5 c6c7a61e2c fix(retry): release a Retry held on a resolved decision (#21267)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails, the task shows a Retry button. Admission
can hold an accepted Retry while a person must answer a pending approval
or question (`decision_pending`).
> - The hold is saved on the deferred Retry receipt. Nothing cleared
that saved reason when the person answered, so the task kept saying
"Retry is waiting for a pending approval or question" when nothing was
left to answer.
> - The deferred-receipt worker drops a refused receipt from its
schedule. No producer hinted it when the decision resolved, so the held
Retry did not start until some unrelated event rescanned the company.
> - A Retry was also held by a confirmation that was already open when
the failed turn was admitted. That turn had already run past the
confirmation, so the Retry added a gate that the original turn did not
have.
> - A person's comment on the task started a continuation, but the held
Retry receipt stayed deferred next to it.
> - This pull request makes a resolved decision release the held Retry
at once, keeps the projection from showing a resolved decision as the
wait, stops holding a Retry on a decision that predates the failed turn,
and settles a held Retry when a comment or answer continues the same
failed run.
> - The benefit is that Retry always works: a person never waits on a
decision that is already resolved.

## Linked Issues or Issue Description

**What happened?**

A board user pressed Retry on blocked tasks. No run started. The task
said "Retry is waiting for a pending approval or question to be
resolved. Your message is saved." In one task, the only interaction was
a `request_confirmation` that the previous successful run created. The
user's later comment started the turn that failed. The confirmation
stayed pending, so every Retry attached to the same held receipt. After
the user rejected the confirmation and sent a new comment, the task
still showed the same wait. No run started until a server restart
rescanned the deferred receipts about three minutes later. The task
projection said `canRetry: true` with the `decision_pending` wait.

**Expected behavior**

An accepted Retry is never held on a decision that is already resolved.
Accepting or rejecting the decision releases a held Retry right away. A
person's comment on the blocked task starts work, and the held Retry is
settled. The task projection never shows a resolved decision as the
Retry wait.

**Steps to reproduce**

1. Let an agent run on a remote runner create a `request_confirmation`
for the task.
2. Let the runner be lost so the run fails with
`native_continuation_requires_reconciliation`, and let saved work
recovery complete.
3. Press Retry. The Retry receipt is saved with `executionWait.reason =
decision_pending`.
4. Reject or accept the confirmation.
5. Observe that the receipt keeps `decision_pending`, the task keeps
showing the wait, and the Retry does not start until an unrelated
rescan.

**Paperclip version or commit**

Staging Core `57aeda190b`. Fixed on top of `origin/master` `53608bd1d6`.

**Deployment mode**

Hosted cloud (staging), remote sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (native runtime).

## What Changed

- `hasPendingContinuationDecision` (`explicit-native-continuation.ts`)
is now the single check for a decision that holds a person's
continuation. Admission and the task projection both use it.
- For a Retry, a pending interaction that was created before the failed
source run was created does not hold the Retry. That turn was already
admitted while the decision was open. Decisions that the failed run
created (for example a plan review) still hold the Retry.
- `releaseDecisionHeldExecution` (`deferred-execution-wait.ts`) removes
the saved `decision_pending` wait from deferred receipts on the task and
hints each receipt to the deferred-receipt worker. It runs in the
resolution transaction of every interaction resolution path (through
`notifyChatConfirmationResolution`) and in question answers.
- The task projection reads the live decision state. A saved
`decision_pending` wait whose decision is resolved is not shown. This
also repairs receipts saved before this change.
- When a comment or an interaction answer continues a failed run,
`retireTaskRetriesContinuedBy` (`task-retry.ts`) settles held Retry
receipts for the same source run as `task_retry_superseded`.

## Verification

- Red/green: `cd server && npx vitest run
src/services/governed-wait-runner-loss.test.ts`. The five new tests fail
on `origin/master` sources and pass with this change (17/17 pass):
  - releases the held Retry as soon as the person chooses accept
  - releases the held Retry as soon as the person chooses reject
- never shows a resolved confirmation as the Retry's wait, and both
projections offer the same Retry
- a person's comment starts work and settles the held Retry once nothing
is left to answer
- starts a Retry at once when the confirmation was already open before
the failed turn started
- The existing tests that keep a Retry or a message on hold while a plan
review, governed tool action, secret proposal, or connection
authorization is pending still pass.
- Related suites: deferred execution wait, explicit native continuation,
interaction resolution and service, task interruption presentation and
query, task retry, deferred receipt work, interaction routes.

## Risks

- Medium-low. A Retry now starts when the only pending interaction is
older than the failed run. That turn already ran with the interaction
open, so the Retry gets no new authority. Governed tool actions still
need their own approval before the tool runs.
- Interaction resolution now also updates deferred receipts on the same
task. The update is limited to receipts with a saved `decision_pending`
wait. A receipt that is still blocked by another pending decision is
held again on its next admission.
- Issue-linked approvals (`approvals` table) do not yet hint the held
receipt when they resolve. The projection no longer shows them as the
wait after they resolve.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:58:48 -06:00