Commit Graph
7818 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 2b69cdfcd6 fix(evals): grade the personal-secret setup reply by its rendered link (#21344)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that a chat request reaches the correct
saved outcome
> - The personal-secret case checks that chat sends the person to their
own secure value entry, and never asks for the value in chat
> - The case judged the reply's wording with regular expressions, and it
found the settings route by splitting the reply's markdown into tokens
> - A valid reply failed because its link was followed by a colon, so
the tokenizer read the route as `/COM/company/settings/secrets):`
> - This pull request grades the rendered link instead: it finds the
visible link, clicks it, and requires the person's own-value entry to
appear
> - The benefit is that the case measures what a person can actually do,
and the reply's wording is left to human review

## Linked Issues or Issue Description

**What happened?**
`chat-features.runner-opencode.local.secrets.personal-secret` failed
with "Actual reply must direct the supported own-value settings path for
this definition". The retained screenshot and DOM snapshot show a
correct reply. The turn rendered as two messages, and both linked **My
secrets** to `/COM/company/settings/secrets` and said **Set my value**.
The wording check accepted the reply. The link check returned nothing.
It extracted routes from the reply's markdown with a regular expression
and removed only `.,;!?])}` from the end of each token. In the final
message the link was followed by a colon (`[My
secrets](/COM/company/settings/secrets): choose`), so the token was
`/COM/company/settings/secrets):`, which does not match the settings
path. An earlier passing run had `,` after the link, which the check
removed.

**Expected behavior**
A reply passes when its rendered message has a visible link that a
person can click to reach their own value entry for the required
definition. Wording is not graded by keywords.

**Steps to reproduce**
Grade a setup reply whose markdown link to the secrets settings is
followed by `:`. The old `personalSecretSetupURL` returns `undefined`
for the retained reply, although its wording check returns `true`.

## What Changed

- Add
`tests/runner-e2e/chat-catalog-domains/external-personal-secret-link.ts`.
It finds a visible link in the reply's rendered message
(`#comment-<id>`) whose target is this company's
`/company/settings/secrets`. An optional `definition` query must name
the required definition. Other origins, companies, paths, credentials,
fragments and query keys are refused. It clicks the link and requires
the person's own-value entry to be visible. A link that names the
definition opens that definition's detail sheet, so the check requires
**Set my value** in the sheet, which must show
`PERSONAL_PUBLISHING_KEY`. Otherwise it selects **My secrets** if that
tab is not active and requires a visible **Set value** button on the row
bound to the required definition (`my-secret-<definitionId>`). It then
returns to the conversation and records the proof as
`personal-secret-reply-link.json`.
- In `external-personal-secret.ts`, the setup reply is no longer checked
against wording patterns. It must still be the agent's own comment from
the bound run. The grade uses the recorded link proof, which must belong
to a reply of the initial run (a turn can post several messages). The
reply link is now followed before the human-entry phase.
- The confirmation and question paths are unchanged. When a question or
confirmation card already carries the handoff, the reply link is not
checked, so an optional link cannot fail those paths.
- No scenario YAML changed, so the TestOps manifest needs no entry.
`chat-catalog-descriptors.json` lists the new source file for this case.

## Verification

- `external-personal-secret-link.test.ts`:
  - An ungated unit test covers the link-target rules.
- A Chromium test (gated like the other secret-route browser tests by
`PAPERCLIP_CHAT_EVAL_SECRET_ROUTE_CANARY=1`, run locally) uses the
retained two-message reply. These cases pass: the retained reply; a link
carrying `?definition=` that opens the definition's detail sheet; a page
where the **My secrets** tab is selected after the click.
- These cases fail: no link (wording only); a plain-text path; a
wrong-company link; a foreign link; a wrong-page link; a hidden link; a
link that reaches another definition's row; a definition link whose
sheet shows another key; a link that reaches no own-value entry.
- Red check: the old `personalSecretSetupURL` returns `undefined` for
the retained reply, while `personalSecretGuidance` returns `true`.
- The personal-secret test files, catalog consistency and contract tests
pass (210 tests). `pnpm test:e2e:runner:typecheck` passes.
- A narrow paid run of the case on this branch is linked in a comment.

## Risks

- Low risk. The change is in eval code only.
- A reply that points to the settings page without a clickable link (for
example, text menu directions only) no longer passes this path. That
matches the stated contract: the person reaches their entry through a
link in the reply. Question and confirmation cards are unchanged.
- The company-secret case uses the same markdown tokenizer and could hit
the same trailing-colon problem. It is not changed here.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 04:26:29 -06:00
DottaandClaude Opus 5.5 70be453c38 test(e2e): accept either order when a person's message and the fallback settle a legacy hold (#21275)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is cut off, either a person's message or the
fresh-continuation fallback's disposition worker can settle the task's
recovery hold, and the task continues.
> - The e2e spec `legacy-failure-continuation` "message" case asserted
an internal field that depends on which of the two settles the hold
first.
> - When the fallback's worker wins, it stamps the cut-off run as
superseded. The spec then fails, even though the person sees the same
outcome.
> - This pull request makes the spec assert what a person sees, not
which path won.
> - The benefit is that the spec stops flaking for every PR and in the
release source check.

## Linked Issues or Issue Description

Refs #21135, #21250

**What happened?**
`tests/e2e/legacy-failure-continuation.spec.ts` › "legacy startup hold:
message reaches a new agent response" failed on #21250 at 0f47702b50.
The same tree had passed at 1791534795. Line 153 expected the source
run's `resultJson` to be `null`, and it received `{
freshContinuationSupersededAt }`. Every user-visible assertion before it
passed:
- the task is done with no execution blocker;
- there is no "Task recovery" banner;
- there is exactly one new run;
- the agent answered the follow-up once.

Cause: since #21135, the fallback's disposition worker can settle the
hold before the person's message admission does. Both orders are valid
and end the same way. When the fallback wins, it marks the dead source
run as superseded.

**Expected behavior**
The spec passes whichever path settles the hold first.

## What Changed

- `tests/e2e/legacy-failure-continuation.spec.ts`: the source run must
stay `failed` and must not be replayed. Its `resultJson` may be `null`,
or may carry only the superseded stamp. Any other content still fails
the spec.

Why both orders are acceptable (decision (a), not a product ordering
fix):
- `freshContinuationSupersededAt` has one effect: it revokes the old
run's write authority (`agent-run-cancellation.ts`,
`middleware/auth.ts`). The old run is already dead (`process_lost`), so
nothing changes.
- In the wake queue, the stamp only skips the native finalization gate
and the reconciliation incident, and this run is legacy. No UI code
reads it, so it adds no banner and no Retry.
- When the fallback wins, the opening note says the previous attempt was
cut off. That is accurate here, because the server restarted during
startup.
- The spec already proves the person sees one answer, one new run, and
no banner.

## Verification

- Not run locally. The race does not happen on demand, so a local pass
would prove nothing about it, and the shared Mac was loaded. The CI e2e
shard for this PR runs the spec on the final revision.
- No product code changed.

## Risks

- Test-only change. The spec still fails on any other `resultJson`
content, on a second run, or on a visible banner.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 04:23:54 -06:00
DottaandClaude Opus 5.5 566843820c fix(recovery): a queued continuation is not shown blocked, recent temporary failures continue, and notices say what a person can do (#21336)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, recovery continues the task, shows the
person a notice, and sweeps for tasks that nothing will continue.
> - On staging, a task stayed "Blocked / needs attention" for 24 minutes
while its Retry's run was queued.
> - Two tasks escalated to the board after a sandbox resume timeout, a
temporary failure, and nobody continued them.
> - Recovery notices asked a "Board operator" to inspect evidence and
repair sandboxes. Recovery must work without that.
> - A children-completed system wake could start on a parent that was
already closed.
> - This pull request fixes those four gaps.
> - The benefit is that interrupted tasks continue on their own, and
notices tell a person only what happened and the one thing they can do.

## Linked Issues or Issue Description

Refs #21250, #21257 (merged), #21258, #21234

**What happened?**
1. A task whose continuation was queued was shown as blocked (one
staging task). Execution recovery blocked the task at 23:51:28, and
restart recovery's Retry queued the successor at 23:51:50 through
`admitExplicitNativeContinuation`. That path resolves the interruption's
holds, but it never lifted the block those holds owned. The run started
at 00:15, so the task was queued for 24 minutes and shown blocked the
whole time. This was not a race: the issue's `updatedAt` equals
`blockedTransitionAt`, so nothing wrote to the issue after the Retry.
2. Board-owned stranded holds after a temporary failure had no continuer
(two staging tasks). A person's Retry failed while resuming the saved
sandbox (`reusable_sandbox_resume_failed`, cause `transport_timeout`,
`providerWorkStarted=false`), and stranded recovery escalated the task
to the board. The sweep leaves a blocked task that has any active hold
to the hold's owner. The fresh-continuation fallback takes only
execution-reconciliation holds. So nobody continued these tasks.
3. Recovery notices addressed a "Board operator" and asked people to
inspect evidence and repair runtimes, sandboxes or workspaces.
4. An `issue_children_completed` system wake could start on a parent
that was already done or cancelled.

**Expected behavior**
A task with a queued continuation is not shown blocked. A recent
temporary failure is continued automatically. A notice says what
happened and the one thing a person can do. A system wake does not start
on a closed task.

**Steps to reproduce**
1. Fail a run so that execution recovery blocks the task, then press
Retry while the agent's queue is full. The task stays Blocked until the
run starts.
2. Make a saved-sandbox resume time out on a person's Retry. The task
escalates to the board and stays blocked.

## What Changed

- **Queued continuation is not shown blocked**
(`explicit-native-continuation.ts`,
`recovery/assigned-work-invariant.ts`,
`recovery/assigned-work-sweep.ts`):
- `admitExplicitNativeContinuation` (a person's Retry, restart
recovery's Retry, or a saved message) moves the task from Blocked back
to To do when the replaced interruption set the block. That means the
hold's own block receipt, or a block written at or after the source run
ended with no later decision. A declared unblock action or an open
blocker keeps the block.
- New invariant violation `blocked_with_live_run`: the task is blocked,
has a queued or running run, and is not waiting on anyone else. Under
the row lock, the sweep lifts that block without starting a run or
spending the cap. Its suspect pre-filter now lets blocked tasks with
live runs through.
- **Recent temporary failures are continued** (new
`recovery/stranded-hold-continuation.ts`,
`recovery/assigned-work-sweep.ts`, `fresh-continuation-fallback.ts`,
`recovery/stranded-notice.ts`):
- The assigned-work sweep is the single owner of board-owned
`stranded_assigned_issue` holds whose failed run ended within the last
24 hours with a restart-class code, or with a temporary setup failure
before provider work (for example a sandbox resume refused for a
timeout).
- Under the row lock it resolves the hold (outcome `restored`, owner
system), restores the task's status from before the block, and queues
one continuation under the usual per-task cap and spacing.
- Older escalations and lasting causes (access denied, lost sandbox,
unknown) stay with their owner. The fallback explicitly never takes a
`stranded_assigned_issue` hold.
- **Plain recovery notices** (`recovery/service.ts`,
`recovery/stranded-notice.ts`, `recovery/successful-run-handoff.ts`):
see the wording table below.
- **Closed parent** (`heartbeat/queue.ts`): `issue_children_completed`
is added to the claim-time cancel for system continuations on a done or
cancelled task. A person's saved message still runs, and so does
onboarding's completion report (`onboardingCompletion: true`), which is
meant to run on a done parent.
- **Review fixes** (Greptile, first round):
- A person's block is never lifted by a Retry or saved message. The
block-time inference applies only when no hold carries a block receipt,
and a block whose activity entry was written by a person is kept. This
is the same signal the sweep uses (`attributeBlock`, so `blockedBy:
"user"` is a wait).
- The sweep's suspect query lets a blocked task with a live run through
the queued/claimed-wake filter, so a continuation queued by
`enqueueWakeup` (a queued wake bound to its run) reaches the
`blocked_with_live_run` repair.
- The saved-sandbox and workspace-setup timeout notices promise an
automatic retry only when the sweep will continue that hold. Both use
one decision (`strandedRunCauseIsTemporary`, inside
`strandedHoldContinuesAutomatically`). Otherwise the notice says to
press Retry.

### Old → new wording

| Where | Old | New |
|---|---|---|
| service.ts hold nextAction, successful run with missing state | Board
operator: inspect the run evidence, then explicitly choose a valid issue
disposition... | The agent finished without saying whether the task is
done. Mark the task done if it is, or press Retry to let the agent
continue from the saved work. |
| service.ts, process_lost / default | Board operator: inspect the retry
history / inspect the evidence, repair the runtime if appropriate... |
This attempt could not be finished. Press Retry to start it again from
the saved work. (`RETRY_FROM_SAVED_WORK_NEXT_ACTION`) |
| service.ts, provider_quota | Wait for provider quota recovery, then
retry the original assignee; do not wake a takeover owner. | The AI
provider's usage limit was reached. The same agent continues the task
once the limit resets. |
| service.ts, environment provider | No agent work started. Retry the
task once the environment provider is available. | The sandbox service
was unavailable, so no agent work started. Press Retry once it is
available again. |
| service.ts, inactivity | Board operator: inspect the inactivity
evidence... | The agent stopped responding, so this attempt was ended.
Press Retry to start it again from the saved work. |
| service.ts, worktree branch incoherence | Board operator: repair the
source task git worktree branch incoherence... | The task's workspace no
longer matches its branch, so the agent did not start. Press Retry to
start it again. |
| service.ts, base materialization | Board operator: repair the project
workspace repository URL or clone access... | Paperclip could not copy
the project's repository. Restore access to the repository in the
project settings, then press Retry. |
| service.ts, workspace link | Board operator: repair the source task
workspace link, project workspace cwd, or git checkout... | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| service.ts, base ref | Check the starting branch and repository
access, then repair the task's branch... | The starting branch for this
task's workspace was not found in the repository. Set a starting branch
that exists, then press Retry. |
| service.ts, sandbox plugin | Board operator: the sandbox provider
plugin named in the run failure is not ready; ... explicitly retry... |
The plugin that starts this agent's sandbox is not running. To fix it,
&lt;remedy&gt;, then press Retry. |
| service.ts, missing secret | Board operator: bind the missing
secret(s) named in the run failure... | A secret this agent needs is not
set. Add it in the agent's settings, then press Retry. |
| service.ts, review participant | Board operator: repair the failed
review participant path... | The reviewer did not finish the review.
Choose a reviewer, or press Retry to ask the same reviewer again. |
| service.ts failure summary | Latest retry failure details were
withheld from the issue thread; inspect the linked run for evidence. |
The failure details are kept with the run. |
| service.ts nested recovery comment | the assigned recovery owner or
board operator should fix the runtime/adapter problem... | press Retry
on the original task to start it again from the saved work, then close
this task. |
| service.ts recovery-issue stop comment | the current recovery owner
should inspect the failed run evidence, restore a live execution path...
| press Retry to start this task again from the saved work. |
| service.ts deliberate-wait exhaustion hold and comment | Inspect the
evidence and choose whether to repair, retry... / Next action: repair
the liveness disposition or request an explicit source-owner decision. |
The agent stopped without saying what happens next with this task. Mark
the task done if it is, or press Retry... / Next action: mark the task
done if it is, or press Retry... |
| stranded-notice REMOTE_WORKSPACE_RESTORE_NEXT_ACTION | Open the failed
run... inspect and move it aside, then retry | The saved work could not
be put back into the sandbox, so no agent work started. The saved
version was kept. Press Retry to start again from it. |
| stranded-notice scan notices (timeout, plugin worker, saturated,
output limit, failed, cancelled) | Check repository access and server
load / Check the worker status / Inspect the failed run... | "&lt;what
happened&gt;, so no agent work started. Press Retry ..." |
| stranded-notice workspace validation seed | ...Moving it to `blocked`
so the workspace link, cwd, or git checkout can be repaired | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| stranded-notice base ref seed | ...blocked until its workspace
configuration or repository access is repaired / Check that the
configured base ref exists... | starting branch not found / Set a
starting branch that exists in the repository, then press Retry. |
| stranded-notice stale shared holder | Check whether the other run is
still working and stop it if it is stuck... | Stop the other run on this
task, or choose a separate workspace for this task, then press Retry. |
| stranded-notice secret default | required secret/env bindings are
missing. Moving it to `blocked` so an operator can bind... | The agent
did not start because a secret it needs is not set. Add the secret in
the agent's settings, then press Retry. |
| stranded-notice review seeds (2) | ...so the board can inspect the
evidence, repair the reviewer runtime... | The reviewer ... did not
finish / cannot take work. Choose a reviewer, or press Retry ... |
| stranded-notice default body | could not restore a live execution
path... visible for intervention | This attempt could not be finished
automatically. Press Retry to start it again from the saved work. |
| stranded-notice metadata | Board decision required / "Inspect the
evidence, then retry ... repair the execution path ..." / "The recovery
owner should either restore a live execution path..." | Waiting for you
/ `RETRY_FROM_SAVED_WORK_NEXT_ACTION` / The assigned agent continues the
task from the saved work |
| successful-run-handoff.ts | Board decision required / inspect the
evidence, then retry the original owner... | Waiting for you / Mark the
task done if it is, or press Retry to let the agent continue from the
saved work |
| stranded-notice, saved sandbox resume timeout and workspace setup
timeout | asked to inspect the run or the provider | No agent work
started; Paperclip tries again automatically. |

Kept on purpose:
- `buildImmediateExecutionPathRecoveryNoticeSeed` ("No live execution
path" title, "retried continuation ... visible for intervention" body).
The UI recognizes this exact text
(`ui/src/components/task-chat/adjacent-errors.ts` regex and title,
`ui/src/lib/system-notice-humanizer.ts`) and humanizes it, so changing
it needs a UI change in the same PR.
- The UI's own "Board decision required"
(`IssueRecoveryActionCard.tsx`).
- Recovery-issue comments that still contain internal codes
(`stranded_issue_recovery`).
- The agent-facing disposition instruction in
`successful-run-handoff.ts`.

### Deliberately changed test expectations

Each of these asserted the old wording or the old behavior:
- `server/src/services/recovery/stranded-notice.test.ts`: stale shared
holder; base ref body and next action; scan timeout metadata; plugin
worker metadata; secret default body; owner row is now "Waiting for
you".
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: owner row
is now "Waiting for you" (3 places); failure-summary line;
board-escalation nextAction expectations (review participant,
process_lost, missing disposition, deliberate wait, default); sandbox
plugin; workspace link and validation comments; missing secret action
and comment; review participant comment.
- `server/src/__tests__/assigned-work-invariant.test.ts`: the old "left
to the hold owner" test now uses a 30-hour-old run. A recent temporary
failure is now continued, which is the new behavior.
- `server/src/services/heartbeat/queue.test.ts`:
`issue_children_completed` is now in the claim-time cancel list.
- `server/src/services/heartbeat/run-completion.test.ts`: the
restore-failure next action now says "The saved version was kept".
- `server/src/__tests__/issue-recovery-actions.test.ts` and
`server/src/__tests__/heartbeat-workspace-branch-containment.test.ts`:
the branch-incoherence next action now says the "workspace no longer
matches its branch".
- `server/src/services/recovery/stranded-notice.test.ts`: the timeout
notice is checked for both an automatic and a manual retry.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `task-retry.test.ts` "lifts the failure's block once the Retry's run
is queued..."; invariant unit test and sweep integration test for
`blocked_with_live_run`.
- sweep integration test "continues a recent board escalation of a
temporary setup failure...". A recent one is continued; a 30-hour-old
one and `provider_access_denied` are left with their owner. Plus a
`stranded-notice.test.ts` case.
- the wording expectations above.
- `heartbeat/queue.test.ts`.

Review-fix tests, each red before its fix:
- `task-retry.test.ts` "lifts the failure's block...": new `newer` case
(the hold's receipt names an older status version) and `person` case (a
person's activity entry set the block). Both stay blocked.
- `__tests__/assigned-work-invariant.test.ts` "a recovery block whose
task already has a queued run...": now seeds a queued wake bound to the
queued run, as `enqueueWakeup` does, and asserts `listSuspects` returns
the task.
- `heartbeat-dependency-scheduling.test.ts` "dispatches only a verified
completed onboarding report" (it failed in CI on the first push) passes
again.

Evidence:
- Item 1: the staging task's activity log. The issue's `updatedAt`
equals `blockedTransitionAt`; nothing wrote after the Retry.
- Item 2: the two staging runs ended at 12:28Z and 14:23Z on Nov 12,
inside the 24-hour window.
- Ruled out, so there is no change for it: a third staging task is
waiting on a real person's gate. A pending `request_confirmation` was
created before the fallback run failed.

Local runs, on master after #21257 (e063587edd) plus these commits
(server `tsc --noEmit` also passes):
- `stranded-notice.test.ts`, `recovery/assigned-work-invariant.test.ts`,
`task-retry.test.ts` and `__tests__/assigned-work-invariant.test.ts`: 4
files plus `heartbeat/queue.test.ts`, 329 tests passed.
- `heartbeat-process-recovery.test.ts` and `recovery-wake-herd.test.ts`:
475 tests passed.

Code comments that the original commits added were removed to follow
CONTRIBUTING; the explanations are in this description.

## Risks

- The sweep now resolves board-owned stranded holds for recent temporary
failures. It is bounded to failures within the last 24 hours with a
restart-class or temporary-setup cause, one continuation per task under
the usual cap and spacing. Lasting causes stay with their owner.
- Lifting a block on explicit continuation is limited to blocks the
replaced interruption set. A declared unblock action or an open blocker
keeps the block.
- Wording only changes server text. The UI-recognized "No live execution
path" notice is unchanged.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 04:04:20 -06:00
DottaandClaude Opus 5.5 cb94108484 fix(native): finish the run from a provider turn a timed-out start left running (#21338)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run talks to a remote Runner through a controller. A
deploy or a same-run retry can reattach a new controller to the live
Runner.
> - A slow turn start can make the first controller give up with
"runnerd did not report the provider turn identity". Codex still runs
that turn.
> - The retry reattaches the Runner and adopts that turn, because the
Runner accepted it for this run's latest turn.start.
> - If the adopted turn finishes while recovery still runs, the saved
checkpoint has no active turn. The runtime then tries to start a fresh
turn, and the run fails with "session cannot start another turn". The
Runner is lost and the task needs a Retry.
> - This pull request makes the runtime treat a terminal that the
recovered session newly settled as the adopted provider terminal when
the saved checkpoint bound no turn.
> - The benefit is that a turn which a timed-out start left running
finishes the same run on the reattached Runner, with its session
context, and with no second turn/start.

## Linked Issues or Issue Description

**What happened?**
On staging, Codex runs on a remote Runner lost the Runner after a
same-run reattachment. Each chain started with a turn.start that timed
out ("runnerd did not report the provider turn identity") while Codex
ran the turn anyway. After the reattachment, the run failed with a
protocol integrity fault (thread_binding_mismatch,
tool_binding_mismatch, or native_event_replay_conflict "Provider
authoritative notification item/started did not name its execution
owner"). The task then fell back to a fresh session and lost its
context. A local reproduction with the real runnerd and the fake Codex
app server shows the same trigger also fails when the abandoned turn
finishes during the reattachment: the runtime tries a new turn and fails
with "session cannot start another turn".

**Expected behavior**
A turn that a timed-out turn.start left running belongs to the same run.
The reattached run adopts it, binds its late notifications, and finishes
from its outcome. Recovery does not kill the Runner and does not need a
manual Retry. A real cross-run or cross-task binding violation still
fails closed.

**Steps to reproduce**
1. Start a native Codex run on a remote Runner whose provider is slow to
emit turn/started after it accepts turn/start.
2. Let the controller give up with "runnerd did not report the provider
turn identity" while the Runner stays alive.
3. Let the same-run retry reattach the Runner.
4. Let the provider turn start and finish after the reattachment.
5. See the run fail instead of finishing from the adopted turn.

**Paperclip version or commit**
master at 22c9300dac (after #21321 and #21329)

**Deployment mode**
Paperclip Cloud staging, remote Runner in a Daytona sandbox

**Agent adapter(s) involved**
Codex (codex_app_server through paperclip-runnerd)

## What Changed

- `native-session-runtime.ts`: a terminal that the recovered session
newly settled now counts as the adopted provider terminal when the saved
checkpoint had no active turn. The runtime consumes that terminal
instead of starting a fresh turn.
- New `runnerd-abandoned-turn-start-reattach.test.ts` with the real
runnerd and the fake Codex app server. Each attempt runs in a fresh
module graph, as each attempt runs in its own executor process in
production. The first attempt times out on turn.start against an
unreachable remote Runner and leaves it alive.
- Red/green: the provider turn starts and finishes only after the
reattachment. The run now succeeds from `provider-turn-1` with one
turn/start and no duplicate Runner launch. Before the fix it failed with
"session cannot start another turn".
- Guard: output the abandoned turn reported while no controller was
attached binds to the reattached run's adopted turn with no
`session.failed`, and the Runner stays alive.

## Verification

- `npx vitest run
src/live/runnerd-abandoned-turn-start-reattach.test.ts` in
`packages/paperclip-runner`: red without the runtime change ("session
cannot start another turn"), green with it.
- `npx vitest run src/native-session-runtime.test.ts
src/live/runnerd-restart-continuation.test.ts
src/live/runnerd-abandoned-turn-start-reattach.test.ts
src/live/runnerd-turn-start-timeout-retention.test.ts
src/live/runnerd-codex-transport-cut-turn-start.test.ts
src/drivers/codex/codex-app-server-driver.terminal-replay.test.ts
src/drivers/codex/codex-app-server-driver.recovery.test.ts`: 7 files,
386 tests passed.
- `tsc -p tsconfig.test.json --noEmit` in `packages/paperclip-runner`:
no new errors (the existing `src/eval/workflow-harness.ts` errors need
the eval kernel build).

## Risks

- The runtime now adopts a newly settled terminal when the saved
checkpoint bound no turn. The driver records a turn terminal only for a
turn that it bound as active (the persisted active turn, the
uncheckpointed disposition turn, or this run's own accepted turn). A
turn of another run or task is never bound, so it still fails closed in
the driver.
- The staging chains failed with binding-mismatch codes. The local
reproduction reaches the same trigger and the adoption path, but the
fake provider cannot hold a live turn while it answers other requests,
so the exact staging codes did not reproduce locally. Late items that
reach a reattached controller before it knows the provider turn identity
can still fail closed. That path needs a separate fix if it recurs after
this change.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5) in Claude Code, with tool use
(shell, file edits, browser reads of staging run events).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21321, #21329

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:43:41 -06:00
DottaandClaude Opus 5.5 d16becac04 test(evals): fix cohort-10 secrets entry, steering graders, private-task settle and queue brief (#21296)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks real chat journeys against saved
product state, not reply wording
> - Full cohort 10 failed six cases where the product worked but the
eval was out of date or raced
> - #21225 redesigned Settings → Secrets, and #21243 changed when a
queued comment counts as delivered
> - The private secret steps, two steering graders, one final run check
and one fixture did not follow
> - This pull request updates those eval layers and keeps each one able
to fail a broken product
> - The benefit is that these six cases measure the product again
instead of the stale harness

## Linked Issues or Issue Description

Refs #21225
Refs #21243

**What happened?**
In full cohort 10 (OpenCode, local), six cases failed for eval reasons:

- `secrets.configure-company-secret` and `secrets.personal-secret`: the
private browser step waited for the old "Create secret" dialog and its
Provider field, and for the old personal form text. #21225 renamed the
dialog to "New secret", removed Provider, moved Key under "More
options", and moved the definition name into the "Set your value for …"
heading.
- `steering.steer-active` and `steering.discard-queued-message`:
Morgan's running turn read the queued comments through task history and
cited them in its accepted result. Under #21243 that is delivery, so no
continuation ran. The graders still required one more run and timed out.
- `tasks.private-task`: the final run check read the ledger while the
CEO's `chat_task_completed` follow-up was still running.
- `steering.queue-follow-up`: the brief was only a reference string, so
the continuation asked a question instead of building.

**Expected behavior**
Each case grades the product's current contract on saved state, and
still fails a broken product.

**Steps to reproduce**
Run the six cases at `origin/master` with the OpenCode local profile.
See the cohort-10 evidence for each case.

## What Changed

- `external-private-browser-worker.ts`: the company step opens "New
secret", finds the dialog by its "Who provides the value" radiogroup,
selects Organization, opens "More options" for a requested key, and
picks Paperclip-encrypted storage only when the "Stored in" picker is
shown. The saved secret must now have provider `local_encrypted`. The
personal step binds the form by the heading "Set your value for Personal
publishing credential". Credential handling is unchanged.
`manual-private-*.ts` is unchanged.
- `external-company-secret-create-dialog.test.ts`: the canary page uses
the new dialog. It adds a several-vaults variant (valid), a
wrong-storage result and the retired "Create secret" form (both
rejected). Source anchors follow `Secrets.tsx`.
- `external-personal-secret-entry-dialog.test.ts`: checks that the
worker heading matches what `SetMyUserSecretDialog.tsx` renders, and
rejects another definition and the "Replace your value" form.
- New `runtime-cited-delivery.ts`: `owedAfterTurn` (queued comments the
turn did not read at the current revision and cite) and
`continuationCarriesOwed` (one continuation carries each owed comment
once, with its text and the earlier-read warning, and none that the turn
handled).
- `runtime-active-steering.ts` (oracle `steer-active.v9`): records the
queued comments (person and any CEO relay) and the worker's receipts.
Delivery passes if every queued comment was handled in turn with no
continuation, or one continuation carries exactly the owed ones. Run
count is 2 plus one when anything is owed. The website the worker
registered last must still render blue.
- `runtime-discard.ts` (oracle `runtime-discard.v7`, `minimumRunCount:
4`): the same rule for the font note and the later contact request. The
run that handled the contact request must save its labelled section. The
discarded request must never be cited. The gate no longer says "Leave
saved follow-ups for their ordinary continuation", because that
contradicts the #21243 contract.
- `internal-private-delegation.ts` (oracle `private-task.v3`):
`verifyResult` waits for the delegated work and its completion follow-up
to settle, in the pattern of `identity-plan-review.ts`. A failed or
cancelled run still stops the wait. A worker still running at the
deadline stays bounded active work.
- `runtime-queue.ts` (oracle `runtime-queue.v4`): the gate gives a
complete website brief (product, page, hero copy, features, pricing,
company details). It does not contain the word "contact", so the brief
does not complete the queued requirement.
- Oracle bumps for the secrets cases:
`company-secret-controller-baseline.v22`, `personal-secret.v21`.
`chat-catalog-descriptors.json` follows all bumps and source files. No
scenario YAML changed, so the TestOps manifest is unchanged.

## Why each new check still catches a broken product

- Secrets: the old check matched removed UI text, so it failed a working
product. The new check uses roles and labels, and still fails a missing
form, a failed POST, wrong saved metadata and storage outside Paperclip.
- Steering: the old check required a continuation, which #21243 removes
when the turn cites the comment. The new check still fails an uncited
read with no continuation (the lost-comment bug), a citation without a
current-revision read, a cited comment that was not applied (no blue
page or no saved label), a replay of a handled comment, and any extra
run.
- Private task: the old check read the ledger at a random point after
acceptance. The case still fails if the follow-up fails or is cancelled.
- Queue brief: the grader is unchanged. The brief test fails if the
brief leaks the contact requirement.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog tests/runner-e2e/chat-catalog-domains`:
285 files, 3948 tests pass.
- Calibration uses the retained cohort-10 receipts. The steer-active
worker read and cited both queued comments. The discard worker read and
cited the font note and the later contact request. Both now pass.
Reworded-valid variants (target handled in turn, relay or font note owed
to one continuation) pass. The broken variants listed above fail.
- `PAPERCLIP_CHAT_EVAL_SECRET_ROUTE_CANARY=1` secrets canaries: 3 of 3
pass. RED: the old worker against the new dialog page fails
(`locator.fill` timeout).
- `pnpm test:e2e:runner:typecheck`: pass. `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 of 10.
`node tests/runner-e2e/source-sdk-public-inputs.mjs
e0b618b02a --check`: `changed: false`.
- Paid run: Runner Full-Stack E2E 43368487002 on throwaway branch
`chat-eval/full10-paid` (the six cases). The result follows in a
comment.

## Risks

- Steering: a run that applies a comment but does not cite it still
needs one continuation. This matches the product contract.
- The steer-active gate still asks the worker to keep queued
requirements for normal delivery. Both paths are accepted, so the gate
cannot cause a false pass.
- The private-task settle adds up to 120 seconds inside the original
observation budget.
- Low risk overall: eval-only changes. No product code changed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:38:12 -06:00
DottaandClaude Opus 5.5 43aabb7080 fix(runner): keep configured secret values out of OpenCode output, storage and model requests (#21330)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can configure a secret that Paperclip delivers to an
agent's commands as an environment variable
> - The value must stay out of what the agent runtime stores and what it
sends to the model
> - With the OpenCode runtime, a command that printed the variable put
the value in OpenCode's tool output. OpenCode stored that output in its
session database and sent it to the model. Paperclip's own run log
already redacted it
> - Paperclip did not tell the runner which configured values are
secrets, so the runner could not protect them without also hiding
ordinary settings
> - This pull request sends secret names (never values) to the runner,
filters exact secret values before OpenCode captures any tool output,
redacts them from model requests, and tells the agent how to check a
secret without printing it
> - The benefit is that a configured secret stays out of model context
and out of runtime storage, even when an agent prints it

## Linked Issues or Issue Description

**What happened?**
The `chat-features.runner-opencode.local.secrets.personal-secret` eval
failed because the persisted-state scan found a configured secret value
in OpenCode's session database. The person entered the value through the
normal secret entry form. Nobody pasted it in chat. The agent ran a
shell command that printed the environment variable it was given.
Paperclip's run log redacted the value. OpenCode stored the raw output
in its database and included it in the next model request.

**Expected behavior**
A configured secret must never reach model context or runtime
persistence. Ordinary configured values must not be redacted.

**Steps to reproduce**
Configure a secret environment variable for an OpenCode agent. Have the
agent run a command that prints the variable. Then search the OpenCode
session data and the model requests for the value.

## What Changed

- **Secret names cross to the runner.**
`configuredEnvironmentProjection` takes the controller's secret key
names and adds `PAPERCLIP_SECRET_ENV_KEYS`. This marker holds only the
names of configured entries that came from secrets. It travels with the
configured projection across the TypeScript and Rust runner boundaries
(`configured-environment.ts`, `configured_environment.rs`). The server
passes `secretKeys` in `runtime-execution.ts`.
- **Command output is filtered before OpenCode reads it.** For a launch
that has configured secrets, the runner sets OpenCode's `shell` to a
small POSIX wrapper (`drivers/opencode/output-custody.ts`). The wrapper
runs the shell OpenCode would use anyway. It sends stdout and stderr
through an `awk` filter that replaces exact secret values. OpenCode
never receives the raw bytes, so its streamed previews, its database and
its model requests stay clean. The wrapper keeps the exit status. It
does not wait for background processes. The values come from the
environment that OpenCode already holds and are never written to disk.
- **Other tool results are rewritten before they are saved.** A local
OpenCode plugin rewrites completed non-shell results (file reads,
searches, MCP results) in `tool.execute.after`. OpenCode runs that hook
before it stores or returns the result. The wrapper refuses to run any
command until the plugin has started for the current launch (per-launch
nonce). So no command can write a secret to a file that a later read
returns without filtering.
- **Model requests are redacted as a second layer.** The OpenCode
provider proxy removes exact secret values from the string values of
each parsed model request. Field names and other literals are never
changed. A quick byte scan for each value, raw and JSON-escaped, sends
the original bytes unchanged when no value is present.
- **The agent gets guidance.** Native task prompts list the configured
secret names. They say to check a secret with `[ -n "$VAR" ]` and never
to print, echo, log or write a secret value.
- **No added startup delay.** The wrapper is content-addressed under the
runtime directory and reused across launches, so it is not rewritten on
every launch. OpenCode's plugin-dependency install is marked as already
done, so loading the plugin does not add wait time to the first tool
call.
- **Only exact values are redacted.** Command output is filtered line by
line, so each line of a multi-line value that has at least 4 characters
is redacted too. If a configured secret has no line of 4 or more
characters, the launch is refused with
`opencode_output_custody_unavailable` instead of running unprotected.

## Verification

-
`packages/paperclip-runner/src/drivers/opencode/output-custody.test.ts`
(10 tests): the wrapper filters stdout, stderr and a partial last line;
keeps exit codes (7, and 143 for SIGTERM); filters each line of a
multi-line value; refuses to run before plugin readiness or with an old
nonce; returns at once when a background process keeps running; leaves
ordinary values unchanged. The plugin rewrites nested results.
Model-request redaction covers raw and JSON-escaped values and never
changes field names or literals (a secret equal to `true` leaves
`"stream": true` intact). Secrets the filter cannot match are refused.
No value is written to disk.
- `opencode-server-driver.test.ts`: one saved session launches three
times. First with no secret marker (older controller), then with the
marker, then with no marker again (rollback). The first launch is
unchanged, the second applies custody, and the third launch still
starts. This test fails without the driver change.
- `configured-environment.test.ts`: the marker names only secret
entries, crosses the runner spawn and OpenCode boundaries, reads as "no
secrets" when absent, and is refused when malformed. Rust:
`secret_name_marker_crosses_only_with_its_projection`.
- `server/src/__tests__/configured-secret-guidance.test.ts`: guidance
content, and no guidance when there are no secrets.
- Real OpenCode 1.18.34 qualification (`PAPERCLIP_OPENCODE_QUALIFY=1
node --test scripts/qualify-opencode-runtime.test.mjs`). A fake model
asks for a command that prints the secret and writes it to a file, then
a read of that file. Unprotected baseline: the value is in the session
database and in model requests (red). With custody: it is in neither,
and the exit status is kept (green).
- **Latency** (local, macOS, real OpenCode, 8 runs each): boot 629 ms vs
697 ms baseline; first model request after the prompt 678 ms vs 698 ms
(no regression); first command 139 ms vs 191 ms; each later command +9
ms (36 vs 27 ms). Proxy cost per request when no secret is present (the
normal case): 33 µs at 50 KiB, 164 µs at 300 KiB, 0.55 ms at 1 MiB, 2.3
ms at 4 MiB. When a secret is present: 91 µs, 0.48 ms, 1.6 ms and 6.8
ms. The scan runs only for launches that have configured secrets, and it
adds nothing to OpenCode's own startup.
- A narrow paid run of
`chat-features.runner-opencode.local.secrets.personal-secret` on this
branch is linked in a comment.

## Risks

- **Previous-release story.** The marker is new and optional. A runner
that receives no marker (older controller) launches exactly as before.
An older runner drops the marker at its environment boundary. The change
saves no new run or session state. OpenCode configuration is regenerated
on every launch, so sessions saved before this change resume under
custody (covered by the cutover test). Warm-process reuse keys include
the marker, so a warm process from before the deploy is not reused for a
run that has secrets.
- **Commands go through the wrapper.** Output arrives line by line.
Output written after the shell exits is dropped, which matches how
OpenCode stops reading at exit. If `awk`, `mkfifo` or `rm` is missing,
the launch is refused with `opencode_output_custody_unavailable` and
does not run unprotected.
- **Scope.** Only exact values are redacted. An agent that encodes a
value (for example base64) is out of scope. The model-request layer
applies when Paperclip routes the model through its proxy (OpenRouter
and Paperclip connections).

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1113.0-canary.5
2026-11-13 03:23:56 -06:00
DottaandClaude Opus 5.5 0986c3eb3a fix(native): let a same-run reattachment take over its own retained remote Runner cleanup (#21328)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 keeps an unreachable remote Runner alive when the failed
attempt closes, so that the same-run reattachment can adopt it
> - The same close also leaves the session in the in-process cleanup
quarantine. The quarantine refuses every new session of the task, and
that includes the reattachment that the Runner was kept for
> - Close retries cannot settle it. After the first close fails, the
Runner transport returns the same rejected close for every retry
> - On staging, this was the largest failure class: 14 Codex runs in one
hour failed with `native_session_cleanup_quarantined`, while only 5 runs
succeeded. Each task then needed Retry
> - This pull request lets the same run's reattachment to that Runner,
in the same remote sandbox, take over its own retained cleanup
> - The benefit is that the task continues on its own, with no spent
attempt, no failed run and no Retry

## Linked Issues or Issue Description

Refs #20923, #21031

**What happened?**
A remote native run (Codex on Daytona) failed a turn with a recoverable
error, for example `runnerd did not report the provider turn identity`
or `native_finalization_missing: session returned no semantic result`.
The close then logged `runner did not prove durable suspension before
checkpoint` and `remote runner left alive unsettled for same-run
reattachment`. Each later same-run reattachment was refused at
admission:
- When the close error was retryable, admission waited out the grace and
failed with `native_session_cleanup_quarantined: prior session cleanup
remains incomplete`. That spent a same-run attempt. After the attempt
budget, the run failed. All close retries replayed the same failed close
within milliseconds (`runner transport close requested: native session
quarantined cleanup recovery`), so they could never settle the
quarantine.
- When the close error was `native_session_close_unrecoverable`
(operator-required), the next attempt failed at once with
`native_session_cleanup_quarantined: the previous session is still
shutting down; Paperclip finishes the cleanup automatically and the
saved work is kept.` That code is a permanent failure.

"Paperclip finishes the cleanup automatically" did not happen in the
process. Only a server restart (a deploy) removed the in-memory
quarantine, and by then the run had failed and the task waited for
Retry. Staging restarts every 20 to 45 minutes, so attempts often end
mid-cleanup and meet this gate.

**Expected behavior**
The same run's reattachment adopts the Runner that its earlier attempt
kept for it. It does not wait for a close that cannot settle, and it
does not spend an attempt. The task continues without Retry. Other runs
of the task and other sandboxes stay gated.

**Steps to reproduce**
1. Start a remote native run, and let its turn fail with a recoverable
transport error while the Runner cannot prove suspension.
2. The close keeps the Runner alive and quarantines the session.
3. The server schedules the same-run reattachment 30 s later.
4. The reattachment fails with `native_session_cleanup_quarantined`, and
the task needs Retry.

**Paperclip version or commit**
Deployed Core `c730872974` on staging. Reproduced on `master` at
`49f0030f06`.

**Deployment mode**
Hosted cloud (staging), Daytona remote sandboxes.

**Agent adapter(s) involved**
`paperclip_runner` native runtime with the Codex app server backend.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
`executeNativeSession` takes a new option `retainedRunnerReattachment`.
With a remote cleanup scope, admission retires quarantine entries that
belong to the same run in the same remote scope. Removal stops later
close retries. A close attempt that is still in flight stays an
admission owner until it settles, so the old and new controllers do not
overlap. Other runs, other sandboxes and local backends keep the gate.
- `server/src/services/native-runtime/native-session-executor.ts`: sets
`retainedRunnerReattachment` only for a `reattach_remote_runner`
recovery of the same run.
- `doc/execution-semantics.md`: records the takeover.
- Tests:
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: four
new cases. An operator-required close with a retained Runner. A
retryable close whose retries replay one failure. A close that a deploy
interrupts while it is in flight (the reattachment waits for it, then
takes over). Controls: another run, a run without the takeover option
and a local backend stay refused.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
asserts that the option is set for a same-run remote reattachment and is
not set for a fresh sandbox run.

## Verification

- Red on `master` (runtime source restored): 3 of the 4 new runtime
cases fail with exactly the two staging errors (`the previous session is
still shutting down…` and `prior session cleanup remains incomplete`).
The control case passes.
- Red on `master` (executor source restored): the 3 executor assertions
fail.
- Green with this change:
- `cd packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts` (294 passed)
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "exact
Stop cleanup|fresh sandbox|retirement"` (46 passed)
- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/completed-turn-cleanup-hold-settle.test.ts` (61 passed)
- `npx tsc --noEmit` in `packages/paperclip-runner` and `server` passes.
- Staging evidence: runs `7b7ded99` and `27c487ef` show the sequence
above in their events and logs.

## Risks

- The takeover applies only when the server recovers the same run with
`reattach_remote_runner` in the same remote cleanup scope. This is the
case that #20923 keeps the Runner alive for. After a deploy the
quarantine is already gone and the same reattachment runs today, so the
takeover gives the in-process retry the same behavior as a restart.
- A close attempt that is in flight is never overlapped. The
reattachment waits for it within the existing admission grace.
- The reattachment still verifies Runner ownership and its controller
generation before it adopts the Runner. If the Runner is gone, the
existing relaunch or reconciliation path handles it.
- No persisted shape changes. Saved state from the previous release is
not affected because the quarantine lives only in process memory.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:15:40 -06:00
DottaandClaude Opus 5.5 d1d71a7755 test(restart-chaos): staging twin cells, follow-up/queue/subtask grading, and full Daytona cleanup (#21325)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness (`tests/restart-chaos/`) proves that a
release swap (two servers that overlap, then the old one stops) does not
break running agent work
> - The staging twin runs that swap with two local servers and real
Daytona sandboxes, and it is the gate before any hosted proof
> - A loaded twin wave (48 tasks, 20 runs plus 20 queued, 2 swaps)
reported 31 per-task failures and 2 extra turns, but the product did the
correct thing in almost all of them
> - Three harness defects made the noise: the graders called a required
follow-up run an extra turn, called a healthy queued run an orphan, and
called a parent that waited for its subtasks a stall. Cleanup also left
archived Daytona sandboxes in the shared org on every wave
> - This pull request lands the twin harness work and fixes the three
grader defects and the cleanup gap, each with a red/green harness test
> - The benefit is a stop-rule grade that shows only real failures
(here: 1 Retry banner) and makes the follow-up wait visible, and a wave
that leaves no sandbox behind

## Linked Issues or Issue Description

Refs #21175 and #21227 (the server's comment and child-result delivery
rules; this harness takes their outcome from the server's wake record
and does not re-check them).

**What happened?**

Wave `w-20261113T052703-s7301` (loaded twin L1, on Daytona) graded 31
tasks FAIL and listed 2 extra turns. The release-swap grade was 82 / 82
tasks completed without human action.

- t007 and t010 (OpenCode): a person posted a follow-up comment during
the first run (05:44:56 and 05:49:21). OpenCode takes no input mid-turn.
The first run finished `done` at 05:53 and said "No board comments
arrived". The deferred wake was promoted, and the second run answered
the comment after 16 to 17 minutes in the 20-deep queue. The graders
counted the second run as an extra turn and as a `completion_once`
duplicate.
- t021 to t048: each waited about 483 s queued for a slot in the
deliberate 20-deep queue. U1 allows 5 minutes queued plus 3 minutes
without an owner, so all 28 failed `no_owner_no_action`.
- t001: a parent ended its turn `yielded` to wait for its two subtasks.
The observer saw no run for 3 minutes, called it a stall, and rested it
`in_progress`. The parent completed later through its
`issue_children_completed` wake. R1, R3, R5 and U1 failed.
- Cleanup: every Daytona wave left 1 to 8 archived `paperclip-create-*`
setup sandboxes and reported `remainingSandboxes: []`. Daytona does not
delete archived sandboxes.

**Expected behavior**

- A run that starts with a person's comment that no earlier run received
is required. Only a run that starts with no undelivered input is an
extra turn.
- A run queued while every slot of its agent is taken is healthy for the
declared queue time. A run queued while its agent has a free slot still
fails.
- A parent that waits for open subtasks is in a named wait, not a stall.
- Cleanup deletes the sandboxes that this wave created and fails when
one remains.

**Steps to reproduce**

1. Run the L1 loaded twin (command in the README section "Staging twin
cells").
2. Read `staging-mix-summary.md` and `SUMMARY.md`.

## What Changed

- Cherry-picked the 8 twin harness commits from
`fastloop/staging-twin-loaded`: staging-mix under the release-swap
driver on Daytona, `--server-from` on Daytona, probe agents verified
while the first tasks run, image matched to the `--server-from`
checkout, and the knobs `RESTART_CHAOS_K15_KILL_OLD_AFTER_SECONDS` /
`_KILL_OLD_SWAPS`, `RESTART_CHAOS_K15_FREEZE_NEW_SECONDS` /
`_FREEZE_SWAPS`, `RESTART_CHAOS_MIX_BANNER_REST_SECONDS` and
`RESTART_CHAOS_MIX_FIRST_SLOT_SECONDS`. A misplaced doc block in
`supervisor.ts` is moved back to its function. No code comments are
added.
- Cleanup (`daytona-sweep.ts`): after the environment delete, the
harness deletes every sandbox that carries both of this wave's UUID
labels (company and environment), archived ones included.
`assertHarnessMayMutate` still refuses any sandbox without both labels
and any `fleet-sbx-*` snapshot. The `paperclip-create-*` name refusal is
removed, because every plugin sandbox (the fleet's included) has that
name, so the labels are the real ownership proof. A sandbox left after
deletion fails the wave (`undeletedSandboxes`, warm-shutdown verdict,
`ttft-baseline`).
- Follow-ups (`follow-up-delivery.ts`, `wake-evidence.ts`,
`staging-mix-retry.ts`, `staging-mix.ts`): delivery comes from the
server's own record. A comment that the server judged delivered never
becomes a run. A run that the server started from a wake carrying a
person's comment posted after the first run started (a retry included)
is required, not an extra turn. `completion_once` keeps the first
completion as the baseline and does not count a later completion by such
a run (truth-table test). The grader does not re-check the server's
delivery rule; the server's tests cover it. The summary adds `extra
turns: N` and `follow-up answered in a later run: n, queued wait
p50/max, posted to answered p50/max`.
- Declared queue (`universal.ts`, `agent-slots.ts`,
`wave-grade-inputs.ts`): with `--mix-queue` above 0, U1 allows
ceil(queue / runs) × (longest turn + 5 min) queued (27 minutes for L1).
Only queued time while the run's agent has a free slot counts against
the 5-minute bound. Slot use comes from a new
`wake-evidence/agent-slots.json` (run start/finish and each agent's
`maxConcurrentRuns`).
- Subtask wait (`child-wait.ts`, `observer.ts`, `grader.ts`): an idle
parent after a `yielded` or `blocked` turn, until its
`issue_children_completed` wake, is a named wait (`waitingOnChildren`)
for R5 and U1. The observer no longer gives up on a parent while a
subtask is open. An older record that rested during that wait takes the
completion from the wake evidence.
- Review follow-ups: the first staging-mix slot moves earlier without
adding a swap; each sandbox delete pass has a deadline; a frozen
incoming server is always resumed (`server-freeze.ts`); the first
completion is always the baseline for `completion_once` (truth-table
test); a follow-up run that failed before its turn keeps the answer owed
for its retry; `agent-slots.json` saves the server's effective run cap
(`maxConcurrentRunsSql`); the wave-end cleanup flow moved to
`daytona-cleanup.ts` (`main.ts` stays under 1,000 lines).
- README: one-command entries for the twin cell (G, G2, G3) and L1, the
knobs, the stop-rule grading, the new summary lines, the U1 queue rule,
the subtask wait and the cleanup rule.

## Verification

- Red/green tests (red = new tests against the previous source):
- Cleanup, `daytona-sweep.test.ts`: red 4 failed (for example "expected
[] to deeply equal [ 'archived', 'archiving', 'started' ]": nothing was
deleted). Green: pass.
- Follow-ups, declared queue, subtask wait (`staging-mix-retry.test.ts`,
`universal.test.ts`, `child-wait.test.ts`, `wake-evidence.test.ts`): red
6 failed (for example the t010 run listed as an extra turn, U1 fail for
the t030 queue shape, t001 "rested_not_done"). Green: pass. The tests
use the real timings of t001, t010 and t030.
- Full harness suite: `npx vitest run --config
tests/restart-chaos/vitest.config.ts`: 71 files, 811 tests pass. `npx
tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean.
- `pnpm verify` exits 2 (the harness README is a global CI input).
Required CI supplies the broad coverage.
- One-off offline analysis (a scratch script, not part of this PR): the
saved wave `w-20261113T052703-s7301` regraded with these graders, from a
copy of its results:

| | Before | After |
| --- | --- | --- |
| Per-task FAIL | 31 (t001 R1, t007 and t010 `completion_once`, t021 to
t048 U1) | 0 |
| Extra turns | 2 (t007, t010) | 0 |
| Retry banner shown | 1 (t020, 3 s, completed without a click) | 1
(t020) |
| Follow-up answered in a later run | not shown | 4: t001, t007, t010,
t012. Queued wait p50 983 s, max 1739 s. Posted to answered p50 1420 s,
max 2255 s |

t001 and t012 started their follow-up run before completion, so they
were never extra turns. They are listed because the wait is real. This
wave predates `agent-slots.json`, so the regrade used only the declared
queue bound for U1.
- No Daytona wave ran for this PR. There is no live proof here that the
cleanup deletes archived sandboxes in the real org or that the new
evidence file is written on a live wave. The delete path uses the same
`DELETE /sandbox/<id>` call as the Daytona SDK's `deleteSandbox`.

## Risks

- The harness now deletes Daytona sandboxes in an org that the Cloud
fleet shares. The guard is the wave's two freshly minted UUID labels,
filtered on the server and checked again on every row, plus the
`fleet-sbx-*` refusal. A sandbox without both labels is never deleted.
- When the server wrongly re-delivers a comment that a finished run
already received, the harness counts that run as a required follow-up,
not as an extra turn. The server's own delivery tests catch that class,
not this grader. `completion_once` still fails when that run also
reports completion and is not the first.
- The declared queue bound is generous (27 minutes for L1). A stuck
queue with full slots for longer than that still fails. A queue with a
free slot fails after 5 minutes when slot evidence exists. Without slot
evidence only the declared bound applies.
- Test harness only. No product code changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:03:08 -06:00
DottaandClaude Opus 5.5 22c9300dac fix(recovery): a run that fails before the agent is asked anything continues by itself on a fresh computer (#21297)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox opens a provider session and then
waits at a launch gate before the first prompt goes to the provider
> - On remote OpenCode sandboxes the gate saves the provider launch
files; when that save fails, no prompt was ever sent
> - Today that failure ends as `native_session_interrupted` with
ambiguous recovery evidence, so the task goes blocked and the user must
press Retry
> - Nothing was sent to the provider, so nothing can be duplicated: the
system can safely continue by itself
> - This pull request types that failure, proves it from the Runner turn
ledger at the failure point, and continues the task on a fresh sandbox
in the existing interrupted-preparation lane
> - The benefit is that a setup failure before the first prompt no
longer needs a human, and the user sees plain words instead of internal
codes

## Linked Issues or Issue Description

Refs #21287
Refs #21304

**What happened?**
On a remote OpenCode sandbox, the provider launch-files gate failed
before the first prompt (for example
`native_provider_artifacts_transfer_timeout`). The run ended blocked
with a Retry banner. The run error showed internal codes and raw byte
counts to the user.

**Expected behavior**
A failure that happens before any prompt reaches the provider continues
automatically on a fresh sandbox. The task shows as working while the
continuation is pending. The visible text is plain and has no internal
codes or raw numbers.

**Steps to reproduce**
1. Use an OpenCode agent on a Daytona sandbox environment with
controller persistence.
2. Arm the crash point `native.provider_artifacts.before_capture` in
fail mode (or let the launch-file upload time out).
3. Start a task. The first run fails at the gate before any prompt.
4. On master the task goes blocked with Retry. With this change a
successor starts on a new sandbox and the task completes.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- New typed cause `native_failed_before_provider_turn`
(`server/src/services/native-runtime/failed-before-provider-turn.ts`).
The executor raises it only when the launch gate fails, the run was not
cancelled, and the Runner turn ledger proves that no turn command was
issued. The proof is taken at the failure point, before the session
closes. In all other cases the original error is rethrown and today's
path stays.
- The executor quarantines the failed run's session root, so the
successor starts in a new root.
- `heartbeat/predecessor-retirement-fallback.ts` accepts the new cause:
it records the proof on `runnerProfileJson.failedBeforeProviderTurn`,
marks the run's reusable sandbox leases for replacement, and schedules
the successor in the existing `interrupted_preparation` lane (same cap
of 12 and the same backoff).
- `remote-workspace-unchanged-proof.ts` accepts the runner process,
runner command and pre-turn events only when that proof record is
present. It still refuses any turn evidence, an unsettled save by
another run on the same workspace, and setup writes. When the proof
refuses, the run's epoch is abandoned unapplied: the files are kept and
recorded, and the next run restores the last committed save into a fresh
root.
- When the proof refuses and the epoch is abandoned, the replacement
mark stays. The next acquisition replaces the sandbox. #21304's
retention hold makes `retireReplacedReusableSandboxLease` park the old
lease as `retained` instead of `pending_cleanup`, so the task continues
on a fresh sandbox and the sandbox that holds the abandoned files is
kept. When the proof holds (only the Runner's own session folder was
written), the old sandbox is replaced and cleaned up as usual.
- `task-interruption-presentation.ts` shows a pending
`interrupted_preparation` retry as `working` with no Retry until it is
due (plus the dispatch grace).
- The visible run error is now plain: "Setting up the computer took too
long, so the task is continuing on a fresh one." (for a setup timeout)
or "The computer could not be set up before the agent started, so the
task is continuing on a fresh one." The technical cause code stays in
`runnerProfileJson.failedBeforeProviderTurn.technicalCause`, the server
log and the run log.
- New crash point `native.provider_artifacts.before_capture` (fail mode
throws the typed transfer timeout at the gate).
`tests/runner-e2e/source-sdk-public-inputs.json` is refreshed for the
changed registry file.
- `tests/restart-chaos/main.ts`: `RESTART_CHAOS_EXTERNAL_CRASH_CONTROL`
lets an outside driver arm fail-mode crash points without a restart
fault.

## Verification

Local (targeted):
- `server/src/__tests__/heartbeat-native-predispatch-retry.test.ts`: 8/8
pass. New cases: fresh remote sandbox and fresh plugin-task sandbox
continuation (now also asserts the plain visible text with no
identifier-looking tokens, and the technical cause in the proof record),
the ambiguous hold when the ledger cannot prove no turn, and backoff
plus the lane cap.
- `server/src/services/remote-workspace-unchanged-release.test.ts`:
27/27 pass. New cases: hand-on when the ledger proves no turn; refusal
on a recorded turn, an unsettled predecessor save, and setup writes. Two
more cases follow the marked lease through the next acquisition's
replacement decision (`readReplaceBeforeReuseMarker` +
`reusableSandboxReplacementPlan` + `retireReplacedReusableSandboxLease`)
and then through `sweepPendingCleanupLeases`. "replaces the marked
sandbox when the failed run provably left its files unchanged": the old
lease goes to `pending_cleanup` and the sweep destroys it. "continues on
a fresh sandbox and parks the one that holds abandoned files, which
cleanup leaves alone": the old lease is `retained` with the
abandoned-files reason and is held by the receipt, the sweep does not
destroy it, and the file is still there.
- Red without the product change (product files reset to the base, tests
kept): 8 failed, 25 passed of 33. All 8 failures are the new cases. The
visible-text assertion alone was red on the previous commit: the error
read "The computer could not be prepared before the agent started
(native_provider_artifacts_transfer_timeout: the upload of 536870912
bytes did not finish within 120 s). The task continues on a fresh
computer."
- `task-interruption-presentation.test.ts`,
`native-provider-artifact-gate.test.ts`,
`crash-point-call-sites.test.ts`: 79/79 pass.
`packages/shared/src/crash-points.test.ts`: 16/16.
`tests/restart-chaos/crash-points.test.ts`: 12/12.
- `tsc --noEmit` clean for `server`, `packages/shared` and
`tests/restart-chaos`.

Daytona (local servers, Daytona sandboxes, image `sha256:787c3e9c…`,
these commits rebased onto a298ca7fbc, gate failures forced with the
crash point):
- F1, wave `w-20261113T042929-s2301`: 4 OpenCode tasks, one forced
failure on each first run. 4/4 failed runs recorded cause
`native_failed_before_provider_turn`. Each successor (retry reason
`interrupted_preparation`) was created 0.0–0.1 s after the failure and
started 1–3 s later, each on a NEW sandbox. Successor first agent event
at 32.8–56.6 s. 4/4 tasks done hands-off. 0 Retry banners. 0 blocked. No
operator wording. No reconciliation codes.
- F3, wave `w-20261113T043934-s2303`: 1 task, 3 consecutive forced
failures. Retries started 7 s, 35 s and 57 s after creation (backoff).
The 4th run succeeded. 0 Retry banners while pending. Not proven: that
retries 2 and 3 in F3 got fresh sandboxes (proven in F1 only).
- F2 (what bootstrap writes before the first turn): the whole sandbox
workspace was sampled every 5 s on all 4 failed runs, right after the
workspace was ready and at the gate failure. Bootstrap wrote nothing to
project files. All 14 new paths are under
`<workspace>/.paperclip-runtime/paperclip-runner/sessions/<session-sha>/`
(runner state, identity and lineage, `runtime-context.json`, the
instructions copy `AGENTS.md`, OpenCode config, data and log). The only
deletion was the staged agent-files `AGENTS.md` (or a
`.paperclip-upload-<uuid>` temp file). The successor never reuses them:
new session root, new workspace generation, restored from the snapshot.
3 of the files are byte-identical when recreated; the rest are
run-specific and recreated by the successor's own bootstrap. The project
workspace is not a git checkout there. Raw manifests are kept with the
PR author.
- The Daytona runs used the error text before the plain-wording commit.
The wording change is covered by the unit test above.

## Risks

- Scope: this applies only where the gate exists: remote OpenCode
sandboxes with controller persistence. Local runs cannot hit it.
- The proof is the Runner's in-memory turn ledger, read at the failure
point. In unit tests it is mocked. If a turn may have been sent, today's
ambiguous hold stays (tested).
- At the lane cap (12) the run stops with a working Retry instead of
waiting longer. A later change can move this onto the shared hourly
limit.
- A direct "is the workspace dirty" check by the Runner is not added,
because it needs a new Runner command. The unchanged proof instead
relies on the recorded evidence and refuses on any turn evidence,
unsettled saves or setup writes.
- Whether a replacement mark gives a fresh sandbox depends on
`reusableSandboxReplacementPlan`. When it refuses, the sandbox is
resumed, but the session root and workspace root are still fresh.
- Refused case (abandoned files): the task continues on a fresh sandbox,
in a fresh session root and a fresh workspace root. The old files stay
on the parked sandbox, recorded in the abandoned-epoch receipt and
unapplied. That sandbox is kept under #21304's hold until the files are
recovered, discarded, or the retention period ends; this PR does not
change that hold.
- The changes in `heartbeat/` and `remote-workspace-unchanged-proof.ts`
were cleared with their owner under these conditions: evidence-scoped,
latest committed epoch only, files kept when unproven, negatives tested.
- `server/src/services/fresh-continuation-fallback.ts` is deliberately
untouched.
- Upgrade safety: runs saved by an older build have no proof record, so
the unchanged proof keeps its previous behavior for them.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use (shell, file edits, test runs). Earlier commits in this branch
came from Claude sessions in the same fix loop.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:48:51 -06:00
Nicky LeachandClaude Opus 5.5 41998d551c feat(plugins): send the claimant's credential on plugin task resume (#21335)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugin environment drivers run agent tasks on remote computers, and
a warm follow-up run resumes a parked Runner through the `resume` task
operation
> - `resume` carries only the durable identity and a bootstrap ticket,
which is enough while the parked Runner process is still alive
> - A provider sometimes has to start the Runner again to resume the
session: after it moved the parked session to a new sandbox, or after
the computer rebooted
> - A Claude Runner cannot start without its credential, and today only
the portable handoff path (`handoffCredential`) delivers one
> - This pull request lets a provider opt in to receiving the claimant
run's harness credential on `resume`
> - The benefit is that a provider can continue a parked session on a
new computer without Core orchestrating a handoff

## Linked Issues or Issue Description

Refs #21289
Related: #21305

**What existing behavior does this improve?**
Warm resume for plugin task providers that restart the Runner to resume
a session.

**Subsystem affected**
Plugin SDK environment task schema, the server's plugin Runner task
owner, and the plugin environment task docs.

**Current behavior**
`submit` can carry the harness credential (`credential`), but `resume`
cannot. A provider that must start the Runner again on resume has no
credential for a Claude task.

**Proposed behavior**
A provider that returns `resumeCredentialVersion: 1` from `describe`
receives `credential` on `resume` whenever Core would send it on
`submit`, resolved for the claimant run. Other providers see no change.

**Reason and benefit**
Providers can restart a Claude Runner on resume without the Core-run
handoff protocol.

**Breaking changes**
None. The field is opt-in: the SDK worker parses operations strictly, so
Core sends it only to providers that advertise the capability.

## What Changed

- `resume` gains an optional `credential`, with the same schema and
validation as `submit`'s.
- `describe` gains an optional `resumeCredentialVersion: 1` capability.
- The plugin Runner task owner adds the credential on `resume` when the
provider advertised the capability and Core resolved a credential, using
the same resolution as `submit` (`preparePluginRunnerCredentials`).
After a warm rebind, the claimant run's credential is sent.
- `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` documents the field and its
handling rules.
- The runner-e2e SDK public-input pin is updated.

`handoffCredential` on the session handoff path is unchanged.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts` (68 passed): `resume` accepts and
rejects the same 11 credential cases as `submit`; the capability is
optional and rejects other versions; a resume credential reaches the
plugin through the serialized worker RPC.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts` (151 passed): credential sent with
the capability and a resolved credential; omitted without a credential
or without the capability; the rebound claimant's credential is sent;
the credential appears in no metadata write and not in the sanitized
error, even when the worker error echoes it.
- `native-session-executor.test.ts -t "uses a task-only"` (5 passed):
the executor forwards the capability to the owner.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts` (46 passed).
- `tsc --noEmit` for `packages/plugins/sdk` and `server`.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- No server-side provider check on resume, unlike `submit`: the lease
has no reliable harness at resume time. Protection relies on the owner,
whose rebind refuses a changed harness, account or credential identity.
- The credential is the claimant run's current one, which can be newer
than the one that started the parked Runner. That is intended.
- A provider must advertise the capability only after its SDK pin
includes this schema; roll Core out first.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:40:46 +00:00
DottaandClaude Opus 5.5 e063587edd fix(recovery): repair stale pending recovery states and tell the fresh run what the cut-off run already did (#21257)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The assigned-work invariant and its periodic sweep find open tasks
that nothing will continue, and continue them once.
> - The fresh-continuation fallback continues a cut-off run in a new
run, with an opening note about what the cut-off run did.
> - In release-swap waves, the invariant believed some "pending" states
forever, so stalled tasks were reported OK and never repaired.
> - The opening note did not say what the cut-off run had already
created or read, so fresh runs created subtasks again and posted DONE
twice.
> - This pull request bounds the stale "pending" states and adds the
cut-off run's work to the opening note.
> - The benefit is that a stalled task gets repaired within minutes, and
a fresh run does not repeat completed work.

## Linked Issues or Issue Description

Refs #21135, #21203, #21234

**What happened?**
1. The invariant counted a task as pending without limit when its
projection said "system recovery working", or when a wake was deferred
behind a run that had already ended, or when a wake was claimed but no
run was ever created. In release-swap waves, an undelivered continuation
and follow-ups that were never promoted were reported OK for 17 minutes
or more and not repaired. The suspect listing also excluded every task
with a deferred wake, so the periodic sweep did not find them.
2. A fresh continuation's opening note listed only actions with unknown
outcomes. Subtasks, comments and documents that the cut-off run had
completed were not mentioned, and follow-ups it had already read were
not mentioned. Continuations created the same subtasks again and posted
the DONE comment twice.

**Expected behavior**
A stale pending state stops counting after a bounded time, and the sweep
continues the task once. The fresh run knows what the cut-off run
already created and read.

**Steps to reproduce**
1. Run a release-swap wave where a run is cut off after it created
subtasks and posted a comment, and where a follow-up waits behind it.
2. See the continuation create the subtasks again, or see the task stay
"working" with no run while the invariant reports it OK.

## What Changed

- Invariant bounds
(`server/src/services/recovery/assigned-work-invariant.ts`,
`server/src/services/recovery/assigned-work-sweep.ts`):
- "System recovery working" counts as pending only for
`SYSTEM_RECOVERY_WORKING_BOUND_MS` (17 min) after the task's last run
ended: the fallback's progress bound (15 min) plus one disposition
re-check.
- A deferred wake with no live run counts as pending only for 5 minutes
after it was created. A claimed wake with no run counts only for 5
minutes after it was claimed.
- The suspect listing uses the same bounds, so the periodic sweep finds
these tasks.
- A held continuation (paced delivery from #21234) counts as the task's
pending wake only for `PENDING_CONTINUATION_BOUND_MS` (7 min) after it
was recorded: the pacing admits a held continuation within 5 minutes
even when the queue is full (`RECOVERY_CONTINUATION_MAX_HOLD_MS`), plus
2 minutes of 15-second delivery passes. Inside the bound the task still
shows "working" with no Retry, as #21234 intends.
- The sweep's wake still goes through admission, which hands a held task
to the fallback. It never starts a second writer.
- A continuation whose delivery is stuck is repaired, not just no longer
counted (`server/src/services/recovery-continuation-bounds.ts`,
`execution-recovery-resolution.ts`, `fresh-continuation-fallback.ts`):
- One module owns `RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 min) and
`PENDING_CONTINUATION_BOUND_MS` (that plus 2 min). The invariant, wake
admission and delivery all read them from there.
- Delivery re-drives first. A continuation held past 5 minutes was
already admitted when the agent's queue was full, but the delivery page
skipped every record of an agent that had another recovery run queued,
at any age. That exclusion now applies only within the hold bound.
- Admission never replaces the recorded continuation. The recorded
continuation alone carries the isolation and fresh-session decision. A
wake that `hasPendingFreshContinuation` coalesces into it now also
notifies the delivery queue. Delivery then re-drives the same record,
with the same isolation key, `forceFreshSession`, and one-successor
idempotency key.

### Where the isolation decision is read, and every path that starts the
successor

Run preparation reads the decision in one place:
`readFreshWorkspaceReadiness`
(`server/src/services/fresh-workspace-continuation.ts`), called from
`heartbeat/workspace-preparation.ts`. It reads
`contextSnapshot.freshContinuationIsolation`, which names the record,
and loads it through `readFreshContinuationReadiness`. Only a run that
carries that key gets a new isolated workspace.

| Path that can start the successor | How it keeps the isolation | Test
|
|---|---|---|
| Delivery worker (`deliverReconciledExecutions`) | Sends the record's
own wake with `freshContinuationIsolation` and `forceFreshSession` |
`fresh-continuation-fallback.test.ts` "gives the continuation a new
isolated workspace while the old local writer may be alive" |
| Re-drive after the bound | The same delivery wake, admitted past the
hold bound even when the queue is full or another recovery run is queued
| "re-drives an isolated continuation pending past the bound itself; a
repair wake never replaces it"; `recovery-wake-herd.test.ts` "re-drives
a continuation held past the hold bound..." |
| Sweep / monitor / timer wake | Never becomes the successor: coalesced
into the record, which it re-drives | "records a pending continuation
for an ordinary continuation wake and delivers it once"; the re-drive
test above |
| Saved input (a person's message) | Waits behind the isolated
continuation and is promoted when that run is released (#21250) |
`postgres.test.ts` "... (released, continuation pending)" (#21250) |
| A person's Retry started before delivery | Delivery counts the Retry's
run as the successor. The Retry run does not carry
`freshContinuationIsolation`. On a task with a workspace binding, the
fallback cleared the binding, so a new workspace is prepared. On a
projectless task this path is **not yet covered** | "never starts a
second run when a person's Retry already continued the task" (one
successor only). Isolation on this path is a known gap, reported for
follow-up |
- Opening note (`server/src/services/fresh-continuation-fallback.ts`):
- Names the subtasks, comments and document saves that the cut-off run
created, and asks the agent not to create them again. The record keeps
the list as `createdObjects`.
- Lists the person's messages that the cut-off run had read (through
`taskHistoryReadsForRun`, from #21203) before what that run then posted,
so the agent answers a re-delivered follow-up once. Only the revision
that was read counts as read. A message a person edited after the read
is listed as changed ("treat them as new").
- The note is written when the continuation is recorded. Nothing is
added before `turn.start`.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `server/src/services/recovery/assigned-work-invariant.test.ts` (unit)
and `server/src/__tests__/assigned-work-invariant.test.ts` (sweep
integration): stale "working", a stale deferred wake, and a claimed wake
with no run become violations past their bound and stay pending inside
it.
- `server/src/services/fresh-continuation-fallback.test.ts`: "names the
subtasks, comments and documents the cut-off run already created" and
"names the follow-up the cut-off run had read and its own final
comment".
- `fresh-continuation-fallback.test.ts` "re-drives an isolated
continuation pending past the bound itself; a repair wake never replaces
it" (projectless task, writer possibly alive). It was red on b76111e45f,
where the sweep wake started a run of its own without the isolation key.
- `recovery-wake-herd.test.ts` "re-drives a continuation held past the
hold bound even while another recovery run of its agent stays queued".
Red without the delivery change (`expected "vi.fn()" to be called 1
times, but got 0 times`).
- `fresh-continuation-fallback.test.ts` "does not list a follow-up
edited after the cut-off run read it as already read; it is marked as
changed". Red before the fix (the note said "had already read").
- `server/src/__tests__/recovery-wake-herd.test.ts` "reads a task whose
continuation is held back as waiting for its agent...": extended. A held
continuation recorded past the bound no longer counts as a pending wake.
Red without the bound (`expected 1 to be +0`), green with it.

Local command, on this branch based on current master (includes #21222
and #21234):
- `cd server && pnpm exec vitest run
src/__tests__/assigned-work-invariant.test.ts
src/services/recovery/assigned-work-invariant.test.ts
src/__tests__/recovery-wake-herd.test.ts
src/services/fresh-continuation-fallback.test.ts`: 4 files, 144 tests
passed.

CI on cfd42b0516: "General tests (server (1/12))" failed on two
timeouts, and I judged both unrelated to this diff:
- `deferred-execution-receipt-work.test.ts` "dispatches task Retry past
a repeatedly invalidated backlog into real process output": "Expectation
did not complete before its deadline".
- `voice-session-store.integration.test.ts` "retains both spoken sides
once on the public call task and preserves quarantine": it timed out in
15 s, and again on retry.

Why I treat them as unrelated:
- This PR does not touch the task-Retry dispatch, deferred receipt or
voice paths. Nothing in that path calls the sweep, the invariant or the
opening note.
- The same deferred-receipt test fails the same way locally on plain
`origin/master`. On this loaded shared Mac, the real-process wait never
completes.
- Both files passed in the same shard on #21250's CI an hour earlier
(run 43351247913).
- No existing test expectation was changed.

Local release-swap cells:
- R1 (6 tasks, 3 adapters, subtasks and follow-ups, 2 swaps): 4/4
hands-off, 0 banners, 0 duplicates, 0 invariant violations.
- Hour-long release-swap wave (w-20261112T233630-s7201) on the branch
that holds these fixes and the related recovery fixes: 11/11 tasks
finished hands-off (master: 9/11).

## Risks

- A task whose recovery is really working for more than 17 minutes after
its last run gets one sweep wake. Admission hands a held task to the
fallback, so this does not start a second writer.
- The opening note is longer when the cut-off run created many objects.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:35:36 -06:00
DottaandClaude Opus 5.5 7e7452e88f fix(native): keep a reattached Runner alive when its provider recovery deadline ends (#21321)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs on remote sandbox Runners. A same-run reattachment
adopts a live Runner after a failed attempt or a server deploy.
> - On staging, a remote Codex Runner is often gone when the next
attempt reattaches (`process_verification` / `process_not_running`). The
task then continues in a fresh session and loses its context.
> - The deploy does not stop the Runner. The logs show `controller
authority detached for restart; durable runner left alive`, and the new
controller adopts the same Runner after the deploy.
> - Our own cleanup stops it. The adopted Runner answers its recovery
reads slowly while its in-flight turn runs. The 900 s provider recovery
deadline then aborts the recovery, and the abort closes the transport
with `turn.stop` and `runner.suspend` and signals the process.
> - This pull request makes the recovery deadline detach from the
Runner, as a recovery read timeout already does. The server marks the
failure as an unanswered adopted Runner, so the next attempt replaces
the Runner from its saved journal and resumes the same provider thread.
> - The benefit is that a deadline no longer kills a live Runner and no
longer forces a fresh session. The task keeps its session context.

## Linked Issues or Issue Description

Refs #21314, #21272, #21161, #21023

**What happened?**
Codex runs on Daytona remote Runners failed often on staging. In three
sampled chains (attempt 2 of each), the same-run reattachment adopted
the live Runner, authenticated in under 1 s, and then waited the full
deadline: `native session provider recovery timed out after 900000ms`
(909 s, 930 s and 915 s). The deadline cleanup sent `turn.stop` and
`runner.suspend` (`provider turn stop timed out before runner
suspension`). The next attempt found the Runner gone 25–180 s later
(`process_not_running`, `runnerLifecycle: ready`, empty stderr) and
continued in a fresh session. During the 900 s wait, the in-flight turn
was still running Paperclip tools (`tool.execute` spans at 04:43 in a
recovery that started at 04:29).

**Expected behavior**
A remote Runner survives a recovery deadline and a control-plane deploy.
The next attempt continues the same provider session. A Stop still stops
the Runner.

**Steps to reproduce**
1. Start a Codex run on a remote Runner and let a turn start.
2. Fail the attempt so that its close keeps the Runner (for example, a
turn start that outlives its bound).
3. Let the same-run reattachment adopt the Runner, and make the Runner
stop answering recovery commands.
4. Let the provider recovery deadline expire.
5. See that the Runner process is gone and the next attempt reports
`process_not_running`.

**Paperclip version or commit**
master at b555087f0f (after #21314).

**Deployment mode**
Hosted staging (cloud), remote Daytona sandbox Runners, deploys every
20–45 minutes.

**Agent adapter(s) involved**
Native Codex (`paperclip_runner`, Codex app-server through
`paperclip-runnerd`).

## What Changed

- Add `NativeProviderRecoveryDeadlineError` (same message as before) and
use it for the provider recovery deadline in the native session runtime.
- In the Codex driver, a recovery aborted by this deadline detaches the
controller (no `turn.stop`, no suspend, no signal). A Stop and every
other abort still close and stop the Runner.
- On the server, `nativeRecoveryCommandUnanswered` recognizes the
deadline. The failure records `adoptedRunnerUnanswered`, so the next
claim carries `replaceUnansweredRunner`. That attempt fences the old
Runner lineage, launches a fresh Runner from the saved journal and
resumes the same provider thread (the #21272 path). It does not adopt
the slow Runner again for another 900 s.
- Update `doc/execution-semantics.md`.

## Verification

- Real `paperclip-runnerd` + fake Codex
(`runnerd-codex-transport-remote-retention.test.ts`): a retained Runner
is adopted, authenticates, then stops answering. A deadline abort leaves
it alive with no pending `turn.stop`/`runner.suspend`/`runner.shutdown`.
A Stop abort still stops it.
- Red/green: without the fix, the real-runnerd deadline test fails
(`expected 'gone' to be 'alive'`), the driver unit test fails (detach
called 0 times), and the server classification test fails. With the fix,
all pass.
- Measurements on real runnerd: the deadline cleanup took 10.0 s and
killed the Runner before. Now it returns in 0 ms and the Runner stays
alive. The replacement over the kept Runner's journal resumes the same
provider thread in about 2.2 s with one `thread/start` and one
`turn/start` (no replay).
- `npx vitest run
src/live/runnerd-codex-transport-remote-retention.test.ts
src/drivers/codex/codex-app-server-driver.recovery.test.ts
src/native-session-runtime.test.ts` (368 passed).
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "never
answered its recovery read"` (passed).
- `tsc --noEmit` for `packages/paperclip-runner` and `server` (clean).

## Risks

- A deadline now leaves a live Runner behind. The next attempt replaces
it through the launch fence, which fails closed when it cannot prove the
old lineage gone. This is the same risk as the existing recovery read
timeout detach.
- When the provider attempt budget is already spent, the kept Runner is
not replaced by this run. This also matches the existing timeout detach.
- Not in this PR: the upstream trigger. The first attempt abandons a
slow turn start (`runnerd did not report the provider turn identity` or
`turn.start timed out`) while Codex later runs that turn. Two sampled
chains then stopped the Runner on
`protocol_failure:tool_binding_mismatch` / `thread_binding_mismatch`
after a deploy reattachment. These need a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:30:22 -06:00
DottaandClaude Opus 5.5 4fe34df748 Do not show Retry on a parent with open subtasks while its queued message is settled (#21320)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The task view shows a "Retry" notice when a task needs a person to
start it again.
> - The product rule is that a task that continues by itself never shows
this notice, not even for a few seconds.
> - A parent task that waits for its subtasks showed the notice for 3.4
seconds after its run succeeded.
> - The cause is the queued-message grace in the task interruption
presentation: it does not apply to `blocked` tasks.
> - #21200 kept `blocked` out of that grace on purpose, so this pull
request adds only the exact observed shape: a parent with open subtasks
and a queued task message.
> - The benefit is that a parent that waits for its subtasks no longer
shows a false failure, and every other blocked task keeps its Retry.

## Linked Issues or Issue Description

Refs #21264, #21200 (added the queued-message grace and kept `blocked`
out of it)

**What happened?**
A Codex parent task ran across a two-server release swap. Its run waited
for two subtasks and then succeeded with the disposition `blocked`. The
task status became `blocked`. A user comment (a deferred wake, status
`deferred_issue_execution`) had been queued 13 minutes earlier, while
the run was active. For 3.4 seconds after the run finished, the task
presentation was `state: "failed"`, `canRetry: true`. Then the release
drain cancelled the wake as "Queued messages were already delivered to a
completed run", and the presentation changed to `resolved`. The subtasks
finished later. Their `issue_children_completed` wake continued the
parent, and it completed without help.

The cause is in `projectTaskInterruptionState` in
`server/src/services/task-interruption-presentation.ts`. The
`queuedDuringRun` check shows a succeeded run with user input queued
before it finished as `working` for up to 2 minutes. It applies only to
`todo`, `in_progress` and `in_review`. For a `blocked` task the
presentation fell through to the last line: `failed` with `canRetry:
true`.

The workspace finalization deferral
(`native_workspace_finalization_busy`) in the same window does not
change the presentation. The presentation does not read it.

**How this differs from what #21200 narrowed**
#21200 first applied the grace to every status. Its commit `a3fae9bb67`
removed `blocked` because input queued on a blocked task is not promoted
by itself, and because a recovery that gives up hands its hold to the
board and must end with Retry. This change does not add `blocked` back.
It adds one narrow case, and all of these must be true:
- The latest run **succeeded**. A failed, interrupted or given-up run is
not affected.
- The task is `blocked` **and has open subtasks of its own** (not `done`
or `cancelled`). Their completion wakes the parent again
(`issue_children_completed`).
- The pending item is an ordinary **task comment** wake (`commentId` or
`wakeCommentIds` in its payload), not a Retry request
(`taskRetry.sourceRunId`).
- The comment was queued before the run finished, the run finished less
than 2 minutes ago, and saved-work recovery is not exhausted. These are
the existing bounds.

A blocked task that waits for a person or a board decision has no open
subtasks, so it keeps Retry as before.

**Expected behavior**
After the run succeeds, the parent shows `working` with no Retry while
its queued message is settled. Then it shows `resolved` (message already
delivered) or `working` (message promoted to a new run). If nothing
settles the message in 2 minutes, Retry shows.

**Steps to reproduce**
1. Assign a parent task that creates subtasks and waits for them.
2. Post a comment on the parent while its run is active.
3. Let the run succeed with the disposition `blocked`.
4. Read the task presentation in the first second after the run
finishes. It shows `failed` with Retry.

**Paperclip version or commit**
master `bbafa78fe2`. The wave that found it ran on `62d673cc3f`.

## What Changed

- `server/src/services/task-interruption-presentation.ts`:
- The batched wait read also reads whether the wake carries a task
comment (`queuedComment`).
- One company-scoped read finds which `blocked` tasks with a succeeded
latest run and a queued comment have open subtasks
(`waitingOnOpenSubtasks`). It runs only for those tasks.
- The `queuedDuringRun` grace also applies to such a task. The `todo`,
`in_progress` and `in_review` behavior does not change.
- `server/src/__tests__/task-interruption-automatic-path.test.ts`, new
embedded-Postgres tests:
- The exact observed state: a `blocked` parent with two open subtasks, a
run that succeeded 1 second ago, a user comment wake queued 13 minutes
earlier. Shows `working`, no Retry.
- The same parent after the wake is cancelled as already delivered:
`resolved`.
- The same parent after the wake is promoted to a new run: `working` on
the new run.
- Guard: the same parent after the 2-minute grace: `failed` with Retry.
- Guard: a `blocked` task with a queued comment and no subtasks, or only
`done` or `cancelled` subtasks (waiting for a person or a board
decision): `failed` with Retry, unchanged.
- Guard: a parent with open subtasks whose queued wake carries no
comment: `failed` with Retry, unchanged.
- Guard: a parent with open subtasks and a queued comment whose run
failed and whose recovery hold went to the board: `failed` with Retry.
- Guard: a parent with open subtasks and a queued comment whose
saved-work recovery is exhausted: `failed` with Retry.
- A parent whose run succeeded with no queued message stays `resolved`
when its subtasks are open, blocked or cancelled.
- The existing #21200 assertion (a `blocked` task with queued input and
no subtasks shows Retry) is kept unchanged.

## Verification

- `cd server && npx vitest run
src/__tests__/task-interruption-automatic-path.test.ts`
- Red on master `bbafa78fe2`: "shows a parent whose run just succeeded
waiting on its subtasks as working while its queued message is settled"
returns `state: "failed"`, `canRetry: true`. This is the value that the
wave recorded. The other 15 tests pass on master.
  - Green with this change: 16 of 16.
- A plain `blocked` inclusion fails 3 of the guards: the #21200
assertion, "still offers Retry on a blocked task with a queued message
and no open subtasks" and "still offers Retry on a parent with open
subtasks when its queued wake carries no message". This shows that the
narrow condition is necessary.
- `task-interruption-presentation.test.ts`,
`task-interruption-query.test.ts`: green (116 tests with the file
above).
- `cd server && npx tsc --noEmit -p .`: 0 errors.
- Not verified: no new chaos wave. The flash was seen once in a loaded
release-swap wave. The server test reproduces its saved state.

## Risks

- Low risk. The change is presentation only. It does not change wake
promotion, dispatch or Retry.
- A parent with open subtasks and a stranded queued comment now shows
`working` for up to 2 minutes after its run succeeded, and then Retry.
Before, it showed Retry at once.
- One extra indexed read (`issues_company_parent_idx`) runs only for
blocked tasks with a succeeded latest run and a queued comment.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edits, test execution). Extended
reasoning was available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:27:22 -06:00
DottaandClaude Opus 5.5 d9a4bb23fe fix(runner): wait for a slow Codex turn start while it reports progress (#21329)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Codex agents run through the Paperclip runner, and the server-side
transport starts each provider turn
> - After Codex accepts a turn, the transport waits a fixed 30 s for
that turn's `turn/started` before it reports the turn identity
> - Codex starts its MCP servers before it starts the turn, so a healthy
cold start in a loaded sandbox often takes longer than 30 s
> - The transport then failed a live turn with `runnerd did not report
the provider turn identity`, and the abandoned turn's late notifications
broke the reattachment (`did not name its execution owner`,
`runner_harness_state_mismatch: resumed_not_suspended`)
> - This pull request sizes the start window to the measured cold start
and extends it while the provider keeps reporting startup progress
> - The benefit is that slow but healthy Codex starts succeed on the
first attempt, and a dead start still fails at a bound

## Linked Issues or Issue Description

**What happened?**

On staging, Codex runs on Daytona failed at turn start with
`native_session_interrupted: runnerd did not report the provider turn
identity`. The run log shows the same sequence each time. The runner
accepts the turn. Codex then reports startup progress (a bubblewrap
config warning, `account/updated`, and `mcpServer/startupStatus/updated`
failures for an MCP server that is not logged in) for about 30 s. The
transport gives up 30 s after acceptance. A few seconds later, Codex
sends the turn's `turn/started`, which now arrives at a closed turn
(`protocol_failure:turn_binding_mismatch`). The same-run reattachment
then hits that orphaned turn. It fails with `Provider authoritative
notification item/started did not name its execution owner.` or
`runner_harness_state_mismatch: resumed_not_suspended`.

Of 198 recent runs in the staging company, 52 contain this turn identity
failure. In 68 successful starts, submit to `turn/started` took p50 25
s, p75 37 s, p90 50 s, p95 54 s and max 76 s. The fixed 30 s wait sits
near the median.

**Expected behavior**

A slow but healthy Codex start succeeds. A start that makes no progress,
whose Runner exits, or that the runner fails still fails at a bound and
recovers through the normal same-run resume.

**Steps to reproduce**

1. Start a Codex turn whose provider accepts the turn but sends
`turn/started` after the 30 s default window (on staging: several MCP
servers and a loaded sandbox).
2. Let the provider report startup notices during the wait, and let the
runner not prove the accepted turn live in time (on staging no turn was
adopted).
3. See `runnerd did not report the provider turn identity`, then the
reattachment conflict.

The new tests reproduce this with the fake Codex app server
(`--turn-started-delay-ms` with `--turn-started-progress-ms`) and a
runner snapshot that cannot prove liveness.

**Paperclip version or commit**

Staging Core `c730872974` (`2026.1106.0+2253`). Base of this PR:
`origin/master`.

**Deployment mode**

Authenticated, public, managed cloud (staging). Remote runner on
Daytona.

**Agent adapter(s) involved**

`paperclip_runner` with the Codex provider (`gpt-6.1-sol`).

## What Changed

- `runnerd-codex-transport.ts`: the default wait for an accepted turn's
`turn/started` is now 90 s (measured max 76 s), not 30 s. An explicit
`turnStartTimeoutMs` keeps its old bound.
- The wait restarts its window each time the runner commits a provider
notice, or a runner diagnostic that names the accepted turn. The total
wait stops at 180 s (option `turnStartProgressCeilingMs`, so tests can
lower it). A provider relaunch after the accepted `turn.start` does not
extend it.
- The adoption check (runner snapshot proves the accepted turn live) now
runs before each window end, not only before the first one. A Runner
exit and a runner failure of the pending turn still fail at once.
- `fake-codex-app-server`: new `--turn-started-progress-ms` flag sends
`mcpServer/startupStatus/updated` failures while it delays
`turn/started`.
- `doc/execution-semantics.md`: documents the start window, the progress
rule and the staging measurements.

## Verification

- Red on `origin/master` transport (with the new fake flag): `keeps
waiting for an accepted Codex turn while its startup keeps reporting
progress` failed after 4.6 s, and `gives a silent Codex cold start the
measured default start window` failed after 30.9 s. Both failed with
`runnerd did not report the provider turn identity`.
- Green: `npx vitest run
src/live/runnerd-codex-transport-deadlines.test.ts` passes 9/9. This
includes a ceiling test: provider notices every 1 s and no
`turn/started` fail at an 8 s test ceiling, and the test goes red with
the ceiling cap removed. It also includes the existing adoption,
no-adoption-within-bound and runner-failure tests, and a new test that a
silent start still fails within its bound.
- `npx vitest run` on the adoption, cut-turn-start, stall,
turn-start-timeout-retention, remote-retention, restart-continuation and
controller-outage transport suites passes 48/48.
- `npx vitest run src/live/runnerd-codex-transport.test.ts
src/live/live-session.test.ts src/live/turn-stream.test.ts`: 391/392
pass. The one failure (`stops a lost runnerd's provider and escaped tool
job from its lineage journal (journal=true)`, 1 s, process-lineage
timing under full-suite load) passes when run alone, both on this branch
and on `origin/master`.
- `tsc --noEmit` for `packages/paperclip-runner` and `cargo fmt --check`
pass.

## Risks

- A silent dead start (no Runner exit, no provider output, no runner
failure) now fails after 90 s, not 30 s. The Runner exit and
runner-failure paths still fail at once, so this applies only to a
provider that stops without a trace.
- A provider that keeps sending notices but never starts the turn waits
at most 180 s.
- No schema, API or UI change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, browser reads of staging run data).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Refs #21272, #21120

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:21:37 -06:00
Nicky LeachandClaude Fable 5.2 314b893678 fix(native-runtime): deliver assigned gateway tools to every native harness through the PRP relay (#21274)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can have app connections. Paperclip gives the agent the
tools of those apps through an MCP gateway, with its approval and audit
rules.
> - Native harnesses reach the gateway in two ways today. In the relay,
the server keeps the gateway token and offers the tools over the
authenticated Runner channel. In the direct way, the server gives the
gateway URL and a bearer token to the harness.
> - Which way a harness uses comes from a list of special cases that
grew one harness at a time. Local Codex and all ACPX agents use the
direct way. The others use the relay.
> - The direct way puts a run-scoped token in a provider process. A
process that the agent starts can read it. A provider process that
serves more than one run can hold a revoked token.
> - This pull request replaces the list with one rule. Each native
harness gets its assigned gateway tools through the relay.
> - The benefit is one behavior for all native harnesses, and no gateway
token in a provider process.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #13942 (the relay), #17149 (ACPX accepts relayed tools) and #19912
(the design for turn-scoped ACPX MCP authority).

**What existing behavior does this improve?**

How a native harness reaches the tools of its assigned MCP gateway.

**Subsystem affected**

Native runtime in the server (`native-session-executor.ts`,
`native-session-resume.ts`). One Runner test file. One Rust test.

**Current behavior**

- `relayAssignedMcp` is true for plugin tasks, Cursor Cloud, OpenCode,
Codex on a remote target, and Dot.
- For local Codex and for each ACPX agent (Claude, Codex, Cursor, Pi,
Kimi, Grok, Copilot, Antigravity), Core puts
`PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` and
`PAPERCLIP_NATIVE_MCP_TOKEN` in the Runner environment. The ACPX host
gives them to the provider as an HTTP MCP server. The local Codex home
gets them in `config.toml`.
- The token is run-scoped. Core revokes it when the run ends.

**Proposed behavior**

- A native run that has an assigned gateway uses the relay. The
condition names no provider.
- Core builds the gateway tools on the control plane, offers them with
the other Runner tools, dispatches each call over PRP, and authorizes
each call again through the gateway.
- Core removes the three `PAPERCLIP_NATIVE_MCP_*` variables from each
Runner environment.
- The tool contract fingerprint has the relay marker for each target,
not only for remote targets.

**Reason and benefit**

- All native harnesses have the same behavior.
- No provider process holds a gateway token.
- A sandbox does not need to reach the gateway URL of the host.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
- The list of special cases is gone. Core validates the binding and
builds the relayed tools when a binding is present.
- Core does not list the gateway for a review run, Grok Bot or Muse. The
tool authority never offered app tools to those. A gateway error can
therefore not fail those runs.
- OpenCode keeps its `on_demand` catalog. That option depends on the
provider, not on the relay.
  - Core removes the three variables from each Runner environment.
- `server/src/services/native-runtime/native-session-resume.ts`: the
relay marker in the tool contract fingerprint is unconditional.
- `doc/execution-semantics.md`: the text about the tool contract
fingerprint now states the rule, the tool names and the cutover.
- App tool schemas. New module
`packages/paperclip-runner/src/drivers/tool-input-schema.ts` compiles
each schema by itself, in its declared dialect (draft-07, 2019-09 or
2020-12), with no shared `$id` registry. Core uses it when it builds the
relayed tools (`assigned-mcp-tools.ts`). Core does not offer a tool
whose schema has an unsupported dialect or does not compile, and writes
a log entry with the tool name and the reason. The run does not fail.
The ACPX tool bridge and the OpenCode MCP bridge use the same function.
The tools of Paperclip itself stay strict.
- Large arguments. The ACPX sidecar sends the arguments of a tool call
to runnerd with no display limit. The limit is the tool-call admission
limit of runnerd, 256 KiB. A larger call fails with a tool error that
gives the limit. The sidecar does not dispatch it.
- Tests in `native-session-executor.test.ts`,
`native-session-resume.test.ts`, `runtime-context-materializer.test.ts`
and `acpx_provider_backend.rs`.

Not changed in this pull request: the direct code path. It is still in
the Runner, but it receives no binding. #21311 deletes it.

## Behavior that each harness sees

| Harness | Before | After |
|---|---|---|
| Codex on a remote target, OpenCode, Cursor Cloud, Dot, plugin tasks |
Relay | No change |
| Local Codex | Direct: a `paperclip-assigned` server in `config.toml` |
Relay. App tools are dynamic tools named `app_<name>_<digest>` |
| ACPX agents | Direct: a second HTTP MCP server | Relay. App tools are
on the `paperclip` bridge as `mcp__paperclip__app_<name>_<digest>` |
| ACPX with a preconfigured home | No gateway | No change |
| Grok Bot, Muse | Got the variables and ignored them | Do not get the
variables |

- A relayed tool name has a maximum of 64 characters. With the
`mcp__paperclip__` prefix it has a maximum of 80.
- Permission behavior keeps the same intent. For Claude in the
`approve-reads` and `approve-paperclip` modes, app tools ask for
approval, as before. For the other ACPX agents the permission decision
does not use the server name.
- In a planning run or an ask run, the relay offers only tools with read
risk. The direct gateway offered all tools. This is stricter.
- When the catalog is too large, each harness gets the two compact tools
`paperclip_search_assigned_tools` and `paperclip_call_assigned_tool`.
- The four helper tools for resources and prompts
(`paperclip_list_resources`, `paperclip_read_resource`,
`paperclip_list_prompts`, `paperclip_get_prompt`) are not in the relay.
No harness loses them: a run token permits only `tools/list` and
`tools/call` (`run-preparation.ts`), so these four tools already fail
with `gateway_token_action_denied` on the direct path.

## State saved by the previous release

- **Local sessions of each provider start again one time.** The tool
contract fingerprint changes for local targets. The next run sees a
different fingerprint, starts a new provider session with the full task
context in the same workspace, and keeps the old saved state. This is
the existing rotation for a tool contract change. Codex needs it,
because Codex cannot add tools to a thread that it resumes. For the
local providers that already used the relay, the restart is not
necessary. It is accepted so that the rule names no provider.
- **Remote sessions continue.** Their fingerprint does not change. For
ACPX, the attach of the next run replaces the saved tool set.
- **A kept Codex home.** Paperclip writes `config.toml` again at each
start and recovery. The old gateway server and its token are removed.
- **A deploy lands during a run, and the Runner survives.** The
controller attaches to the same Runner for the same run. The old
provider keeps its direct gateway connection. The controller detach
keeps the gateway token of that run, and Paperclip revokes it when the
run ends.
- **A deploy lands during a run, and the Runner is dead.** The Runner
starts again for the same run with its saved tool set and no direct
gateway. The assigned app tools are not available for the remainder of
that one run. The next run gets them through the relay. This applies to
each harness that moves to the relay. It has no test.
- Paperclip refuses no saved state and discards none.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`: `1508 passed
| 7 skipped`, 1 failed. The failed test is "archives a damaged prior
epoch before completing guarded same-task replacement with real runnerd"
(`ENOENT … authority-epochs`). It fails in the same way on `master` at
53608bd1d6 and with the product change removed, on macOS.
- `cd server && npx tsc --noEmit -p tsconfig.json`: no errors.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/runtime-context-materializer.test.ts`: `18 passed`. `npx tsc
--noEmit -p tsconfig.test.json`: no errors.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes. `cargo test -p paperclip-runner-core --lib
acpx_provider_backend`: `50 passed`.

- `npx vitest run src/drivers/tool-input-schema.test.ts
src/drivers/runner-tool-bridge.schemas.test.ts
src/drivers/runner-tool-bridge.test.ts
src/cli/acpx-runtime-sidecar.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: `21 passed (21)` files, `377 passed` tests.
`pnpm run typecheck:typescript`: no errors.
- `cargo test -p paperclip-runner-core --test acpx_event_payload --test
tool_input_schema_fixtures`: 10 and 3 passed.
- Schema tests use one shared fixture
(`tool-input-schema-fixtures.json`) for the TypeScript compiler, the
tool bridge and the Rust `begin_call`: a draft-07 schema, a second
schema with the same `$id`, 2019-09, 2020-12, draft-04, an unknown
dialect, a `$ref` with no target, and an invalid keyword. Before the fix
the bridge failed with `no schema with key or ref
"http://json-schema.org/draft-07/schema#"`.
- A tool schema whose `$id` is the URI of a meta-schema cannot damage
the shared compiler. Before this fix, `removeSchema` deleted that
meta-schema, and later schemas failed to compile for each run in the
process. A test with 7 reserved ids failed before the fix and passes
now.
- Core limits the compile work of each run start. A schema over 512 KiB
is not compiled, and one listing compiles a maximum of 768 KiB of
schemas. These are the limits of the Runner. A tool over a limit is
served only through the search and call tools. Core keeps the outcome
for each schema digest (1,024 entries) and yields to the event loop
before each compile. Four tests failed before this change.
- A sidecar test sends a tool call with 200 KiB of arguments and
receives them byte for byte, refuses a call with 300 KiB with the named
error, and keeps the replacement of a lone surrogate.

Tests, each seen to fail without the product change:

- "keeps assigned MCP credentials on the control plane for %s" now runs
for remote Codex, local Codex, OpenCode and ACPX Claude, Pi, Kimi,
Copilot and Cursor. It checks that Core builds the relay, offers the
tool, dispatches a call, and gives no `PAPERCLIP_NATIVE_MCP_*` variable
to the Runner. Without the change, the five new harness cases fail.
- "relays the assigned gateway when a deploy lands mid-run…" runs for
Claude and Pi.
- The fingerprint tests keep the old local fingerprint as a retained
value and check that the new one is different. The remote fingerprints
for Codex and OpenCode do not change.
- "drops a retained direct gateway server when the relayed launch has no
binding": a Codex home with the old `config.toml` entry loses that entry
at the next start and at a recovery of the same run.
-
`next_run_attach_offers_relayed_gateway_tools_to_a_direct_gateway_checkpoint`
runs for Claude and Pi.

Not run: a live run with a real provider and a real app connection for
each harness. Not run: `pnpm verify` and the full suite. CI supplies
that coverage.

## Risks

- **An ACPX app call has a limit of 256 KiB for its arguments.** The
direct gateway accepted a request body of 10 MB. Codex and OpenCode
permit 480 KiB. To get nearer to 10 MB, runnerd needs a larger admission
limit for tool calls or a protocol that sends arguments in parts. That
is not in this pull request.
- **A tool with a draft-04 or draft-06 schema is not offered.** Ajv 8
does not support draft-04, and the TypeScript and Rust validators must
agree. On the direct path the provider got such a tool.
- **Core compiles app schemas on its event loop.** Before this pull
request Core did not compile them. With the limits above, one valid
schema near 512 KiB can hold the loop for about 0.65 s, and one listing
for about 1.2 s in total (measured on a developer machine). An app that
changes its schemas at each listing gets no help from the cache. A
worker thread with a timeout would remove this; it is not in this pull
request.
- **Tool names change for local Codex and all ACPX agents.** A saved
agent instruction that names a gateway tool directly does not match the
new name.
- **Each local native session starts again one time after the deploy.**
The provider conversation of that session does not continue. The task
context and the workspace stay.
- **A run that is in flight at deploy, with a dead Runner, has no app
tools until it ends.**
- **Names of 80 characters are not tested live** with Kimi, Copilot,
Grok, Antigravity and Codex through ACPX. Pi already shortens a name
longer than 64 characters.
- **Managed Claude and AWS AgentCore now receive relayed app tools.**
This is not tested live.
- **A gateway token expires after 1 hour.** The relay keeps the token on
the controller for the whole run, as OpenCode and remote Codex do today.
A longer run loses its app tools on both paths. This pull request does
not change that.
- **The gateway URL must have the path `/mcp/gateways/<id>`.** With a
path prefix in the API base URL, a run with an assigned gateway fails
with `assigned native MCP gateway path is invalid`. This was true for
the relayed harnesses before. It is now true for all.
- No test covers the rule that Core does not list the gateway for Grok
Bot and Muse.

## Follow-up

The direct path is now unreachable. #21311 deletes it. That pull request
is in the same stack, above the pull requests that change the same ACPX
host files.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff and ran the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated each harness
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 00:16:58 -08:00
DottaandClaude Opus 5.5 fde0328ff5 fix(recovery): deliver every pending continuation within a bounded time (#21324)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Execution recovery continues a task in a fresh run after a restart
or a lost run left it unfinished
> - #21234 paced these continuations: at most 1 per agent and 5 per
company per pass, and a continuation waits while its agent already has a
queued recovery run or a full queue
> - A continuation that waited 5 minutes must enter the queue anyway,
but that bound applied only to the full-queue check. The "agent already
has a queued recovery run" check was in the page query, and it had no
bound.
> - On staging, an agent had queued recovery runs that did not start for
hours. Its pending continuation was never read again, and its task
showed "working" with nothing running or queued
> - This pull request applies the 5-minute bound to both checks, stops
one undeliverable continuation from using its agent's or company's slot,
and stops the task view from showing "working" for a continuation that
is overdue
> - The benefit is that every pending continuation enters the queue
within a bounded time, also after a restart, and the pacing still
protects people from recovery herds

## Linked Issues or Issue Description

Refs #21234, #21135

**What happened?**

On staging, a task had a fresh-continuation hold with
`continuationDelivery: "pending"` since about 03:30Z. More than 3.5
hours later, no run had started for the task. The task view showed the
interruption state `working` with `canRetry: false`, so the user saw
nothing wrong. The task's agent had 20 concurrency slots and 11 running
runs. It also had queued `issue_continuation_needed` runs from 22:54Z
that had not started. The delivery pass reads pending continuations with
a query that leaves out every agent that has a queued recovery run
(`issue_recovery_action_restored` or `issue_continuation_needed`). That
condition had no time bound, so the pass never read this continuation
again. The 5-minute admission bound was never reached. Across the
company, 203 open tasks showed `working` while the API showed no queued
or running run for them. 162 queued runs were more than 1 hour old.

**Expected behavior**

Every pending continuation enters its agent's queue within a bounded
time, also when the agent's queue holds recovery runs that do not start,
and also after a server restart. Pacing still limits how many
continuations enter a queue at the same time. A task does not show
"working" when its continuation is overdue and nothing runs or is
queued.

**Steps to reproduce**

1. Give an agent a concurrency limit of 1.
2. Put one queued `issue_recovery_action_restored` run on that agent
that does not start.
3. Record a pending fresh continuation for another task of the same
agent.
4. Wait more than 5 minutes, and let the 15-second continuation pass
run.
5. See that the continuation stays `pending`, the task shows `working`,
and no run starts.

**Paperclip version or commit**

master at bbafa78fe2 (includes #21234, ade7e0643f)

**Deployment mode**

Hosted cloud (staging)

**Agent adapter(s) involved**

paperclip_runner (native runtime). The problem is in the shared
continuation delivery pass, so it applies to all adapters.

## What Changed

- `deliverReconciledExecutions` in
`server/src/services/execution-recovery-resolution.ts`: a continuation
held for 5 minutes or more is read even when its agent has a queued
recovery run. Before, only the full-queue check let it through.
- Each page of the pass now reads at most one candidate per agent
(`row_number() over (partition by agent)`), so one agent's backlog
cannot fill the page budget and hide other agents' continuations.
- A continuation counts against the company's limit only when its wake
puts a run in the queue. A continuation that was invalidated or replaced
by a newer run does not use its agent's turn.
- When an agent refuses a wake (no run, or an error), the pass skips
that agent for the rest of the pass and stamps the row with
`continuationDeliveryAttemptedAt`. Overdue rows always rank before rows
that are not overdue. Among overdue rows, the turn is the later of the
hold time and the last attempt, so a refused row goes behind rows held
before that attempt and ahead of rows held after it. One refused row
cannot pin its agent, and new arrivals cannot keep an older refused row
waiting.
- New module `server/src/services/recovery-continuation-pacing.ts` holds
the pacing constants and the "held since" rule. It adds
`RECOVERY_CONTINUATION_DELIVERY_BOUND_MS` (15 minutes). The old
constants are still exported from `execution-recovery-resolution.ts`.
- `taskInterruptionPresentations` in
`server/src/services/task-interruption-presentation.ts` shows a pending
continuation as "continuing" only within the delivery bound. After that,
the task shows its real state instead of `working`.
- No change to the schedule: the continuation duty runs only while
pending continuations exist, and it checks the database on start, so a
restart does not lose pending rows.

## Verification

- `pnpm exec vitest run src/__tests__/recovery-wake-herd.test.ts` (in
`server/`): 15 tests pass. New tests:
- A continuation held past the bound is delivered while the agent's
queue holds recovery runs that never start.
- After a restart with a continuation pending past the bound and a full
queue, a new schedule is due within one pass, the pass delivers the
continuation, and the schedule then goes idle.
- Continuations whose wake returns no run do not use the per-company
count. Another continuation of the same agent is delivered on the next
pass.
- An overdue continuation is refused once while the agent's queue is
full. On the next pass, it is delivered even though new continuations
keep arriving for the same agent.
- One agent refuses a backlog of 210 overdue continuations. Another
agent's continuation is delivered in the same pass, and the refusing
agent gets one wake attempt in that pass.
- A task whose continuation is pending past the delivery bound with
nothing running does not show `working`. A fresh pending continuation
still shows `working` with `canRetry: false`.
- Red/green: with the two source files reverted to master (and only the
new constant exported), all 4 first new tests fail. The backlog test
fails on the first PR head (200 refused wake attempts in one pass). The
temporary-refusal test fails on the second PR head (the overdue row
stays pending). With this change, all 15 pass.
- Related suites pass: fresh-continuation fallback, workspace recovery
fallback, native safe replacement, native workspace reconstruction, task
interruption presentation, queued-run dispatch order, execution recovery
resolution, assigned-work invariant, execution control work, task
interruption automatic path and query.
- `node scripts/typecheck-layers.mjs` (in `server/`) passes.

## Risks

- A continuation that waited 5 minutes now enters a queue that already
has recovery runs. The per-agent (1) and per-company (5) limits per pass
still apply, and the queue still starts person-initiated work first.
- An agent that refuses wakes gets one wake attempt per pass. Its
refused rows rotate through `continuationDeliveryAttemptedAt`, and they
do not block other agents.
- After 15 minutes without delivery, the task no longer shows `working`.
With this fix, delivery happens well within that bound, so this is a
safety net.
- No schema or migration changes. No new polling.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, and read-only staging
API requests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:12:13 -06:00
Nicky LeachandClaude Opus 5.5 2483cb46c2 feat(plugins): deliver configured environment to plugin Runner tasks (#21280)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent gets environment variables from its execution environment,
its own configuration, its project, and its routine. Secrets are
resolved for each run.
> - A plugin environment can run the agent as a native Runner task on
its own host, through the environment task contract.
> - That contract has no field for the agent's variables. Runs on a
plugin environment get only the variables that the plugin sets itself.
> - This pull request adds a versioned contract field for the values,
sends them on a session handoff, lets Codex commands inherit the
variables in an external sandbox, and lets a plugin protect the
variables that it sets itself.
> - The benefit is that plain and secret variables work on plugin
environments in the same way as on local environments.

## Linked Issues or Issue Description

Refs #21233. That pull request by @lucktastic finds the same gap and
proposes the same capability and field names
(`configuredEnvironmentVersion: 1`, `configuredEnvironment`). This pull
request uses the same names, so providers implement one contract. It
differs in these places:

- Codex commands get the variables through an external-sandbox denylist,
not through key names sent at `prepare`. A retained Runner does not need
a new preparation when a key is added or removed.
- A session handoff `resume` gets the values, because it starts a new
Runner on another computer. Value changes on the same computer need no
new field: Core's process reuse key includes the values, so Core starts
a new task.
- A driver can declare `protectedEnvironmentKeys`. Paperclip rejects or
warns when one is saved.
- The run log names skipped variables only for an older provider.

Thank you to @lucktastic for the diagnosis and the contract shape.

**What happened?**
An agent has environment variables, plain and secret. Its runs on a
plugin task environment do not get them. `PAPERCLIP_CONFIGURED_ENV_KEYS`
in the run lists only the variables that the plugin sets.

**Expected behavior**
Each run gets the agent's configured variables.

**Steps to reproduce**
1. Set a plain and a secret environment variable on an agent.
2. Assign the agent to a plugin environment that provides Runner tasks.
3. Start a run. Read the names in `PAPERCLIP_CONFIGURED_ENV_KEYS` in a
command that the agent runs.
4. The agent's names are not in the list.

## What Changed

- **SDK contract** (`packages/plugins/sdk/src/environment-tasks.ts`):
- A provider advertises `configuredEnvironmentVersion: 1` on `describe`.
- `submit` gets `configuredEnvironment`. `resume` gets it only for a
session handoff.
- Names, count, and sizes have the same limits as the Runner projection.
- **Server**:
- `createRunnerdBackendWithinSessionClaim` gives the run's configured
projection to the plugin task owner. The projection has no provider
credentials and no generated runtime keys.
- `createPluginRunnerTaskOwner` sends the fields only to a provider with
the capability. An older provider gets no new field, and the run log
names the variables that were not set. The log never prints values.
- **Codex** (`codex-security-config.ts`):
- An externally sandboxed launch uses `inherit="all"`,
`ignore_default_excludes=true`, and an `exclude` list of Codex's own
credentials, instead of an `include_only` list fixed at preparation
time.
- A test reads the Runner's Codex environment keys, and fails if a new
key is not classified.
  - Host launches keep the bounded allowlist.
- **runnerd** (`codex_provider.rs`): `runtime_notification_codex_args`
still grants the notification files to read-only profiles when no
allowlist is present. Plan mode in an external sandbox needs these
grants.
- **Protected keys**:
- An environment driver can declare `protectedEnvironmentKeys` in its
manifest.
- Paperclip rejects a protected key on an agent or environment that uses
the driver. The 422 has code `protected_environment_variable`.
- On a project, a routine, or another agent, Paperclip saves the key.
The response has a `configurationWarnings` notice that names the
environments. The project and routine pages show the notice as a toast.
The agent pages already show configuration warnings.
- **Docs**: `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` describes the
provider obligations.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the reviewed
SDK source pin.

## Verification

- `packages/plugins/sdk`: `vitest run tests/environment-tasks.test.ts`
(38 pass). Accepts bounded values. Rejects bad names, NUL bytes, too
many variables, and oversized values.
- `server`: `vitest run
src/services/native-runtime/plugin-runner-task.test.ts` (95 pass).
Values go only to a provider with the capability, never in `prepare`.
`resume` sends values only for a handoff.
- `server`: `vitest run src/services/protected-environment-keys.test.ts
src/__tests__/project-routes-env.test.ts` (10 pass). Rejection, notices,
and the project route response.
- `server`: all route suites that import the project, routine,
environment, or agent routes (45 files, 1689 pass).
- `packages/shared`: `vitest run src/validators/plugin.test.ts` (64
pass).
- `packages/paperclip-runner`: `vitest run src/drivers/codex
src/configured-environment.test.ts
src/live/runner-provider-launch.test.ts` (444 pass).
- `packages/paperclip-runner/runner`: `cargo test -p
paperclip-runner-core --lib runtime_notification` (9 pass, 2 ignored as
before).
- `pnpm typecheck` in `packages/shared`, `packages/plugins/sdk`,
`packages/paperclip-runner`, `server`, and `ui`.
- Not run locally: `pnpm verify --full` and browser suites. Required CI
covers the final revision.

## Risks

- **Existing providers:** low risk. Core sends no new field to a
provider without the capability. Their runs behave as before, with one
added log line when variables were skipped.
- **Codex in an external sandbox:** commands now see every variable in
the Runner environment except Codex's own credentials. The execution
environment builds that environment from an allowlist. If the Runner
gives Codex a new credential, the drift test fails until the key is
classified.
- **Saving env:** new rejections happen only when an installed driver
declares `protectedEnvironmentKeys`. No driver declares them today.
- **Providers:** they must keep the values transient and never save or
log them. The doc lists these obligations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:11:23 +00:00
DottaandClaude Opus 5.5 e73fc71331 fix(environments): continue on a new sandbox when an idle runner sandbox never starts (#21323)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on the next wake
> - On staging, Daytona left two idle sandboxes in the `archiving` state
for more than three hours, so no resume call could succeed
> - The native runner leases had no verified harness backup, so the run
never replaced the sandbox, even though the task files were saved
> - Each retry spent six minutes on the same stuck sandbox, the bounded
retries ran out, and the task waited for a person to press Retry
> - This pull request continues on a new sandbox when the provider has
proven it cannot start the old one and the workspace is saved, and it
keeps the old sandbox
> - The benefit is that the task recovers by itself and a sandbox that
may hold unsaved runner state is never destroyed

## Linked Issues or Issue Description

Refs #20626, #21113

**What happened?**
Eight Codie runs on staging failed in one hour with
`reusable_sandbox_resume_failed`: `Reusable sandbox lease "<id>" could
not be resumed (cause=transport_timeout); the lease was preserved and no
replacement was created.` They came in two clusters (05:59 to 06:00Z and
06:32 to 06:38Z). Each run took about six minutes, so the 330-second
idle resume budget from #21113 was applied. All runs were already
`transient_failure_retry` wakes.

Investigation of one instance (lease `c23d6120…`, run `d16c272e…`):
- The lease was `released` at 02:37Z with a confirmed stop-and-retain.
No live run held it, so the lease was idle and the extended budget
applied.
- A read-only Daytona API read at 06:5xZ showed the sandbox in state
`archiving` (desired state `archived`), last updated 03:38Z, one hour
after the stop (the auto-archive interval is 60 minutes). The second
sandbox (`e0e6cfe5…`) showed the same state since 03:35Z. Daytona does
not start a sandbox while it archives, so every resume call timed out.
More time or a new HTTP connection could not help.
- The task workspace was clean and had a committed saved snapshot.
- The agent uses `paperclip_runner`. The last run on the sandbox failed
while it saved the runner session, so the lease had no
`nativeHarnessBackup` stamp. `reusableSandboxReplacementIsSafe` requires
a verified harness backup for a runner lease, so the run never replaced
the sandbox. It failed strictly, the bounded retries (2) ran out, and
the task waited for a manual Retry.

**Expected behavior**
When an idle sandbox never starts through the whole provider start
deadline and the task files are saved, the run continues on a new
sandbox automatically. A sandbox that may hold unsaved runner state is
not destroyed.

**Steps to reproduce**
1. Run a `paperclip_runner` task on a reusable Daytona sandbox. Let the
last run end without a harness backup (for example, the runner exits
while it saves the session). The workspace is clean with a committed
snapshot.
2. Let Daytona start to archive the stopped sandbox and get stuck in
`archiving`.
3. Wake the task. Every `environmentResumeLease` call times out, the run
fails after about 330 seconds with `reusable_sandbox_resume_failed`, and
the retries fail the same way.

**Paperclip version or commit**
Staging Core `0c04ff76eb` (current `master` has the same code path).

**Deployment mode**
Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases
on.

**Agent adapter(s) involved**
`paperclip_runner` (native runner) on a Daytona sandbox environment.

**Root cause**
`server/src/services/reusable-sandbox-resume-replacement.ts`: a runner
lease without a verified harness backup was never replaceable, for any
failure. The only other path was to retry the same sandbox, which cannot
work while the provider is stuck. Replacement also always retired the
old lease to `pending_cleanup`, so allowing it without a backup would
have destroyed the only copy of the runner session state.

## What Changed

- `reusable-sandbox-resume-replacement.ts`:
`reusableSandboxReplacementIsSafe` becomes
`reusableSandboxReplacementPlan`. It returns `retire` (same rules as
before), `keep`, or `null`. It returns `keep` for a runner lease without
a verified harness backup only when the caller passes
`providerCannotStart`. All other conditions still apply: idle lease,
same workspace, clean workspace, committed snapshot, this run is the
writer, and no access refusal.
- Add `keepReplacedReusableSandboxLease`. After the replacement lease is
recorded, one guarded write marks the old lease with
`reusableSandboxKeptAfterReplacement`. The lease keeps its status and
its sandbox. Nothing destroys it.
- `environment-runtime.ts`: set `providerCannotStart` only when the idle
start deadline was extended and the run then used all of it on timeouts.
Lease selection skips a lease with the keep marker, so later runs do not
spend time on it and obsolete-lease cleanup does not destroy it.
- The replacement records `replacement` / `resume_failed` with no
harness backup. The native executor already starts a fresh session in a
new sandbox in this case (`freshReplacement`), and workspace preparation
restores the committed snapshot.
- The keep plan needs every resume call to time out. A rate limit, a
reset, or any other failure in between keeps the strict behavior.
- `reusable-sandbox-kept-marker.ts`: the marker key and a SQL guard.
Every automatic teardown path skips a marked lease: retained-orphan
listing and settle, `destroyReusableSandboxLease` (obsolete cleanup,
expiry, run destroy), `destroyClaimedSandboxLease` (issue and workspace
teardown), recovered-workspace cleanup, and abandoned-file release. Only
an operator-consented environment delete still destroys it.
- `doc/execution-semantics.md`: document the rule.

## Verification

- Red before the fix: the new staging-shape test in
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`
(released runner lease, no harness backup, clean committed snapshot,
every resume times out) failed with the staging error `could not be
resumed (cause=transport_timeout); the lease was preserved and no
replacement was created.`
- Review round: "keeps an idle sandbox with no harness backup when some
resume calls failed for another reason", "never destroys a sandbox kept
after replacement through the run or workspace teardown paths", and the
retained-orphan case "keeps a warm sandbox while a replacement kept it"
each failed before the fix and pass after it.
`environment-retained-orphan-sandbox.test.ts`,
`abandoned-workspace-files-hold.test.ts`,
`workspace-recovery-preparation-gates.test.ts`,
`workspace-recovery-admission.test.ts`: all pass.
- Green after the fix: 22/22 pass in the replacement suite. The run gets
a new sandbox within 330 seconds, makes no destroy call, the old lease
stays `released` with the keep marker, and the next run resumes the new
sandbox only. With unsaved work or no saved snapshot, the run still
fails strictly and does not mark the old lease. Earlier tests (a refusal
without a backup, a live owner, a dirty workspace with a timeout) are
unchanged.
- `pnpm exec vitest run src/__tests__/environment-runtime.test.ts
src/services/environment-runtime-reusable-lease-probe.test.ts
src/__tests__/reusable-leases-default.test.ts`: 215 pass.
- `tsc -p server/tsconfig.json --noEmit` passes.
- Not run: a live replacement on staging. It is verified after deploy on
the next wake of the affected tasks.

## Risks

- A runner task that hits this path starts a fresh runner session. The
files come from the saved snapshot, and the task, comments, and run
history are kept. Before this change, the task stopped and waited for a
person.
- The kept sandbox stays with the provider (archived storage) until
someone removes it. This is the cost of never destroying unsaved runner
state.
- The rule needs the full extended start deadline (about 330 seconds) of
timeouts, so a slow but healthy start still resumes the same sandbox.
- No schema change. Old leases follow the same rule.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1113.0-canary.4
2026-11-13 02:06:25 -06:00
Nicky LeachandClaude Fable 5.2 503a736195 feat(ci): read a registry build cache in the Runner image build (#21310)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - `.github/workflows/runner-image.yml` builds the Runner alone and
publishes it from `master` as `ghcr.io/paperclipai/paperclip-runner`,
one time for each set of inputs
> - The build used no cache, so every run compiled the daemon again. The
first publication took 25m46s from push to tag
> - The toolchain and the daemon change only with the Rust sources. Most
commits that change the Runner's inputs do not touch them
> - This pull request lets the build legs read a registry build cache
for those stages. Only a new job can write the cache, and only for a
`master` push
> - A weekly run builds with no cache, so the cache cannot hide a broken
build for long
> - The benefit is about 3 minutes less on each build leg when the Rust
sources did not change, with no change to the published image

## Linked Issues or Issue Description

Refs #21230 (publication of the Runner image), #21229 and #21204 (the
build this change makes faster). No issue exists for this change.

**What existing behavior does this improve?**

The Runner image build in `.github/workflows/runner-image.yml`. It
builds the same image in less time.

**Current behavior**

Each build leg starts from an empty builder. It installs the Rust
toolchain and compiles the daemon in every run. In the first `master`
publication (run 43365670428) the amd64 leg spent 7m22s in the build
step. The toolchain and the cargo build took 3m26s of that.

**Proposed behavior**

The build legs read
`ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>` for the stages
up to the compiled daemon. A new `cache` job writes that cache after a
publication. A scheduled run builds with no cache one time each week.

**Reason and benefit**

The stages before the last one read the Rust sources, the protocol files
and the toolchain pin. In the last 30 days, 438 `master` commits changed
an input of the image, and at most 181 of them changed a file those
stages read. For the other commits, and for each pull request that does
not touch those files, the build step is about 3 minutes shorter on each
leg.

**Breaking changes**

None. The `Dockerfile` and each file in the image identity are
unchanged. The published digest for a set of inputs does not change.

This is part 4 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build (this pull
request)

This pull request uses no code from parts 1 to 3. It is stacked on them
because all four parts edit `doc/runner-image-contract.md`.

## What Changed

- The build legs pass `--cache-from
type=registry,ref=ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>`
to the first build command. They do not log in. Their permissions stay
`contents: read` and `packages: read`.
- A new `cache` job writes the cache with `--cache-to
type=registry,...,mode=max` and `--output type=cacheonly`. It runs only
for a `master` push in the canonical repository, for inputs that had no
image. Its permissions are `contents: read` and `packages: write`. It
has no `id-token` and no `attestations` permission.
- The `cache` job needs the `publish` job, and no job needs the `cache`
job. A cache write cannot delay or fail a publication.
- The `cache` job builds the target `runner-artifacts-runnerd` and no
later stage. The last stage takes the commit as a build argument, so no
other commit can reuse it. The install and its dependency lifecycle
scripts are in that stage, so the job with the write token does not run
them.
- The `cache` job has one concurrency group for each architecture, a job
time limit and a step time limit.
- A `schedule` trigger starts a weekly build. It passes `--no-cache` to
the first command and reads no cache. It runs all checks and publishes
nothing. A fork's schedule runs nothing.
- `server/src/__tests__/runner-image-workflow.test.ts` pins the new
security properties: which jobs hold `packages: write`, that the build
legs have no credential and cannot write the cache, the conditions of
the `cache` job, that the `publish` job does not depend on it, and that
a scheduled run cannot publish or write the cache.
- `doc/runner-image-contract.md` has a "Build cache" section.

## Verification

Tests, on macOS (Node 26.12.0) and in a `node:24.22.3-bookworm`
container (Linux arm64, Node 24.22.3):

- `pnpm exec vitest run src/__tests__/runner-image-workflow.test.ts
src/__tests__/runner-image-identity.test.ts` in `server/`: 55 of 55
tests pass on both.
- `node --test .github/scripts/tests/*.test.mjs`: 807 of 807 tests pass
on both. The container needs `jq`.
- `pnpm verify` exits 2 for this change, because a workflow file is a
global CI input. Required CI supplies the broad coverage.
- Each new assertion fails when the workflow breaks its rule. I checked
seven such edits, for example: the `publish` job needs the `cache` job,
the `cache` job gets `id-token: write`, a build leg gets `--cache-to`.

Cache mechanics, on linux/arm64 with a local registry, BuildKit v0.34.2
(the version the hosted runners use), and an empty builder for each
build:

| Build of both variants | Time | Result |
| --- | --- | --- |
| No cache, cache ref does not exist | 260 s | An importer error in the
log, exit 0 |
| One changed TypeScript file, with the cache | 149 s | Toolchain and
cargo steps cached |
| Unchanged inputs and a new commit, with the cache | 129 s | Toolchain
and cargo steps cached |
| Each stage in the cache (a whole-build cache, which this change does
not write) | 16 s | Both OCI archives complete |
| `--no-cache` on the first command | 272 s | No cached step in the
first command. The second command reuses all of it |

- The images built from the cache pass `runner-image-layout.mjs check`
and `runner-image-oci.mjs check-layers`. The daemon has the same SHA-256
as the daemon of a cold build.
- A cache ref that does not exist, a host that refuses the connection, a
name that does not resolve, and the real
`ghcr.io/paperclipai/paperclip-runner-buildcache:arm64` (which does not
exist yet) each give an importer error in the log and a complete build
with exit 0.
- The `cache` job command on an empty builder with the cache present
takes 6 s. It downloads no layer.

Not tested before merge:

- The `cache` job. It runs only on `master`.
- The scheduled run. It starts only from the default branch.
- The cache package. It does not exist. I did not test a private
package, which answers an anonymous read with an authorization error. I
expect the same importer error and a cold build.
- A fully cached build on the hosted runners. The local test used the
same BuildKit version and a different buildx version (0.37.0, and the
runners have 0.38.1).

## Risks

- **A second job can write packages.** Before this change, only the
`publish` job held `packages: write`, and it runs no build. Now the
`cache` job holds it too, and it runs BuildKit on the repository's
`Dockerfile`. GitHub cannot limit that token to one package, so it can
also write to `ghcr.io/paperclipai/paperclip-runner`. The limits are:
the job runs only for a `master` push in the canonical repository, the
token goes only to the registry login, the build steps run inside
BuildKit without it, the job does not run the install, and the job
cannot sign. An image that it wrote has no attestation.
- **Part of a published image can come from an earlier run.** The daemon
in a published image can come from a step that an earlier `master` run
compiled and cached. The attestation names the run that published, not
the run that compiled. BuildKit reuses a step only for the same
instruction and the same file content, but a party that can write the
cache package can replace what a later build reuses. Write access to the
cache package must stay as narrow as write access to the image package.
- **A cached download is not repeated.** The toolchain installer, the
toolchain and the crates are not downloaded again until the inputs of
their step change. The weekly cold build finds a download that is gone.
It does not compare its output with the cache.
- **A registry failure after the cache import fails the leg.** A cache
that cannot be read at the start is not a failure. A layer download that
fails later is a failure. Run the job again.
- **The cache package starts private.** GitHub creates it on the first
write. A maintainer must make it public one time. Until then each build
runs cold and nothing fails.
- **The first cache write waits for an input change.** The merge of this
pull request changes no input of the image, so its `master` run builds
nothing and writes no cache.
- **A cancelled cache job marks its run as cancelled.** When three cache
writes for one architecture wait at the same time, GitHub cancels the
middle one. The image of that run is published.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, through Claude Code.
Tool use and shell execution. It wrote the change, ran the local builds
and tests, and wrote this description.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:46:15 -08:00
Nicky LeachandClaude Opus 5.5 4f4d823cfc feat(environments): forward the warm idle window to plugin lease acquisition (#21305)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugin environment drivers run agent tasks on remote computers, and
warm leases keep a parked Runner between runs for an idle window that
Core records
> - Some providers stop an idle computer on their own schedule (for
example a sandbox auto-stop interval)
> - If that provider timer is longer than the Runner idle window, the
provider keeps paying for a computer whose Runner is already gone; if it
is shorter, a warm resume can find the computer stopped
> - The plugin cannot align the two because it never sees the idle
window at acquire
> - This pull request forwards the warm idle window to
`environmentAcquireLease` as `runnerLifecyclePolicy`
> - The benefit is that a provider can stop its computer just after the
Runner idle window, without Core knowing anything about the provider's
placement

## Linked Issues or Issue Description

Refs #21289
Refs #21066

**What existing behavior does this improve?**
Warm lease setup for plugin environment drivers.

**Subsystem affected**
Server environment runtime (`createPluginEnvironmentDriver`) and the
plugin SDK acquire parameters.

**Current behavior**
Core records the warm lifecycle policy (`runnerTaskLifecyclePolicy`,
including `idleTimeoutMs`) on the lease, but `environmentAcquireLease`
does not receive it, so a provider with its own idle shutdown has to
guess.

**Proposed behavior**
For a warm lease, `environmentAcquireLease` receives
`runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs }`, the same value
Core records. Other lifecycle modes do not receive the field.

**Reason and benefit**
A provider can align its idle shutdown with the Runner idle window.

**Breaking changes**
None. The field is optional and additive.

## What Changed

- `PluginEnvironmentAcquireLeaseParams` has a new optional
`runnerLifecyclePolicy`.
- Core sends it only for warm leases.
- `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` documents the field.
- The runner-e2e SDK public-input pin is updated for the SDK change.

An earlier revision of this PR also made Core try
`environmentResumeLease` before a session handoff. That part is removed:
placement is the plugin's decision, so Core should not decide to retry
the same computer before moving the session. The plugin will own that
inside its own resume instead.

## Verification

- `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed.
- `vitest run src/__tests__/environment-runtime.test.ts -t "portable
task placement|delegates plugin environment leases with acquisition
budget"` passed (5 cases). The portable test asserts the first warm
acquire receives `runnerLifecyclePolicy` and the lease records
`runnerTaskLifecyclePolicy`; the non-warm delegation test asserts the
field is absent. Making the forwarding unconditional fails both
delegation cases.
- The full suite was not run locally; required CI covers it.

## Risks

- A plugin that does not know the field ignores it.
- Rollback: revert this pull request. No migration.

Note: the branch name still reflects the earlier scope (resume before
handoff).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:45:31 -08:00
DottaandClaude Opus 5.5 86f9555695 fix(native): give a Runner whose first unanswered read came on the last attempt one replacement attempt (#21322)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner, and a same-run retry
reattaches that Runner after an interruption
> - #21272 made the next attempt replace a Runner that left its recovery
read unanswered, instead of adopting it again
> - On staging, three runs after #21272 went live still failed: their
first unanswered read came on the third and last attempt, so no next
attempt existed
> - This pull request gives such a run one more attempt, used only for
that replacement
> - The benefit is that the replacement from #21272 can run when a
restart used up the middle attempt

## Linked Issues or Issue Description

Refs #21272, Refs #21161

**What happened?**

After #21272 (497ccba8f1) went live on staging at 05:34Z, three Codex
runs still failed with `native_session_recovery_failed:
HarnessCapabilityUnavailableError: read is unavailable: Error:
provider_transport_failed: PRP command session.snapshot timed out
(runner connected; pending ahead 0; last runner event seq … 55–84s
ago)`. The run events of all three show the same sequence:

1. Attempt 1 fails at 06:24 with `runnerd did not report the provider
turn identity`.
2. Attempt 2 starts at about 06:25. The 06:24–06:25 server restart ends
it with `local_runtime_api_listener_unavailable`. This attempt sends no
recovery read.
3. Attempt 3 adopts the same Runner. Its first recovery read gets no
answer. Attempt 3 is the last of three, so the run ends.

`adoptedRunnerUnanswered` was recorded on attempt 3, but no next attempt
read it. The replacement from #21272 never ran.

**Expected behavior**

A Runner whose first unanswered read comes on the last attempt is still
replaced once by a fresh Runner from the saved journal. The provider
turn is not sent again. The run does not end only because a restart used
the middle attempt.

**Steps to reproduce**

1. Start a Codex run in a remote Daytona Runner. Let attempt 1 fail at
turn start, so the Runner stays alive.
2. Restart the server during attempt 2, so that attempt fails without a
recovery read.
3. Let the Runner leave its recovery read unanswered on attempt 3.
4. See the run end with `session.snapshot timed out` and no replacement.

**Paperclip version or commit**

Staging with #21272 (497ccba8f1). Fixed on `master` after `bbafa78fe2`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`).

## What Changed

- `nativeUnansweredRunnerReplacementDisposition` (executor): when
attempt 3 fails because its adopted live Runner left a recovery read
unanswered, and that attempt was not itself a replacement, the failure
stays `retryable_failure` (`native_session_interrupted`, retry in 30 s)
instead of `native_session_retry_exhausted`.
- `claimNativeRestartRecoveries`: a failed remote retry at attempt 3
with `adoptedRunnerUnanswered` is not treated as an exhausted budget.
The claim is `reattach_remote_runner` at provider attempt 4 with
`replaceUnansweredRunner`.
- `nativeAdmittedProviderAttemptLimit` (executor admission gate): a
`reattach_remote_runner` claim with `replaceUnansweredRunner` is
admitted up to provider attempt 4. Every other claim keeps the limit of
3.
- `verifyRemoteRunnerOrSettledExit`: the replacement of a live Runner is
allowed up to attempt 4. An exited Runner keeps the fail-closed path.
- Bound: the extra attempt is a replacement (`replaceUnansweredRunner`),
so it never gets another one. Attempt 4 is exhausted by the existing
rule.
- `doc/execution-semantics.md` describes the extra attempt.

## Verification

- End to end (embedded Postgres):
`native-remote-retry-admission.test.ts` › "executes the one replacement
attempt after the last attempt's first unanswered recovery read, and no
further one". It seeds attempt 3 with the staging failure, claims it,
and executes the claim through `executePaperclipNativeSession`. The
provider is reached once, the run then ends at attempt 4, and no further
claim is made. With the old `nextAttempt > 3` gate this test fails
before the provider is reached.
- Red on `master`, green with the fix (7 tests, including "admits
provider attempt 4 at execution only for the replacement claim"):
- `native-runner-restart-recovery.integration.test.ts` › "gives a Runner
whose first unanswered read came on the last attempt one replacement
attempt: true" (embedded Postgres; the coordinator is seeded at attempt
3 with the staging message). On `master` no reattach claim was made.
- `native-session-executor.test.ts` › "bounds the replacement after …"
(4 cases: the last attempt's first unanswered read is retryable; a
replacement that also went unanswered, another failure, and attempt 4
stay terminal) and "replaces the live Runner on the extra replacement
attempt".
- Focused suites: `native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation`: 1737 passed, 21 skipped.
- `pnpm typecheck` in `server` passes.
- Not run locally: the container-gated remote Runner suites and the full
suite (CI covers it). Why the remote Runner stops answering is a
separate investigation and is not changed here.

## Risks

- Low to medium. This adds at most one provider attempt per run, and
only after an unanswered recovery read on a live adopted Runner. The
attempt replaces the Runner through the launch fence, which proves the
old Runner and its lineage gone or fails closed. No provider turn is
sent again.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events),
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:27:35 -06:00
DottaandClaude Opus 5.5 49f0030f06 fix(heartbeat): rank a person's joined comment first and refill free run slots (#21277)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts each agent's queued runs, up to the
agent's `maxConcurrentRuns`
> - #21234 starts person-initiated runs before system recovery runs, and
it reads "person-initiated" from the run's own wake request
> - When a person comments on a task that already has a queued system
run, the comment joins that run. The run keeps its system wake, so
dispatch ranks the person's message as system work
> - Dispatch is event driven. The queued-run re-drive backstop covers
only agents with no live run, so an agent with several slots keeps a
free slot empty until one of its runs finishes, and runs can take hours
> - This pull request makes dispatch read the person wakes that joined
each queued run, and makes the re-drive backstop cover an agent that
still has a free slot
> - The benefit is that a person's comment starts first, and free slots
fill within about a minute instead of waiting for an unrelated run to
finish

## Linked Issues or Issue Description

Refs #21234, #20986

**What happened?**

On staging, one agent had `maxConcurrentRuns` 20, 18 running runs and 76
queued runs at 03:12Z. No run started on that agent from 03:01:33Z to
03:16:57Z (15.4 minutes). From 03:11:10Z, after a run finished, at least
2 slots were free. Other agents in the same company started runs during
that time. Runs started again only after the next deploy restarted the
server.

Five queued runs came from a person: comments at 02:43Z, 02:44Z and
02:45Z, and Retries at 02:46Z, 02:49Z and 03:01Z. The comment runs had
the wake reasons `issue_reopened_via_comment` and `issue_commented`, the
invocation source `automation`, and no person on the run's own wake
request.

**Expected behavior**

A run that carries a person's comment ranks as person-initiated, also
when the comment joined an existing queued run. When an agent has a free
slot and a queued run is ready, the run starts promptly, without waiting
for another run on the agent to finish.

**Steps to reproduce**

1. Give an agent `maxConcurrentRuns` 3. Start 2 runs on it.
2. Queue a third run while no dispatch signal reaches the queue (for
example, a claim declines it while a gate is still closed, and the gate
then clears).
3. See that `listUndispatchedQueuedRuns` returns nothing, because the
agent has a live run. The run stays queued until one of the 2 running
runs finishes.
4. Separately, queue a system run on a task, then post a person's
comment on the task so that the comment wake coalesces into that queued
run. Queue older system runs on other tasks of the same single-slot
agent.
5. See that dispatch starts the older system runs first.

**Root cause**

1. `coalesceIntoActiveExecutionRun`
(`server/src/modules/wake-queue/adapters/postgres.ts`) merges the
person's comment context into the queued run and records the person's
wake as a separate `coalesced` row with `runId` set to that run. The
run's own `wakeupRequestId` still points to the system wake.
`queuedRunDispatcherForAgent` (`server/src/services/heartbeat/queue.ts`)
only read the wake behind `wakeupRequestId`. The task routes do record
the comment author as the requester (`requestedByActorType: "user"`).
The person was lost only at dispatch time.
2. `listUndispatchedQueuedRunPage`
(`server/src/services/queued-run-dispatch-liveness.ts`) skipped every
agent with any `running` or `scheduled_retry` run. That rule assumes one
slot per agent. For an agent with several slots, a lost or declined
dispatch is not re-driven until a run on that agent finishes.

This change is not in `claimQueuedRun`, the fresh-isolated preflight,
the wake-queue promoter or `fresh-continuation-fallback.ts`.

**Paperclip version or commit**

Staging Core `ddc0aacad8` (contains #21234 `ade7e0643f`). This branch is
based on master `6cb0dd0c6b`.

**Deployment mode**

Authenticated, public, Paperclip Cloud managed staging.

**Agent adapter(s) involved**

`paperclip_runner` (Codex native runtime). The fix is
adapter-independent.

## What Changed

- `queuedRunDispatcherForAgent` also reads person wakes whose `runId` is
a queued run of the agent (by `run_id IN (queued run ids)`, so an
earlier saved comment that a run adopted is included). A run with such a
wake ranks as person-initiated and is never counted as system recovery.
- `listUndispatchedQueuedRuns` now returns a queued run when its agent
has fewer `running` plus `scheduled_retry` runs than its
`maxConcurrentRuns`, and its task has no `running` or `scheduled_retry`
run. The re-drive backstop and the ownership I7 check use this query.
- The query counts each agent's live runs once, in one grouped subquery
that is joined to the queued runs.
- Migration 0476 adds `agent_wakeup_requests_company_run_idx`
(`company_id`, `run_id`, partial on a non-null run) with `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`. It is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` and classifies as
`online_compatible`. `doc/DATABASE.md` describes it.
- Added `maxConcurrentRunsSql()` to `timer-policy.ts`. It reads the same
limit as `parseHeartbeatPolicy`, including the single-run runner
providers, which now share one constant.

## Verification

- Red/green: `server/src/__tests__/recovery-wake-herd.test.ts` "starts a
queued run that a person's comment joined ahead of older system work".
It fails on master (`expected null not to be null`: the older system run
started first) and passes with this change.
- Red/green:
`server/src/__tests__/ownership-reconciliation-report.test.ts`
"re-drives a queued run while its agent still has a free slot, and not
once every slot is running". It fails on master (`expected [] to deeply
equal [ Array(1) ]`) and passes with this change.
- Red/green: "starts a queued run that adopted a person's comment saved
before every queued run". It fails with a `requestedAt` bound and passes
with the `run_id` lookup.
- Red/green: "counts an agent's live runs in one grouped aggregate, not
once per queued run" runs EXPLAIN on the page query. It fails with a
per-row correlated count and passes with the grouped subquery.
- New: "uses the agent's single-run limit for a runner provider that
runs one turn at a time".
- Two existing tests now set `maxConcurrentRuns: 1` so that they keep
testing "the agent's only slot is busy".
- `npx vitest run src/__tests__/recovery-wake-herd.test.ts
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/ownership-reconciliation-report.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/services/heartbeat/dispatch-work.test.ts`: 71 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: no errors.

## Risks

- The re-drive backstop now also covers multi-slot agents with live
runs. Each queued run is re-driven at most on its existing backoff (1
minute, then doubling to 15 minutes). A queued run whose task already
has a live run is not re-driven.
- The ownership I7 report can now list a queued run on a multi-slot
agent that has a free slot. This is the stall that this change targets.
- One more indexed read per dispatch pass (`agent_wakeup_requests` by
company and run id). The new index is built concurrently, so the
previous release keeps writing during the build.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, read-only staging API reads in a
browser tab).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:17:57 -06:00
DottaandClaude Opus 5.5 74096c19f4 fix(evals): admit the in-flight chat completion follow-up on a settled source (#21313)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that a chat request reaches the correct
saved outcome
> - The `source-settled-with-active-work` run policy checks that the CEO
chat is settled while delegated work can continue
> - After a delegated task finishes, the product starts its normal
`chat_task_completed` follow-up on the source chat
> - The policy required every source run to be `succeeded` and the
source to be `waiting`, so a re-grade during that follow-up failed
> - This pull request admits that one product follow-up while it is in
flight, and keeps every other source run strict
> - The benefit is that the eval grades the saved private-delegation
outcome, not the timing of the reload

## Linked Issues or Issue Description

**What happened?**
The OpenCode cell of
`chat-features.runner-opencode.local.tasks.private-task` failed with
"Independent oracle survives browser reload: expected true, received
false". The saved state was correct. The CEO answered, Morgan saved the
private draft in the private project, and the unrelated member could not
read it. The grade passed for its full 5-second stability window.
Morgan's run then finished, and 73 ms later the product started the CEO
`chat_task_completed` follow-up on the source chat. The harness reloaded
the page and re-graded while that follow-up was running. The only failed
check was `catalog-settled-source-separate-work`, because it required
all source runs to be `succeeded`.

**Expected behavior**
The case already counts this follow-up in its run bound
(`expectedRunCount: 3`, `minimumRunCount: 2`). A queued or running
product follow-up on an answered source must not fail the case.

**Steps to reproduce**
Run the private-task case. Let the worker finish within about 5 seconds
of the first passing grade. The post-reload re-grade sees the follow-up
as `running` and fails.

## What Changed

- Add `chatCatalogSourceCompletionFollowUp` to
`tests/runner-e2e/chat-catalog-run-policy.ts`. It identifies the lead's
`chat_task_completed` run on the source whose every completion update
names separate work in the run ledger.
- In the `source-settled-with-active-work` check, admit at most one such
follow-up while it is `queued`, `running` or `succeeded`. Accept a
source `conversationState` of `active` only while that follow-up is in
flight.
- Keep the other rules strict. The original answer must exist and must
have succeeded. Any other active, failed or extra source run still
fails. The declared run-count bound still applies.
- Also admit the follow-up before its turn is prepared. The server
claims the run first and saves the completion updates after that, so a
queued run, or a running run in preparation, has only its delivery IDs.
It qualifies only while queued or running, when automation woke it, and
after a separate worker run in the ledger succeeded.
- Add a red/green test that reproduces the observed race and covers the
negative cases.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-run-policy.test.ts`: the new test fails
before the change (1 failed, 13 passed) and passes after it (14 passed).
The queued-snapshot cases were added after review.
- The related runner-e2e tests (`chat-catalog-run-policy`,
`chat-catalog-readiness`, `chat-catalog-failure-retirement`,
`runtime-dependency`, `internal`) pass: 200 tests.
- `pnpm test:e2e:runner:typecheck` passes.
- Paid run
[43374419951](https://github.com/paperclipai/paperclip/actions/runs/43374419951)
on the first commit of this branch:
`chat-features.runner-opencode.local.tasks.private-task` passed.

## Risks

- Low risk. The change is in eval grading only. It does not change
product code.
- The check accepts a follow-up that is still running at grade time. A
later failure of that follow-up is not observed by this case. Completion
delivery is covered by its own completion cases.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:04:35 -06:00
Nicky LeachandClaude Fable 5.2 23c2343015 feat(server): read a retained provider pack from its Runner image (#21308)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner executes agent work in a sandbox. It needs a provider
pack of agent tools, and the server uploads that pack when the sandbox
does not have it
> - A retained session keeps the pack it started with. When that pack is
not the server's own, the server downloads the pack's transport archives
from a registry (#19779)
> - Today the server looks for them in the application image tagged
`sha-<commit that built the pack>`
> - `master` now publishes the Runner one time for each set of inputs,
as `ghcr.io/paperclipai/paperclip-runner`. A pack from that image names
the commit that published the image, and few commits get an application
image, so that tag often does not exist
> - This pull request reads such a pack from its Runner image, and keeps
the old source for every other pack and for a Runner image that cannot
be read
> - The benefit is that a retained session can still get its pack when
the pack came out of the published Runner image

## Linked Issues or Issue Description

Refs #19779 (the retained pack source), #21228 (the pack field
`runnerInputsDigest`), and #21204, #21229, #21230 (the Runner image). No
issue exists for this change.

**Problem or motivation**

`retained-provider-pack-oci.ts` reads
`ghcr.io/paperclipai/paperclip:sha-<runnerSourceRevision>`. It needs
that image to exist, to carry the pack, and to have one layer with a
known build-history line. For a pack from a published Runner image,
`runnerSourceRevision` is the commit that published the image.
`docker.yml` writes `sha-<commit>` only when its `master` run for that
commit completes, and a newer push can replace a waiting run. The
restart-chaos harness has the same assumption in another place: it
requires the pack of a sandbox image to be built by the commit in the
image's `org.opencontainers.image.revision` label.

**Proposed solution**

A pack that names its Runner image in `payload.runnerInputsDigest` is
read from `ghcr.io/paperclipai/paperclip-runner:inputs-<digest>`. A
sandbox image can name the commit that built its Runner in a label of
its own. No application image carries a pack with that field yet, and no
image carries that label, so behaviour today does not change.

**Alternatives considered**

- Keep the application image as the only source and require an
application image for every Runner publication. Publication would then
depend on `docker.yml`, which serves one ref at a time.
- Take a fixed number of layers ("the image has four, take the last
two"). A server in the field cannot be changed when the image gets
another layer. The server now uses the rule that holds for any number of
layers: the full image is the slim image plus transport layers.
- Fall back to the application image for every failure of the Runner
image. A Runner image that exists and does not match the pack is a
fault, and a second source would hide it.

**Roadmap alignment**

This is build and runtime infrastructure. It adds no product feature and
does not overlap an item in `ROADMAP.md`.

This is part 3 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image (this
pull request)
4. #21310: a registry build cache for the Runner image build

This pull request uses no code from parts 1 and 2. It is stacked on them
because all four parts edit `doc/runner-image-contract.md`.

## What Changed

- `server/src/services/native-runtime/retained-provider-pack-oci.ts`,
for a pack whose `payload.runnerInputsDigest` is 64 hexadecimal
characters:
- The server reads the full index `inputs-<digest>` of the Runner image.
The index must carry `io.paperclip.runner.inputs-digest` equal to the
pack's value, `org.opencontainers.image.revision` equal to the pack's
`runnerSourceRevision`, and a valid digest in
`io.paperclip.runner.slim-digest`.
- It reads the slim index by that digest, never by the slim tag, and
checks the bytes it gets against the digest.
- For the pack's platform, the layer digests of the slim image must be
the first layer digests of the full image, in the same order. The layers
after them are the transport layers. There must be at least one and at
most eight, and their total size has the existing limit.
- The server downloads only those layers. It checks each against its
descriptor and refuses every entry that is not a transport archive, the
receipt or the `provider-pack.archives` directory. A file may not be in
two layers.
- When the Runner image cannot be read, the server reads the application
image as before. That covers a registry error status for the token, for
either index or for either manifest: for example 404 for a tag or a slim
index that does not exist. Nothing is written before that point.
- A Runner image that can be read and does not match fails with
`runner_remote_provider_source_invalid`. The server does not try the
application image then.
- The same file, for every other pack: the old path, unchanged in what
it checks. Its code moved into functions that both paths use (`request`,
`pullToken`, `platformImage`, `receiveTransportLayer`).
- The downloaded layer file is removed as soon as it is unpacked.
Before, the importer removed it after its checks, so
`retained-provider-pack-source.ts` loses that one line.
- The cache record gets an optional `runnerImage` field with the inputs
digest and the digests of the transport layers. `transportLayerDigest`
is the last of them and `transportLayerBytes` their total. An older
server ignores the new field.
- `tests/restart-chaos/daytona-image.ts`: when the image has a non-empty
label `io.paperclip.runner.source-revision`, the pack must be built by
that commit. Otherwise the pack must be built by the commit in
`org.opencontainers.image.revision`, as before. The receipt field
`providerPackRunnerSourceRevision` records the pack's commit.
- `doc/runner-image-contract.md` has a new section, "Reading the
transport archives alone". `tests/restart-chaos/README.md` describes the
label.

One choice needs a reviewer's eye. When the full index can be read and
the slim index cannot (a registry error status), the server falls back
to the application image; it does not fail. Without the slim manifest
the server cannot tell which layers are transport layers, so it has not
accepted any byte of the Runner image, and the old source has its own
complete checks. A slim index that is read and has other bytes than its
digest still fails.

## Verification

Linux (`node:24.22.3-bookworm`, GNU tar 1.34, zstd 1.5.4, the worktree
mounted read-only):

- In `server/`: `retained-provider-pack-source.test.ts` (34 tests, 15 of
them new), `provider-pack-archive.test.ts`,
`remote-provider-pack-verification.test.ts`,
`remote-provider-pack.test.ts`: 127 tests pass in 4 files.
- `node node_modules/vitest/vitest.mjs run --config
tests/restart-chaos/vitest.config.ts
tests/restart-chaos/daytona-image.test.ts`: 25 tests pass, 5 of them
new.

macOS (Node 26.12.0):

- `daytona-image.test.ts`: 25 tests pass.
- `retained-provider-pack-source.test.ts`: the 15 new tests pass. Six
older tests fail, with and without this change. They run the real
uploader, which needs GNU tar, and this host has bsdtar.
- `tsc` on the two changed server files and the test, and on the two
restart-chaos files, with the settings of their test projects: no error.

The new cases:

- Read from the Runner image with 6 registry requests (two indexes, two
manifests, two layers), none to the application image and none for the
slim tag.
- Take every layer after the slim image's: one transport layer, and
three.
- Fall back when the tag does not exist, when the token is refused, and
when the slim index cannot be reached.
- Refuse, with no request to the application image: an index for other
inputs, an index for another revision, no slim annotation, a slim
annotation that is a tag, a slim index with other bytes than its digest,
a slim image that is not a prefix of the full image, a slim image equal
to the full image, nine transport layers, and a transport layer that
holds a pack file.
- Four deliberate faults in the reader each make a test fail: the slim
index bytes are not checked, no prefix check, the slim annotation is not
validated, and no bounds on the number of transport layers.

A real run, in the same Linux container (`linux/arm64`), with no
credential:

- The script took `provider-pack/provider-pack.json` of `linux/arm64`
from the published image
`ghcr.io/paperclipai/paperclip-runner:inputs-0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`.
Pack digest
`sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f`,
`runnerSourceRevision` `25e9a9c2e9277f3be8649be8c809cc81bde6f0c5`.
- `resolveProviderPackUploadSource`, with another pack as the server's
own, returned `{ archiveOnly: true }` after 23 seconds. That function
calls `receiveRetainedProviderPack` and then the importer's checks: the
manifest inside the overlay archive is the selected manifest, every
archive matches the receipt, and the content of every file archive
matches the digest of its file.
- The server code made 9 requests: one anonymous token request, the full
index, the `arm64` manifest, the slim index `sha256:f63ea436…4f3c` by
digest, its `arm64` manifest, and two layers, each with one redirect. No
request went to the application image.
- The cache record:

  ```json
  {
    "schema": "paperclip.retained-provider-pack-source.v1",
    "sourceRevision": "25e9a9c2e9277f3be8649be8c809cc81bde6f0c5",
"providerPackDigest":
"sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f",
"imageIndexDigest":
"sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162",
"imageManifestDigest":
"sha256:73dbe864fbab6208616f0aa6f1610a7ece4ef22f68a73d30901cbaea60139589",
"imageConfigDigest":
"sha256:740e48463d5caf0bfead4b923cdbd8b220acb93d058419c3db9d00c7f1831d65",
"transportLayerDigest":
"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf",
    "transportLayerBytes": 426602696,
    "runnerImage": {
"inputsDigest":
"0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca",
      "transportLayerDigests": [

"sha256:6a340088ca2cfa85f10f0385d9c8f5cdccb22c3a2bbb2062620b1d90616ba5b5",

"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf"
      ]
    },
"receiptSha256":
"2afdd26645845369d9e851b1e6e2884eb6c3884ea7c4866a1eeb2efb9fb230d7",
    "verifiedTarPaths": true
  }
  ```

- `uploadVerifiedProviderPack` then installed the pack from that source
through a local stand-in for the sandbox, with no further registry
request, in 4 seconds. The installed tree is 1,249,060,841 bytes. Its
`provider-pack.json` is the selected manifest, all 7 artifacts of the
manifest have the digest the manifest names, and the pack's own Node
prints `v24.22.3`.

Never run:

- The same real run for `linux/amd64`, whose transport layers are about
1 GB.
- A session in a real sandbox. The real run used a local stand-in for
the sandbox.
- The fallback against the real registry. Fixtures cover it.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The old path was moved into shared functions. Its checks are the same,
and its 19 existing tests pass on Linux, but a review must read that
code as changed.
- The layer file of the old path is now removed before the importer's
checks, not after them. The importer does not read it.
- The reader depends on two rules of the image contract: the full index
names its slim index, and the full image starts with the slim image's
layers. A Runner image that breaks either rule is refused, with no
fallback.
- The fallback covers a registry error status only. A network error ends
the import, as it does today.
- The label `io.paperclip.runner.source-revision` is new and no image
carries it yet. The change to the sandbox image that sets it must use
this name.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:02:19 -08:00
DottaandClaude Opus 5.5 0477f622d6 fix(recovery): keep the sandbox that holds abandoned unsaved files out of replacement and cleanup (#21304)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in remote sandboxes, and the workspace files live in
those sandboxes until a save
> - When a run cannot resume exactly, the fresh continuation fallback
(#21135) abandons the old workspace epoch and keeps its unsaved files
"recorded and unapplied" on the old sandbox
> - The abandonment clears the dirty flag, the writer and the location,
so the workspace looks clean. No code reads the abandonment receipt
> - A later failed resume can then replace the sandbox, and cleanup can
destroy it. That sandbox holds the only copy of the unsaved files
> - This pull request adds one shared hold. The task continues on a new
sandbox, and the old sandbox is parked until its files are saved,
discarded, or kept for 7 days
> - The benefit is that the fallback keeps its promise. Abandoned files
are not lost

## Linked Issues or Issue Description

Refs #21135
Refs #21297

**What happened?**
After the fresh continuation fallback abandoned a workspace epoch, the
receipt in `remoteWorkspaceAbandonedEpochs` was write-only.
`reusableSandboxReplacementIsSafe` saw a clean workspace with a
committed save and allowed replacement.
`retireReplacedReusableSandboxLease` then moved the old lease to
`pending_cleanup`, and the cleanup sweep destroyed the sandbox with the
unsaved files. `nativeWorkspaceExportPendingForLease` (the check that
the sweep, the retained orphan settlement and the scoped destroy paths
use) also answered "nothing pending".

**What did you expect?**
A sandbox that holds abandoned, unsaved files stays until the files are
saved, a person discards them, or a stated retention period ends.

**Steps to reproduce**
1. Abandon a remote workspace epoch whose location names sandbox L.
2. Let the next run fail to resume L.
3. The replacement decision says safe, L goes to `pending_cleanup`, and
the sweep destroys it.

## What Changed

- New `server/src/services/remote-workspace-abandoned-files-hold.ts`
holds the single predicate for "this lease holds abandoned files". It is
one SQL fragment (`abandonedWorkspaceFilesHeldSql`) plus a TypeScript
wrapper (`abandonedWorkspaceFilesHeldOnLease`). A lease is held while a
workspace in its company has an abandoned-epoch receipt whose
`location.providerLeaseId` equals the lease's provider lease id, and
that receipt has no `released` object. A receipt written by the current
release (no new fields) counts as unreleased.
- Replace and park. A failed resume of a held sandbox still lets the run
replace it, so the task continues at once on a new sandbox restored from
the last committed save. `retireReplacedReusableSandboxLease` parks the
old lease as `retained` with the `abandonedWorkspaceFilesParked` marker
(failure reason `abandoned_workspace_files_held`) instead of
`pending_cleanup`. `reusableLeaseCanBeResumed` and
`reusableLeaseCanBeCleanedUp` skip a parked lease, so later runs never
select it again. The check is a metadata read with no query, so the run
path does no extra work.
- Every path that can destroy a sandbox now honours the hold:
- `nativeWorkspaceExportPendingForLease` answers pending for a held
lease. This covers the pending cleanup sweep (defer), the orphaned
active lease sweep, the retained orphan settlement `heldBy`,
`destroyClaimedSandboxLease` (terminal workspace reaper
`destroyRemoteWorkspaceLease`, `destroyReusableSandboxLeases` and the
environment delete path, and run lease release with `destroy`), the
`destroyRunLease` dispatcher, and the plugin sandbox teardown retry.
- The run lease settlement (`environment_run_lease_active`) is the one
caller that ignores the hold (`{ destructive: false }`). It never
destroys a sandbox. It stops the sandbox and keeps it, or it moves a
lease without a sandbox to `pending_cleanup`, where the sweep parks a
held lease. If it waited for the hold, a dead run's lease would stay
`active` and block the retry for the whole retention period.
- `destroyReusableSandboxLease` parks the lease and returns without a
provider call. This covers obsolete reusable lease cleanup, the destroy
after an expired or mismatched resume, release with `expired`, and the
sandbox driver's `destroyRunLease`.
- The `retryPendingSandboxTeardown` dispatcher refuses a destructive
teardown of a held lease. Stop-only cleanup still runs, because a stop
keeps the filesystem.
- The retained orphan listing (`alwaysHeldSql`) uses the same SQL
fragment, so held orphans are not polled.
- Release. `releaseAbandonedWorkspaceFiles` marks a receipt `released`
with a reason (`recovered`, `discarded`, `retention_expired`) and an
actor. It moves the parked leases for that sandbox to `pending_cleanup`
when no other unreleased receipt holds them, and it writes a
`workspace.abandoned_files_released` activity entry on the task in plain
language. No export path for abandoned epochs exists today. The
`recovered` reason is the hook for one; this pull request adds no new
export mechanism.
- Retention expiry. New receipts carry `retainUntil` (7 days). An old
receipt uses `abandonedAt` plus 7 days. `nextEnvironmentCleanupAt`
includes the earliest retention end of any unreleased receipt. The
shared cleanup worker therefore wakes at that real deadline, and a
restart rebuilds it from the database. At the deadline the sweep runs
`expireAbandonedWorkspaceFileHolds`. This releases each due receipt as
an attributed system decision (actor `workspace-retention`) and logs the
held receipts and parked leases for each company. Each expiry always
writes a task entry: "Unsaved files from an interrupted run were kept
for N days, and their retention period has ended." The entry adds "They
are scheduled for removal." only when a parked lease was queued for
cleanup. A separate entry, "... kept for N days and have now been
removed.", is written only after the provider confirms the delete. Each
receipt is released in its own transaction. A failed receipt is logged
and skipped, and it is retried after a backoff (5 minutes, doubling, at
most 6 hours). The other receipts are still processed. The entries
reference a run or a task only if it still exists. If the task was
deleted, the decision goes to the server log only.
- Upgrade. A `pending_cleanup` lease (not in flight, not stop-only) that
an unreleased receipt holds goes back to `retained` and is parked by the
sweep. A sandbox that a release before this change already destroyed
cannot be recovered. Its receipt ends at its retention time like any
other receipt.
- Receipt trim (review P1). `abandonUnverifiedRemoteWorkspaceEpoch`
trims only released receipts and keeps the last 20 of them. It never
drops an unreleased receipt. A new abandonment of the same sandbox and
root replaces the unreleased receipt for that root. So a workspace holds
at most one unreleased receipt for each distinct sandbox and root
abandoned in the last 7 days.

## Verification

- `server/src/__tests__/abandoned-workspace-files-hold.test.ts` (19
tests). It follows the lease through the real acquisition path with a
failed resume, the real `abandonUnverifiedRemoteWorkspaceEpoch`, the
real `sweepPendingCleanupLeases`, and `settleRetainedOrphanSandbox`:
- `gives the next run a new sandbox at once and parks the held one,
which cleanup and later runs leave alone`
- `pulls a held lease already queued for cleanup back to retained, and
leaves it on the provider`
- `treats a receipt written before release fields existed as unreleased`
  - `records a retention expiry on the receipt`
- `hands the parked lease to normal cleanup once the files are
recovered` / `... discarded`
- `removes the files only after the retention period, and says removed
only after the provider confirms it`
  - `does not report removal when the provider delete fails`
- `records the end of retention without scheduling removal when another
receipt still holds the sandbox`
- `records the end of retention when it expires before the lease was
ever parked`
- `wakes the cleanup worker at the receipt's retention end, also after a
restart, and then goes quiet`
- `releases a receipt whose run was deleted and records it without the
run reference`
- `releases a receipt whose task was deleted without writing a task
entry`
- `keeps expiring later receipts when one fails, and retries the failed
one after a backoff`
- `keeps a held sandbox when the retained orphan settlement reviews it`
- `lists the sandbox for the retained orphan settlement again once the
receipt is released`
- `keeps holding the first sandbox after many later abandonments on
another sandbox`
- `replaces and cleans the sandbox as before when the receipt names a
different sandbox`
- `replaces and cleans the sandbox as before when no epoch was
abandoned`
- Red on the base revision (62d673cc3f): the park, upgrade, old-receipt,
expiry, many-abandonments and scheduling tests fail. The controls pass.
- Green: 19 of 19. Also
`environment-runtime-reusable-resume-replacement.test.ts` 17/17,
`environment-retained-orphan-sandbox.test.ts` 32/32,
`heartbeat-pending-cleanup-sweep.test.ts` 51/51,
`fresh-continuation-fallback.test.ts` 58/58.
- `tsc --noEmit -p server/tsconfig.json` and a tsc pass over the new
test file: clean. `environment-cleanup-work.test.ts` +
`heartbeat-pending-cleanup-sweep.test.ts`: 80/80.

- No existing test expectation was changed. Three CI failures on bb43f52
came from this change. Two came from the hold blocking the run lease
settlement: `ownership-reconciliation-report.test.ts > releases the
lease and frees the workspace for the retry once a preparation-only
epoch ends` and `remote-workspace-save-concurrency.test.ts > lets a
retry prepare the workspace again after its first writer was killed
during preparation`. The third, `native-session-executor.test.ts >
restores a warm backup into a created lease only when a settled restore
hold retained the stopped source`, came from a mocked database that has
no `execute`; the predicate now runs through `select`. All three pass
locally now. So do `heartbeat-orphaned-active-lease-sweep`,
`environment-run-lease-ownership`, `native-workspace-warm-reuse` and
`remote-workspaces.integration`.

## Risks

- Cost of a parked sandbox. Parking makes no provider call and changes
only the lease row. A sandbox that could not be resumed is not running.
A reusable sandbox that was released is already stopped and kept. In
both cases the cost is stopped storage for up to 7 days. One case is
different: if a destructive path finds a held warm sandbox that is still
running (`retained` after a keep-running release), the sandbox stays
running until the receipt is released. The expiry log line counts parked
leases for each company, so this cost can be seen.
- Run lease release for an `ephemeral` lease calls the provider's
release. Some providers delete the sandbox on that call. A held sandbox
normally has a reusable lease, so this pull request does not gate that
call.
- No schema change. The receipt is JSON in
`execution_workspaces.metadata`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:00:20 -06:00
Nicky LeachandClaude Fable 5.2 e363b0e52d fix(docker): keep local build output out of the Runner image's build context (#21312)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as the image `ghcr.io/paperclipai/paperclip-runner`, one time
for each set of Runner inputs (#21204, #21229, #21230)
> - The name of a published image is a digest of every file the Runner
build receives from the build context
> - `.dockerignore` keeps the Runner's own build output
(`packages/paperclip-runner/dist`) out of the context, but not the eval
kernel's (`packages/paperclip-eval-kernel/dist`), which the Runner's
build also writes
> - A tree where the Runner was built therefore has another inputs
digest than a clean checkout of the same commit
> - The Runner package's Vite builds write five more directories with
the same problem
> - This pull request adds the missing ignore rules, and a test that
finds the next such directory
> - The benefit is that a developer's tree and a clean checkout of the
same commit name the same Runner image

## Linked Issues or Issue Description

Refs #21204, which added the image identity. No issue exists for this
change.

**Problem or motivation**

The inputs digest covers every file the build receives under
`packages/paperclip-eval-kernel`. The Runner's build compiles the eval
kernel and writes `packages/paperclip-eval-kernel/dist`. After a local
Runner build, that directory is in the build context, so the digest of
the tree differs from the digest of a clean checkout. A lookup of the
published Runner image for such a tree finds nothing.

CI checkouts are clean, so no published image is affected.

**Proposed solution**

Add `packages/paperclip-eval-kernel/dist` and
`packages/paperclip-runner/dist-*` to `.dockerignore`, next to the rule
for the Runner's own `dist`. Add a test that derives every build-output
directory inside the image's inputs and requires a rule for each.

**Alternatives considered**

- Exclude the directory in the identity script only. Then the digest
would not describe what the build receives, which is the property the
identity is built on.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This pull request stands alone. It is based on `master` and needs no
other open pull request.

## What Changed

- `.dockerignore`: two rules.
- `packages/paperclip-eval-kernel/dist`: the Runner's own build writes
it.
- `packages/paperclip-runner/dist-*`: the output of the Runner package's
Vite builds (`dist-browser`, `dist-issue-thread`, `dist-scenarios`,
`dist-sdk`, `dist-standalone`). The Runner's `build` script does not
write them, but a tree where one of those builds ran had the same digest
drift. No tracked path matches the rule, and no later `!` rule
re-includes those directories.
- `server/src/__tests__/runner-image-identity.test.ts`: one new test.
For each package among the identity's inputs it collects the `outDir` of
every tsconfig a script compiles with `tsc -p`, the `target` directory
beside every Cargo manifest a script names, and the `outDir` of every
Vite configuration. It requires that the last `.dockerignore` rule
matching each directory is an exclusion. Today that is eight
directories. A new output directory inside the inputs fails the test
until it has a rule.
- No image build reads that directory from the host. The Runner stages,
the `build` stage of the application image and `docker/daytona-runner`
all compile the kernel themselves (`tsc -p`, not incremental), and the
eval kernel is a development dependency of the Runner only.

## Verification

The inputs digest, as the `runner-artifacts-identity` build target
reports it, on a `linux/arm64` Docker host:

| Tree | Without the rule | With the rule |
| --- | --- | --- |
| clean export at `62d673cc3f` | `0e6662ad…` | `0e6662ad…` |
| the same, with an eval-kernel `dist` | `e4cf3095…` | `0e6662ad…` |
| clean export at `bc71c23e0a` | `88fe2929…` | `88fe2929…` |
| the same, with an eval-kernel `dist` | `ac331275…` | `88fe2929…` |
| clean export at `731b52453c` | `88fe2929…` | `88fe2929…` |
| the same, with two files in `dist-browser` | `44fe867d…` | `88fe2929…`
|
| the same, with one file in each of the five Vite directories | not
measured | `88fe2929…` |

- `0e6662ad…` is the digest of the image that `master` published at
`25e9a9c2e9`, so the rule does not change the digest of a clean
checkout.
- A worktree after a real `pnpm --filter
@paperclipai/paperclip-eval-kernel build` also gave the clean digest
with the rule.
- `runner-image-identity.test.ts` (42 tests) and
`tests/runner-e2e/daytona-image.test.ts` (19 tests), which reads
`.dockerignore`, pass on macOS and in `node:24.22.3-bookworm`.
- The new test fails on the `.dockerignore` of `master` and names
`packages/paperclip-eval-kernel/dist`. It fails without the `dist-*`
rule and names `packages/paperclip-runner/dist-browser`. It fails
without either of the two older Runner rules, and with a trailing `!`
rule that re-includes one of the directories.

Never run:

- Any image build with the new rules beyond the identity target,
including `.github/docker-context-checks.Dockerfile`. On a CI checkout
the directory does not exist, so the rules remove nothing there.
- `linux/amd64`.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- **The sandbox image's content id changes one time.** `.dockerignore`
is an input of that id (`tests/runner-e2e/daytona-image-content.ts`). At
`62d673cc3f` the id goes from `be208823…` to `1e13a151…`. The next
end-to-end run builds a new sandbox image, and the restart-chaos harness
needs an image built from this tree. The image's contents do not change.
- Low otherwise: the rules remove directories that no image build reads.
The `Dockerfile`, `docker/daytona-runner/Dockerfile` and
`.github/docker-context-checks.Dockerfile` name none of them and run no
Vite build.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:55:23 -08:00
DottaandClaude Opus 5.5 bbafa78fe2 fix(retry): offer Retry only when admission can take it, and start held saved-work Retries (#21315)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run fails and its saved work is held for recovery, the task
shows a Retry button. Admission then decides if the Retry can start a
new turn.
> - The task view, the task's execution blocker, and admission each
computed their own answer. The task view said `canRetry: true`. The
blocker said `canRetry: false`, and its own `workspaceRecovery` part
said `canRetry: true`.
> - Admission refused these Retries with no reason. The receipt kept the
generic `execution_recovery` wait ("Retry is waiting for execution
checks"), and each later click joined the same held receipt. Nothing
ran.
> - The refusals had real causes. A preparation proof rejected
host-written workspace notices in the run log. A repair proof did not
accept a snapshot that the failed preparation only restored and never
changed.
> - This pull request fixes both proofs, names every admission refusal
of a held Retry, and makes the task view and the blocker read one shared
Retry answer.
> - The benefit is that Retry starts work in these cases, and Retry is
offered only when admission can take it.

## Linked Issues or Issue Description

**What happened?**

On hosted staging, several blocked tasks showed a Retry button. Pressing
Retry was accepted but nothing ran. The task said "Retry is waiting for
execution checks. Your message is saved." The task view had
`interruption.canRetry: true` and `retryWait.reason:
execution_recovery`. The task's `executionBlocker` had `canRetry:
false`.

- Saved-work recovery failed with `source_unavailable`. The legacy
preparation had failed before the provider started
(`executionRecovery.providerWorkStarted: false`), its sandbox epoch was
abandoned, and its sandbox was destroyed. The blocker contradicted
itself: `workspaceRecovery.canRetry: true`, top-level `canRetry: false`.
- Saved work showed "recovered", but `workspaceRepairRequired` stayed
`true`. The failed preparation had restored a snapshot that an earlier
run produced. The repair proof accepted only a snapshot that the run
itself saved.

**Expected behavior**

The task view, the execution blocker, and admission give the same Retry
answer. A Retry that admission can take starts work. When saved work
cannot be recovered, Retry continues in a fresh session from the
recorded work. Every admission refusal of a held Retry has a named
reason. No Retry is offered for a refusal that another click cannot
change.

**Steps to reproduce**

1. Let a legacy `paperclip_runner` preparation fail before the provider
starts, after the host writes its workspace notices ("Execution
workspace reuse freshness action ...", "Skipping saved session resume
...") to the run log.
2. Let saved-work recovery end with `source_unavailable` and
`use_alternative_strategy`.
3. Press Retry.
4. Observe that the Retry receipt is held with `executionWait.reason =
execution_recovery` and no run starts.

Or:

1. Let a legacy preparation restore the task's saved snapshot and fail
with `setup_failed` before the provider starts.
2. Let saved-work recovery complete.
3. Observe that the blocker keeps `workspaceRepairRequired: true`, and
that Retry is held with `execution_recovery`.

**Paperclip version or commit**

Staging on `origin/master` `62d673cc3f` (after #21267 and #21295).

**Deployment mode**

Hosted cloud (staging), remote Daytona sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (legacy runtime runs, native agent).

Refs #21267, #21295.

## What Changed

- `hasVerifiedPreparationLog` (`verified-preparation-log.ts`) now
accepts a log made only of host notices that the host writes before it
hands the turn to an adapter: the mentioned-skills line (first record
only, checked against the saved skill selection) and the workspace
freshness and session-reset notices. Every record must have the same
attempt, sequential `seq`, `stdout`, and the excerpt and watermark must
match. Any other line still keeps the run held.
- `verifiedUnchangedRestoredSnapshot`
(`remote-workspace-snapshot-reuse-receipt.ts`) proves that a failed
preparation's saved state is the snapshot it restored. It needs
`executionStage: preparing`, `providerWorkStarted: false`, exactly one
succeeded restore of that snapshot at that epoch and lease, and no
snapshot save by the run. `verifiedWorkspaceRepairReceipt` uses it next
to the existing reuse proof.
- `savedWorkOffersRetry` (`workspace-recovery-receipt.ts`) is the one
rule for Retry on a saved-work hold. Retry is offered when the saved
work is recovered and verified, or when recovery is exhausted in a state
that only a Retry restarts (a new recovery generation, or the
fresh-session fallback after `source_unavailable`). Retry is not offered
while recovery re-arms by itself (a stop recheck or a newer capture
policy), or while a copyback owner's stop is unconfirmed. The blocker
uses this rule for the top-level `canRetry` and for
`workspaceRecovery.canRetry`, so the two parts cannot contradict.
- `task-retry-eligibility.ts` lists the held-Retry refusals that another
click cannot change. The task view and the blocker both use it.
- Admission (`explicit-native-continuation.ts`) names each refusal of a
held request that was silent before: `previous_execution_unverified`,
`execution_settling`, `cleanup_pending`, and `remote_cleanup`. It names
a refusal only when an execution blocker holds the task, so a request
without a hold still takes the normal path.
- The task view now reads the blocker's answer for a task with a
saved-work hold. It shows no Retry when the blocker offers none, and it
names the recovery wait.

## Verification

- Red/green: `cd server && npx vitest run
src/services/saved-work-retry-eligibility.test.ts`. The tests fail on
`origin/master` sources and pass with this change. They are built from
the staging shapes:
- offers the same Retry in the task view and the blocker when saved work
is no longer available
- offers no Retry in either view while saved-work recovery waits for an
unconfirmed file copy to stop
- continues in a fresh session from the recorded work when saved work is
no longer available
- verifies the repair of a restored snapshot that the failed preparation
never changed
- keeps the repair unverified when the failed preparation may have
changed the restored snapshot
- names a refusal that another click cannot change and offers no Retry
for it in either view
- names the admission refusal of a held Retry and offers Retry only when
admission would take it
- `npx vitest run src/services/verified-preparation-log.test.ts`: new
cases accept the host notices and keep a provider line, a different
attempt, or a late skills line held.
- Related suites: deferred execution wait, execution blocker read,
explicit native continuation, governed wait runner loss, legacy
preparation retry hold, remote workspace snapshot reuse, task
interruption presentation, task retry, and all `workspace-recovery-*`
suites.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Medium-low. The preparation-log proof accepts more host lines. Each
accepted line has a fixed host format, and the run must still pass every
other preparation proof (events, leases, no provider evidence).
- The restored-snapshot proof applies only when the run recorded that
the provider never started and it never saved its own snapshot.
- Admission now records a named wait instead of the generic one for some
held requests. It changes no admission decision.
- The task view calls `getExecutionBlocker` for tasks that offer Retry
and have a saved-work hold. These tasks are few, so the extra reads are
small.
- Not covered here: a copyback owner on a replaced deployment still
needs its stop confirmed before saved work can be copied out of a full
sandbox. The task view now shows that wait instead of a Retry that
admission refuses.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:52:25 -06:00
DottaandClaude Opus 5.5 b555087f0f fix(native): continue again when a deploy interrupts an automatic continuation (#21314)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner (Daytona) that the
server controls; a deploy restarts the server while runs are in flight
> - When the restart's reattachment proves the Runner process gone
mid-turn, #21039 makes the server request one automatic Retry that
continues in a fresh session from the saved work, without replaying the
turn
> - Staging deploys every 20 to 45 minutes, and Codex runs often take
longer, so that automatic continuation is often interrupted by the next
deploy too
> - The hook treated that second loss as "automatic Retry already used"
and left the task blocked for a person
(`native_continuation_requires_reconciliation`)
> - This pull request gives an automatic continuation one more
fresh-session continuation each time a server restart or deploy took its
controller, and a deploy spends no automatic budget
> - The benefit is that a task interrupted by repeated deploys continues
by itself, and no provider turn that may have run is sent again

## Linked Issues or Issue Description

Refs #21039, Refs #21052, Refs #21272, Refs #21023, Refs #21120, Refs
#21161

**What happened?**

On staging, the largest failure class for a Codex agent on Daytona was
`runner_remote_process_identity_unavailable` with error code
`native_continuation_requires_reconciliation`: 18 runs in 3 hours. The
error name is misleading. It is the shared code for every refused remote
reattachment (`REMOTE_RUNNER_RECOVERY_ERROR_CODE`). In every sampled run
the refusal was `process_verification` / `process_not_running`: the
Runner process was gone, its stderr log was empty, and the last turn
event was `turn.submitted` or `turn.accepted` (unsettled). It was not a
missing identity file.

Sampled 12 such runs. In 9 the #21039 hook worked: it recorded
`native.interrupted_turn_outcome_unknown` and requested the
fresh-session Retry. The tasks that stayed blocked had one shape:

1. A run lost its Runner after a deploy. The hook requested the
automatic Retry (key `task-retry:restart-interrupted-turn:<run>`).
2. The Retry run started a fresh session. The next deploy came during
its turn (a graceful `SIGTERM` shutdown transition, then "Recovering the
existing native runner process after server restart").
3. The same-run reattachment found that Runner gone too
(`process_not_running`).
4. The hook returned `automatic_retry_used`, because the run's Retry
intent came from the hook. The task stayed blocked.

Examples: a `task_retry` run created 40 s after its predecessor failed
(the hook's Retry), interrupted by the 05:17 deploy, failed at 05:18,
task blocked. A second chain (assignment, then the hook's Retry) failed
the same way at 05:12.

**Expected behavior**

A deploy is not the task's failure. When a server restart or deploy took
the controller of an automatic continuation and its Runner is proven
gone, the server continues the task again in a fresh session from the
saved work. It never replays the turn. A loss without restart evidence
keeps the existing one-time rule.

**Steps to reproduce**

1. Start a long Codex run in a remote Daytona Runner.
2. Deploy the server during its provider turn so that the reattachment
finds the Runner gone. The server requests the automatic Retry.
3. Deploy again during the Retry run's turn so that its Runner is also
lost.
4. See the Retry run fail with
`runner_remote_process_identity_unavailable` and the task stay blocked
with no new run.

**Paperclip version or commit**

Staging Core `25e9a9c2e9`. Fixed on `master` after `731b52453c`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `restart-interrupted-turn.ts`: `lostControllerToRestart` finds restart
evidence on the run: a `native.recovery.transition` with `restartKind:
"graceful"` (shutdown), or a restart claim (`reattach_remote_runner`,
`reattach_existing_runner`, `resume_dead_runner`,
`bootstrap_incomplete`) without the `same_run_retry` trigger.
- Both `automatic_retry_used` refusals (a person's Retry that the hook
requested, and a system retry of owed work) now continue when the run
has restart evidence. A deploy interruption spends no budget, so a task
that only deploys interrupt never runs out. Each continuation needs a
new deploy during its run, so deploys bound it. A loss without restart
evidence keeps the one-time rule. All other gates are unchanged: proven
Runner exit, unknown-turn record, the issue still assigned and open,
idempotency per run, and the Retry's admission gates.
- `doc/execution-semantics.md` describes the rule.

## Verification

- Red before the fix, green after:
`server/src/services/runner-lost-mid-turn-continuation.test.ts` › "an
automatic continuation that the next deploy interrupted". The test seeds
the staging sequence: a lost run, the hook's Retry run with a
`turn.submitted` turn, the shutdown and restart transitions with staging
payloads, then a same-run reattachment that proves the Runner gone. On
the old code the two continuation tests return `automatic_retry_used`.
With the fix they request one fresh-session Retry, record the unknown
turn outcome once, recover the saved work and queue the successor with
`forceFreshSession: true`.
- A chain of 12 deploy continuations, each with a system retry that
inherits its Retry receipt, still continues (red on the first revision
of this PR, which had a budget of 8). Losses without deploy evidence
stay refused (with and without earlier deploy continuations).
- `npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts` 23/23 pass.
`governed-wait-runner-loss.test.ts` and
`workspace-recovery-retry-copyback.test.ts` 103/103 pass.
- `pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify` exits 2 (a global doc input changed), so required CI
supplies the broad coverage.

## Risks

- Low to medium. A task whose Runner is lost on every deploy now
continues after each deploy instead of once. Each continuation is a full
run, so it costs tokens. The deploy count bounds it, because each
continuation needs a new restart during its run.
- Not fixed here: why the reattached Runner is gone after a deploy. In
the sampled runs the reattachment after the restart hung in `native
session provider recovery timed out after 900000ms`, and the next
attempt found the process gone with empty stderr. That needs its own
investigation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, browser read-only API GETs on staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:41:32 -06:00
Nicky LeachandClaude Fable 5.2 03f9b83e52 feat(ci): report the Runner image on master and build the app image's Runner path on pull requests (#21307)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work, and `master` now
publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`
> - The application image will soon copy that Runner instead of
compiling its own. Part 1 of this stack adds the resolver that picks the
Runner for a build
> - Two gaps make that next change risky. The resolver has never run in
CI on `master`, so its lookup time and how often a commit has a
published Runner are unknown. And `docker.yml` has no pull-request
trigger, so a change to how the application image gets its Runner first
runs after it merged
> - This pull request adds a report-only job that runs the resolver on
every `master` push, and a pull-request check that builds the Runner
path of the application image
> - The benefit is real data from `master` before any build depends on
the resolver, and a failed check on the pull request instead of a failed
image build on `master`

## Linked Issues or Issue Description

Refs #21204, #21229 and #21230 (the Runner image). No issue exists for
this change.

**Problem or motivation**

The resolver decides whether a build copies the published Runner. On
`master`, a commit that changes the Runner's inputs is ahead of its
publication for some minutes, and nobody knows yet how often an
application build will meet that. The application image itself is built
only on `master` and on tags. The change that removes the Runner's
compile stages from that image will edit the `Dockerfile`, and today
nothing builds those stages on a pull request.

**Proposed solution**

Add one job to `docker.yml` that only reports, and one new workflow that
only checks. Neither changes what an image contains, and no existing job
reads either of them.

**Alternatives considered**

- Run the resolver inside the build legs. A failure there could fail the
image build. A job of its own, with `continue-on-error`, cannot.
- Give the lookup the job token. The Runner package is public, so an
anonymous lookup works and the job needs no credential.
- Build `production` in the pull-request check. It adds three `COPY`
instructions and a readability check to the two stages that this check
builds, at the cost of the tool layer and two more copies of several
gigabytes.
- Trigger the check on every Runner source change. `runner-image.yml`
already builds the Runner on those pull requests, and CI builds the
server and the UI.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 2 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path (this pull request)
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build

## What Changed

- `.github/workflows/docker.yml`: new job `runner_image_report`.
  - It runs only for a `master` push in this repository.
- It runs `runner-image-resolve.mjs inputs`, then `resolve`, and the
result goes to the job summary with the line "Report only: no build read
this result."
- It has `continue-on-error: true`, declares no `outputs`, and no job
needs it. Its failure does not fail the run.
- It holds no credential. Its only permission is `contents: read`, the
checkout does not keep its token, and the lookup is anonymous. It does
not log in to a registry and writes to none.
- Time limits: 8 minutes for the job, 5 for the step that builds, 2 for
the lookup.
- It is the first job in the file. Existing tests cut the other jobs out
of the file by name, up to the next job or to the end of the file, so no
other place leaves those tests unchanged.
- `.github/workflows/docker-app-check.yml` (new): on a pull request,
builds the `build` and `runner-provider-pack` targets with
`PAPERCLIP_BUILD_COMMIT` set, for `linux/amd64` only.
- `permissions: {}` at the top, `contents: read` for the job, no
credential, no `pull_request_target`.
- It exports nothing (`--output type=cacheonly`) and pushes nothing. It
reads the public build cache that `docker.yml` writes and never writes
it.
- It refreshes the lockfile in the build context first, as `docker.yml`
does.
  - Time limits: 60 minutes for the job, 45 for the two builds.
  - Trigger paths, and why:

    | Path | Reason |
    | --- | --- |
    | `.github/workflows/docker-app-check.yml` | the check itself |
| `Dockerfile`, `.dockerignore` | the recipe, and the files it receives
|
| `server/scripts/prepare-runner-vendor.mjs` | the step of `build` that
reuses or rebuilds the Runner |
| `packages/paperclip-runner/scripts/build-inputs.mjs` | the build stamp
that step compares with the files the image build received |

- `server/src/__tests__/docker-runner-workflows.test.ts` (new, 90
lines): only security properties. The report job has no credential and
one read permission, nothing can read it, and its two steps have time
limits. The check has no credential, cannot publish, and exports
nothing. In both, every action is pinned by commit and no shell script
holds an expression.
- `doc/runner-image-contract.md` and `doc/DOCKER.md` name the job and
the check.

## Verification

macOS (Node 26.12.0):

- In `server/`: `docker-runner-workflows`, `runner-image-workflow`,
`docker-build-stamp`, `container-init-reaping`,
`cloud-image-bundled-plugins`, `docker-image-layer-reuse` and
`cloud-image-sentry` tests: 52 tests pass in 7 files.
- `tests/runner-e2e` workflow tests (`workflow-security`,
`private-trust-workflow-security`, `workflow-image-only`,
`paid-runner-routing`): 27 tests pass in 4 files.
- `node --test .github/scripts/tests/*.test.mjs
scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: 962 of 962 pass.
- Both workflow files parse as YAML.
- `docker buildx build --target runner-artifacts-context --cache-from
type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-amd64
--output type=cacheonly .` succeeds, so both flags are accepted and the
cache is readable.

Linux (`node:24.22.3-bookworm`, with `zstd` and `jq` added): the same
three sets pass: 52, 27 and 962 of 962.

The two commands the report job runs were run by hand, with no
credential, on clean exports of the tree. For the inputs `0e6662ad…d1ca`
the answer was `published`, with the two index digests that `master`
published. For a commit whose image was not yet published the answer was
`inline`, `absent`. Part 1 has the output.

With the job at the end of `docker.yml`, two existing tests failed. They
are the reason for its place: `docker-canary-promotion.test.mjs` reads
the last job to the end of the file, and
`release-verify-workflow.test.mjs` allows no BuildKit configuration
after `merge-and-push`.

Never run:

- Either workflow. Only GitHub can run them. This pull request starts
the check by its own path; the report job first runs on `master`.
- The two builds of the check (`--target build`, `--target
runner-provider-pack`) on a local host. Their time and disk use on a
hosted runner are not known.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The check builds the whole `build` stage, with the UI and server
builds. Its duration is not measured. It runs only when one of five
paths changes, and its first real run is on this pull request.
- `docker.yml` holds one concurrency slot for each ref until all its
jobs end. The report job normally ends long before the build legs. A
stalled job can hold the slot for at most 8 minutes when the legs end
sooner.
- `scripts/watch-release-source.mjs` warns about every job of the image
run that waited more than 60 seconds for a runner. The report job is one
more job that it can name.
- `scripts/merge-group-image.mjs` refuses to promote a merge-queue image
for a push that changes `docker.yml`. The push that lands this change is
such a push. The merge queue is not enabled.
- The lookup has 2 minutes. The registry client retries a failing
request for longer than that, so a registry outage ends the step by its
time limit, and the job then reports nothing. The run stays green.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:29:33 -08:00
Nicky LeachandClaude Fable 5.2 ff91bfe2dc fix(runner): withhold Claude's self-scheduling tools from Paperclip ACPX sessions (#21276)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip owns scheduling. An agent uses Paperclip tools such as
task monitors and routines to continue work later.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude Code offers its own scheduling tools to the model:
`CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`. With these
tools the provider can wake itself. Paperclip does not see or govern
that work.
> - Today the Runner stops the Claude process when a turn ends, so these
tools have little effect. A later change keeps the Claude process alive
between turns.
> - This pull request removes the four tools from Claude sessions that
Paperclip starts through ACPX.
> - The benefit is that a Claude agent schedules work only through
Paperclip.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #19912 (the design for turn-scoped ACPX MCP authority).

**What existing behavior does this improve?**

The set of tools that Claude Code offers the model in a Paperclip ACPX
session.

**Subsystem affected**

Paperclip Runner, ACPX provider (Claude).

**Current behavior**

A Paperclip ACPX Claude session offers the model 26 built-in tools. Four
of them are `CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`.

**Proposed behavior**

- The Claude settings that Paperclip writes into the session home deny
the four tools. Claude Code then does not offer them.
- Each Claude launch sets `CLAUDE_CODE_DISABLE_CRON=1`. This stops the
cron scheduler of Claude Code and removes the three cron tools.
- All other tools stay. `SendMessage`, `ListAgents`, `Workflow`, `Agent`
and `TaskStop` do not change.

**Reason and benefit**

The provider cannot schedule its own wake-ups. Paperclip tools stay the
only way to schedule work.

## What Changed

- `packages/paperclip-runner/src/drivers/acpx/permission-policy.ts`: new
constant `CLAUDE_SELF_SCHEDULING_TOOLS` with the four tool names.
- `packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.ts`: the
Claude `settings.json` now always has `permissions.deny` with the four
tools, in each permission mode. The `allow` rules do not change.
- `packages/paperclip-runner/src/drivers/acpx/environment.ts`:
`createSanitizedAcpxSpawnInput` sets `CLAUDE_CODE_DISABLE_CRON=1` for
each Claude launch. The host cannot override it.
- New test `claude-scheduling-tools.claude-acp.test.ts`. It prepares the
session home with the production `prepareAcpxRuntimeSandbox`. It then
runs the pinned `claude-agent-acp` and the pinned Claude Code client
against a local fake Anthropic API. It reads the tools that the client
offers the model.
- Unit tests in `environment.test.ts` and `runtime-sandbox.test.ts`. One
assertion in `runtime-host.test.ts` now expects the `deny` list.
- No change to the `claude-agent-acp` patch, the lockfile or a pinned
digest. No change to Codex.

## State saved by the previous release

- A session that the previous release saved continues with no change.
The new restriction is not part of a saved identity or digest.
- Each time Paperclip starts Claude for a session, it writes its own
`settings.json` again with the deny list and sets
`CLAUDE_CODE_DISABLE_CRON`. This includes a resumed session and a start
after an upgrade.
- A resumed session keeps its identity, its home and its transcript. It
loses the four tools from its next start.
- A Claude process that the previous release started keeps its tools
until it stops. Paperclip does not refuse it and does not classify it
differently. Its next start gets the restriction.
- No refusal code is necessary, because no saved state becomes
incompatible.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/acpx/claude-scheduling-tools.claude-acp.test.ts`
- Without the product change: `2 failed` (`expected [ Array(26) ] to not
include 'CronCreate'`).
  - With the product change: `2 passed`.
- `npx vitest run
src/drivers/acpx/claude-scheduling-tools.claude-acp.test.ts
src/drivers/acpx/environment.test.ts
src/drivers/acpx/runtime-sandbox.test.ts
src/drivers/acpx/runtime-host.test.ts`: `4 passed (4)` files, `147
passed` tests.
- `npx tsc --noEmit -p tsconfig.json` in `packages/paperclip-runner`: no
errors.
- Tools offered after the change, isolated home: the same 26 tools
without `CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`.
- `npx vitest run src/drivers/acpx/runtime-host.test.ts -t "previous
release"`: the new test "resumes a Claude session saved by the previous
release and withholds self-scheduling tools on relaunch" fails without
the product change and passes with it. The test saves a session, puts
its home into the shape of the previous release, and resumes it through
`AcpxRuntimeHost.open`.
- `npx tsc --noEmit -p tsconfig.test.json` in
`packages/paperclip-runner`: no errors.
- The fixture in `claude-scheduling-tools.claude-acp.test.ts` now stops
the process group of the provider and waits for it before it removes the
fixture directory.
- Not run: a live run with a real Claude account, and a remote Runner.
Not run: `pnpm verify` and the full suite. CI supplies that coverage.

## Risks

- **A preconfigured home still offers `ScheduleWakeup`.** In that mode
Paperclip does not write the Claude settings of the user, so the deny
list does not apply. The environment variable removes only the three
cron tools. To remove `ScheduleWakeup` there, the Runner must pass
`disallowedTools` to the session. ACPX 0.22 does not forward that
option. This is a follow-up.
- **The new test runs the real pinned Claude Code client.** It needs the
platform binary, as `runner-tool-bridge.claude-schema.test.ts` does
today. It uses no credentials and no network.
- **A prompt or skill that tells Claude to use these tools no longer
works.** No Paperclip instruction refers to them.
- The prepared harness home and remote Runners use the same code path.
They are not tested separately.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff and ran the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated the options
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:17:24 -08:00
DottaandClaude Opus 5.5 bcadbde449 feat(runner): record the OpenRouter host that served each OpenCode model call (#21298)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agents can use models through OpenRouter, and the Runner
proxies those model calls
> - OpenRouter routes one model across many hosts, and these hosts use
different quantizations (fp4, mxfp4, fp8)
> - Chat evaluations with identical prompts behave very differently
between runs, but the run log does not record which host served a call
> - The run usage shows `provider: "deepseek"` and `biller:
"openrouter"`, and `providerRequestId` is always `null`
> - This pull request records the serving host and the OpenRouter
generation ID for each model call in the run log
> - The benefit is that we can compare evaluation cohorts by host, and
the cost event names a real OpenRouter generation

## Linked Issues or Issue Description

**Subsystem affected**
Runner OpenCode provider proxy, runnerd usage normalization, and native
run usage accounting.

**Problem or motivation**
OpenRouter's default routing sends a request to one of many hosts. Each
host can use a different quantization. Chat evaluation results change a
lot between runs with identical prompts. Paperclip does not record which
host served a call, so we cannot compare results by host.
`usageJson.providerRequestId` is always `null` for OpenCode runs.

**Proposed solution**
Every OpenRouter completion response has a top-level `provider` field
and an `id` (the generation ID). Each streamed chunk has them too. The
Runner proxy reads them from the first chunk after it forwards the
bytes. The Runner attaches them to the usage of that call. The values go
to the run log, and the run's first generation ID goes to
`usageJson.providerRequestId`.

**Alternatives considered**
A new PRP event type such as `provider.upstream.served` would need a new
event enum value. An older control plane rejects an unknown event type.
A `harness.diagnostic` event shows in the task chat as a diagnostic. The
usage payload already flows to every consumer, and older consumers
ignore extra fields. Thus this change uses the usage payload.

**Additional context**
Run-log change only (AGENTS.md section 7). The data stays in
`heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch
`packages/shared/src/telemetry/` or OpenTelemetry.

## What Changed

- New `createUpstreamServedTap`
(`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`).
It is a pass-through stream. It pushes each chunk downstream first, then
it reads `provider` and `id` from the first SSE `data:` event (or from a
JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of
a JSON body. It skips encoded bodies and unknown content types. Any
parse problem or recorder error stops the scan and leaves the response
unchanged.
- `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The
OpenCode session keeps the most recent served call. Each
`item.completed` usage payload and `session.usage()` now carry
`upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint
does not include these fields, so the number of usage events does not
change.
- The OpenCode app-server proxy adds the two fields to
`thread/tokenUsage/updated`. Rust `usage.reported` now fills its
existing `providerRequestId` field and adds an optional
`upstreamProvider`. The TypeScript runnerd transport keeps both fields
when it rehydrates the usage notification. `usage.schema.json` documents
`upstreamProvider` (the protocol manifest and schema bundle are
regenerated).
- Server native turn accounting keeps the first generation ID the run
reports. The usage checkpoint and the native adapter result carry it as
`providerRequestId`, so `usageJson.providerRequestId` and the final cost
event name it.
- `doc/run-log-events.md` documents the fields in a new "OpenRouter
serving host" section.

## Verification

- Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's
stream through unchanged and records the host that served it on the
turn's usage". The test sends a streamed fixture with a comment line, a
host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks
that the bytes are identical and that the turn's usage names `Novita`
and the generation ID. The test fails without the proxy wiring.
- `upstream-served.test.ts`: streamed fixture, chunk split inside a
multibyte character, non-streamed JSON, malformed chunks, a truncated
JSON body, an encoded body, the 64 KiB limit, and a recorder that
throws.
- `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts`
(rehydration), Rust
`provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`,
and server `native-usage-accounting.test.ts` (first generation ID stays
stable across turns).
- Local: OpenCode driver, OpenCode proxy, and runnerd transport suites
(31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes
(one timing test failed once under load and passed alone). Runner and
server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks
pass.
- Latency: a local benchmark proxied a 400-chunk SSE stream 200 times
with the tap and 200 times without it. The median time to first byte was
292 µs with the tap and 259 µs without it (+33 µs, which is the extra
stream stage). The total stream time changed from 491.4 ms to 492.5 ms.
The tap spends about 5.5 µs of CPU to inspect the first chunk, after it
has forwarded that chunk. OpenRouter time to first token is hundreds of
milliseconds.
- Paid evidence: one single-case run of
`chat-features.runner-opencode.local.skills.fork-read-only` on this
branch. The result is in a comment on this pull request.

## Risks

- Low risk. All new fields are optional and additive. An older control
plane ignores the extra usage fields. `usage.schema.json` is not bound
by the event schema, and no validator enforces it on `usage.reported`.
- Saved state from the previous release: this change adds no persisted
run, session, checkpoint, or launch state. A saved session resumes
unchanged. A run that is in flight during a deploy reports usage without
the new fields until its Runner is new. `usageJson.providerRequestId`
then stays `null`, as before.
- `providerRequestId` on the run and its cost event is the run's first
generation ID, not each call's ID. The value is stable, so a late usage
receipt still matches the posted charge. Per-call IDs and hosts are in
the run log.
- If OpenCode starts a new model call before it reports the usage of the
previous call, a usage event can name the newer call. OpenCode reports a
message's usage when its step ends, so this is not expected.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code as an agent with tool use (shell, file edits, test runs). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:15:53 -06:00
DottaandClaude Opus 5.5 c730872974 fix(recovery): a chat turn waiting on the person is not stranded work (#21302)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In a chat, the agent's turn ends and the conversation waits for the
person's next message. The stranded-work sweep must leave that wait
alone
> - The sweep recognizes a finished chat turn by
`finalizationReasonCode: "conversation_turn_finished"` on the run record
> - Run completion re-reads the run through the 64 KiB read projection.
When the stored result is larger, that projection drops the finalization
fields, and completion wrote the reduced copy back
> - The chat turn then lost its finalization reason. Its reply was not
presented, and the sweep queued
`issue.productive_terminal_continuation_recovery` 30 s later. That is an
extra run while the agent waits on a person
> - This pull request keeps the full stored result through run
completion, and makes the sweep read the committed decision from its
durable rows
> - The benefit is that a turn waiting on a person is never treated as
stranded work

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `app-permissions.new-quarantined-action`, the
CEO's chat turn yielded with a `response_wake` continuation and waited
for a person to allow an app action. The stored run result was 113,937
bytes. Its saved record had the read projection's `truncated`,
`truncationReason: oversized_result_json` and `originalSizeBytes` fields
and no `finalizationReasonCode` or `decisionId`. The run's presentation
decision was `interaction_owns_conversation_response`, so no reply
comment was written. 30 s later the sweep queued
`issue.productive_terminal_continuation_recovery`, and the case failed
its run limit.

**Steps to reproduce**
1. Finish a native chat turn that yields `response_wake`, so
finalization records `conversation_turn_finished`.
2. Let the stored run result be larger than 64 KiB when run completion
reads it.
3. Run completion writes back the reduced copy without
`finalizationReasonCode`.
4. Run the stranded-work sweep. It queues a productive continuation run.

**Expected behavior**
The run record keeps its finalization fields, and the sweep leaves the
waiting chat turn alone. No new run starts until the person replies.

## What Changed

- `server/src/services/heartbeat/run-completion.ts`: run completion
reads the run with `unsafeFullResultJson` before it merges the native
result. It no longer writes the read projection's reduced copy back.
- `server/src/services/native-runtime/native-response-wake-wait.ts`:
`hasCommittedNativeConversationTurnWait` checks the run's committed
finalization decision (`conversation_turn_finished`) through
`native_run_finalizations` and `status_decisions` for the conversation's
own agent.
- `server/src/services/recovery/service.ts`: the sweep's
finished-chat-turn skip also uses that durable check, so runs already
saved with the reduced copy are recognized.
- Tests: `run-completion.test.ts` keeps the finalization record when the
stored result is larger than the read projection.
`heartbeat-process-recovery.test.ts` finalizes a real chat turn with the
eval's yielded result, saves the reduced run record, and expects two
sweeps to start no run.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts -t "larger than the read
projection"` — fails on master (the saved result is the truncated copy)
and passes here.
- `cd server && npx vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "chat turn that
yielded"` — fails on master (`continuationRequeued: 1`) and passes here.
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts
src/__tests__/heartbeat-process-recovery.test.ts` (614 passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- Run completion now reads the full stored result for one run. This is
one row; the read projection still serves every API read.
- The sweep check adds one indexed query for chat issues whose latest
run succeeded and whose run record lacks the reason.
- Previous-release saved state: no persisted shape changes. Runs already
saved with the reduced copy are covered by the durable decision check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
nightly/v2026.1113.0-nightly.0 canary/v2026.1113.0-canary.3
2026-11-13 00:11:30 -06:00
Nicky LeachandClaude Fable 5.2 f79e2c5d84 feat(ci): add the Runner image resolver and a no-rebuild mode for the Runner vendor step (#21306)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - `master` now publishes the Runner alone as the image
`ghcr.io/paperclipai/paperclip-runner`, one time for each set of Runner
inputs (#21204, #21229, #21230)
> - The application image still compiles its own Runner on every build.
The next steps make it copy the published Runner instead
> - Before a build can do that, it must answer one question the same way
as every other build: "does a published Runner exist for exactly this
tree?" It must also fail clearly when the Runner it was given does not
match the tree
> - This pull request adds the script that answers the question, and a
mode of the Runner vendor step that fails instead of rebuilding
> - The benefit is that the later change to the image build is small,
and that a wrong or unreachable answer can only mean "build the Runner
yourself"

## Linked Issues or Issue Description

Refs #21204, #21229 and #21230 (the Runner image). No issue exists for
this change.

**Problem or motivation**

Three builders make images that contain the Runner: `docker.yml`, the
merge-queue build and the release preview build. Each one would have to
repeat three steps to find the published Runner of its tree: check the
lockfile, compute the inputs digest in the build, and look up
`inputs-<digest>`. Three copies can decide differently. A build that
copies a Runner built elsewhere also has no Rust toolchain. If the
copied Runner does not match the tree, `prepare-runner-vendor.mjs`
starts a rebuild, and the build fails late at `cargo` with a message
that hides the cause.

**Proposed solution**

Add `scripts/runner-image-resolve.mjs`, the one place that decides. Add
`PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` to
`prepare-runner-vendor.mjs`. Nothing calls the resolver and nothing sets
the variable in this pull request, so no workflow job changes.

**Alternatives considered**

- One command that builds and looks up. Two commands keep a registry
credential, when a caller has one, out of the step that builds.
- A wait for the publication of the commit. The decision whether the
application build waits is open, so the resolver has no wait mode. It
reports `absent` and stops.
- A build argument that names the expected inputs. No `Dockerfile`
declares one yet. The change that makes the application image copy the
Runner will add both ends together.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 1 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode (this pull request)
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build

## What Changed

- `scripts/runner-image-resolve.mjs` (new, 185 lines), with two commands
that share a directory:
- `inputs --dir <dir>` exports the `runner-artifacts-context` target,
runs the lockfile check on that export, and builds the
`runner-artifacts-identity` target when the lockfile is current. It
reads no credential. A lockfile that is behind stops there: no image is
published for such a tree.
- `resolve --dir <dir>` looks up `inputs-<digest>` and prints `mode`,
`reason`, `inputs_digest`, `image`, `slim_image`, `build_contexts` and
`labels`. It also writes them to `$GITHUB_OUTPUT` and a short report to
`$GITHUB_STEP_SUMMARY`.
- `mode` is `published` only for a complete publication whose full index
names these inputs and both platforms. Both images are pinned by index
digest. The slim digest is the one the full index names in
`io.paperclip.runner.slim-digest`; the slim tag is never read on its
own.
- Every other answer is `inline`, with the last four values empty:

  | `reason` | Meaning |
  | --- | --- |
  | `lock-behind` | `pnpm-lock.yaml` is behind the Runner's manifests |
  | `absent` | no tag exists |
| `incomplete`, `broken` | a publication stopped, or its two tags do not
belong together |
| `mismatch` | the full index names other inputs, or lacks a platform |
| `unknown` | the registry refused, failed or could not be reached, or
the lookup threw |

- The resolver reuses `runnerLockState`, `readRunnerImageManifest`,
`runnerPublicationState` and `runnerImageTags` from
`scripts/runner-image-publish.mjs`. That file now exports
`environmentCredentials`; it has no other change.
- `server/scripts/prepare-runner-vendor.mjs`: with
`PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1`, the step reuses a current
build as before. When the build is not current, it prints the reason,
starts no rebuild, leaves the stamp in place and exits with status 1. It
also fails when `PAPERCLIP_RUNNER_VENDOR_REBUILD=1` is set at the same
time. Without the variable the step is the same as before.
- `doc/runner-image-contract.md` has a new section, "Choosing the Runner
for an image build". `doc/DEVELOPING.md` describes the new variable.
- The resolver is not an input of the Runner image, so the inputs digest
does not change.

## Verification

macOS (Node 26.12.0), in `server/`:

- `node node_modules/vitest/vitest.mjs run
scripts/prepare-runner-vendor.test.mjs
src/__tests__/runner-image-resolve.test.ts
src/__tests__/runner-image-publish.test.ts
src/__tests__/runner-image-identity.test.ts
src/__tests__/runner-image-workflow.test.ts
src/__tests__/server-package-build-script.test.ts`: 179 tests pass in 6
files. 27 of them are new resolver tests and 3 are new tests of the
vendor step, whose end-to-end test also has a new case.
- `tsc` on the new test file, with the settings of
`server/src/__tests__/tsconfig.json`: no error.

Linux (`node:24.22.3-bookworm`, GNU tar 1.34, the worktree mounted
read-only): the same files, 179 tests pass.

Real runs, on clean exports of the tree (`git archive`), with no
credential, on a `linux/arm64` Docker host:

- On `master` at `62d673cc3f`: `inputs` gave `lock=current` and
`inputs_digest=0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`.
`resolve` gave `mode=published`,
`image=ghcr.io/paperclipai/paperclip-runner@sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162`
and
`slim_image=ghcr.io/paperclipai/paperclip-runner@sha256:f63ea43666ed708f1b0e5d2ade8bdc002885bf04252b9ed6cee21f7920e54f3c`.
These are the two indexes that `master` published for those inputs.
- The same tree with one line added to a Runner source file: another
digest, `mode=inline`, `reason=absent`.
- On `master` at `bc71c23e0a`, which changed the Runner's inputs, before
its image was published: digest
`88fe2929321b4706c77f11a098572ce990fec4449d505544c62ef3a84c28308a`,
`mode=inline`, `reason=absent`.
- Four deliberate faults in the resolver each make a test fail: no index
check, only `absent` is inline, the identity is built for a lockfile
that is behind, and the slim image is taken by tag.

Never run:

- A lookup with a registry credential. The package is public, so no
caller needs one now.
- Any workflow. No workflow calls the resolver in this pull request.
- `linux/amd64`.
- The full test suite, the full typecheck and `pnpm build`. `pnpm verify
--explain` reports exit 2, "broader verification required", so CI must
supply them.

## Risks

- Low for existing behaviour: nothing calls the resolver, and the vendor
step is the same unless the new variable is set.
- The resolver does not verify the attestation of the image. The
workflow that consumes the image must do that from the digest the
resolver reports.
- The resolver reports both build contexts, `runner-artifacts` and
`runner-artifacts-slim`. A build that reads only one stage gets one
context it does not use.
- A `broken` publication is `inline`, not a failure. The Runner image
workflow fails for that state.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:03:15 -08:00
Nicky LeachandPaperclip 731b52453c fix(environments): verify workspaces on run leases and release settled tasks (#21289)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core owns run leases and delegates remote execution to environment
plugins.
> - A retry must verify its old checkout on the placement that will run
the task.
> - A completed task must still release its provider resources.
> - This change uses the existing lease for workspace verification and
always calls the release hook after settlement.
> - Saved settlement proof lets cleanup retry without repeating
execution work.

## Linked Issues or Issue Description

Refs #21237. That PR records stop proof before lease release. This PR
keeps provider cleanup after settlement and changes workspace
verification sequencing. Both touch task completion; preserve both
changes if the merge order requires conflict resolution.

**What happened?**

Workspace verification ran before a portable task had a lease, so a
provider could not verify the checkout on its selected placement. Core
also skipped the provider release hook after successful task completion.
This could leave resources allocated after the run ended.

**Expected behavior**

Verify the old checkout through the selected run lease. A confirmed
refusal can use the authorized fresh-workspace fallback. A transport
failure must remain retryable. Call the provider release hook after
settlement and retry failed cleanup without changing the run result.

**Steps to reproduce**

1. Retry a task whose prior checkout uses plugin-managed storage and
portable task placement.
2. Verify that the plugin receives the same task lease for verification
and preparation.
3. Complete a task, fail the provider release once, and reconstruct the
cleanup controller.
4. Confirm that cleanup calls only the release hook and retains the
completed run outcome.

## What Changed

- Acquire the logical run lease before provider workspace verification
and reuse it for preparation and execution.
- Preserve the existing session and workspace choices. Recompute
workspace-dependent inputs only after a definite refusal; acquire
credentials and managed runtime inputs once.
- Save the exact refusal and lease before cleanup. Use a
workspace-scoped creation key only for the resulting fresh attempt.
- Replay interrupted verification on the same unlaunched lease. Keep
transport errors retryable and reject replay after execution starts.
- Always call the provider release hook after task settlement. Pass
refreshed settlement proof and route failed releases through the
existing cleanup reaper.
- Add database regressions for call order, restart, retained writers,
refusal, transport errors, and settlement-only cleanup replay.

`plugin-workspaces.ts` grows from 997 to 1,001 lines. The four added
lines extend the existing workspace transaction and lease callbacks.
Keeping the checks there preserves the ownership assertion, provider
result validation, and binding update in one operation. Extracting only
the verification branch would split that transaction without removing a
separate responsibility.

## Verification

- Server TypeScript check passed: `tsc -p server/tsconfig.json
--noEmit`. The complete `server/scripts/typecheck-layers.mjs` check also
passed, including all test-tree layers.
- The first CI typecheck caught a readonly array in a test fixture that
the narrower source check did not include. Commit 8d9ec19e7a corrects
that fixture; all layered checks pass afterward.
- Added a public heartbeat refusal regression in 7c598a4da3. All ten
session-preparation tests pass; independent execution of the refusal
case also passes. It checks successful completion, the saved fresh
workspace, a new conversation, and one credential acquisition.
- Nine database/service suites passed 335 tests: plugin warm tasks,
plugin runner tasks, startup cleanup recovery, credential writeback,
plugin workspaces, session preparation, plugin environments, retry
workspace continuity, and heartbeat workspace preparation.
- After the final restart change, both targeted refusal/restart tests
passed. The primary agent independently reran those cases and the final
server typecheck. After integrating current master, the primary agent's
independent rerun passed all 237 tests across the warm-task,
plugin-workspace, session-preparation, and heartbeat-workspace suites.
- `pnpm verify` returned exit 2 and requested broad CI coverage because
the plugin execution contract documentation changed. No full local
Core-suite pass is claimed.
- Previous-revision CI was green on 8d9ec19e7a: 59 checks passed and 3
were skipped. The first run failed the unchanged legacy-continuation
browser assertion and credential-cleanup server shard. Each passed on
one targeted retry with no code changes. The credential-cleanup
integration suite also passed all 21 tests locally on Node 22. The
browser failure concerned an existing continuation metadata marker; the
server failure cascaded after a cleanup error left a test spy installed.
The successful retries do not establish the cause of the first failures.
- Local database fixtures use simulated provider workers. No live
provider or deployed environment was changed.

- Additional verification for 59c7a1dae8: the layered server typecheck
passed. Independent execution passed all 265 tests across durable task
retries, workspace preparation, and public heartbeat preparation. The
new shutdown regression preserves the fresh Retry choice, waits for
parent cleanup, and prevents the successor from inheriting the parent
lease reference. Same-run recovery retains its receipt. Required CI is
green on 59c7a1dae8: 58 checks passed and 3 were skipped. Standard
review scored 5/5 with no unresolved comments. The separate Superagent
Security Scan is neutral/inconclusive because its workflow failed; a
direct rerun request returned HTTP 404. That scan is not claimed as
passed.

## Risks

- Providers must use the supplied task lease for verification and return
a definite `binding_mismatch` only when the checkout is unconfirmed.
Transport errors retain the retry decision.
- A refused checkout waits for confirmed release before a fresh
allocation. Failed deletion remains owned by the existing cleanup
reaper.
- Task settlement and physical resource release stay separate. Retained
handoff leases do not enter this release path.
- No migration or SDK schema change is required.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) implemented the changes and tests.
The primary Codex agent independently reviewed the connected Core and
plugin paths and ran checks. A separate GPT-6.1 Sol agent reviewed
lifecycle and restart boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue using the bug template labels
- [x] I have not referenced internal or private project details
- [x] My branch name describes the change
- [x] I have added relevant tests and documentation
- [x] I have considered and documented risks
- [x] All required CI gates are green
- [x] Greptile is 5/5 with no unresolved comments
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 21:56:02 -08:00
DottaandClaude Opus 5.5 61ab829d5d fix(queue): another agent's comment no longer hides a pending approval (#21303)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person approves a new plan while the agent is busy, the
approval waits in the task's message queue. Its Interrupt control stops
the current attempt and starts a fresh one with the approved plan
> - The task view shows one queue at a time. A saved card response (an
approval) gives way to any later queued message, so it never hides the
person's own later messages
> - Another agent's comment (here, the CEO's direction with `resume:
true`) is also a queued message. It took over the view, and the approval
and its Interrupt control disappeared
> - The person could then not interrupt for the plan they approved
> - This pull request lets only the person's own later messages hide a
pending approval. A queue that holds only agent comments no longer hides
it
> - The benefit is that the Interrupt control for an approved plan stays
available until the person uses it or writes a newer message

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `steering.interrupt-with-direction`, the person
approved a saved simpler plan while Morgan was busy. The approval queued
with `requiresFreshSession`, and its Interrupt control showed. The CEO
then posted its own direction comment on the task at 04:03:21. That
comment queued as a separate deferred wake. `GET /queued-comments` then
returned only the CEO comment's queue. The approval's Interrupt control
(`task-chat-queued-interrupt-<approval id>`) never rendered, the click
timed out, and no interrupt request was sent. The approval was not lost:
its deferred wake was still saved. Only the view and its control were
hidden.

**Steps to reproduce**
1. Approve a plan on a busy task, so the approval waits for a fresh
session.
2. Let another agent comment on the task, so its comment queues after
the approval.
3. Read `GET /api/issues/:id/queued-comments`. The approval is gone from
the queue.

**Expected behavior**
The approval stays the visible queue with its Interrupt control. A newer
message from the person still takes the view, as before.

## What Changed

- `server/src/routes/issues.ts`: `findQueuedCommentWake` returns a later
message queue only when it holds a message from a person. A queue with
only agent comments yields to a pending saved response. With no saved
response it is still returned, as before.
- `server/src/__tests__/issue-queued-comments-routes.test.ts`: the eval
ordering (a fresh-session approval, then another agent's comment) keeps
the approval as the queue, and a later person message in that queue
still takes the view.

## Verification

- `cd server && npx vitest run
src/__tests__/issue-queued-comments-routes.test.ts -t "another agent's
comment queues after it"` — fails on master (the CEO comment's queue is
returned) and passes here.
- `cd server && npx vitest run
src/__tests__/issue-queued-comments-routes.test.ts` (all tests passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- While a saved response waits, queued agent-only comments are not shown
in the composer queue. They stay saved and are delivered by the normal
deferred-wake path, as they were before the person approved anything.
The person's own messages are never hidden.
- One indexed comment lookup per queued message wake during the queue
read.
- No persisted shape or API contract change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:53:54 -06:00
DottaandClaude Opus 5.5 99d9e608f4 fix(heartbeat): hold the new owner's run while the reassigned owner's run settles (#21301)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person reassigns a task, the old owner's native run is
cancelled and the new owner's run is queued
> - The old run's executor keeps closing its provider session for
several seconds after the run is marked cancelled
> - #21031 holds a queued run while the task's previous session close is
pending. That hold reads the runner's cleanup gate, which exists only
after the close begins
> - The stale-lock sweep cleared the task lock as soon as the old run
was terminal. The new owner's run was then claimed before the close
began, met the close as it escalated, and failed with
`native_session_cleanup_quarantined`
> - This pull request keeps the new owner's run queued while the old
owner's stopped run still settles in this process, and wakes it when
that executor finishes
> - The benefit is that a reassignment never fails the new owner's run
and needs no Retry

## Linked Issues or Issue Description

Refs #21031

**What happened?**
In the paid chat eval `tasks.edit-properties`, the old owner's run
`6cf2ee38` was cancelled for reassignment at 04:03:47.611. Its process
stopped only at 04:03:57. The stale-lock sweep cleared the task lock at
04:03:47. The new owner's run was claimed at 04:03:48.626. The runner
admission waited for the old close, which escalated to operator-required
at 04:03:56, and the new run failed with
`native_session_cleanup_quarantined`.

Why #21031 did not apply: its hold reads the runner's in-memory cleanup
gate. At claim time the old executor had not started its session close,
so no gate existed. The existing `predecessor_settling_in_process` hold
covers only the same agent, and a reassignment changes the agent. #21260
does not touch this path: its new checkpoint runs only before a turn in
a recovered session, and both sessions here were new.

**Steps to reproduce**
1. Run a native task for agent A.
2. Reassign it to agent B. A's run is cancelled, but its executor is
still settling.
3. The stale-lock sweep clears the task lock.
4. B's queued run is claimed before A's session close begins.

**Expected behavior**
B's run stays queued until A's executor has finished. It starts by
itself after that, then #21031's gate covers any close that is still
settling.

## What Changed

- `server/src/services/heartbeat/queue.ts`: the
`predecessor_settling_in_process` hold in `claimQueuedRun` also holds
for another agent's native run on the same task when that run is
terminal but its executor is still active in this process. A live run of
another agent (for example a reviewer) does not hold it.
- `server/src/services/heartbeat/queue.ts`:
`notifyOtherAgentsQueuedOnSettledIssue` sends a queued-run dispatch
signal when a native executor settles and another agent has a queued run
on the same task. It does nothing when no run waits.
- `server/src/services/heartbeat.ts`: the executor calls it after its
cleanup, next to the existing same-agent `startNextQueuedRunForAgent`.
- `server/src/services/heartbeat/queue.test.ts`: the exact ordering
(cancelled old-owner run still in the executor set, lock cleared, new
owner's run queued), a control for a live run of another agent, and the
dispatch wake.

## Verification

- `cd server && npx vitest run src/services/heartbeat/queue.test.ts -t
"reassigned owner|another agent's live run|next owner once"` — the hold
and the wake tests fail on master and pass here.
- `cd server && npx vitest run src/services/heartbeat/queue.test.ts
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/__tests__/heartbeat-workspace-busy.test.ts` (all passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- A queued run on a task waits while another agent's terminal native run
on that task is still settling in this process. That wait ends when the
executor finishes, and the new wake re-drives dispatch. The queued-run
liveness redrive still covers a lost signal.
- The wake is event-driven: one indexed query when a native executor
settles, and one notification only when another agent waits on that
task. It adds no polling.
- Previous-release saved state: no persisted shape changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:53:49 -06:00
DottaandClaude Opus 5.5 f82ec65800 fix(runs): a person's Stop wins over lease-release teardown (#21300)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can stop an agent run with Stop (cancel). The run must then
end as cancelled
> - A run that a person stops just as it starts tears down its
environment lease at the same moment
> - The lease-release teardown wrote `interrupted`
(`lease_released_before_terminal`). It recognized only an acknowledged
Stop, and it decided from a run snapshot that it read before the Stop
request landed
> - An interrupted run looks like an infrastructure failure, and it is a
recovery candidate. That is against the person's decision
> - This pull request makes a recorded person Stop win over the teardown
in both orderings
> - The benefit is that a person's Stop always ends the run as cancelled

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `steering.stop-versus-pause`, the fixture
cancelled a resumed run as it reached native running. The Stop request
was recorded at 04:02:11.049. The teardown wrote `interrupted` with
`lease_released_before_terminal` at 04:02:11.083. The native
cancellation intent was still `pending`, so the teardown did not see it
as a Stop. The eval needs `cancelled`.

**Steps to reproduce**
1. Start a native run.
2. Record a person Stop request on the run (the cancel route does this
first).
3. Let the lease-release teardown run before the cancel finalizer writes
`cancelled`.
4. The run ends `interrupted`.

**Expected behavior**
The run ends `cancelled`. This is true when the Stop is recorded before
the teardown reads the run, and when it is recorded between that read
and the write.

## What Changed

- `server/src/services/acknowledged-native-stop.ts`:
`hasRecordedUserStopRequest` and `recordedUserStopRequestSql` recognize
a person Stop that the cancel route recorded (`cancelledByActorType:
"user"`, a user id, and a requested or acknowledged execution
cancellation or a startup cancellation fence).
- `server/src/services/heartbeat/run-lifecycle.ts`:
`terminalizeRunOnLeaseRelease` re-reads the run before it decides. A
recorded person Stop now gives `cancelled`. The `interrupted` write is
fenced on "no person Stop recorded", so a Stop that lands between the
read and the write makes the write fail. The function then reads again
and writes `cancelled`.
-
`server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts`:
two orderings (Stop recorded before the teardown read; Stop recorded
after the read, with the stale snapshot) and a control (an agent-only
request still reports `interrupted`).

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-lease-release-terminalization.test.ts` —
both new ordering tests fail on master (`interrupted`,
`lease_released_before_terminal`) and pass here.
- `cd server && npx vitest run
src/__tests__/heartbeat-run-lease-release-terminalization.test.ts
src/__tests__/heartbeat-run-terminalize-before-release.test.ts
src/services/heartbeat/run-lifecycle.test.ts` (67 passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- Low. Only the lease-release teardown changes, and only when a person
Stop is recorded on the run. The cancel path already wrote `cancelled`
for the same run when it won the race.
- The teardown now makes one more read of the run row. This is one
indexed primary-key read per teardown.
- Previous-release saved state: no persisted shape changes. Runs saved
by the previous release carry the same Stop fields.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:52:13 -06:00
DottaandClaude Fable 5.2 bc71c23e0a fix(test): remove the Pi fault calibration file race and halve its runtime (#21286)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite checks the Pi provider fault helper. The
helper kills the exact Pi child process on Linux with a pidfd.
> - `tests/runner-e2e/pi-provider-fault.test.ts` failed in 3 unrelated
PRs on 2026-11-12. One failure passed on a rerun of the same commit.
> - The failures had two causes. A fixture file race gave a JSON parse
error. A CPU-heavy calibration ran past the 25 s outer timeout on the
slower shared fleet.
> - This pull request removes the file race and removes about half of
the calibration work. It also states the outer timeout margin.
> - The benefit is a deterministic test that runs in about 3.4 s instead
of about 5.9 s on Linux.

## Linked Issues or Issue Description

**What happened?**
The test "calibrates the exact Pi fault helper with metadata negatives
and native Linux pidfd" failed intermittently in the "Runner support
suite" lane. There were two errors:
- Job 131195539083: `test_real_descriptor_launch_then_exact_child_pidfd
(descriptor=7)` failed at `json.loads(ready.read_text())` with
`JSONDecodeError: Expecting value: line 1 column 1 (char 0)`.
- Jobs 131420813114 and 131420614853: `spawnSync /usr/bin/python3
ETIMEDOUT` after 25 s. Both jobs ran on the `paperclip-public-pr-x64`
fleet (m5n.xlarge, 4 cores, load average about 4). In those jobs, every
file in the lane ran about 1.8x slower than on the hosted `ubuntu-24.04`
image.

**Expected behavior**
The test passes on every run of the same commit.

**Steps to reproduce**
1. Error A: add a delay between file creation and write in the fixture
child (for example `openSync`, then `Atomics.wait` for 500 ms, then
`writeSync`). Run `/usr/bin/python3 -B
tests/runner-e2e/pi-provider-fault.test.py` on Linux. All four real
calibrations fail with the same `JSONDecodeError`.
2. Error B: run the suite on a loaded 4-core runner. The Python suite
needs more than 25 s.

**Paperclip version or commit**
`master` at 9b613562b7.

## What Changed

- Root cause A: the fixture child wrote `child.json` and `exit.json`
with `writeFileSync`. That call creates the file and then writes it. The
test polls only `exists()` and then parses. A reader could see the
created but empty file. Now the fixture writes `<name>.tmp` and renames
it into place, so the file appears only with complete content. This also
fixes the same race on `exit.json`.
- Root cause B: a profile of one calibration showed that about 1.0 s of
1.28 s was spent in `inspect()`. Each `inspect()` call reads and SHA-256
hashes the runner executable and the distribution Node executable. Both
were copies of the 123 MB Node binary. Each calibration calls
`inspect()` five times, and the suite runs four calibrations, so the
suite hashed about 5 GB. Polling sleeps were only 0.09 s. The synthetic
runner root now is a copy of `/bin/sh` that runs a short script. It is
still a distinct, pinned ELF, and `/proc/<pid>/exe` still resolves to
it. The script keeps the same launch shape (direct path, or
`/proc/self/fd/3` or `/proc/self/fd/7` with the executable descriptor
opened only for the wrapper child). It keeps the same SIGTERM cascade.
The distribution Node, the wrapper and the Pi child are unchanged.
- The outer `spawnSync` timeout is now a named constant (45 s) with its
margin in a comment. The Vitest test timeout derives from it (+5 s), so
a slow Python run reports its own output instead of a Vitest timeout.
- No assertion is removed, skipped or weakened. The fault helper
(`pi-provider-fault.py`) is unchanged.

## Verification

All Linux runs use Docker `node:24-bookworm` (Python 3.11, kernel 6.12)
on an Apple Silicon host.

- Red (master + 500 ms create-to-write delay): `Ran 21 tests`, `FAILED
(errors=4)`. Each of the four real calibrations failed with
`json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char
0)`.
- Green (this branch + the same 500 ms delay between creating and
writing the temporary file): `Ran 21 tests in 7.788s`, `OK`.
- Timing, `python3 -B pi-provider-fault.test.py`, 3 runs each:
- Unlimited CPU: master 6.77 / 5.88 / 5.75 s; this branch 3.45 / 3.34 /
3.43 s.
- `--cpus=1`: master 6.16 / 6.08 / 5.53 s; this branch 3.37 / 3.34 /
3.04 s.
- macOS: `pnpm exec vitest run --config
tests/runner-e2e/vitest.config.ts pi-provider-fault` passes. macOS skips
the 3 Linux-only calibrations, as before.
- `pnpm verify` exits 2 because a `.py` file is a global CI input.
Required CI supplies the broad coverage.
- Not run: the test on an x64 GitHub-hosted or fleet runner before this
PR's CI. I could not reproduce the fleet's load locally.

## Risks

- Low risk. The change is test-only.
- The runner root fixture now is a shell instead of Node. The fault
helper checks only the root's executable identity, hash, arguments,
process group and working directory. A copied shell satisfies these
checks in the same way.
- If `/bin/sh` is bash instead of dash, the script still works. It uses
only POSIX `trap`, `kill`, `wait` and `$!`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, Docker, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (not
applicable: test-only change)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:19:29 -06:00
Nicky LeachandClaude Opus 5.5 e65a3095a7 fix(runner): quiesce the Core-dialed route once the Runner commits its suspend (#21245)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Core runs agent sessions through the Paperclip Runner. With provider
ingress, Core dials the Runner over a PRP WebSocket.
> - When Core closes a session, it asks the Runner to suspend. The
Runner suspends, reports the result, and exits.
> - Core's ingress connection did not know that this exit was expected.
It treated the drop as a transport loss and redialed the Runner. Each
redial got a 404 until the close released the route.
> - The release (park) path already quiesces the connection before the
Runner goes away. The suspend path did not.
> - This pull request makes the suspend close quiesce the route in the
same way, once the Runner commits its suspend.
> - The benefit is that a clean suspend no longer looks like a lost
Runner. A drop before the suspend is committed still redials and
recovers as before.

## Linked Issues or Issue Description

**What happened?**
After a provider-ingress session closed cleanly, Core redialed the
Runner's ingress endpoint several times and got 404 each time. The
Runner state was `suspended`, `runner.drain` and `runner.suspend` had
both completed, and the Runner had exited with code 0. The redials
followed the outbound reconnect backoff (about 250, 500, 1000 and 2000
ms) and stopped when the registration release closed the outbound.

**Expected behavior**
After Core commits to `runner.suspend` and the Runner reports it
completed, the Runner's exit is expected. Core must not redial or
classify the drop as a transport loss. The release path already works
this way: it calls `outbound.quiesce()` before it releases the task.

**Steps to reproduce**
1. Open a session through a Core-dialed route (plugin provider ingress
or the `listen_ws` driver).
2. Close the session. Core drains, queues `runner.suspend`, and the
Runner suspends and exits.
3. See that the outbound reconnect loop redials the gone Runner until
the registration release closes it.

**Paperclip version or commit**
`master` at b4ca19f3a8.

**Agent adapter(s) involved**
Not adapter-specific (core bug). It affects any Core-dialed Runner
route: plugin provider ingress and the `listen_ws` driver.

**Additional context**
The redials do not fail the session. The plugin outbound uses a 7-minute
recovery grace, and `#failTransport` ignores failures after
`#closeAndSettle` sets `#closed`. The new test confirms this: without
the fix, the close still settles, but the route redials. This PR removes
the misleading redial. It does not change how a run's own failure is
classified.

## What Changed

- `RunnerExecutionDriver.controlPlaneRegistration` results can now
include an optional `quiesce()`. Core calls it when it has committed the
Runner's exit. The route then stops liveness and reconnect, and it keeps
the established receive/ACK path.
- `DurablePrpControlPlane.onRunnerExitCommitted(listener)` is new. The
authority calls its listeners after it saves a completed
`runner.suspend` or `runner.shutdown` result and before it sends the
acknowledgement. The Runner exits when it gets that acknowledgement, so
a route quiesced here never sees the exit as a loss.
- `awaitRunnerSuspensionBarrier` calls a new `onExitCommitted` callback
once, when its suspend command is `completed`. It also calls the
callback when it reuses a completed receipt and the durable state is
still `suspended`. A rejected, failed or unanswered suspend never calls
it.
- `DurablePrpCodexTransport` registers each control-plane route through
one `#registerControlPlane` helper. The helper records each route's
`quiesce` until that route is released. The suspend close subscribes to
`onRunnerExitCommitted` while its barrier runs and quiesces all of these
routes from that notification. The barrier's own `onExitCommitted`
covers a completed receipt that was acknowledged before this close.
Quiescing clears the tracked routes, so it runs once per close. The
helper copies property descriptors, so getters such as the `listen_ws`
driver's lazy `failure` stay live.
- The plugin task registration (`plugin-runner-task.ts`) and the
`listen_ws` driver registration (`native-runner-execution-driver.ts`)
implement `quiesce` with `outbound?.quiesce()`. This is the same call
that the release path makes.

## Verification

- New transport test with a real runnerd:
`packages/paperclip-runner/src/live/runnerd-codex-transport-suspend-route.test.ts`.
The test route acts like a Core-dialed provider ingress: on a socket
drop that it was not told to expect, it records a redial into a 404.
Before the fix the events were `connected, redial: 404, release`. After
the fix they are `connected, quiesce after runner.suspend completed,
release`. The close settles, and the durable Runner state is
`suspended`. A second case delays the durable-state read by 1.5 s, so
the Runner exits while the barrier still reads. That case fails without
the authority hook and passes with it.
- New barrier unit test in `runnerd-codex-transport.test.ts`. It shows
that the exit is committed only after the retried suspend completes,
never on a rejected or unanswered suspend, and on a reused receipt only
when the state is `suspended`.
- New server tests for the plugin task and `listen_ws` registrations.
`quiesce` reaches the active outbound and does not close it. Release
still closes it.
- Existing tests pass. These include the outbound release-path quiesce
test, the half-open and lease-expiry failure tests (a drop before the
commit still fails after its grace), and the plugin release/resume
facade tests.
- `pnpm verify` exited 2 (the change touches files outside the resolved
test graphs), so I ran focused suites. Locally: 8 runner transport files
and the 4 control-plane test files, 667 tests
(`runnerd-codex-transport*.test.ts` for suspension, release/resume,
remote retention, settlement, warm attachment and attachment recovery,
plus the new suspend-route test), and 5 server files, 148 tests (both
plugin Runner integration tests, the plugin task, the execution driver
and the outbound). Required CI supplies the broad coverage.
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Low risk. `quiesce` is optional. Registrations that do not supply it
behave as before.
- The quiesce happens only after the Runner reports a completed suspend.
A socket drop before that point still redials and recovers.
- The quiesced route keeps its receive/ACK path, so the Runner's last
frames before exit are still acknowledged.
- No persisted state or protocol change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:19:24 -08:00
DottaandClaude Opus 5.5 62d673cc3f fix(retry): start a held Retry once a stopped preparation is proven stopped (#21295)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails, a person can press Retry. Admission starts
a new turn from the recorded work only after it proves that the previous
execution stopped.
> - For a legacy runner run that failed while it prepared its sandbox,
the proof reads the run's events and its environment lease.
> - After the automatic retry budget is spent, execution recovery
settles a no-replay hold and appends its own lifecycle event to the run.
The proof accepted only the error and measured events, so this platform
event made the proof fail.
> - The proof also asked for a workspace repair receipt for every run
with a lease, even when the run never recorded a remote workspace and
its sandbox was destroyed. No repair could ever exist, so the proof
could never pass.
> - Admission then refused the Retry without a named reason. The receipt
kept the generic `execution_recovery` wait ("Retry is waiting for
execution checks"), and every later Retry joined the same held receipt.
> - This pull request accepts the recovery hold's own settlement event
when it is bound to a settled hold for the same run, and asks for a
workspace repair only when the run holds prepared files.
> - The benefit is that Retry starts work at once when the previous
preparation is proven stopped, and it continues from the recorded work.

## Linked Issues or Issue Description

**What happened?**

A board user pressed Retry on a blocked task. The Retry was accepted,
but no run started for more than 13 hours. The task said "Retry is
waiting for execution checks. Your message is saved." A second Retry the
next day attached to the same held request and also did nothing. The
execution blocker said `legacy_execution_requires_reconciliation` with
`canRetry: true` and "A person can use Retry on this task ... to
continue once the previous execution has stopped." The previous run was
a legacy `paperclip_runner` run that failed with `setup_failed` while
preparing (`executionRecovery.kind = bootstrap`, `providerWorkStarted =
false`). It had no process identity, no controller lease, and its
Daytona sandbox lease had a `destroyed` termination receipt. Execution
recovery had settled a no-replay hold on it and appended the "Automatic
recovery stopped" lifecycle event.

**Expected behavior**

When the previous execution is proven stopped, a person's Retry starts a
new turn promptly and continues from the recorded work. The task view,
the execution blocker and admission agree that Retry works.

**Steps to reproduce**

1. Let a legacy `paperclip_runner` run fail while it prepares a remote
sandbox (`setup_failed`, bootstrap evidence with `providerWorkStarted =
false`) on its third automatic attempt.
2. Let the sandbox lease be released with a `destroyed` termination
receipt.
3. Let execution recovery settle the
`legacy_execution_requires_reconciliation` hold. It appends the
"Automatic recovery stopped" lifecycle event to the run.
4. Press Retry on the task.
5. Observe that the Retry receipt keeps `executionWait.reason =
execution_recovery` and no run starts. Press Retry again and observe
that it attaches to the same held receipt.

**Paperclip version or commit**

Staging Core `53608bd1d6`. Fixed on top of `origin/master` `c4f639ed1b`.

**Deployment mode**

Hosted cloud (staging), Daytona remote sandbox.

**Agent adapter(s) involved**

`paperclip_runner` (legacy runtime).

Related: #21267 (a Retry held on a resolved decision,
`decision_pending`). This pull request does not change its files.

## What Changed

- `readFailedPreparationSettlement` (`failed-preparation-settlement.ts`)
accepts one trailing host lifecycle event: the no-replay settlement
event that execution recovery appends. The event must have the exact
payload shape and must name a resolved
`legacy_execution_requires_reconciliation` hold whose evidence binds the
same run. Any other event still fails the proof.
- `failedNativePreparation` (`cancelled-native-startup.ts`) asks for a
verified workspace repair only when the run holds prepared files: a
recorded remote workspace, a workspace restore failure, or required
workspace recovery. A run with no recorded remote workspace and a
stopped sandbox has nothing to repair. The lease termination receipt,
the bootstrap evidence and the settled-event proof are still required.
- New regression test `legacy-preparation-retry-hold.test.ts` uses the
exact shape of the staging run.

## Verification

- Red/green: `cd server && npx vitest run
src/services/legacy-preparation-retry-hold.test.ts`.
- "starts the person's Retry once the stopped preparation's sandbox is
destroyed" fails on `origin/master` (the receipt keeps
`executionWait.reason = execution_recovery`, no run) and passes with
this change (one queued successor with `previousRunId` set to the failed
run and the Retry request id).
- The Retry stays held when the proof does not hold: a settlement event
bound to another run's hold, a provider execution event, a lease without
a termination receipt, and a recorded remote workspace without a repair
receipt.
- Related suites pass (299 tests): `task-retry`,
`saved-bootstrap-continuation`, `interrupted-preparation-hold-settle`,
`workspace-recovery-preparation`,
`workspace-recovery-preparation-gates`, `heartbeat-session-preparation`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low. The change only widens two proofs for a run that never started
provider work: the platform's own settlement event is accepted only when
it binds a settled hold for the same run, and the repair receipt is
skipped only when the run recorded no files to repair.
- Retry still starts a new turn. It never replays the stopped run.
- Not in scope: two other tasks show the same generic wait with a
different cause on the saved-work side. Their blocker says `canRetry:
false` while the task view offers Retry. One has saved-work recovery
that failed with `source_unavailable`. The other shows recovered work
without a verified repair receipt. They need a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:07:17 -06:00
DottaandClaude Fable 5.2 db3d7223e0 fix(verify): report stale SDK public input pins locally, before CI (#21283)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite proves the plugin SDK public types against
pinned source. The pins are in
`tests/runner-e2e/source-sdk-public-inputs.json`.
> - The pins cover every file in `packages/shared/src` and
`packages/plugins/sdk/src`, and the SDK build inputs (for example
`pnpm-lock.yaml`).
> - A PR that edits a pinned file and does not refresh the registry
fails three files in the "Runner support suite" lane.
> - Between 2026-11-12 10:27Z and 2026-11-13 01:14Z, this failure
occurred in 14 CI runs on 12 PRs.
> - The local check, `pnpm verify`, returned exit 2 ("let CI cover it")
for these changes. It did not mention the pins. Authors pushed, and CI
failed.
> - This pull request makes `pnpm verify` find stale pins before the
push. It also makes the opaque fixture error name the file and the fix.
> - The benefit is that authors find and fix the problem in seconds,
locally, instead of after a 6-minute CI lane.

## Linked Issues or Issue Description

**What happened?**
Three files in the "Runner support suite" lane failed together on many
unrelated PRs: `source-sdk-evidence.test.ts`,
`source-sdk-public-inputs.test.mjs` and
`source-sdk-public-type.test.ts`. Two of the three failed with the
opaque error `Reviewed SDK reference source changed`.

**Root cause**
The pins on master are not stale. The tests pass on master (58/58). Each
failing PR edited a pinned input and did not refresh the registry. I
compared the PR heads with their merge base:

- All 14 failing heads (12 PRs) changed a pinned input. None of them
changed `source-sdk-public-inputs.json`.
- All 19 heads in the same window that changed a pinned input and passed
this lane also refreshed the registry. This includes later heads of
three PRs that failed first (#21119, #21121, #21156).
- Two of the 12 PRs are Dependabot PRs that change only `pnpm-lock.yaml`
(#19593, #19594). The lockfile refresh bot already resyncs the pins for
lock changes on master.

**Steps to reproduce**
1. Start from `origin/master`.
2. Add one line to `packages/shared/src/types/heartbeat.ts` and commit
it.
3. Run `node node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts source-sdk-evidence.test.ts
source-sdk-public-inputs.test.mjs source-sdk-public-type.test.ts`.
4. The result is the same as CI: 3 files fail, with the same 4 failures.
5. Run `pnpm verify --explain`. It returns exit 2 and does not mention
the pins.

**Expected behavior**
`pnpm verify` reports the stale pins and the refresh command before the
author pushes.

## What Changed

- `tests/runner-e2e/source-sdk-public-inputs.mjs`: add
`staleSdkPublicInputsInWorkingTree(root)`. It applies the existing drift
guard to the working tree (tracked and untracked files that git does not
ignore). Add `isSdkPublicInput(file)` and the shared failure message
`staleSdkPublicInputsMessage(stale)`. The committed-tree guard keeps its
behavior and now uses a shared comparison.
- `scripts/local-verify.mjs`: when the diff touches a pinned input, the
registry or `report-lock.json`, `pnpm verify` checks the pins first. If
they are stale, it exits 1 and prints the files and the refresh command.
The check reads files only. It does not write the registry.
- `tests/runner-e2e/source-sdk-reviewed-fixture.ts`: the error `Reviewed
SDK reference source changed` now names the file and gives the refresh
command.
- `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: new test for the
working-tree guard (uncommitted edit, untracked file, ignored file). The
CI drift test uses the shared message.
- `doc/DEVELOPING.md`: document the new check under "Incremental local
verification".

No assertion is weakened. No pin is changed. No file under `.github/` is
changed.

## Verification

- Master, without this change: the three files pass (58/58).
- Red: on master plus a one-line edit to
`packages/shared/src/types/heartbeat.ts`, the three files fail with the
CI signature. `pnpm verify --explain` returns exit 2 and does not
mention the pins.
- With this change on the same red commit: `pnpm verify` exits 1 in
about 15 s and prints `tests/runner-e2e/source-sdk-public-inputs.json is
stale for 1 pinned input(s): packages/shared/src/types/heartbeat.ts
(changed). Commit your change, run ... --write, and commit the updated
registry.`
- Green: I followed the printed command (`node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`, then
commit). The three files then pass (59/59, with the new test).
- `node --test scripts/__tests__/local-verify.test.mjs`: 14/14 pass.
- `pnpm verify` on this branch returns exit 2 because
`scripts/local-verify.mjs` is a global CI input. Required CI supplies
the broad coverage.

## Risks

- Low risk. The new check only reads files. It runs only when the diff
touches a pinned input.
- `pnpm verify` now fails (exit 1) for a change that also fails in CI.
Before, it returned exit 2.
- The check needs a working `git ls-files` in the repository. `pnpm
verify` already needs git.
- Dependabot lock PRs still fail this lane until the lockfile bot
refreshes the pins. This PR does not change that path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent with tool use (shell, git,
GitHub CLI). Commits are attributed to Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:03:39 -06:00
DottaandClaude Opus 5.5 0c63f0fdb3 test(skills): budget list read bytes for fixture rows, not the bundled catalog (#21285)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The skill library list reconciles the bundled skill catalog,
folders, and local sources on each read
> - #18023 added a test that limits the `company_skills` bytes one warm
`list()` call reads, so the folder pass and the list query keep their
narrow projections
> - The limit counts every `company_skills` read, so it also counts the
bundled catalog rows that every company gets
> - Each new bundled skill adds about 9.6 KB to that total, so pull
requests that only add skills under `skills/` fail a test about
projections
> - This pull request measures a catalog-only company next to the
fixture company and applies the same limit to the difference
> - The benefit is that the test fails only when the fixture rows are
read too widely, not when the catalog grows

## Linked Issues or Issue Description

Refs #18023 (added the test). Refs #20760, #20764, #20767, #20768,
#20769 (failed on this test).

**What happened?**

`companySkillService.list > keeps folder classification and stored
content while narrowing inventory refresh reads` failed on five open
pull requests that only add bundled skills under `skills/`. The error
was `expected 1163219 to be less than 950000`. Other runs had 983607,
1035956, 982321, and 985561. The test passes on master.

**Expected behavior**

The test must fail when `list()` reads the unused metadata or the
Markdown of company skill rows. It must not fail when the bundled
catalog gets larger.

**Steps to reproduce**

1. Check out master.
2. Run `git checkout <head of #20760> -- skills` to add its 34 bundled
skills.
3. Run `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/company-skills-service.test.ts -t "narrowing inventory
refresh"`.
4. The test fails with `expected 1163504 to be less than 950000`.

**Paperclip version or commit**

`9b613562b70b7bea9e7562ddc1087d4c1b1788b1` (master).

## What Changed

- The test creates a second company that has only the bundled catalog.
- The test lists both companies warm and captures the `company_skills`
response-value bytes of each call.
- The test applies the existing 950,000-byte limit to the difference.
The difference is about 663 KB and comes only from the ten fixture rows.
- The test also checks that the fixture company lists the catalog rows
plus the ten fixture rows.
- No service code changes.

## Verification

Where the bytes come from (master, one warm `list()` call):

| Read | Master catalog | Catalog of #20760 |
| --- | --- | --- |
| Main list query (`company-skills.ts:3324`) | 671,003 B, 20 rows |
691,694 B, 52 rows |
| Bundled reconciliation `getByKey` reads (one per bundled skill) |
165,423 B, 13 calls | 471,771 B, 45 calls |
| Total (old assertion) | 836,465 B, pass | 1,163,504 B, fail |
| Fixture company minus catalog-only company (new assertion) | 663,153
B, pass | 663,166 B, pass |

Red and green:

- Old test, master `skills/` plus the `skills/` tree of #20760: fails,
`expected 1163504 to be less than 950000`.
- New test, same enlarged catalog: passes, difference 663,166.
- New test, master: passes, difference 663,153.

The new test still catches the regression it exists for. I widened each
projection on purpose with the enlarged catalog, ran the test, and
reverted:

- Folder pass reads the full `metadata` column instead of `sourceKind`
only: fails, `expected 1323122 to be less than 950000`.
- Main list query also reads `markdown`: fails, `expected 1183162 to be
less than 950000`.

Commands:

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/company-skills-service.test.ts`: 81 passed.
- `pnpm verify`: not completed. It selected 2,615 of 3,435 test files
(conservative selection), and I stopped it to keep load low on a shared
machine. CI ran the full required checks on the final commit: 54 passed,
4 skipped.

## Risks

- Low risk. The change is in one test only.
- The new test runs one more warm `list()` call, about 0.5 s more.
- The bound is unchanged. A widened projection on the fixture rows still
adds at least 540 KB and fails.
- This test no longer limits the bytes the bundled catalog adds to each
list call. Each bundled skill adds about 9.6 KB per warm call, because
`ensureBundledSkills` calls `getByKey` with the full row for each
bundled skill. That cost is real but separate. A fix should narrow that
read, not let this test limit the catalog size.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (not
applicable: test-only change)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (Superagent Security Scan is
neutral: "Scan inconclusive")
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:03:23 -06:00
DottaandClaude Opus 5.5 c6c7a61e2c fix(retry): release a Retry held on a resolved decision (#21267)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails, the task shows a Retry button. Admission
can hold an accepted Retry while a person must answer a pending approval
or question (`decision_pending`).
> - The hold is saved on the deferred Retry receipt. Nothing cleared
that saved reason when the person answered, so the task kept saying
"Retry is waiting for a pending approval or question" when nothing was
left to answer.
> - The deferred-receipt worker drops a refused receipt from its
schedule. No producer hinted it when the decision resolved, so the held
Retry did not start until some unrelated event rescanned the company.
> - A Retry was also held by a confirmation that was already open when
the failed turn was admitted. That turn had already run past the
confirmation, so the Retry added a gate that the original turn did not
have.
> - A person's comment on the task started a continuation, but the held
Retry receipt stayed deferred next to it.
> - This pull request makes a resolved decision release the held Retry
at once, keeps the projection from showing a resolved decision as the
wait, stops holding a Retry on a decision that predates the failed turn,
and settles a held Retry when a comment or answer continues the same
failed run.
> - The benefit is that Retry always works: a person never waits on a
decision that is already resolved.

## Linked Issues or Issue Description

**What happened?**

A board user pressed Retry on blocked tasks. No run started. The task
said "Retry is waiting for a pending approval or question to be
resolved. Your message is saved." In one task, the only interaction was
a `request_confirmation` that the previous successful run created. The
user's later comment started the turn that failed. The confirmation
stayed pending, so every Retry attached to the same held receipt. After
the user rejected the confirmation and sent a new comment, the task
still showed the same wait. No run started until a server restart
rescanned the deferred receipts about three minutes later. The task
projection said `canRetry: true` with the `decision_pending` wait.

**Expected behavior**

An accepted Retry is never held on a decision that is already resolved.
Accepting or rejecting the decision releases a held Retry right away. A
person's comment on the blocked task starts work, and the held Retry is
settled. The task projection never shows a resolved decision as the
Retry wait.

**Steps to reproduce**

1. Let an agent run on a remote runner create a `request_confirmation`
for the task.
2. Let the runner be lost so the run fails with
`native_continuation_requires_reconciliation`, and let saved work
recovery complete.
3. Press Retry. The Retry receipt is saved with `executionWait.reason =
decision_pending`.
4. Reject or accept the confirmation.
5. Observe that the receipt keeps `decision_pending`, the task keeps
showing the wait, and the Retry does not start until an unrelated
rescan.

**Paperclip version or commit**

Staging Core `57aeda190b`. Fixed on top of `origin/master` `53608bd1d6`.

**Deployment mode**

Hosted cloud (staging), remote sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (native runtime).

## What Changed

- `hasPendingContinuationDecision` (`explicit-native-continuation.ts`)
is now the single check for a decision that holds a person's
continuation. Admission and the task projection both use it.
- For a Retry, a pending interaction that was created before the failed
source run was created does not hold the Retry. That turn was already
admitted while the decision was open. Decisions that the failed run
created (for example a plan review) still hold the Retry.
- `releaseDecisionHeldExecution` (`deferred-execution-wait.ts`) removes
the saved `decision_pending` wait from deferred receipts on the task and
hints each receipt to the deferred-receipt worker. It runs in the
resolution transaction of every interaction resolution path (through
`notifyChatConfirmationResolution`) and in question answers.
- The task projection reads the live decision state. A saved
`decision_pending` wait whose decision is resolved is not shown. This
also repairs receipts saved before this change.
- When a comment or an interaction answer continues a failed run,
`retireTaskRetriesContinuedBy` (`task-retry.ts`) settles held Retry
receipts for the same source run as `task_retry_superseded`.

## Verification

- Red/green: `cd server && npx vitest run
src/services/governed-wait-runner-loss.test.ts`. The five new tests fail
on `origin/master` sources and pass with this change (17/17 pass):
  - releases the held Retry as soon as the person chooses accept
  - releases the held Retry as soon as the person chooses reject
- never shows a resolved confirmation as the Retry's wait, and both
projections offer the same Retry
- a person's comment starts work and settles the held Retry once nothing
is left to answer
- starts a Retry at once when the confirmation was already open before
the failed turn started
- The existing tests that keep a Retry or a message on hold while a plan
review, governed tool action, secret proposal, or connection
authorization is pending still pass.
- Related suites: deferred execution wait, explicit native continuation,
interaction resolution and service, task interruption presentation and
query, task retry, deferred receipt work, interaction routes.

## Risks

- Medium-low. A Retry now starts when the only pending interaction is
older than the failed run. That turn already ran with the interaction
open, so the Retry gets no new authority. Governed tool actions still
need their own approval before the tool runs.
- Interaction resolution now also updates deferred receipts on the same
task. The update is limited to receipts with a saved `decision_pending`
wait. A receipt that is still blocked by another pending decision is
held again on its next admission.
- Issue-linked approvals (`approvals` table) do not yet hint the held
receipt when they resolve. The projection no longer shows them as the
wait after they resolve.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:58:48 -06:00
Pavel-GuseynovandClaude Opus 5.5 0b9a9593d3 fix(tool-gateway): advertise only the MCP capabilities the gateway token may list (#19279)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A named MCP gateway gives a client its assigned tools. Each gateway
token has a list of allowed actions
> - A native run token allows only `tools/list` and `tools/call`
> - `initialize` always advertises the `tools`, `resources` and
`prompts` capabilities, whatever the token allows
> - A client that follows the advertised capabilities calls
`resources/list` and `prompts/list`, and the gateway refuses both with
403 `gateway_token_action_denied`
> - This pull request makes `initialize` advertise only the capabilities
whose list action the token allows
> - The benefit is that a client is not told to call methods that its
token cannot use

## Linked Issues or Issue Description

This continues #17688, which made the same change and was closed
unmerged by its author on 2026-11-02 ("we are no longer contributing
upstream"). That pull request reports that Claude Code requests every
advertised capability at once, drops the HTTP connection after the first
403, and then has no gateway tools for the run. I did not reproduce the
Claude Code part. The gateway part is below.

I searched pull requests for "allowedActions tools/list", "gateway token
allowed actions context tools", "paperclip_list_resources",
"gatewayTokenAllowedActions", "gateway_token_action_denied", "MCP
gateway capabilities resources prompts", "initialize capabilities token"
and "named gateway token actions discovery", and issues for
"paperclip_list_resources" and "gateway token allowedActions". Results:

- #17688 (closed, not merged): this change.
- #14427 (open): filters the four `paperclip_*` context tools in
`tools/list` by the token's allowed actions. It does not change
`initialize`, so the two pull requests do not overlap. They touch
different handlers in server/src/routes/tool-gateway.ts.

No issue matched.

**What happened?**

On master, the `initialize` handler in server/src/routes/tool-gateway.ts
returns `capabilities: { tools: {}, resources: {}, prompts: {} }` for
every token. For a token that allows only `tools/list` and `tools/call`,
a `resources/list` call returns 403 with reason code
`gateway_token_action_denied`.

**Expected behavior**

`initialize` advertises `tools`, `resources` or `prompts` only when the
token allows `tools/list`, `resources/list` or `prompts/list`. A session
with no token action list keeps all three.

**Steps to reproduce**

1. Create a named gateway and a token with `allowedActions:
["tools/list", "tools/call"]`.
2. Send `initialize` with the token. On master the result advertises
`resources` and `prompts`.
3. Send `resources/list` with the same token. The gateway returns 403
`gateway_token_action_denied`.

**Paperclip version or commit**

master at 499712c68b.

**Deployment mode**

Any deployment whose MCP clients use named gateway tokens with
restricted actions, including native runs.

## What Changed

- server/src/routes/tool-gateway.ts: `initialize` reads the session that
`initializeNamedGatewayProtocol` already returns, and advertises each
capability only when the token may list it.
- server/src/__tests__/tool-gateway.test.ts: a new test sends
`initialize` with five tokens (tools only, tools and prompts, tools and
resources, no list action, and no action list) and checks the exact
capabilities. It also checks that the tools-only token gets 403
`gateway_token_action_denied` on `resources/list`.

## Verification

At head e40a1f7d6e, run from server:

```sh
pnpm exec vitest run src/__tests__/tool-gateway.test.ts src/__tests__/tool-access-service.test.ts src/__tests__/tool-gateway-discovery-http.test.ts src/__tests__/mcp-http.test.ts src/__tests__/public-mcp.test.ts src/__tests__/tool-grant-scope.test.ts
```

- With server/src/routes/tool-gateway.ts from master and the test file
from this branch: 1 failed, 706 passed. The new test fails at its first
assertion: the tools-only token gets the tools, resources and prompts
capabilities, where the test expects only tools.
- With this branch: 707 passed.
- Typecheck: `pnpm --filter @paperclipai/server typecheck` reports 29
errors, the same 29 as unmodified master ("Cannot find module
@paperclipai/plugin-sdk" in the sandbox provider plugin packages of my
local install). None is in a touched file, and this pull request adds
none.
- The full upstream test suite runs in CI. I ran only the test files
named above.

## Risks

- Low risk. Only the `initialize` response changes, and only for tokens
that cannot list resources or prompts. Those methods were already
refused for such tokens.
- A client that called `resources/read` or `prompts/get` without listing
first no longer sees the capability advertised for a token that allows
only the read or get action.

## Model Used

- Claude, claude-opus-5-5, with tool use and terminal code execution.
The route change follows #17688.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:52:08 -06:00
DottaandClaude Opus 5.5 8a0f95849d perf(native-runtime): start the first turn without re-copying launch files, and never fail a run on a slow copy (#21287)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode in a remote sandbox must have a durable,
verified copy of the provider launch files before their first turn (the
rule from #19622)
> - Today each new run copies these files out of the sandbox, uploads
them and verifies them again, even when the company already has a ready
artifact with the same files
> - The whole copy runs under one five-minute clock. When eight runs
start together on small sandboxes, the copies compete, and slow copies
fail the run before its first turn
> - This makes cold starts slow (p50 273 s from run start to turn
submitted) and makes half of the first runs fail in a burst
> - This pull request reuses a ready artifact with the same digests,
shares one in-flight copy per company and manifest, and bounds each
network step instead of the whole copy
> - The benefit is that first turns start in about one minute on a cold
burst and in about 20 s when the artifact is already ready, and a slow
copy no longer fails the run

## Linked Issues or Issue Description

Refs #19622

**What happened?**
When 8 OpenCode runs started together on 1 vCPU Daytona sandboxes, 4 of
8 first runs failed before their first turn. Each run copied, uploaded
and verified the same provider launch files. The single 300 s clock over
the whole copy expired. Run start to turn submitted was p50 273 s and
p95 314 s.

**Expected behavior**
A run starts its first turn without a new copy when a ready, verified
artifact with the same files exists. A slow copy does not fail the run.

**Steps to reproduce**
1. Use a company with remote OpenCode agents on Daytona sandboxes with
durable storage.
2. Start 8 runs at the same time on new sandboxes.
3. See the time before the first turn and the failed runs.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- `retainReadyNativeProviderArtifacts` (`native-provider-artifacts.ts`):
reuse a ready, verified company artifact with the same paths and SHA-256
digests through a lookup only. No file is copied, uploaded or verified
again. The lookup (`readyNativeProviderArtifactCandidates`) matches
target, paths and digests in Postgres with jsonb containment on the
company's ready rows (existing partial index), newest first, limit 4. It
does not load or hash other retained versions. No migration.
- `shareNativeProviderArtifactCapture`
(`native-provider-artifact-capture.ts`): one in-flight capture per
company and manifest digest per server. Other runs wait for it, then
reuse its result.
- The single 300 s clock over the whole capture is replaced by a bound
for each network step (upload, verifying download): max(120 s, size at
256 KiB/s). A timeout has the typed cause
`native_provider_artifacts_transfer_timeout`. The capture makes up to 3
attempts from the local copy.
- Stop now interrupts the capture and the wait for a shared capture
(`ActiveNativeSession.admissionAbort`).
- `doc/opencode-cold-recovery.md` describes the new behavior.
- Tests: new `native-provider-artifact-gate.test.ts` (7 tests), 5 new
tests in `native-harness-backup-storage.integration.test.ts`, and new
executor variants `opencode-new-durable-slow-capture` and
`opencode-new-durable-ready` in `native-session-executor.test.ts`.

## Verification

Local (this branch, on master 9b613562b7):

- `npx vitest run
src/services/native-runtime/native-provider-artifact-gate.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
3 files, 1511 passed, 7 skipped.
- `npx tsc --noEmit -p server`: pass.
- Lookup scale: the integration test "loads only the ready artifact
whose launch files match, however many other versions the company keeps"
adds 200 ready non-matching versions. It returns only the matching
artifact. Without the SQL filter it fails (4 rows returned).
- Red/green: with master's `native-session-executor.ts` and the new
tests, `opencode-new-durable-slow-capture` fails with `TimeoutError: The
operation was aborted due to timeout`, and `opencode-new-durable-ready`
fails because the ready artifact is not looked up. With this change,
both pass.

Daytona (same commit content, cherry-picked onto the baseline commit
84a1ee3e11 with #21262; image `sha256:9461cc75…`; 1 vCPU sandboxes;
OpenCode):

| Wave | Shape | First runs OK | Run start → turn submitted | Other |
|---|---|---|---|---|
| Baseline `w-20261113T011027-s1702` | 8 runs started together | 4/8 (4
failed) | p50 273 s / p95 314 s | "only a getter" TypeError ×8,
`bootstrap_not_proven_incomplete` ×8 |
| Fix `w-20261113T031045-s1702` | same seed and shape | 8/8 | p50 66.0 s
/ p95 85.7 s | 8/8 tasks hands-off, 0 Retry banners; first agent event
p50 72.9 s / p95 90.0 s; exactly 1 upload, 7 runs waited on it (~65 s
shared capture); all failure codes 0 |
| Fix `w-20261113T034033-s1702` | 16 tasks, cap 8 | 16/16 | p50 19.1 s /
p95 24.3 s | second batch (artifact already ready, cold sandboxes) gate
time 24, 25, 26, 27, 31, 31, 45, 119 ms; first agent event p50 25.9 s |

## Risks

- The rule from #19622 still holds: a durable, verified artifact exists
before the first turn. Reuse only accepts a ready, verified artifact of
the same company with the same paths and digests.
- The first run per company and manifest per server still pays the full
transfer.
- Two servers can still upload duplicate artifacts for the same company
and manifest. Sharing is per server.
- No test drives Stop through the executor during a capture. Signal
handling is unit-tested only.
- A capture that fails all 3 attempts still ends blocked with a working
Retry. A follow-up PR (pre-turn failures continue automatically) removes
that.
- The documented five-minute deadline per capture is replaced by bounds
per network step. A very large artifact on a slow link can take longer
than five minutes in total, but each step still has a bound.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Default context
window; extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:48:55 -06:00
DottaandClaude Fable 5.2 5d298b1b48 test(server): judge the Linux real-process boundary cases only by the processes they start (#21284)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In `local_trusted` mode, the operator's local processes get implicit
board access. The local board boundary refuses that access to processes
of an agent run
> - #21209 added a Linux rule for a connection whose owner the server
cannot inspect. The server keeps board access only if every same-user
process it cannot inspect belongs to the operator
> - The real-process test "keeps board access for an operator process
the server cannot inspect" used that rule on the whole CI host. Other
processes of the same user on the host can be uninspectable and not the
operator's
> - Since #21209 merged, this test failed in about one in four `PR`
runs, on PRs that did not touch it. 22 failed server-test jobs between
2026-11-12 23:05Z and 2026-11-13 03:06Z
> - This pull request makes the real-process cases judge only the
processes that each case starts. It adds a case that pins the host-wide
rule with a known uninspectable agent-run process
> - The benefit is that the test is stable on shared CI hosts, and the
product rule stays tested

## Linked Issues or Issue Description

Refs #21209

**What happened?**
`server/src/__tests__/local-agent-process-boundary.test.ts` failed in
the "General tests (server)" lane of unrelated PRs. Both cases "keeps
board access for an operator process the server cannot inspect"
(127.0.0.1 and ::1) received `{ type: "none", source: "none" }`. They
expected `{ type: "board", source: "local_implicit" }`. The failures
started with the first runs that included #21209 (merged 22:52Z). They
continued until at least 03:06Z. Two runner image versions failed in the
same way.

**Expected behavior**
The test passes on a CI host that runs other processes.

**Actual behavior**
The boundary looks at every same-user process on the host that it cannot
inspect. When the owner of a connection is hidden, one uninspectable
process that is not clearly the operator's makes the boundary refuse.
Examples are a process of another agent run, a process whose parent is
PID 1, or a process that exits before the server reads its parent chain.
The test did not control which processes were on the host.

**Steps to reproduce**
On Linux, as a non-root user, run `python3 -c 'import ctypes,time;
ctypes.CDLL(None).prctl(4,0,0,0,0); time.sleep(60)' &`. Then make the
operator request from the test. The boundary refuses it.

## What Changed

- The Linux real-process cases use an inspector that wraps the real
Linux inspector. For a hidden owner, it keeps only the candidate
processes that descend from the process that the case started. All other
parts are the real Linux inspector: the TCP table reads, the fd scan,
the recheck, and the ancestry reads.
- Each Linux block starts an idle, uninspectable process with
`PAPERCLIP_RUN_ID`. This process stands for another agent run on the
host.
- A new case, "refuses board access to an operator process the server
cannot inspect while another agent run's process it cannot inspect is
running", uses the unwrapped inspector. It pins the host-wide rule in a
real-process test.
- If the test runner can inspect every process (for example, as root or
with `CAP_SYS_PTRACE`), the idle process is not uninspectable. The
host-wide case then skips. The other cases run as before.
- No product code changed.

## Verification

- Red, Linux (Docker `node:24-bookworm`, non-root user): a script runs
the product module
`server/src/middleware/local-agent-process-boundary.ts` from master with
the same operator request as the test. With no other process, the
operator gets board access 10/10. With one idle uninspectable agent-run
process, it gets 0/10. With one uninspectable process whose parent is
PID 1, it gets 0/10. With short-lived uninspectable processes, it gets
8/10. Agent-run requests are refused 5/5 in all cases.
- Red, CI Linux: in this PR's run, the new host-wide case passes. It
shows that the earlier assertion fails when one uninspectable agent-run
process is on the host.
- Green, CI Linux: in the same run, with that process on the host, both
"keeps board access for an operator process the server cannot inspect"
cases pass. Result: 27 tests passed in the shard that ran the file.
- macOS: `pnpm --dir server exec vitest run
src/__tests__/local-agent-process-boundary.test.ts` gives 15 passed, 12
skipped (Linux-only).
- Not verified: the specific process that was on the CI host in each
failed run. One fresh CI run with a temporary diagnostic found no other
uninspectable process. That run passed. The diagnostic is not in the
final commit.

## Risks

- Low risk. The change is test-only.
- Product note for a maintainer, not changed here: on Linux, an operator
process that the server cannot inspect loses implicit board access when
any other uninspectable same-user process on the machine is not clearly
the operator's. A process of another agent run is an example. This is
the rule from #21209, and the new case pins it. If this is too strict
for operators, that needs a product decision.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, Docker on Linux, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (not
applicable: test-only)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:48:46 -06:00
DottaandClaude Opus 5.5 0c04ff76eb fix(native-runtime): a follow-up that arrives as a warm session closes continues the task (#21279)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent session on a remote sandbox stays warm after a run,
and an idle timeout closes it after about 5 minutes
> - When a person's follow-up comment arrived just as that idle close
began, the new run failed with `native_session_supervisor_busy`
> - The idle close of the finished owner could also fail its final
checkpoint with `native_runner_checkpoint_owner_lost`, because the task
already pointed at the queued follow-up run
> - Then the retry was held for workspace repair and the task stayed
blocked; the environment could not be deleted (409) because its warm
runner could not be stopped
> - This pull request makes the follow-up wait for the idle close that
is in flight, and lets the finished owner save its final checkpoint
while its successor still waits in the queue
> - The benefit is that a follow-up that lands at the warm session's
idle close continues the task with no person involved

## Linked Issues or Issue Description

Refs #18011 (the checkpoint ownership fence this change extends).

**What happened?**

A follow-up comment arrived 295–340 s after the previous run ended,
while the warm session's idle close was in progress. The new run failed
with `native_session_supervisor_busy`. In another case, the idle close
failed its final checkpoint with `native_runner_checkpoint_owner_lost`,
the next run failed
`native_instruction_materialization_handoff_unavailable`, and the retry
was cancelled with `execution_reconciliation_required`. The task stayed
blocked. A hosted instance showed the same class as "previous run's
session on this sandbox could not be retired".

**Expected behavior**

The follow-up run waits for the idle close (inside the existing bounded
release wait) and then starts. The finished owner saves its final
checkpoint and closes. The task completes hands-off.

**Steps to reproduce**

1. Run an opencode agent on a Daytona sandbox with a warm session (idle
timeout 300 s).
2. Let a run finish.
3. Post a comment on the task 298–306 s after the run ended.

**Paperclip version or commit**

master `444dc18d98`.

**Deployment mode**

Local server with Daytona remote sandboxes (restart-chaos harness, light
lane). The same class was seen on a hosted staging instance.

## What Changed

- `claimWarmNativeInstructionCopy` now also waits when an idle close is
in flight for the session scope (`closingWarmNativeSessions`). The wait
uses the same deadline as the existing release wait
(`warmSessionReleaseWait`), so it adds no new wait and nothing before
`turn.start` on the normal path.
- `validateNativeRunnerCheckpointOwner` accepts the finished owner's
final checkpoint (accepted result, no new work) when
`issues.executionRunId` points at a successor run that is `queued` or
`scheduled_retry` for the same company and task and has no
`native_run_finalizations` row.
- Tests: a server test for a follow-up that arrives while the idle
timeout closes the warm session; four new scenarios (`queued_successor`,
`scheduled_retry_successor` × `runner.suspend`/`turn.stop`) and four new
negatives (`queued_wrong_issue`, `queued_new_work` × both commands) in
the early-retirement checkpoint matrix.
- Restart-chaos harness:
`RESTART_CHAOS_MIX_FOLLOW_UP_AFTER_END_MS=<min>-<max>` and
`RESTART_CHAOS_MIX_FOLLOW_UP_COUNT` post the staging-mix follow-up a set
delay after the task's last run ended, so a cell can aim the follow-up
at the warm idle close. The restart-chaos README describes both
variables. A failed post is retried with the same `clientRequestId`, the
grade check `after_end_follow_ups_served` needs an agent reply after
each post and the ledger marker, and the task budget includes the
planned waits.
- Harness: the wave database cleanup
(`tests/runner-e2e/controller-database.ts`) no longer waits forever when
a group signal (for example `timeout` or Ctrl-C) already stopped the
embedded Postgres. Before, `embedded-postgres` `stop()` waited for an
exit event that had already fired, and an interrupted wave hung after
the Daytona R6 sweep.

## Verification

- Fix 1, red without the product hunk: `native-session-executor.test.ts
-t "idle timeout is closing"` fails with `promise rejected "Error:
native_session_supervisor_busy"`. Green with the fix: 1 passed.
- Fix 2, red without the product hunk:
`native-runner-restart-recovery.integration.test.ts` fails 4 of 208
(`queued_successor` and `scheduled_retry_successor` ×
`runner.suspend`/`turn.stop`) with
`native_runner_checkpoint_owner_lost`. Green with the fix: 208 passed,
14 skipped. The two new negative cases pass in both states.
- Harness hang fix, red without the fix: the new
`controller-database.test.ts` case ("finishes cleanup when the database
already stopped because a group signal reached it") times out on cleanup
(`expected 'hung' to be 'cleaned'`). Green with the fix: 5 passed, 1
skipped.
- After the rebase onto master `7154030ef6`, both server test files
together: 1639 passed, 21 skipped.
- Restart-chaos harness tests (`staging-mix`, `staging-mix-retry`,
`observer`): 75 passed, including a test for the after-end retry, the
`after_end_follow_ups_served` grade check and the added budget. `tsc
--noEmit` passes for `server`, `tests/runner-e2e` and
`tests/restart-chaos`.
- `pnpm verify` exits 2 because `tests/restart-chaos/observer.ts` is a
global CI input. Required CI supplies the broad coverage.
- Daytona cell on this branch (4 opencode tasks, warm sandboxes, 1 vCPU,
follow-up comment posted 300.8–308.2 s after each task's last run ended,
image built for master `444dc18d98`): 4/4 tasks done hands-off, 8/8 runs
succeeded, 0 Retry banners, 0 `owner_lost` / `supervisor_busy` / "could
not be retired" / `execution_reconciliation_required` lines in the
server log, environment delete clean, warm-shutdown grade pass, 0
sandboxes left. The harness exited by itself after 1076 s.
- Same cell before the fix (master, 8 tasks): 1/8 tasks blocked
(`native_session_supervisor_busy` →
`native_runner_checkpoint_owner_lost` →
`execution_reconciliation_required`). With fix 1 only: 1/8 still blocked
(`owner_lost` on the idle close). With both fixes: 8/8 done and 0 failed
runs (cut by the 20-minute cap before cleanup), then 4/4 above.
- The cell grade `T:completion_once` fails on all 4 tasks by design: the
timed follow-up comment arrives after the task is done, so the agent
posts a second completion comment.

## Risks

- Fix 1 can delay a follow-up by up to the existing warm release wait
(30 s) while an idle close finishes. Before this change, that run
failed. If the close fails or the wait expires, the claim returns null
and the run uses a fresh instruction copy, as before.
- Fix 2 widens the checkpoint ownership fence from #18011. It does not
reopen what #18011 closed:
- The exemption applies only to the finished owner's final save: the
checkpoint must hold an accepted result and no new work, so no provider
turn can be granted through it.
- The successor must still be unclaimed (`queued` or `scheduled_retry`)
and must have no native coordinator (`native_run_finalizations`) row. No
controller has started it, so nothing can have consumed or depended on
the predecessor's checkpoint. Once the successor is claimed, the
existing preparing/claimed rules from #18011 apply unchanged.
  - The successor must belong to the same company and the same task.
- Cases that must still reject, with tests in the same matrix:
- `queued_wrong_issue`: a queued successor for a different task is
rejected with `native_runner_checkpoint_owner_lost` and the stored
checkpoint does not change.
- `queued_new_work`: a `turn.start` (new work) while the successor is
queued is rejected with `native_runner_checkpoint_owner_lost` and the
stored checkpoint does not change.
- All earlier rejections stay: unclaimed native coordinator,
dispatching, expired or foreign preparation, foreign native preparation,
wrong task, new work, unaccepted result, replaced controller.
- Not changed: an owner whose close fails still blocks environment
delete (409). That is a separate product decision.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Earlier commits on
this branch were written by the same model in an earlier session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:42:50 -06:00
DottaandClaude Fable 5.2 c4f639ed1b fix(ci): re-run cancelled runs that lost a runner, and let runner-loss evidence win (#21291)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - PR CI runs about 50 jobs per run on AWS spot instances. AWS can
reclaim a machine mid-job, and the job dies.
> - `ci-runner-loss-rerun.yml` re-runs such a run once, automatically,
when every failed job lost its runner.
> - In a 17-hour sample, 42 runs lost a runner and were still the PR's
current head. The automation rescued only 14.
> - Two gaps caused most of the misses. A spot reclaim on RunsOn v3.2.3
usually ends the job, and the run, as `cancelled`; the workflow and
script only look at `failure`. Also, the Runner support suite always
carries `gateway_indexing_incomplete` error annotations from a passing
test, so a lost runner there reads as a real failure.
> - On-demand fallback for the fleet was turned off on 2026-11-13, so
reclaims will happen more often. About 20% of runs are expected to lose
at least one job.
> - This pull request makes the re-run act on cancelled runs and jobs,
but only with positive runner-loss evidence, and lets that evidence win
over other error annotations.
> - The benefit is fewer manual "Re-run failed jobs" clicks: an
estimated 34 of 42 instead of 14 of 42 in the sample.

## Linked Issues or Issue Description

No public issue exists.

**What happened?**
Run
[43285064435](https://github.com/paperclipai/paperclip/actions/runs/43285064435)
lost the runner of `ci / General tests (server (8/12))`. The job
concluded `cancelled` (its log has `##[error]The runner has received a
shutdown signal`). The run concluded `cancelled`, so the re-run
workflow's `if:` skipped it.

Run
[43325855069](https://github.com/paperclipai/paperclip/actions/runs/43325855069)
lost the runners of `ci / Runner support suite` and `ci / General tests
(server (2/12))`. Both logs have the shutdown signal. The re-run bot
([decision
log](https://github.com/paperclipai/paperclip/actions/runs/43327873358))
said "1 of 1 failed jobs did not lose their runner", with evidence
`gateway_indexing_incomplete`. It ignored the cancelled job. It read the
annotation as a real failure.

**Expected behavior**
Both runs get one automatic re-run of their failed jobs.

**Steps to reproduce**
The new tests replay trimmed snapshots of both runs
(`.github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json`
and `run-43325855069.json`) through the script.

In a 17-hour sample on 2026-11-12, 42 runs lost a runner and were still
the PR's current head. Only 14 were re-run automatically. 16 stayed red
because the run concluded `cancelled`. 4 stayed red because an error
annotation in the lost job was read as a test failure.

**Additional context**
The `gateway_indexing_incomplete` annotation does not come from the
shutdown. `scheduled-gateway-indexing.test.ts` prints `::error` lines
while it passes, 84 seconds before the shutdown signal in this run.
Attempt 2 of the same run passed with the same two failure annotations.
So, before this change, a lost runner in the Runner support suite could
never be re-run.

## What Changed

- `ci-runner-loss-rerun.yml`: the job also runs when the PR run
concluded `cancelled`. The trigger type, `permissions`, the repository,
path, event and attempt-1 conditions do not change.
- `runner-loss-rerun.mjs`, `eligibleRun`: accepts `completed/cancelled`
as well as `completed/failure`.
- `runner-loss-rerun.mjs`, `decide()`: classifies jobs that concluded
`cancelled` as well as `failure`. It still skips the `ci / verify` and
`ci / e2e` aggregates, and a failed or cancelled routing gate still
blocks the re-run.
- `decide()` now runs `staleReason` (open PR, current head, no newer
run) before it reads any annotation or log, and again just before the
re-run, as on master. A run cancelled by a new push has about 40
cancelled jobs; it now costs two API listings and no log reads. The
final check closes the window where a push lands during the log reads: a
re-run of the old run would join the PR's `cancel-in-progress`
concurrency group (`pr-<number>` in `pr-trusted.yml`) and cancel the new
commit's CI.
- Following CONTRIBUTING.md ("Do not submit a PR that includes code
comments"), this PR adds no new code comments. The reasons for each rule
are in this description and in `doc/DEVELOPING.md`.
- `classifyJob`: runner-loss evidence wins. A lost-communication
annotation means lost. A cancellation annotation plus a shutdown signal
in the log means lost, even if the job also has other error annotations.
A cancelled job without this evidence (for example, a person cancelled
the run) counts as a real failure and blocks the re-run.
- Fixtures: two new trimmed snapshots from the example runs (jobs with
`id`, `name`, `status`, `conclusion`; failure annotations; `##[error]`
log lines only). The two older fixtures get the real annotations and log
error lines of their cancelled jobs. Those jobs were spot-reclaimed too
(each log has the shutdown signal), but the old code ignored them.
- `doc/DEVELOPING.md`: the runner-loss paragraph describes the new
rules.

## Verification

- Red first. Against the unchanged script, the new tests fail for the
reasons above: 43285064435 gives `run is completed/cancelled, not
completed/failure`; 43325855069 gives `1 of 1 failed jobs did not lose
their runner` with evidence `gateway_indexing_incomplete`, the same
decision the bot logged.
- Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`:
25 of 25 pass.
- Push during log reads: the PR head moves after the first log read.
With the final freshness check removed, this test fails (the script
re-runs); with it, there is no re-run and the reason is `no open pull
request still has this run's head commit`.
- Guard cases, all with no re-run and no POST:
- Superseded run: 52 jobs cancelled, the PR head moved. The test asserts
zero annotation and log reads.
- Person cancelled the run: cancelled jobs with only "The operation was
canceled." in the annotation and log, and some never-started jobs with
no annotation.
- Mixed: one cancelled lost job and one real test failure (`1 of 2
failed jobs did not lose their runner`).
- Once only: a cancelled run on attempt 2 is refused after one API read.
The workflow condition test still requires `run_attempt == 1`.
  - Cancelled routing gate: left for a person.
- Policy job command: `node --test '.github/scripts/tests/*.test.mjs'`:
806 of 806 pass.
- `pnpm verify`: exit 2 ("Global CI input changed:
.github/scripts/runner-loss-rerun.mjs"). It selects no local Vitest
files and asks for required CI coverage. I did not run `pnpm verify
--full`; required CI on this PR supplies that coverage.
- This workflow runs on `workflow_run` from the default branch. This
PR's own CI cannot exercise the new behavior. The fixture tests are the
evidence. I did not dry-run the script against live runs.

## Risks

- A test that really fails, and then a spot reclaim of the same job a
minute later, now gets one wasted re-run. Attempt 2 then fails for real.
It cannot loop: the workflow `if:` and `eligibleRun` both accept attempt
1 only.
- A run a person cancels on the current head now starts the re-run job.
The job reads the annotations, and the log of each job with a
cancellation annotation, and then does not re-run. This costs some API
calls on a GitHub-hosted runner.
- A concurrency cancel by a label event that does not request CI (the
newer run is `skipped`) passes the head check. Its cancelled jobs have
no runner-loss evidence, so it is not re-run.
- Trust: no change to `permissions`, the trigger, the accepted
repository, event, workflow path, attempt, or the routing-gate rule. A
cancelled run reuses its attempt-1 gate output, the same as a failed run
does today.
- Estimated effect, from a 17-hour sample: 14 of 42 rescued before; 16
cancelled runs and 4 runs with error-annotation noise become eligible
(about 34 of 42). The other 8 stay skipped on purpose (a real test
failure in the same run, a person re-ran first, or the PR closed).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent. Model ID claude-opus-5-5, with
tool use (shell, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:31:50 -06:00
Awssandcommitperclip[bot] af1c685f5f fix(secrets): return 422 for secret proposals missing a required field (#15348)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can propose a company secret or a secret binding through
the API.
> - The proposal route reads the request body and passes it to the
service.
> - The service trimmed required string fields before it checked them.
> - A request that omitted name or justification threw a TypeError.
> - The route then returned 500 instead of a validation error.
> - This pull request checks the required fields first and returns 422.
> - The benefit is that a client can tell a bad request from a server
fault.

## Linked Issues or Issue Description

Closes #15334

## What Changed

- Check `name` and `justification` in `createSecret` before `trim`, and
return 422 when either is missing.
- Check `justification` in `createBinding` the same way.
- Keep the existing empty-value and malformed-value checks and the
optional `key` field unchanged.
- Add a regression test that covers the missing-field cases for both
proposal kinds and keeps the complete-body case at 201.

No documentation change is needed. The OpenAPI document already lists
422 for this route, so the server now matches the documented contract.

## Verification

- Run the route suite. It passes.

`./node_modules/.bin/vitest run
server/src/__tests__/secret-proposals-routes.test.ts` gives 26 passed.

- Revert only the service change and run the new test. It fails with
`expected 500 to be 422`, which proves the test catches the bug. Restore
the change and it passes again.

- Run the server typecheck. It passes.

`pnpm --filter @paperclipai/server run typecheck` exits 0 with no
errors.

- A `POST /api/agents/me/secret-proposals` that omits `justification`
returns 422 after the fix and returned 500 before it. A complete body
still returns 201.

## Risks

- Low risk. The change adds two field checks before an existing `trim`
call. Valid requests still return 201. A missing required field now
returns 422, which the route already documents.

## Model Used

Claude Opus 4.8 (`claude-opus-4-8`), with extended thinking and tool
use. I chose the issue, set the evidence bar, directed each step, and
reviewed the code, the tests, and the results before this PR. The model
traced the route and the service, wrote the fix and the regression test,
and ran the checks under that direction.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
2026-11-12 22:31:42 -06:00
Nicky LeachandClaude Fable 5.2 25e9a9c2e9 feat(ci): publish both variants of the Runner image from master (#21230)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Parts 2 and 3 of this stack build the Runner alone as an image with
two variants, and check it on pull requests. Nothing publishes it
> - Downstream sandbox and VM images can copy the Runner only from a
registry, and they must be able to verify where it came from
> - A published image must be exactly the bytes that were checked, and
its name must never point at something else later
> - This pull request publishes both variants from `master` as
`ghcr.io/paperclipai/paperclip-runner`, one time for each distinct set
of inputs, and attests them
> - Nothing consumes the image in this pull request. The application
image build does not change
> - The benefit is one verified, attested Runner that later changes can
copy by digest

## Linked Issues or Issue Description

Refs #21204 (part 2 of this stack) and #21096 (the merge-queue image
build). No issue exists for this change.

**Problem or motivation**

The Runner image exists only inside a pull-request check. A consumer
cannot pull it. There is no rule for which commit publishes an image, no
protection for its tags, and no way to verify a copied Runner.

**Proposed solution**

Publish from a `master` push in this repository, and from nothing else.
Build only when no image has the digest of the commit's inputs. Let the
build legs hold no write access. Push the bytes that the legs checked by
digest, attest each multi-platform index, and write the tags last. Write
the full tag after the slim tag, so that the full tag means "both
exist".

**Alternatives considered**

- Publish from `docker.yml`. That workflow holds one concurrency slot
for each ref, and the Runner image needs its own signer identity, so
that its attestation cannot stand in for the application image's.
- Resolve the lockfile in the workflow before a `master` build, as
`docker.yml` does. Then the published Runner depends on the state of the
npm registry at run time, and the digest cannot be computed again from a
commit.
- Hand each variant to the publish job as its own OCI archive. The two
archives hold the two shared layers two times: 1.27 GB for each
architecture in place of 0.85 GB.
- Write the full tag first. Then a consumer can see a full image whose
slim image does not exist yet.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 4 of 4. Each part is on top of the one before it:

1. #21228 (merged): an optional field in the provider pack manifest
2. #21204: build and check the Runner image, with no publication
3. #21229: layers by change rate, and the slim variant
4. #21230: publish both variants from `master` (this pull request)

## What Changed

- `.github/workflows/runner-image.yml`: adds the trigger `push` to
`master` and the job `publish`.
- `identity` checks the committed lockfile against the Runner's
manifests, computes the digest in the build context, and asks the
registry whether these inputs are published.
- `build` is the job of part 3. It also records what it built, on every
event. On a canonical `master` push it uploads that. It cannot write to
the registry on any event. The only token in the job can read packages,
and only the lookup step receives it.
- `publish` runs only for a `master` push in this repository. It builds
nothing and runs no dependency code.
- Lockfile rule. A `master` push builds from the committed
`pnpm-lock.yaml` and resolves nothing, so the digest is a function of
the commit. When the committed lockfile is behind the manifests that the
Runner build reads, the run writes a notice, publishes nothing, and
stays green. The commit that merges the lockfile refresh publishes the
image. Pull requests and manual runs resolve the lockfile first, always
build, and never publish.
- Hand-off. Each leg joins its two OCI archives into one OCI layout that
names the images `slim` and `full`. The layout stores the two shared
layers one time. `record` then reads the layout back, each image by its
name, and checks every blob against its digest. The leg's record binds
the layout to both image digests, the architecture, the inputs digest,
and the commit. Those digests are the digests of the archives that part
3 unpacks and checks, so the tree that is checked and the image that is
pushed are the same bytes. The publish job accepts only records for its
own inputs and commit, and only a layout that still has the recorded
bytes.
- `record` runs on pull requests too. Only the upload is for a `master`
push. A pull request therefore runs the hand-off on real build output.
- Publication order. The publish job pushes the four images by digest
(`skopeo copy --preserve-digests`). It makes a candidate index for the
slim variant, then one for the full variant that names the slim index in
the annotation `io.paperclip.runner.slim-digest`. It verifies that each
index names exactly the recorded images, attests both, and then writes
`inputs-<digest>-slim` and, last, `inputs-<digest>`.
- The full tag is the completion marker. Every "is it published" lookup
reads the full tag. `publication-state` reports one of:

  | State | Registry | What a run does |
  | --- | --- | --- |
  | `absent` | Neither tag | Builds and publishes |
| `incomplete` | The slim tag alone | Builds and publishes. It replaces
the slim tag with its own image, then writes the full tag |
| `published` | The full tag, and the slim tag at the digest that the
full index names | Nothing |
| broken | The full tag, and a slim tag that is missing or points at
another image | Fails, with the steps for an operator |
| `unknown` | No clear answer | Builds. The publish job fails before it
writes a final tag |

- Recovery rule. A run that stops between the two tags leaves the state
`incomplete`. It does not block those inputs. The next run never keeps
the earlier slim image: it writes its own slim tag immediately before
its full tag, verifies both against its own index digests, and then
reads the pair back as every later run will see it. The broken state
cannot be repaired by a run, because no run moves a full tag. An
operator must delete the two package versions and re-run the workflow
run of a `master` push with those inputs.
- Pairing rule for consumers, in the contract: resolve the full tag
first, take the slim digest from the annotation of the full index, and
pull the slim image by that digest.
- `verify-index` accepts `linux/arm64` with the variant `v8`, which some
BuildKit versions record, and no other variant.
- `scripts/runner-image-publish.mjs` (533 lines): the commands
`lock-state`, `record`, `verify-legs`, `publication-state` and
`verify-index`. It reads archives with the reader of
`scripts/runner-image-oci.mjs` (607 lines here), which can now read one
named image of a layout. `lock-state` refuses a checkout, because pnpm
can rewrite the lockfile of the tree that it checks. The workflow gives
it an export that no build reads.
- The workflow test pins the build arguments of the image build. Build
arguments are not part of the image identity.
- `doc/runner-image-contract.md` (197 lines): the section "Publication":
the lockfile rule, the order, the pairing rule, the recovery rule, the
create-only tags, the internal `candidate-` tags, the command that
verifies provenance, and the step to make the package public.
`doc/DOCKER.md` says where the image is published.

## Verification

Tests, and the build jobs of the workflow on GitHub. Nothing in this
pull request ran against a registry, and no publish job has run.

- Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and
zstd): `runner-image-publish.test.ts` (75),
`runner-image-workflow.test.ts` (12), the three other Runner image test
files, the provider pack archive tests and the existing Dockerfile
tests: 372 tests pass in 10 files. The same files pass on macOS.
- The tests of the leg record failed on Linux in the first runs of this
pull request: their fixtures named the archive members `./index.json`,
which bsdtar finds and GNU tar does not. The fixtures now name the
members as buildx does, and the publish script uses the one reader of
part 3.
- Leg records: both variants are accepted for this run's inputs and
commit; other inputs, another commit, a changed layout, a record without
a variant, an archive that lacks a blob, a full image that does not
start with the slim image's layers, and a build that is not a
publication build are refused.
- Publication state: all five states, and the half-finished publication
end to end. A slim tag alone lets the next run publish. A full image
that is left with another run's slim tag is reported as broken.
- Index: exactly the recorded images of one variant; the full index must
name its slim index, and the slim index must name none; `v8` is accepted
on arm64 only.
- Registry lookups: a missing tag only when the registry says so; every
refused, failed, or unreachable request is `unknown`.
- Workflow: job permissions; the publish job and the upload are behind
the canonical `master` push; the build steps receive no credential; the
full tag is written last, after a strict lookup and both attestations;
one export and a time limit for each build command; the two build
arguments; actions pinned by commit; no expression inside a shell
script.
- `record` on the two OCI archives of a real `linux/arm64` build from
part 3: the joined layout is 850,149,376 bytes, and both images read
back by name with every blob under its digest. The two archives together
are 1,273,646,592 bytes. The image digests in the record are the digests
that `unpack` reports for the trees that were checked.
- `tests/runner-e2e` workflow tests: 39 pass. `node --test
.github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image
contract scripts: 33 pass. `docker buildx build --check` reports
nothing. The image identity is the same as in part 3
(`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`),
because this pull request changes no input of the image.

GitHub, at this head:

- The `Runner image` workflow passes with these changes: `Build and
check (amd64)` in 18m30s and `Build and check (arm64)` in 7m48s. Each
leg builds both variants, checks them, and writes the leg record.
`Publish and attest` is skipped, as it must be on a pull request.

Never run:

- The publish job. It runs only for a `master` push.
- The publish path: the registry lookups with a workflow token, `skopeo
copy` from `oci-archive:<layout>:<name>` by digest, `docker buildx
imagetools create` with index annotations, and the tag writes.
- The attestations, and the `gh attestation verify` command in the
document.
- `actionlint`. It is not installed on the local host.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- **Manual step.** GitHub creates the `paperclip-runner` package private
on the first publication. A maintainer must make it public one time.
Until then, consumers cannot pull the image without credentials.
- The first `master` runs of the workflow are the first test of the
publish path. A failure before the tags publishes nothing. A failure
between the two tags leaves the `incomplete` state, and the next run
with those inputs repairs it.
- A slim tag can move while its publication is incomplete. A consumer
must resolve the full tag first and take the slim digest from the
annotation of the full index. It must never resolve
`inputs-<digest>-slim` by tag on its own. The contract says so.
- `candidate-` tags are written before the attestations and can name an
index that is not attested. They are internal.
- The broken state needs an operator, and it fails every `master` run
with those inputs until then. It can arise only when someone changes or
deletes a tag outside the workflow.
- **Known residual.** Two `master` runs that both start before a new
digest is published can each build the image. Each build leg and the
publish job look again before they work, and the publish job runs one at
a time for each digest. The cost is runner time, not correctness.
- A commit whose lockfile is behind the Runner's manifests has no
published Runner image until the lockfile refresh merges. A lockfile
that the refresh cannot fix leaves every `master` run green with only a
notice, for as long as it lasts.
- The tags are create-only because the workflow never moves a tag of a
complete publication. The registry does not enforce that. Consumers must
pin the digest and verify the attestation.
- `payload.runnerSourceRevision` in a pack from this image is the commit
that published the image. Code that reads it as the application's commit
must change before it consumes the image.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 20:27:36 -08:00