Commit Graph
2348 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 0986c3eb3a fix(native): let a same-run reattachment take over its own retained remote Runner cleanup (#21328)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 keeps an unreachable remote Runner alive when the failed
attempt closes, so that the same-run reattachment can adopt it
> - The same close also leaves the session in the in-process cleanup
quarantine. The quarantine refuses every new session of the task, and
that includes the reattachment that the Runner was kept for
> - Close retries cannot settle it. After the first close fails, the
Runner transport returns the same rejected close for every retry
> - On staging, this was the largest failure class: 14 Codex runs in one
hour failed with `native_session_cleanup_quarantined`, while only 5 runs
succeeded. Each task then needed Retry
> - This pull request lets the same run's reattachment to that Runner,
in the same remote sandbox, take over its own retained cleanup
> - The benefit is that the task continues on its own, with no spent
attempt, no failed run and no Retry

## Linked Issues or Issue Description

Refs #20923, #21031

**What happened?**
A remote native run (Codex on Daytona) failed a turn with a recoverable
error, for example `runnerd did not report the provider turn identity`
or `native_finalization_missing: session returned no semantic result`.
The close then logged `runner did not prove durable suspension before
checkpoint` and `remote runner left alive unsettled for same-run
reattachment`. Each later same-run reattachment was refused at
admission:
- When the close error was retryable, admission waited out the grace and
failed with `native_session_cleanup_quarantined: prior session cleanup
remains incomplete`. That spent a same-run attempt. After the attempt
budget, the run failed. All close retries replayed the same failed close
within milliseconds (`runner transport close requested: native session
quarantined cleanup recovery`), so they could never settle the
quarantine.
- When the close error was `native_session_close_unrecoverable`
(operator-required), the next attempt failed at once with
`native_session_cleanup_quarantined: the previous session is still
shutting down; Paperclip finishes the cleanup automatically and the
saved work is kept.` That code is a permanent failure.

"Paperclip finishes the cleanup automatically" did not happen in the
process. Only a server restart (a deploy) removed the in-memory
quarantine, and by then the run had failed and the task waited for
Retry. Staging restarts every 20 to 45 minutes, so attempts often end
mid-cleanup and meet this gate.

**Expected behavior**
The same run's reattachment adopts the Runner that its earlier attempt
kept for it. It does not wait for a close that cannot settle, and it
does not spend an attempt. The task continues without Retry. Other runs
of the task and other sandboxes stay gated.

**Steps to reproduce**
1. Start a remote native run, and let its turn fail with a recoverable
transport error while the Runner cannot prove suspension.
2. The close keeps the Runner alive and quarantines the session.
3. The server schedules the same-run reattachment 30 s later.
4. The reattachment fails with `native_session_cleanup_quarantined`, and
the task needs Retry.

**Paperclip version or commit**
Deployed Core `c730872974` on staging. Reproduced on `master` at
`49f0030f06`.

**Deployment mode**
Hosted cloud (staging), Daytona remote sandboxes.

**Agent adapter(s) involved**
`paperclip_runner` native runtime with the Codex app server backend.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
`executeNativeSession` takes a new option `retainedRunnerReattachment`.
With a remote cleanup scope, admission retires quarantine entries that
belong to the same run in the same remote scope. Removal stops later
close retries. A close attempt that is still in flight stays an
admission owner until it settles, so the old and new controllers do not
overlap. Other runs, other sandboxes and local backends keep the gate.
- `server/src/services/native-runtime/native-session-executor.ts`: sets
`retainedRunnerReattachment` only for a `reattach_remote_runner`
recovery of the same run.
- `doc/execution-semantics.md`: records the takeover.
- Tests:
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: four
new cases. An operator-required close with a retained Runner. A
retryable close whose retries replay one failure. A close that a deploy
interrupts while it is in flight (the reattachment waits for it, then
takes over). Controls: another run, a run without the takeover option
and a local backend stay refused.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
asserts that the option is set for a same-run remote reattachment and is
not set for a fresh sandbox run.

## Verification

- Red on `master` (runtime source restored): 3 of the 4 new runtime
cases fail with exactly the two staging errors (`the previous session is
still shutting down…` and `prior session cleanup remains incomplete`).
The control case passes.
- Red on `master` (executor source restored): the 3 executor assertions
fail.
- Green with this change:
- `cd packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts` (294 passed)
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "exact
Stop cleanup|fresh sandbox|retirement"` (46 passed)
- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/completed-turn-cleanup-hold-settle.test.ts` (61 passed)
- `npx tsc --noEmit` in `packages/paperclip-runner` and `server` passes.
- Staging evidence: runs `7b7ded99` and `27c487ef` show the sequence
above in their events and logs.

## Risks

- The takeover applies only when the server recovers the same run with
`reattach_remote_runner` in the same remote cleanup scope. This is the
case that #20923 keeps the Runner alive for. After a deploy the
quarantine is already gone and the same reattachment runs today, so the
takeover gives the in-process retry the same behavior as a restart.
- A close attempt that is in flight is never overlapped. The
reattachment waits for it within the existing admission grace.
- The reattachment still verifies Runner ownership and its controller
generation before it adopts the Runner. If the Runner is gone, the
existing relaunch or reconciliation path handles it.
- No persisted shape changes. Saved state from the previous release is
not affected because the quarantine lives only in process memory.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:15:40 -06:00
Nicky LeachandClaude Opus 5.5 41998d551c feat(plugins): send the claimant's credential on plugin task resume (#21335)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugin environment drivers run agent tasks on remote computers, and
a warm follow-up run resumes a parked Runner through the `resume` task
operation
> - `resume` carries only the durable identity and a bootstrap ticket,
which is enough while the parked Runner process is still alive
> - A provider sometimes has to start the Runner again to resume the
session: after it moved the parked session to a new sandbox, or after
the computer rebooted
> - A Claude Runner cannot start without its credential, and today only
the portable handoff path (`handoffCredential`) delivers one
> - This pull request lets a provider opt in to receiving the claimant
run's harness credential on `resume`
> - The benefit is that a provider can continue a parked session on a
new computer without Core orchestrating a handoff

## Linked Issues or Issue Description

Refs #21289
Related: #21305

**What existing behavior does this improve?**
Warm resume for plugin task providers that restart the Runner to resume
a session.

**Subsystem affected**
Plugin SDK environment task schema, the server's plugin Runner task
owner, and the plugin environment task docs.

**Current behavior**
`submit` can carry the harness credential (`credential`), but `resume`
cannot. A provider that must start the Runner again on resume has no
credential for a Claude task.

**Proposed behavior**
A provider that returns `resumeCredentialVersion: 1` from `describe`
receives `credential` on `resume` whenever Core would send it on
`submit`, resolved for the claimant run. Other providers see no change.

**Reason and benefit**
Providers can restart a Claude Runner on resume without the Core-run
handoff protocol.

**Breaking changes**
None. The field is opt-in: the SDK worker parses operations strictly, so
Core sends it only to providers that advertise the capability.

## What Changed

- `resume` gains an optional `credential`, with the same schema and
validation as `submit`'s.
- `describe` gains an optional `resumeCredentialVersion: 1` capability.
- The plugin Runner task owner adds the credential on `resume` when the
provider advertised the capability and Core resolved a credential, using
the same resolution as `submit` (`preparePluginRunnerCredentials`).
After a warm rebind, the claimant run's credential is sent.
- `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` documents the field and its
handling rules.
- The runner-e2e SDK public-input pin is updated.

`handoffCredential` on the session handoff path is unchanged.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts` (68 passed): `resume` accepts and
rejects the same 11 credential cases as `submit`; the capability is
optional and rejects other versions; a resume credential reaches the
plugin through the serialized worker RPC.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts` (151 passed): credential sent with
the capability and a resolved credential; omitted without a credential
or without the capability; the rebound claimant's credential is sent;
the credential appears in no metadata write and not in the sanitized
error, even when the worker error echoes it.
- `native-session-executor.test.ts -t "uses a task-only"` (5 passed):
the executor forwards the capability to the owner.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts` (46 passed).
- `tsc --noEmit` for `packages/plugins/sdk` and `server`.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- No server-side provider check on resume, unlike `submit`: the lease
has no reliable harness at resume time. Protection relies on the owner,
whose rebind refuses a changed harness, account or credential identity.
- The credential is the claimant run's current one, which can be newer
than the one that started the parked Runner. That is intended.
- A provider must advertise the capability only after its SDK pin
includes this schema; roll Core out first.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:40:46 +00:00
DottaandClaude Opus 5.5 7e7452e88f fix(native): keep a reattached Runner alive when its provider recovery deadline ends (#21321)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs on remote sandbox Runners. A same-run reattachment
adopts a live Runner after a failed attempt or a server deploy.
> - On staging, a remote Codex Runner is often gone when the next
attempt reattaches (`process_verification` / `process_not_running`). The
task then continues in a fresh session and loses its context.
> - The deploy does not stop the Runner. The logs show `controller
authority detached for restart; durable runner left alive`, and the new
controller adopts the same Runner after the deploy.
> - Our own cleanup stops it. The adopted Runner answers its recovery
reads slowly while its in-flight turn runs. The 900 s provider recovery
deadline then aborts the recovery, and the abort closes the transport
with `turn.stop` and `runner.suspend` and signals the process.
> - This pull request makes the recovery deadline detach from the
Runner, as a recovery read timeout already does. The server marks the
failure as an unanswered adopted Runner, so the next attempt replaces
the Runner from its saved journal and resumes the same provider thread.
> - The benefit is that a deadline no longer kills a live Runner and no
longer forces a fresh session. The task keeps its session context.

## Linked Issues or Issue Description

Refs #21314, #21272, #21161, #21023

**What happened?**
Codex runs on Daytona remote Runners failed often on staging. In three
sampled chains (attempt 2 of each), the same-run reattachment adopted
the live Runner, authenticated in under 1 s, and then waited the full
deadline: `native session provider recovery timed out after 900000ms`
(909 s, 930 s and 915 s). The deadline cleanup sent `turn.stop` and
`runner.suspend` (`provider turn stop timed out before runner
suspension`). The next attempt found the Runner gone 25–180 s later
(`process_not_running`, `runnerLifecycle: ready`, empty stderr) and
continued in a fresh session. During the 900 s wait, the in-flight turn
was still running Paperclip tools (`tool.execute` spans at 04:43 in a
recovery that started at 04:29).

**Expected behavior**
A remote Runner survives a recovery deadline and a control-plane deploy.
The next attempt continues the same provider session. A Stop still stops
the Runner.

**Steps to reproduce**
1. Start a Codex run on a remote Runner and let a turn start.
2. Fail the attempt so that its close keeps the Runner (for example, a
turn start that outlives its bound).
3. Let the same-run reattachment adopt the Runner, and make the Runner
stop answering recovery commands.
4. Let the provider recovery deadline expire.
5. See that the Runner process is gone and the next attempt reports
`process_not_running`.

**Paperclip version or commit**
master at b555087f0f (after #21314).

**Deployment mode**
Hosted staging (cloud), remote Daytona sandbox Runners, deploys every
20–45 minutes.

**Agent adapter(s) involved**
Native Codex (`paperclip_runner`, Codex app-server through
`paperclip-runnerd`).

## What Changed

- Add `NativeProviderRecoveryDeadlineError` (same message as before) and
use it for the provider recovery deadline in the native session runtime.
- In the Codex driver, a recovery aborted by this deadline detaches the
controller (no `turn.stop`, no suspend, no signal). A Stop and every
other abort still close and stop the Runner.
- On the server, `nativeRecoveryCommandUnanswered` recognizes the
deadline. The failure records `adoptedRunnerUnanswered`, so the next
claim carries `replaceUnansweredRunner`. That attempt fences the old
Runner lineage, launches a fresh Runner from the saved journal and
resumes the same provider thread (the #21272 path). It does not adopt
the slow Runner again for another 900 s.
- Update `doc/execution-semantics.md`.

## Verification

- Real `paperclip-runnerd` + fake Codex
(`runnerd-codex-transport-remote-retention.test.ts`): a retained Runner
is adopted, authenticates, then stops answering. A deadline abort leaves
it alive with no pending `turn.stop`/`runner.suspend`/`runner.shutdown`.
A Stop abort still stops it.
- Red/green: without the fix, the real-runnerd deadline test fails
(`expected 'gone' to be 'alive'`), the driver unit test fails (detach
called 0 times), and the server classification test fails. With the fix,
all pass.
- Measurements on real runnerd: the deadline cleanup took 10.0 s and
killed the Runner before. Now it returns in 0 ms and the Runner stays
alive. The replacement over the kept Runner's journal resumes the same
provider thread in about 2.2 s with one `thread/start` and one
`turn/start` (no replay).
- `npx vitest run
src/live/runnerd-codex-transport-remote-retention.test.ts
src/drivers/codex/codex-app-server-driver.recovery.test.ts
src/native-session-runtime.test.ts` (368 passed).
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "never
answered its recovery read"` (passed).
- `tsc --noEmit` for `packages/paperclip-runner` and `server` (clean).

## Risks

- A deadline now leaves a live Runner behind. The next attempt replaces
it through the launch fence, which fails closed when it cannot prove the
old lineage gone. This is the same risk as the existing recovery read
timeout detach.
- When the provider attempt budget is already spent, the kept Runner is
not replaced by this run. This also matches the existing timeout detach.
- Not in this PR: the upstream trigger. The first attempt abandons a
slow turn start (`runnerd did not report the provider turn identity` or
`turn.start timed out`) while Codex later runs that turn. Two sampled
chains then stopped the Runner on
`protocol_failure:tool_binding_mismatch` / `thread_binding_mismatch`
after a deploy reattachment. These need a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:30:22 -06:00
DottaandClaude Opus 5.5 d9a4bb23fe fix(runner): wait for a slow Codex turn start while it reports progress (#21329)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Codex agents run through the Paperclip runner, and the server-side
transport starts each provider turn
> - After Codex accepts a turn, the transport waits a fixed 30 s for
that turn's `turn/started` before it reports the turn identity
> - Codex starts its MCP servers before it starts the turn, so a healthy
cold start in a loaded sandbox often takes longer than 30 s
> - The transport then failed a live turn with `runnerd did not report
the provider turn identity`, and the abandoned turn's late notifications
broke the reattachment (`did not name its execution owner`,
`runner_harness_state_mismatch: resumed_not_suspended`)
> - This pull request sizes the start window to the measured cold start
and extends it while the provider keeps reporting startup progress
> - The benefit is that slow but healthy Codex starts succeed on the
first attempt, and a dead start still fails at a bound

## Linked Issues or Issue Description

**What happened?**

On staging, Codex runs on Daytona failed at turn start with
`native_session_interrupted: runnerd did not report the provider turn
identity`. The run log shows the same sequence each time. The runner
accepts the turn. Codex then reports startup progress (a bubblewrap
config warning, `account/updated`, and `mcpServer/startupStatus/updated`
failures for an MCP server that is not logged in) for about 30 s. The
transport gives up 30 s after acceptance. A few seconds later, Codex
sends the turn's `turn/started`, which now arrives at a closed turn
(`protocol_failure:turn_binding_mismatch`). The same-run reattachment
then hits that orphaned turn. It fails with `Provider authoritative
notification item/started did not name its execution owner.` or
`runner_harness_state_mismatch: resumed_not_suspended`.

Of 198 recent runs in the staging company, 52 contain this turn identity
failure. In 68 successful starts, submit to `turn/started` took p50 25
s, p75 37 s, p90 50 s, p95 54 s and max 76 s. The fixed 30 s wait sits
near the median.

**Expected behavior**

A slow but healthy Codex start succeeds. A start that makes no progress,
whose Runner exits, or that the runner fails still fails at a bound and
recovers through the normal same-run resume.

**Steps to reproduce**

1. Start a Codex turn whose provider accepts the turn but sends
`turn/started` after the 30 s default window (on staging: several MCP
servers and a loaded sandbox).
2. Let the provider report startup notices during the wait, and let the
runner not prove the accepted turn live in time (on staging no turn was
adopted).
3. See `runnerd did not report the provider turn identity`, then the
reattachment conflict.

The new tests reproduce this with the fake Codex app server
(`--turn-started-delay-ms` with `--turn-started-progress-ms`) and a
runner snapshot that cannot prove liveness.

**Paperclip version or commit**

Staging Core `c730872974` (`2026.1106.0+2253`). Base of this PR:
`origin/master`.

**Deployment mode**

Authenticated, public, managed cloud (staging). Remote runner on
Daytona.

**Agent adapter(s) involved**

`paperclip_runner` with the Codex provider (`gpt-6.1-sol`).

## What Changed

- `runnerd-codex-transport.ts`: the default wait for an accepted turn's
`turn/started` is now 90 s (measured max 76 s), not 30 s. An explicit
`turnStartTimeoutMs` keeps its old bound.
- The wait restarts its window each time the runner commits a provider
notice, or a runner diagnostic that names the accepted turn. The total
wait stops at 180 s (option `turnStartProgressCeilingMs`, so tests can
lower it). A provider relaunch after the accepted `turn.start` does not
extend it.
- The adoption check (runner snapshot proves the accepted turn live) now
runs before each window end, not only before the first one. A Runner
exit and a runner failure of the pending turn still fail at once.
- `fake-codex-app-server`: new `--turn-started-progress-ms` flag sends
`mcpServer/startupStatus/updated` failures while it delays
`turn/started`.
- `doc/execution-semantics.md`: documents the start window, the progress
rule and the staging measurements.

## Verification

- Red on `origin/master` transport (with the new fake flag): `keeps
waiting for an accepted Codex turn while its startup keeps reporting
progress` failed after 4.6 s, and `gives a silent Codex cold start the
measured default start window` failed after 30.9 s. Both failed with
`runnerd did not report the provider turn identity`.
- Green: `npx vitest run
src/live/runnerd-codex-transport-deadlines.test.ts` passes 9/9. This
includes a ceiling test: provider notices every 1 s and no
`turn/started` fail at an 8 s test ceiling, and the test goes red with
the ceiling cap removed. It also includes the existing adoption,
no-adoption-within-bound and runner-failure tests, and a new test that a
silent start still fails within its bound.
- `npx vitest run` on the adoption, cut-turn-start, stall,
turn-start-timeout-retention, remote-retention, restart-continuation and
controller-outage transport suites passes 48/48.
- `npx vitest run src/live/runnerd-codex-transport.test.ts
src/live/live-session.test.ts src/live/turn-stream.test.ts`: 391/392
pass. The one failure (`stops a lost runnerd's provider and escaped tool
job from its lineage journal (journal=true)`, 1 s, process-lineage
timing under full-suite load) passes when run alone, both on this branch
and on `origin/master`.
- `tsc --noEmit` for `packages/paperclip-runner` and `cargo fmt --check`
pass.

## Risks

- A silent dead start (no Runner exit, no provider output, no runner
failure) now fails after 90 s, not 30 s. The Runner exit and
runner-failure paths still fail at once, so this applies only to a
provider that stops without a trace.
- A provider that keeps sending notices but never starts the turn waits
at most 180 s.
- No schema, API or UI change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, browser reads of staging run data).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Refs #21272, #21120

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:21:37 -06:00
Nicky LeachandClaude Fable 5.2 314b893678 fix(native-runtime): deliver assigned gateway tools to every native harness through the PRP relay (#21274)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can have app connections. Paperclip gives the agent the
tools of those apps through an MCP gateway, with its approval and audit
rules.
> - Native harnesses reach the gateway in two ways today. In the relay,
the server keeps the gateway token and offers the tools over the
authenticated Runner channel. In the direct way, the server gives the
gateway URL and a bearer token to the harness.
> - Which way a harness uses comes from a list of special cases that
grew one harness at a time. Local Codex and all ACPX agents use the
direct way. The others use the relay.
> - The direct way puts a run-scoped token in a provider process. A
process that the agent starts can read it. A provider process that
serves more than one run can hold a revoked token.
> - This pull request replaces the list with one rule. Each native
harness gets its assigned gateway tools through the relay.
> - The benefit is one behavior for all native harnesses, and no gateway
token in a provider process.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #13942 (the relay), #17149 (ACPX accepts relayed tools) and #19912
(the design for turn-scoped ACPX MCP authority).

**What existing behavior does this improve?**

How a native harness reaches the tools of its assigned MCP gateway.

**Subsystem affected**

Native runtime in the server (`native-session-executor.ts`,
`native-session-resume.ts`). One Runner test file. One Rust test.

**Current behavior**

- `relayAssignedMcp` is true for plugin tasks, Cursor Cloud, OpenCode,
Codex on a remote target, and Dot.
- For local Codex and for each ACPX agent (Claude, Codex, Cursor, Pi,
Kimi, Grok, Copilot, Antigravity), Core puts
`PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` and
`PAPERCLIP_NATIVE_MCP_TOKEN` in the Runner environment. The ACPX host
gives them to the provider as an HTTP MCP server. The local Codex home
gets them in `config.toml`.
- The token is run-scoped. Core revokes it when the run ends.

**Proposed behavior**

- A native run that has an assigned gateway uses the relay. The
condition names no provider.
- Core builds the gateway tools on the control plane, offers them with
the other Runner tools, dispatches each call over PRP, and authorizes
each call again through the gateway.
- Core removes the three `PAPERCLIP_NATIVE_MCP_*` variables from each
Runner environment.
- The tool contract fingerprint has the relay marker for each target,
not only for remote targets.

**Reason and benefit**

- All native harnesses have the same behavior.
- No provider process holds a gateway token.
- A sandbox does not need to reach the gateway URL of the host.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
- The list of special cases is gone. Core validates the binding and
builds the relayed tools when a binding is present.
- Core does not list the gateway for a review run, Grok Bot or Muse. The
tool authority never offered app tools to those. A gateway error can
therefore not fail those runs.
- OpenCode keeps its `on_demand` catalog. That option depends on the
provider, not on the relay.
  - Core removes the three variables from each Runner environment.
- `server/src/services/native-runtime/native-session-resume.ts`: the
relay marker in the tool contract fingerprint is unconditional.
- `doc/execution-semantics.md`: the text about the tool contract
fingerprint now states the rule, the tool names and the cutover.
- App tool schemas. New module
`packages/paperclip-runner/src/drivers/tool-input-schema.ts` compiles
each schema by itself, in its declared dialect (draft-07, 2019-09 or
2020-12), with no shared `$id` registry. Core uses it when it builds the
relayed tools (`assigned-mcp-tools.ts`). Core does not offer a tool
whose schema has an unsupported dialect or does not compile, and writes
a log entry with the tool name and the reason. The run does not fail.
The ACPX tool bridge and the OpenCode MCP bridge use the same function.
The tools of Paperclip itself stay strict.
- Large arguments. The ACPX sidecar sends the arguments of a tool call
to runnerd with no display limit. The limit is the tool-call admission
limit of runnerd, 256 KiB. A larger call fails with a tool error that
gives the limit. The sidecar does not dispatch it.
- Tests in `native-session-executor.test.ts`,
`native-session-resume.test.ts`, `runtime-context-materializer.test.ts`
and `acpx_provider_backend.rs`.

Not changed in this pull request: the direct code path. It is still in
the Runner, but it receives no binding. #21311 deletes it.

## Behavior that each harness sees

| Harness | Before | After |
|---|---|---|
| Codex on a remote target, OpenCode, Cursor Cloud, Dot, plugin tasks |
Relay | No change |
| Local Codex | Direct: a `paperclip-assigned` server in `config.toml` |
Relay. App tools are dynamic tools named `app_<name>_<digest>` |
| ACPX agents | Direct: a second HTTP MCP server | Relay. App tools are
on the `paperclip` bridge as `mcp__paperclip__app_<name>_<digest>` |
| ACPX with a preconfigured home | No gateway | No change |
| Grok Bot, Muse | Got the variables and ignored them | Do not get the
variables |

- A relayed tool name has a maximum of 64 characters. With the
`mcp__paperclip__` prefix it has a maximum of 80.
- Permission behavior keeps the same intent. For Claude in the
`approve-reads` and `approve-paperclip` modes, app tools ask for
approval, as before. For the other ACPX agents the permission decision
does not use the server name.
- In a planning run or an ask run, the relay offers only tools with read
risk. The direct gateway offered all tools. This is stricter.
- When the catalog is too large, each harness gets the two compact tools
`paperclip_search_assigned_tools` and `paperclip_call_assigned_tool`.
- The four helper tools for resources and prompts
(`paperclip_list_resources`, `paperclip_read_resource`,
`paperclip_list_prompts`, `paperclip_get_prompt`) are not in the relay.
No harness loses them: a run token permits only `tools/list` and
`tools/call` (`run-preparation.ts`), so these four tools already fail
with `gateway_token_action_denied` on the direct path.

## State saved by the previous release

- **Local sessions of each provider start again one time.** The tool
contract fingerprint changes for local targets. The next run sees a
different fingerprint, starts a new provider session with the full task
context in the same workspace, and keeps the old saved state. This is
the existing rotation for a tool contract change. Codex needs it,
because Codex cannot add tools to a thread that it resumes. For the
local providers that already used the relay, the restart is not
necessary. It is accepted so that the rule names no provider.
- **Remote sessions continue.** Their fingerprint does not change. For
ACPX, the attach of the next run replaces the saved tool set.
- **A kept Codex home.** Paperclip writes `config.toml` again at each
start and recovery. The old gateway server and its token are removed.
- **A deploy lands during a run, and the Runner survives.** The
controller attaches to the same Runner for the same run. The old
provider keeps its direct gateway connection. The controller detach
keeps the gateway token of that run, and Paperclip revokes it when the
run ends.
- **A deploy lands during a run, and the Runner is dead.** The Runner
starts again for the same run with its saved tool set and no direct
gateway. The assigned app tools are not available for the remainder of
that one run. The next run gets them through the relay. This applies to
each harness that moves to the relay. It has no test.
- Paperclip refuses no saved state and discards none.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`: `1508 passed
| 7 skipped`, 1 failed. The failed test is "archives a damaged prior
epoch before completing guarded same-task replacement with real runnerd"
(`ENOENT … authority-epochs`). It fails in the same way on `master` at
53608bd1d6 and with the product change removed, on macOS.
- `cd server && npx tsc --noEmit -p tsconfig.json`: no errors.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/runtime-context-materializer.test.ts`: `18 passed`. `npx tsc
--noEmit -p tsconfig.test.json`: no errors.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes. `cargo test -p paperclip-runner-core --lib
acpx_provider_backend`: `50 passed`.

- `npx vitest run src/drivers/tool-input-schema.test.ts
src/drivers/runner-tool-bridge.schemas.test.ts
src/drivers/runner-tool-bridge.test.ts
src/cli/acpx-runtime-sidecar.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: `21 passed (21)` files, `377 passed` tests.
`pnpm run typecheck:typescript`: no errors.
- `cargo test -p paperclip-runner-core --test acpx_event_payload --test
tool_input_schema_fixtures`: 10 and 3 passed.
- Schema tests use one shared fixture
(`tool-input-schema-fixtures.json`) for the TypeScript compiler, the
tool bridge and the Rust `begin_call`: a draft-07 schema, a second
schema with the same `$id`, 2019-09, 2020-12, draft-04, an unknown
dialect, a `$ref` with no target, and an invalid keyword. Before the fix
the bridge failed with `no schema with key or ref
"http://json-schema.org/draft-07/schema#"`.
- A tool schema whose `$id` is the URI of a meta-schema cannot damage
the shared compiler. Before this fix, `removeSchema` deleted that
meta-schema, and later schemas failed to compile for each run in the
process. A test with 7 reserved ids failed before the fix and passes
now.
- Core limits the compile work of each run start. A schema over 512 KiB
is not compiled, and one listing compiles a maximum of 768 KiB of
schemas. These are the limits of the Runner. A tool over a limit is
served only through the search and call tools. Core keeps the outcome
for each schema digest (1,024 entries) and yields to the event loop
before each compile. Four tests failed before this change.
- A sidecar test sends a tool call with 200 KiB of arguments and
receives them byte for byte, refuses a call with 300 KiB with the named
error, and keeps the replacement of a lone surrogate.

Tests, each seen to fail without the product change:

- "keeps assigned MCP credentials on the control plane for %s" now runs
for remote Codex, local Codex, OpenCode and ACPX Claude, Pi, Kimi,
Copilot and Cursor. It checks that Core builds the relay, offers the
tool, dispatches a call, and gives no `PAPERCLIP_NATIVE_MCP_*` variable
to the Runner. Without the change, the five new harness cases fail.
- "relays the assigned gateway when a deploy lands mid-run…" runs for
Claude and Pi.
- The fingerprint tests keep the old local fingerprint as a retained
value and check that the new one is different. The remote fingerprints
for Codex and OpenCode do not change.
- "drops a retained direct gateway server when the relayed launch has no
binding": a Codex home with the old `config.toml` entry loses that entry
at the next start and at a recovery of the same run.
-
`next_run_attach_offers_relayed_gateway_tools_to_a_direct_gateway_checkpoint`
runs for Claude and Pi.

Not run: a live run with a real provider and a real app connection for
each harness. Not run: `pnpm verify` and the full suite. CI supplies
that coverage.

## Risks

- **An ACPX app call has a limit of 256 KiB for its arguments.** The
direct gateway accepted a request body of 10 MB. Codex and OpenCode
permit 480 KiB. To get nearer to 10 MB, runnerd needs a larger admission
limit for tool calls or a protocol that sends arguments in parts. That
is not in this pull request.
- **A tool with a draft-04 or draft-06 schema is not offered.** Ajv 8
does not support draft-04, and the TypeScript and Rust validators must
agree. On the direct path the provider got such a tool.
- **Core compiles app schemas on its event loop.** Before this pull
request Core did not compile them. With the limits above, one valid
schema near 512 KiB can hold the loop for about 0.65 s, and one listing
for about 1.2 s in total (measured on a developer machine). An app that
changes its schemas at each listing gets no help from the cache. A
worker thread with a timeout would remove this; it is not in this pull
request.
- **Tool names change for local Codex and all ACPX agents.** A saved
agent instruction that names a gateway tool directly does not match the
new name.
- **Each local native session starts again one time after the deploy.**
The provider conversation of that session does not continue. The task
context and the workspace stay.
- **A run that is in flight at deploy, with a dead Runner, has no app
tools until it ends.**
- **Names of 80 characters are not tested live** with Kimi, Copilot,
Grok, Antigravity and Codex through ACPX. Pi already shortens a name
longer than 64 characters.
- **Managed Claude and AWS AgentCore now receive relayed app tools.**
This is not tested live.
- **A gateway token expires after 1 hour.** The relay keeps the token on
the controller for the whole run, as OpenCode and remote Codex do today.
A longer run loses its app tools on both paths. This pull request does
not change that.
- **The gateway URL must have the path `/mcp/gateways/<id>`.** With a
path prefix in the API base URL, a run with an assigned gateway fails
with `assigned native MCP gateway path is invalid`. This was true for
the relayed harnesses before. It is now true for all.
- No test covers the rule that Core does not list the gateway for Grok
Bot and Muse.

## Follow-up

The direct path is now unreachable. #21311 deletes it. That pull request
is in the same stack, above the pull requests that change the same ACPX
host files.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff and ran the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated each harness
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 00:16:58 -08:00
Nicky LeachandClaude Opus 5.5 2483cb46c2 feat(plugins): deliver configured environment to plugin Runner tasks (#21280)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent gets environment variables from its execution environment,
its own configuration, its project, and its routine. Secrets are
resolved for each run.
> - A plugin environment can run the agent as a native Runner task on
its own host, through the environment task contract.
> - That contract has no field for the agent's variables. Runs on a
plugin environment get only the variables that the plugin sets itself.
> - This pull request adds a versioned contract field for the values,
sends them on a session handoff, lets Codex commands inherit the
variables in an external sandbox, and lets a plugin protect the
variables that it sets itself.
> - The benefit is that plain and secret variables work on plugin
environments in the same way as on local environments.

## Linked Issues or Issue Description

Refs #21233. That pull request by @lucktastic finds the same gap and
proposes the same capability and field names
(`configuredEnvironmentVersion: 1`, `configuredEnvironment`). This pull
request uses the same names, so providers implement one contract. It
differs in these places:

- Codex commands get the variables through an external-sandbox denylist,
not through key names sent at `prepare`. A retained Runner does not need
a new preparation when a key is added or removed.
- A session handoff `resume` gets the values, because it starts a new
Runner on another computer. Value changes on the same computer need no
new field: Core's process reuse key includes the values, so Core starts
a new task.
- A driver can declare `protectedEnvironmentKeys`. Paperclip rejects or
warns when one is saved.
- The run log names skipped variables only for an older provider.

Thank you to @lucktastic for the diagnosis and the contract shape.

**What happened?**
An agent has environment variables, plain and secret. Its runs on a
plugin task environment do not get them. `PAPERCLIP_CONFIGURED_ENV_KEYS`
in the run lists only the variables that the plugin sets.

**Expected behavior**
Each run gets the agent's configured variables.

**Steps to reproduce**
1. Set a plain and a secret environment variable on an agent.
2. Assign the agent to a plugin environment that provides Runner tasks.
3. Start a run. Read the names in `PAPERCLIP_CONFIGURED_ENV_KEYS` in a
command that the agent runs.
4. The agent's names are not in the list.

## What Changed

- **SDK contract** (`packages/plugins/sdk/src/environment-tasks.ts`):
- A provider advertises `configuredEnvironmentVersion: 1` on `describe`.
- `submit` gets `configuredEnvironment`. `resume` gets it only for a
session handoff.
- Names, count, and sizes have the same limits as the Runner projection.
- **Server**:
- `createRunnerdBackendWithinSessionClaim` gives the run's configured
projection to the plugin task owner. The projection has no provider
credentials and no generated runtime keys.
- `createPluginRunnerTaskOwner` sends the fields only to a provider with
the capability. An older provider gets no new field, and the run log
names the variables that were not set. The log never prints values.
- **Codex** (`codex-security-config.ts`):
- An externally sandboxed launch uses `inherit="all"`,
`ignore_default_excludes=true`, and an `exclude` list of Codex's own
credentials, instead of an `include_only` list fixed at preparation
time.
- A test reads the Runner's Codex environment keys, and fails if a new
key is not classified.
  - Host launches keep the bounded allowlist.
- **runnerd** (`codex_provider.rs`): `runtime_notification_codex_args`
still grants the notification files to read-only profiles when no
allowlist is present. Plan mode in an external sandbox needs these
grants.
- **Protected keys**:
- An environment driver can declare `protectedEnvironmentKeys` in its
manifest.
- Paperclip rejects a protected key on an agent or environment that uses
the driver. The 422 has code `protected_environment_variable`.
- On a project, a routine, or another agent, Paperclip saves the key.
The response has a `configurationWarnings` notice that names the
environments. The project and routine pages show the notice as a toast.
The agent pages already show configuration warnings.
- **Docs**: `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` describes the
provider obligations.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the reviewed
SDK source pin.

## Verification

- `packages/plugins/sdk`: `vitest run tests/environment-tasks.test.ts`
(38 pass). Accepts bounded values. Rejects bad names, NUL bytes, too
many variables, and oversized values.
- `server`: `vitest run
src/services/native-runtime/plugin-runner-task.test.ts` (95 pass).
Values go only to a provider with the capability, never in `prepare`.
`resume` sends values only for a handoff.
- `server`: `vitest run src/services/protected-environment-keys.test.ts
src/__tests__/project-routes-env.test.ts` (10 pass). Rejection, notices,
and the project route response.
- `server`: all route suites that import the project, routine,
environment, or agent routes (45 files, 1689 pass).
- `packages/shared`: `vitest run src/validators/plugin.test.ts` (64
pass).
- `packages/paperclip-runner`: `vitest run src/drivers/codex
src/configured-environment.test.ts
src/live/runner-provider-launch.test.ts` (444 pass).
- `packages/paperclip-runner/runner`: `cargo test -p
paperclip-runner-core --lib runtime_notification` (9 pass, 2 ignored as
before).
- `pnpm typecheck` in `packages/shared`, `packages/plugins/sdk`,
`packages/paperclip-runner`, `server`, and `ui`.
- Not run locally: `pnpm verify --full` and browser suites. Required CI
covers the final revision.

## Risks

- **Existing providers:** low risk. Core sends no new field to a
provider without the capability. Their runs behave as before, with one
added log line when variables were skipped.
- **Codex in an external sandbox:** commands now see every variable in
the Runner environment except Codex's own credentials. The execution
environment builds that environment from an allowlist. If the Runner
gives Codex a new credential, the drift test fails until the key is
classified.
- **Saving env:** new rejections happen only when an installed driver
declares `protectedEnvironmentKeys`. No driver declares them today.
- **Providers:** they must keep the values transient and never save or
log them. The doc lists these obligations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:11:23 +00:00
DottaandClaude Opus 5.5 e73fc71331 fix(environments): continue on a new sandbox when an idle runner sandbox never starts (#21323)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on the next wake
> - On staging, Daytona left two idle sandboxes in the `archiving` state
for more than three hours, so no resume call could succeed
> - The native runner leases had no verified harness backup, so the run
never replaced the sandbox, even though the task files were saved
> - Each retry spent six minutes on the same stuck sandbox, the bounded
retries ran out, and the task waited for a person to press Retry
> - This pull request continues on a new sandbox when the provider has
proven it cannot start the old one and the workspace is saved, and it
keeps the old sandbox
> - The benefit is that the task recovers by itself and a sandbox that
may hold unsaved runner state is never destroyed

## Linked Issues or Issue Description

Refs #20626, #21113

**What happened?**
Eight Codie runs on staging failed in one hour with
`reusable_sandbox_resume_failed`: `Reusable sandbox lease "<id>" could
not be resumed (cause=transport_timeout); the lease was preserved and no
replacement was created.` They came in two clusters (05:59 to 06:00Z and
06:32 to 06:38Z). Each run took about six minutes, so the 330-second
idle resume budget from #21113 was applied. All runs were already
`transient_failure_retry` wakes.

Investigation of one instance (lease `c23d6120…`, run `d16c272e…`):
- The lease was `released` at 02:37Z with a confirmed stop-and-retain.
No live run held it, so the lease was idle and the extended budget
applied.
- A read-only Daytona API read at 06:5xZ showed the sandbox in state
`archiving` (desired state `archived`), last updated 03:38Z, one hour
after the stop (the auto-archive interval is 60 minutes). The second
sandbox (`e0e6cfe5…`) showed the same state since 03:35Z. Daytona does
not start a sandbox while it archives, so every resume call timed out.
More time or a new HTTP connection could not help.
- The task workspace was clean and had a committed saved snapshot.
- The agent uses `paperclip_runner`. The last run on the sandbox failed
while it saved the runner session, so the lease had no
`nativeHarnessBackup` stamp. `reusableSandboxReplacementIsSafe` requires
a verified harness backup for a runner lease, so the run never replaced
the sandbox. It failed strictly, the bounded retries (2) ran out, and
the task waited for a manual Retry.

**Expected behavior**
When an idle sandbox never starts through the whole provider start
deadline and the task files are saved, the run continues on a new
sandbox automatically. A sandbox that may hold unsaved runner state is
not destroyed.

**Steps to reproduce**
1. Run a `paperclip_runner` task on a reusable Daytona sandbox. Let the
last run end without a harness backup (for example, the runner exits
while it saves the session). The workspace is clean with a committed
snapshot.
2. Let Daytona start to archive the stopped sandbox and get stuck in
`archiving`.
3. Wake the task. Every `environmentResumeLease` call times out, the run
fails after about 330 seconds with `reusable_sandbox_resume_failed`, and
the retries fail the same way.

**Paperclip version or commit**
Staging Core `0c04ff76eb` (current `master` has the same code path).

**Deployment mode**
Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases
on.

**Agent adapter(s) involved**
`paperclip_runner` (native runner) on a Daytona sandbox environment.

**Root cause**
`server/src/services/reusable-sandbox-resume-replacement.ts`: a runner
lease without a verified harness backup was never replaceable, for any
failure. The only other path was to retry the same sandbox, which cannot
work while the provider is stuck. Replacement also always retired the
old lease to `pending_cleanup`, so allowing it without a backup would
have destroyed the only copy of the runner session state.

## What Changed

- `reusable-sandbox-resume-replacement.ts`:
`reusableSandboxReplacementIsSafe` becomes
`reusableSandboxReplacementPlan`. It returns `retire` (same rules as
before), `keep`, or `null`. It returns `keep` for a runner lease without
a verified harness backup only when the caller passes
`providerCannotStart`. All other conditions still apply: idle lease,
same workspace, clean workspace, committed snapshot, this run is the
writer, and no access refusal.
- Add `keepReplacedReusableSandboxLease`. After the replacement lease is
recorded, one guarded write marks the old lease with
`reusableSandboxKeptAfterReplacement`. The lease keeps its status and
its sandbox. Nothing destroys it.
- `environment-runtime.ts`: set `providerCannotStart` only when the idle
start deadline was extended and the run then used all of it on timeouts.
Lease selection skips a lease with the keep marker, so later runs do not
spend time on it and obsolete-lease cleanup does not destroy it.
- The replacement records `replacement` / `resume_failed` with no
harness backup. The native executor already starts a fresh session in a
new sandbox in this case (`freshReplacement`), and workspace preparation
restores the committed snapshot.
- The keep plan needs every resume call to time out. A rate limit, a
reset, or any other failure in between keeps the strict behavior.
- `reusable-sandbox-kept-marker.ts`: the marker key and a SQL guard.
Every automatic teardown path skips a marked lease: retained-orphan
listing and settle, `destroyReusableSandboxLease` (obsolete cleanup,
expiry, run destroy), `destroyClaimedSandboxLease` (issue and workspace
teardown), recovered-workspace cleanup, and abandoned-file release. Only
an operator-consented environment delete still destroys it.
- `doc/execution-semantics.md`: document the rule.

## Verification

- Red before the fix: the new staging-shape test in
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`
(released runner lease, no harness backup, clean committed snapshot,
every resume times out) failed with the staging error `could not be
resumed (cause=transport_timeout); the lease was preserved and no
replacement was created.`
- Review round: "keeps an idle sandbox with no harness backup when some
resume calls failed for another reason", "never destroys a sandbox kept
after replacement through the run or workspace teardown paths", and the
retained-orphan case "keeps a warm sandbox while a replacement kept it"
each failed before the fix and pass after it.
`environment-retained-orphan-sandbox.test.ts`,
`abandoned-workspace-files-hold.test.ts`,
`workspace-recovery-preparation-gates.test.ts`,
`workspace-recovery-admission.test.ts`: all pass.
- Green after the fix: 22/22 pass in the replacement suite. The run gets
a new sandbox within 330 seconds, makes no destroy call, the old lease
stays `released` with the keep marker, and the next run resumes the new
sandbox only. With unsaved work or no saved snapshot, the run still
fails strictly and does not mark the old lease. Earlier tests (a refusal
without a backup, a live owner, a dirty workspace with a timeout) are
unchanged.
- `pnpm exec vitest run src/__tests__/environment-runtime.test.ts
src/services/environment-runtime-reusable-lease-probe.test.ts
src/__tests__/reusable-leases-default.test.ts`: 215 pass.
- `tsc -p server/tsconfig.json --noEmit` passes.
- Not run: a live replacement on staging. It is verified after deploy on
the next wake of the affected tasks.

## Risks

- A runner task that hits this path starts a fresh runner session. The
files come from the saved snapshot, and the task, comments, and run
history are kept. Before this change, the task stopped and waited for a
person.
- The kept sandbox stays with the provider (archived storage) until
someone removes it. This is the cost of never destroying unsaved runner
state.
- The rule needs the full extended start deadline (about 330 seconds) of
timeouts, so a slow but healthy start still resumes the same sandbox.
- No schema change. Old leases follow the same rule.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:06:25 -06:00
Nicky LeachandClaude Fable 5.2 503a736195 feat(ci): read a registry build cache in the Runner image build (#21310)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - `.github/workflows/runner-image.yml` builds the Runner alone and
publishes it from `master` as `ghcr.io/paperclipai/paperclip-runner`,
one time for each set of inputs
> - The build used no cache, so every run compiled the daemon again. The
first publication took 25m46s from push to tag
> - The toolchain and the daemon change only with the Rust sources. Most
commits that change the Runner's inputs do not touch them
> - This pull request lets the build legs read a registry build cache
for those stages. Only a new job can write the cache, and only for a
`master` push
> - A weekly run builds with no cache, so the cache cannot hide a broken
build for long
> - The benefit is about 3 minutes less on each build leg when the Rust
sources did not change, with no change to the published image

## Linked Issues or Issue Description

Refs #21230 (publication of the Runner image), #21229 and #21204 (the
build this change makes faster). No issue exists for this change.

**What existing behavior does this improve?**

The Runner image build in `.github/workflows/runner-image.yml`. It
builds the same image in less time.

**Current behavior**

Each build leg starts from an empty builder. It installs the Rust
toolchain and compiles the daemon in every run. In the first `master`
publication (run 43365670428) the amd64 leg spent 7m22s in the build
step. The toolchain and the cargo build took 3m26s of that.

**Proposed behavior**

The build legs read
`ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>` for the stages
up to the compiled daemon. A new `cache` job writes that cache after a
publication. A scheduled run builds with no cache one time each week.

**Reason and benefit**

The stages before the last one read the Rust sources, the protocol files
and the toolchain pin. In the last 30 days, 438 `master` commits changed
an input of the image, and at most 181 of them changed a file those
stages read. For the other commits, and for each pull request that does
not touch those files, the build step is about 3 minutes shorter on each
leg.

**Breaking changes**

None. The `Dockerfile` and each file in the image identity are
unchanged. The published digest for a set of inputs does not change.

This is part 4 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build (this pull
request)

This pull request uses no code from parts 1 to 3. It is stacked on them
because all four parts edit `doc/runner-image-contract.md`.

## What Changed

- The build legs pass `--cache-from
type=registry,ref=ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>`
to the first build command. They do not log in. Their permissions stay
`contents: read` and `packages: read`.
- A new `cache` job writes the cache with `--cache-to
type=registry,...,mode=max` and `--output type=cacheonly`. It runs only
for a `master` push in the canonical repository, for inputs that had no
image. Its permissions are `contents: read` and `packages: write`. It
has no `id-token` and no `attestations` permission.
- The `cache` job needs the `publish` job, and no job needs the `cache`
job. A cache write cannot delay or fail a publication.
- The `cache` job builds the target `runner-artifacts-runnerd` and no
later stage. The last stage takes the commit as a build argument, so no
other commit can reuse it. The install and its dependency lifecycle
scripts are in that stage, so the job with the write token does not run
them.
- The `cache` job has one concurrency group for each architecture, a job
time limit and a step time limit.
- A `schedule` trigger starts a weekly build. It passes `--no-cache` to
the first command and reads no cache. It runs all checks and publishes
nothing. A fork's schedule runs nothing.
- `server/src/__tests__/runner-image-workflow.test.ts` pins the new
security properties: which jobs hold `packages: write`, that the build
legs have no credential and cannot write the cache, the conditions of
the `cache` job, that the `publish` job does not depend on it, and that
a scheduled run cannot publish or write the cache.
- `doc/runner-image-contract.md` has a "Build cache" section.

## Verification

Tests, on macOS (Node 26.12.0) and in a `node:24.22.3-bookworm`
container (Linux arm64, Node 24.22.3):

- `pnpm exec vitest run src/__tests__/runner-image-workflow.test.ts
src/__tests__/runner-image-identity.test.ts` in `server/`: 55 of 55
tests pass on both.
- `node --test .github/scripts/tests/*.test.mjs`: 807 of 807 tests pass
on both. The container needs `jq`.
- `pnpm verify` exits 2 for this change, because a workflow file is a
global CI input. Required CI supplies the broad coverage.
- Each new assertion fails when the workflow breaks its rule. I checked
seven such edits, for example: the `publish` job needs the `cache` job,
the `cache` job gets `id-token: write`, a build leg gets `--cache-to`.

Cache mechanics, on linux/arm64 with a local registry, BuildKit v0.34.2
(the version the hosted runners use), and an empty builder for each
build:

| Build of both variants | Time | Result |
| --- | --- | --- |
| No cache, cache ref does not exist | 260 s | An importer error in the
log, exit 0 |
| One changed TypeScript file, with the cache | 149 s | Toolchain and
cargo steps cached |
| Unchanged inputs and a new commit, with the cache | 129 s | Toolchain
and cargo steps cached |
| Each stage in the cache (a whole-build cache, which this change does
not write) | 16 s | Both OCI archives complete |
| `--no-cache` on the first command | 272 s | No cached step in the
first command. The second command reuses all of it |

- The images built from the cache pass `runner-image-layout.mjs check`
and `runner-image-oci.mjs check-layers`. The daemon has the same SHA-256
as the daemon of a cold build.
- A cache ref that does not exist, a host that refuses the connection, a
name that does not resolve, and the real
`ghcr.io/paperclipai/paperclip-runner-buildcache:arm64` (which does not
exist yet) each give an importer error in the log and a complete build
with exit 0.
- The `cache` job command on an empty builder with the cache present
takes 6 s. It downloads no layer.

Not tested before merge:

- The `cache` job. It runs only on `master`.
- The scheduled run. It starts only from the default branch.
- The cache package. It does not exist. I did not test a private
package, which answers an anonymous read with an authorization error. I
expect the same importer error and a cold build.
- A fully cached build on the hosted runners. The local test used the
same BuildKit version and a different buildx version (0.37.0, and the
runners have 0.38.1).

## Risks

- **A second job can write packages.** Before this change, only the
`publish` job held `packages: write`, and it runs no build. Now the
`cache` job holds it too, and it runs BuildKit on the repository's
`Dockerfile`. GitHub cannot limit that token to one package, so it can
also write to `ghcr.io/paperclipai/paperclip-runner`. The limits are:
the job runs only for a `master` push in the canonical repository, the
token goes only to the registry login, the build steps run inside
BuildKit without it, the job does not run the install, and the job
cannot sign. An image that it wrote has no attestation.
- **Part of a published image can come from an earlier run.** The daemon
in a published image can come from a step that an earlier `master` run
compiled and cached. The attestation names the run that published, not
the run that compiled. BuildKit reuses a step only for the same
instruction and the same file content, but a party that can write the
cache package can replace what a later build reuses. Write access to the
cache package must stay as narrow as write access to the image package.
- **A cached download is not repeated.** The toolchain installer, the
toolchain and the crates are not downloaded again until the inputs of
their step change. The weekly cold build finds a download that is gone.
It does not compare its output with the cache.
- **A registry failure after the cache import fails the leg.** A cache
that cannot be read at the start is not a failure. A layer download that
fails later is a failure. Run the job again.
- **The cache package starts private.** GitHub creates it on the first
write. A maintainer must make it public one time. Until then each build
runs cold and nothing fails.
- **The first cache write waits for an input change.** The merge of this
pull request changes no input of the image, so its `master` run builds
nothing and writes no cache.
- **A cancelled cache job marks its run as cancelled.** When three cache
writes for one architecture wait at the same time, GitHub cancels the
middle one. The image of that run is published.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, through Claude Code.
Tool use and shell execution. It wrote the change, ran the local builds
and tests, and wrote this description.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:46:15 -08:00
Nicky LeachandClaude Opus 5.5 4f4d823cfc feat(environments): forward the warm idle window to plugin lease acquisition (#21305)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugin environment drivers run agent tasks on remote computers, and
warm leases keep a parked Runner between runs for an idle window that
Core records
> - Some providers stop an idle computer on their own schedule (for
example a sandbox auto-stop interval)
> - If that provider timer is longer than the Runner idle window, the
provider keeps paying for a computer whose Runner is already gone; if it
is shorter, a warm resume can find the computer stopped
> - The plugin cannot align the two because it never sees the idle
window at acquire
> - This pull request forwards the warm idle window to
`environmentAcquireLease` as `runnerLifecyclePolicy`
> - The benefit is that a provider can stop its computer just after the
Runner idle window, without Core knowing anything about the provider's
placement

## Linked Issues or Issue Description

Refs #21289
Refs #21066

**What existing behavior does this improve?**
Warm lease setup for plugin environment drivers.

**Subsystem affected**
Server environment runtime (`createPluginEnvironmentDriver`) and the
plugin SDK acquire parameters.

**Current behavior**
Core records the warm lifecycle policy (`runnerTaskLifecyclePolicy`,
including `idleTimeoutMs`) on the lease, but `environmentAcquireLease`
does not receive it, so a provider with its own idle shutdown has to
guess.

**Proposed behavior**
For a warm lease, `environmentAcquireLease` receives
`runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs }`, the same value
Core records. Other lifecycle modes do not receive the field.

**Reason and benefit**
A provider can align its idle shutdown with the Runner idle window.

**Breaking changes**
None. The field is optional and additive.

## What Changed

- `PluginEnvironmentAcquireLeaseParams` has a new optional
`runnerLifecyclePolicy`.
- Core sends it only for warm leases.
- `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` documents the field.
- The runner-e2e SDK public-input pin is updated for the SDK change.

An earlier revision of this PR also made Core try
`environmentResumeLease` before a session handoff. That part is removed:
placement is the plugin's decision, so Core should not decide to retry
the same computer before moving the session. The plugin will own that
inside its own resume instead.

## Verification

- `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed.
- `vitest run src/__tests__/environment-runtime.test.ts -t "portable
task placement|delegates plugin environment leases with acquisition
budget"` passed (5 cases). The portable test asserts the first warm
acquire receives `runnerLifecyclePolicy` and the lease records
`runnerTaskLifecyclePolicy`; the non-warm delegation test asserts the
field is absent. Making the forwarding unconditional fails both
delegation cases.
- The full suite was not run locally; required CI covers it.

## Risks

- A plugin that does not know the field ignores it.
- Rollback: revert this pull request. No migration.

Note: the branch name still reflects the earlier scope (resume before
handoff).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:45:31 -08:00
DottaandClaude Opus 5.5 86f9555695 fix(native): give a Runner whose first unanswered read came on the last attempt one replacement attempt (#21322)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner, and a same-run retry
reattaches that Runner after an interruption
> - #21272 made the next attempt replace a Runner that left its recovery
read unanswered, instead of adopting it again
> - On staging, three runs after #21272 went live still failed: their
first unanswered read came on the third and last attempt, so no next
attempt existed
> - This pull request gives such a run one more attempt, used only for
that replacement
> - The benefit is that the replacement from #21272 can run when a
restart used up the middle attempt

## Linked Issues or Issue Description

Refs #21272, Refs #21161

**What happened?**

After #21272 (497ccba8f1) went live on staging at 05:34Z, three Codex
runs still failed with `native_session_recovery_failed:
HarnessCapabilityUnavailableError: read is unavailable: Error:
provider_transport_failed: PRP command session.snapshot timed out
(runner connected; pending ahead 0; last runner event seq … 55–84s
ago)`. The run events of all three show the same sequence:

1. Attempt 1 fails at 06:24 with `runnerd did not report the provider
turn identity`.
2. Attempt 2 starts at about 06:25. The 06:24–06:25 server restart ends
it with `local_runtime_api_listener_unavailable`. This attempt sends no
recovery read.
3. Attempt 3 adopts the same Runner. Its first recovery read gets no
answer. Attempt 3 is the last of three, so the run ends.

`adoptedRunnerUnanswered` was recorded on attempt 3, but no next attempt
read it. The replacement from #21272 never ran.

**Expected behavior**

A Runner whose first unanswered read comes on the last attempt is still
replaced once by a fresh Runner from the saved journal. The provider
turn is not sent again. The run does not end only because a restart used
the middle attempt.

**Steps to reproduce**

1. Start a Codex run in a remote Daytona Runner. Let attempt 1 fail at
turn start, so the Runner stays alive.
2. Restart the server during attempt 2, so that attempt fails without a
recovery read.
3. Let the Runner leave its recovery read unanswered on attempt 3.
4. See the run end with `session.snapshot timed out` and no replacement.

**Paperclip version or commit**

Staging with #21272 (497ccba8f1). Fixed on `master` after `bbafa78fe2`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`).

## What Changed

- `nativeUnansweredRunnerReplacementDisposition` (executor): when
attempt 3 fails because its adopted live Runner left a recovery read
unanswered, and that attempt was not itself a replacement, the failure
stays `retryable_failure` (`native_session_interrupted`, retry in 30 s)
instead of `native_session_retry_exhausted`.
- `claimNativeRestartRecoveries`: a failed remote retry at attempt 3
with `adoptedRunnerUnanswered` is not treated as an exhausted budget.
The claim is `reattach_remote_runner` at provider attempt 4 with
`replaceUnansweredRunner`.
- `nativeAdmittedProviderAttemptLimit` (executor admission gate): a
`reattach_remote_runner` claim with `replaceUnansweredRunner` is
admitted up to provider attempt 4. Every other claim keeps the limit of
3.
- `verifyRemoteRunnerOrSettledExit`: the replacement of a live Runner is
allowed up to attempt 4. An exited Runner keeps the fail-closed path.
- Bound: the extra attempt is a replacement (`replaceUnansweredRunner`),
so it never gets another one. Attempt 4 is exhausted by the existing
rule.
- `doc/execution-semantics.md` describes the extra attempt.

## Verification

- End to end (embedded Postgres):
`native-remote-retry-admission.test.ts` › "executes the one replacement
attempt after the last attempt's first unanswered recovery read, and no
further one". It seeds attempt 3 with the staging failure, claims it,
and executes the claim through `executePaperclipNativeSession`. The
provider is reached once, the run then ends at attempt 4, and no further
claim is made. With the old `nextAttempt > 3` gate this test fails
before the provider is reached.
- Red on `master`, green with the fix (7 tests, including "admits
provider attempt 4 at execution only for the replacement claim"):
- `native-runner-restart-recovery.integration.test.ts` › "gives a Runner
whose first unanswered read came on the last attempt one replacement
attempt: true" (embedded Postgres; the coordinator is seeded at attempt
3 with the staging message). On `master` no reattach claim was made.
- `native-session-executor.test.ts` › "bounds the replacement after …"
(4 cases: the last attempt's first unanswered read is retryable; a
replacement that also went unanswered, another failure, and attempt 4
stay terminal) and "replaces the live Runner on the extra replacement
attempt".
- Focused suites: `native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation`: 1737 passed, 21 skipped.
- `pnpm typecheck` in `server` passes.
- Not run locally: the container-gated remote Runner suites and the full
suite (CI covers it). Why the remote Runner stops answering is a
separate investigation and is not changed here.

## Risks

- Low to medium. This adds at most one provider attempt per run, and
only after an unanswered recovery read on a live adopted Runner. The
attempt replaces the Runner through the launch fence, which proves the
old Runner and its lineage gone or fails closed. No provider turn is
sent again.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events),
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:27:35 -06:00
DottaandClaude Opus 5.5 49f0030f06 fix(heartbeat): rank a person's joined comment first and refill free run slots (#21277)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts each agent's queued runs, up to the
agent's `maxConcurrentRuns`
> - #21234 starts person-initiated runs before system recovery runs, and
it reads "person-initiated" from the run's own wake request
> - When a person comments on a task that already has a queued system
run, the comment joins that run. The run keeps its system wake, so
dispatch ranks the person's message as system work
> - Dispatch is event driven. The queued-run re-drive backstop covers
only agents with no live run, so an agent with several slots keeps a
free slot empty until one of its runs finishes, and runs can take hours
> - This pull request makes dispatch read the person wakes that joined
each queued run, and makes the re-drive backstop cover an agent that
still has a free slot
> - The benefit is that a person's comment starts first, and free slots
fill within about a minute instead of waiting for an unrelated run to
finish

## Linked Issues or Issue Description

Refs #21234, #20986

**What happened?**

On staging, one agent had `maxConcurrentRuns` 20, 18 running runs and 76
queued runs at 03:12Z. No run started on that agent from 03:01:33Z to
03:16:57Z (15.4 minutes). From 03:11:10Z, after a run finished, at least
2 slots were free. Other agents in the same company started runs during
that time. Runs started again only after the next deploy restarted the
server.

Five queued runs came from a person: comments at 02:43Z, 02:44Z and
02:45Z, and Retries at 02:46Z, 02:49Z and 03:01Z. The comment runs had
the wake reasons `issue_reopened_via_comment` and `issue_commented`, the
invocation source `automation`, and no person on the run's own wake
request.

**Expected behavior**

A run that carries a person's comment ranks as person-initiated, also
when the comment joined an existing queued run. When an agent has a free
slot and a queued run is ready, the run starts promptly, without waiting
for another run on the agent to finish.

**Steps to reproduce**

1. Give an agent `maxConcurrentRuns` 3. Start 2 runs on it.
2. Queue a third run while no dispatch signal reaches the queue (for
example, a claim declines it while a gate is still closed, and the gate
then clears).
3. See that `listUndispatchedQueuedRuns` returns nothing, because the
agent has a live run. The run stays queued until one of the 2 running
runs finishes.
4. Separately, queue a system run on a task, then post a person's
comment on the task so that the comment wake coalesces into that queued
run. Queue older system runs on other tasks of the same single-slot
agent.
5. See that dispatch starts the older system runs first.

**Root cause**

1. `coalesceIntoActiveExecutionRun`
(`server/src/modules/wake-queue/adapters/postgres.ts`) merges the
person's comment context into the queued run and records the person's
wake as a separate `coalesced` row with `runId` set to that run. The
run's own `wakeupRequestId` still points to the system wake.
`queuedRunDispatcherForAgent` (`server/src/services/heartbeat/queue.ts`)
only read the wake behind `wakeupRequestId`. The task routes do record
the comment author as the requester (`requestedByActorType: "user"`).
The person was lost only at dispatch time.
2. `listUndispatchedQueuedRunPage`
(`server/src/services/queued-run-dispatch-liveness.ts`) skipped every
agent with any `running` or `scheduled_retry` run. That rule assumes one
slot per agent. For an agent with several slots, a lost or declined
dispatch is not re-driven until a run on that agent finishes.

This change is not in `claimQueuedRun`, the fresh-isolated preflight,
the wake-queue promoter or `fresh-continuation-fallback.ts`.

**Paperclip version or commit**

Staging Core `ddc0aacad8` (contains #21234 `ade7e0643f`). This branch is
based on master `6cb0dd0c6b`.

**Deployment mode**

Authenticated, public, Paperclip Cloud managed staging.

**Agent adapter(s) involved**

`paperclip_runner` (Codex native runtime). The fix is
adapter-independent.

## What Changed

- `queuedRunDispatcherForAgent` also reads person wakes whose `runId` is
a queued run of the agent (by `run_id IN (queued run ids)`, so an
earlier saved comment that a run adopted is included). A run with such a
wake ranks as person-initiated and is never counted as system recovery.
- `listUndispatchedQueuedRuns` now returns a queued run when its agent
has fewer `running` plus `scheduled_retry` runs than its
`maxConcurrentRuns`, and its task has no `running` or `scheduled_retry`
run. The re-drive backstop and the ownership I7 check use this query.
- The query counts each agent's live runs once, in one grouped subquery
that is joined to the queued runs.
- Migration 0476 adds `agent_wakeup_requests_company_run_idx`
(`company_id`, `run_id`, partial on a non-null run) with `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`. It is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` and classifies as
`online_compatible`. `doc/DATABASE.md` describes it.
- Added `maxConcurrentRunsSql()` to `timer-policy.ts`. It reads the same
limit as `parseHeartbeatPolicy`, including the single-run runner
providers, which now share one constant.

## Verification

- Red/green: `server/src/__tests__/recovery-wake-herd.test.ts` "starts a
queued run that a person's comment joined ahead of older system work".
It fails on master (`expected null not to be null`: the older system run
started first) and passes with this change.
- Red/green:
`server/src/__tests__/ownership-reconciliation-report.test.ts`
"re-drives a queued run while its agent still has a free slot, and not
once every slot is running". It fails on master (`expected [] to deeply
equal [ Array(1) ]`) and passes with this change.
- Red/green: "starts a queued run that adopted a person's comment saved
before every queued run". It fails with a `requestedAt` bound and passes
with the `run_id` lookup.
- Red/green: "counts an agent's live runs in one grouped aggregate, not
once per queued run" runs EXPLAIN on the page query. It fails with a
per-row correlated count and passes with the grouped subquery.
- New: "uses the agent's single-run limit for a runner provider that
runs one turn at a time".
- Two existing tests now set `maxConcurrentRuns: 1` so that they keep
testing "the agent's only slot is busy".
- `npx vitest run src/__tests__/recovery-wake-herd.test.ts
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/ownership-reconciliation-report.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/services/heartbeat/dispatch-work.test.ts`: 71 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: no errors.

## Risks

- The re-drive backstop now also covers multi-slot agents with live
runs. Each queued run is re-driven at most on its existing backoff (1
minute, then doubling to 15 minutes). A queued run whose task already
has a live run is not re-driven.
- The ownership I7 report can now list a queued run on a multi-slot
agent that has a free slot. This is the stall that this change targets.
- One more indexed read per dispatch pass (`agent_wakeup_requests` by
company and run id). The new index is built concurrently, so the
previous release keeps writing during the build.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, read-only staging API reads in a
browser tab).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:17:57 -06:00
Nicky LeachandClaude Fable 5.2 23c2343015 feat(server): read a retained provider pack from its Runner image (#21308)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner executes agent work in a sandbox. It needs a provider
pack of agent tools, and the server uploads that pack when the sandbox
does not have it
> - A retained session keeps the pack it started with. When that pack is
not the server's own, the server downloads the pack's transport archives
from a registry (#19779)
> - Today the server looks for them in the application image tagged
`sha-<commit that built the pack>`
> - `master` now publishes the Runner one time for each set of inputs,
as `ghcr.io/paperclipai/paperclip-runner`. A pack from that image names
the commit that published the image, and few commits get an application
image, so that tag often does not exist
> - This pull request reads such a pack from its Runner image, and keeps
the old source for every other pack and for a Runner image that cannot
be read
> - The benefit is that a retained session can still get its pack when
the pack came out of the published Runner image

## Linked Issues or Issue Description

Refs #19779 (the retained pack source), #21228 (the pack field
`runnerInputsDigest`), and #21204, #21229, #21230 (the Runner image). No
issue exists for this change.

**Problem or motivation**

`retained-provider-pack-oci.ts` reads
`ghcr.io/paperclipai/paperclip:sha-<runnerSourceRevision>`. It needs
that image to exist, to carry the pack, and to have one layer with a
known build-history line. For a pack from a published Runner image,
`runnerSourceRevision` is the commit that published the image.
`docker.yml` writes `sha-<commit>` only when its `master` run for that
commit completes, and a newer push can replace a waiting run. The
restart-chaos harness has the same assumption in another place: it
requires the pack of a sandbox image to be built by the commit in the
image's `org.opencontainers.image.revision` label.

**Proposed solution**

A pack that names its Runner image in `payload.runnerInputsDigest` is
read from `ghcr.io/paperclipai/paperclip-runner:inputs-<digest>`. A
sandbox image can name the commit that built its Runner in a label of
its own. No application image carries a pack with that field yet, and no
image carries that label, so behaviour today does not change.

**Alternatives considered**

- Keep the application image as the only source and require an
application image for every Runner publication. Publication would then
depend on `docker.yml`, which serves one ref at a time.
- Take a fixed number of layers ("the image has four, take the last
two"). A server in the field cannot be changed when the image gets
another layer. The server now uses the rule that holds for any number of
layers: the full image is the slim image plus transport layers.
- Fall back to the application image for every failure of the Runner
image. A Runner image that exists and does not match the pack is a
fault, and a second source would hide it.

**Roadmap alignment**

This is build and runtime infrastructure. It adds no product feature and
does not overlap an item in `ROADMAP.md`.

This is part 3 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image (this
pull request)
4. #21310: a registry build cache for the Runner image build

This pull request uses no code from parts 1 and 2. It is stacked on them
because all four parts edit `doc/runner-image-contract.md`.

## What Changed

- `server/src/services/native-runtime/retained-provider-pack-oci.ts`,
for a pack whose `payload.runnerInputsDigest` is 64 hexadecimal
characters:
- The server reads the full index `inputs-<digest>` of the Runner image.
The index must carry `io.paperclip.runner.inputs-digest` equal to the
pack's value, `org.opencontainers.image.revision` equal to the pack's
`runnerSourceRevision`, and a valid digest in
`io.paperclip.runner.slim-digest`.
- It reads the slim index by that digest, never by the slim tag, and
checks the bytes it gets against the digest.
- For the pack's platform, the layer digests of the slim image must be
the first layer digests of the full image, in the same order. The layers
after them are the transport layers. There must be at least one and at
most eight, and their total size has the existing limit.
- The server downloads only those layers. It checks each against its
descriptor and refuses every entry that is not a transport archive, the
receipt or the `provider-pack.archives` directory. A file may not be in
two layers.
- When the Runner image cannot be read, the server reads the application
image as before. That covers a registry error status for the token, for
either index or for either manifest: for example 404 for a tag or a slim
index that does not exist. Nothing is written before that point.
- A Runner image that can be read and does not match fails with
`runner_remote_provider_source_invalid`. The server does not try the
application image then.
- The same file, for every other pack: the old path, unchanged in what
it checks. Its code moved into functions that both paths use (`request`,
`pullToken`, `platformImage`, `receiveTransportLayer`).
- The downloaded layer file is removed as soon as it is unpacked.
Before, the importer removed it after its checks, so
`retained-provider-pack-source.ts` loses that one line.
- The cache record gets an optional `runnerImage` field with the inputs
digest and the digests of the transport layers. `transportLayerDigest`
is the last of them and `transportLayerBytes` their total. An older
server ignores the new field.
- `tests/restart-chaos/daytona-image.ts`: when the image has a non-empty
label `io.paperclip.runner.source-revision`, the pack must be built by
that commit. Otherwise the pack must be built by the commit in
`org.opencontainers.image.revision`, as before. The receipt field
`providerPackRunnerSourceRevision` records the pack's commit.
- `doc/runner-image-contract.md` has a new section, "Reading the
transport archives alone". `tests/restart-chaos/README.md` describes the
label.

One choice needs a reviewer's eye. When the full index can be read and
the slim index cannot (a registry error status), the server falls back
to the application image; it does not fail. Without the slim manifest
the server cannot tell which layers are transport layers, so it has not
accepted any byte of the Runner image, and the old source has its own
complete checks. A slim index that is read and has other bytes than its
digest still fails.

## Verification

Linux (`node:24.22.3-bookworm`, GNU tar 1.34, zstd 1.5.4, the worktree
mounted read-only):

- In `server/`: `retained-provider-pack-source.test.ts` (34 tests, 15 of
them new), `provider-pack-archive.test.ts`,
`remote-provider-pack-verification.test.ts`,
`remote-provider-pack.test.ts`: 127 tests pass in 4 files.
- `node node_modules/vitest/vitest.mjs run --config
tests/restart-chaos/vitest.config.ts
tests/restart-chaos/daytona-image.test.ts`: 25 tests pass, 5 of them
new.

macOS (Node 26.12.0):

- `daytona-image.test.ts`: 25 tests pass.
- `retained-provider-pack-source.test.ts`: the 15 new tests pass. Six
older tests fail, with and without this change. They run the real
uploader, which needs GNU tar, and this host has bsdtar.
- `tsc` on the two changed server files and the test, and on the two
restart-chaos files, with the settings of their test projects: no error.

The new cases:

- Read from the Runner image with 6 registry requests (two indexes, two
manifests, two layers), none to the application image and none for the
slim tag.
- Take every layer after the slim image's: one transport layer, and
three.
- Fall back when the tag does not exist, when the token is refused, and
when the slim index cannot be reached.
- Refuse, with no request to the application image: an index for other
inputs, an index for another revision, no slim annotation, a slim
annotation that is a tag, a slim index with other bytes than its digest,
a slim image that is not a prefix of the full image, a slim image equal
to the full image, nine transport layers, and a transport layer that
holds a pack file.
- Four deliberate faults in the reader each make a test fail: the slim
index bytes are not checked, no prefix check, the slim annotation is not
validated, and no bounds on the number of transport layers.

A real run, in the same Linux container (`linux/arm64`), with no
credential:

- The script took `provider-pack/provider-pack.json` of `linux/arm64`
from the published image
`ghcr.io/paperclipai/paperclip-runner:inputs-0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`.
Pack digest
`sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f`,
`runnerSourceRevision` `25e9a9c2e9277f3be8649be8c809cc81bde6f0c5`.
- `resolveProviderPackUploadSource`, with another pack as the server's
own, returned `{ archiveOnly: true }` after 23 seconds. That function
calls `receiveRetainedProviderPack` and then the importer's checks: the
manifest inside the overlay archive is the selected manifest, every
archive matches the receipt, and the content of every file archive
matches the digest of its file.
- The server code made 9 requests: one anonymous token request, the full
index, the `arm64` manifest, the slim index `sha256:f63ea436…4f3c` by
digest, its `arm64` manifest, and two layers, each with one redirect. No
request went to the application image.
- The cache record:

  ```json
  {
    "schema": "paperclip.retained-provider-pack-source.v1",
    "sourceRevision": "25e9a9c2e9277f3be8649be8c809cc81bde6f0c5",
"providerPackDigest":
"sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f",
"imageIndexDigest":
"sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162",
"imageManifestDigest":
"sha256:73dbe864fbab6208616f0aa6f1610a7ece4ef22f68a73d30901cbaea60139589",
"imageConfigDigest":
"sha256:740e48463d5caf0bfead4b923cdbd8b220acb93d058419c3db9d00c7f1831d65",
"transportLayerDigest":
"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf",
    "transportLayerBytes": 426602696,
    "runnerImage": {
"inputsDigest":
"0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca",
      "transportLayerDigests": [

"sha256:6a340088ca2cfa85f10f0385d9c8f5cdccb22c3a2bbb2062620b1d90616ba5b5",

"sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf"
      ]
    },
"receiptSha256":
"2afdd26645845369d9e851b1e6e2884eb6c3884ea7c4866a1eeb2efb9fb230d7",
    "verifiedTarPaths": true
  }
  ```

- `uploadVerifiedProviderPack` then installed the pack from that source
through a local stand-in for the sandbox, with no further registry
request, in 4 seconds. The installed tree is 1,249,060,841 bytes. Its
`provider-pack.json` is the selected manifest, all 7 artifacts of the
manifest have the digest the manifest names, and the pack's own Node
prints `v24.22.3`.

Never run:

- The same real run for `linux/amd64`, whose transport layers are about
1 GB.
- A session in a real sandbox. The real run used a local stand-in for
the sandbox.
- The fallback against the real registry. Fixtures cover it.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The old path was moved into shared functions. Its checks are the same,
and its 19 existing tests pass on Linux, but a review must read that
code as changed.
- The layer file of the old path is now removed before the importer's
checks, not after them. The importer does not read it.
- The reader depends on two rules of the image contract: the full index
names its slim index, and the full image starts with the slim image's
layers. A Runner image that breaks either rule is refused, with no
fallback.
- The fallback covers a registry error status only. A network error ends
the import, as it does today.
- The label `io.paperclip.runner.source-revision` is new and no image
carries it yet. The change to the sandbox image that sets it must use
this name.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:02:19 -08:00
DottaandClaude Opus 5.5 b555087f0f fix(native): continue again when a deploy interrupts an automatic continuation (#21314)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner (Daytona) that the
server controls; a deploy restarts the server while runs are in flight
> - When the restart's reattachment proves the Runner process gone
mid-turn, #21039 makes the server request one automatic Retry that
continues in a fresh session from the saved work, without replaying the
turn
> - Staging deploys every 20 to 45 minutes, and Codex runs often take
longer, so that automatic continuation is often interrupted by the next
deploy too
> - The hook treated that second loss as "automatic Retry already used"
and left the task blocked for a person
(`native_continuation_requires_reconciliation`)
> - This pull request gives an automatic continuation one more
fresh-session continuation each time a server restart or deploy took its
controller, and a deploy spends no automatic budget
> - The benefit is that a task interrupted by repeated deploys continues
by itself, and no provider turn that may have run is sent again

## Linked Issues or Issue Description

Refs #21039, Refs #21052, Refs #21272, Refs #21023, Refs #21120, Refs
#21161

**What happened?**

On staging, the largest failure class for a Codex agent on Daytona was
`runner_remote_process_identity_unavailable` with error code
`native_continuation_requires_reconciliation`: 18 runs in 3 hours. The
error name is misleading. It is the shared code for every refused remote
reattachment (`REMOTE_RUNNER_RECOVERY_ERROR_CODE`). In every sampled run
the refusal was `process_verification` / `process_not_running`: the
Runner process was gone, its stderr log was empty, and the last turn
event was `turn.submitted` or `turn.accepted` (unsettled). It was not a
missing identity file.

Sampled 12 such runs. In 9 the #21039 hook worked: it recorded
`native.interrupted_turn_outcome_unknown` and requested the
fresh-session Retry. The tasks that stayed blocked had one shape:

1. A run lost its Runner after a deploy. The hook requested the
automatic Retry (key `task-retry:restart-interrupted-turn:<run>`).
2. The Retry run started a fresh session. The next deploy came during
its turn (a graceful `SIGTERM` shutdown transition, then "Recovering the
existing native runner process after server restart").
3. The same-run reattachment found that Runner gone too
(`process_not_running`).
4. The hook returned `automatic_retry_used`, because the run's Retry
intent came from the hook. The task stayed blocked.

Examples: a `task_retry` run created 40 s after its predecessor failed
(the hook's Retry), interrupted by the 05:17 deploy, failed at 05:18,
task blocked. A second chain (assignment, then the hook's Retry) failed
the same way at 05:12.

**Expected behavior**

A deploy is not the task's failure. When a server restart or deploy took
the controller of an automatic continuation and its Runner is proven
gone, the server continues the task again in a fresh session from the
saved work. It never replays the turn. A loss without restart evidence
keeps the existing one-time rule.

**Steps to reproduce**

1. Start a long Codex run in a remote Daytona Runner.
2. Deploy the server during its provider turn so that the reattachment
finds the Runner gone. The server requests the automatic Retry.
3. Deploy again during the Retry run's turn so that its Runner is also
lost.
4. See the Retry run fail with
`runner_remote_process_identity_unavailable` and the task stay blocked
with no new run.

**Paperclip version or commit**

Staging Core `25e9a9c2e9`. Fixed on `master` after `731b52453c`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `restart-interrupted-turn.ts`: `lostControllerToRestart` finds restart
evidence on the run: a `native.recovery.transition` with `restartKind:
"graceful"` (shutdown), or a restart claim (`reattach_remote_runner`,
`reattach_existing_runner`, `resume_dead_runner`,
`bootstrap_incomplete`) without the `same_run_retry` trigger.
- Both `automatic_retry_used` refusals (a person's Retry that the hook
requested, and a system retry of owed work) now continue when the run
has restart evidence. A deploy interruption spends no budget, so a task
that only deploys interrupt never runs out. Each continuation needs a
new deploy during its run, so deploys bound it. A loss without restart
evidence keeps the one-time rule. All other gates are unchanged: proven
Runner exit, unknown-turn record, the issue still assigned and open,
idempotency per run, and the Retry's admission gates.
- `doc/execution-semantics.md` describes the rule.

## Verification

- Red before the fix, green after:
`server/src/services/runner-lost-mid-turn-continuation.test.ts` › "an
automatic continuation that the next deploy interrupted". The test seeds
the staging sequence: a lost run, the hook's Retry run with a
`turn.submitted` turn, the shutdown and restart transitions with staging
payloads, then a same-run reattachment that proves the Runner gone. On
the old code the two continuation tests return `automatic_retry_used`.
With the fix they request one fresh-session Retry, record the unknown
turn outcome once, recover the saved work and queue the successor with
`forceFreshSession: true`.
- A chain of 12 deploy continuations, each with a system retry that
inherits its Retry receipt, still continues (red on the first revision
of this PR, which had a budget of 8). Losses without deploy evidence
stay refused (with and without earlier deploy continuations).
- `npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts` 23/23 pass.
`governed-wait-runner-loss.test.ts` and
`workspace-recovery-retry-copyback.test.ts` 103/103 pass.
- `pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify` exits 2 (a global doc input changed), so required CI
supplies the broad coverage.

## Risks

- Low to medium. A task whose Runner is lost on every deploy now
continues after each deploy instead of once. Each continuation is a full
run, so it costs tokens. The deploy count bounds it, because each
continuation needs a new restart during its run.
- Not fixed here: why the reattached Runner is gone after a deploy. In
the sampled runs the reattachment after the restart hung in `native
session provider recovery timed out after 900000ms`, and the next
attempt found the process gone with empty stderr. That needs its own
investigation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, browser read-only API GETs on staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:41:32 -06:00
Nicky LeachandClaude Fable 5.2 03f9b83e52 feat(ci): report the Runner image on master and build the app image's Runner path on pull requests (#21307)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work, and `master` now
publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`
> - The application image will soon copy that Runner instead of
compiling its own. Part 1 of this stack adds the resolver that picks the
Runner for a build
> - Two gaps make that next change risky. The resolver has never run in
CI on `master`, so its lookup time and how often a commit has a
published Runner are unknown. And `docker.yml` has no pull-request
trigger, so a change to how the application image gets its Runner first
runs after it merged
> - This pull request adds a report-only job that runs the resolver on
every `master` push, and a pull-request check that builds the Runner
path of the application image
> - The benefit is real data from `master` before any build depends on
the resolver, and a failed check on the pull request instead of a failed
image build on `master`

## Linked Issues or Issue Description

Refs #21204, #21229 and #21230 (the Runner image). No issue exists for
this change.

**Problem or motivation**

The resolver decides whether a build copies the published Runner. On
`master`, a commit that changes the Runner's inputs is ahead of its
publication for some minutes, and nobody knows yet how often an
application build will meet that. The application image itself is built
only on `master` and on tags. The change that removes the Runner's
compile stages from that image will edit the `Dockerfile`, and today
nothing builds those stages on a pull request.

**Proposed solution**

Add one job to `docker.yml` that only reports, and one new workflow that
only checks. Neither changes what an image contains, and no existing job
reads either of them.

**Alternatives considered**

- Run the resolver inside the build legs. A failure there could fail the
image build. A job of its own, with `continue-on-error`, cannot.
- Give the lookup the job token. The Runner package is public, so an
anonymous lookup works and the job needs no credential.
- Build `production` in the pull-request check. It adds three `COPY`
instructions and a readability check to the two stages that this check
builds, at the cost of the tool layer and two more copies of several
gigabytes.
- Trigger the check on every Runner source change. `runner-image.yml`
already builds the Runner on those pull requests, and CI builds the
server and the UI.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 2 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path (this pull request)
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build

## What Changed

- `.github/workflows/docker.yml`: new job `runner_image_report`.
  - It runs only for a `master` push in this repository.
- It runs `runner-image-resolve.mjs inputs`, then `resolve`, and the
result goes to the job summary with the line "Report only: no build read
this result."
- It has `continue-on-error: true`, declares no `outputs`, and no job
needs it. Its failure does not fail the run.
- It holds no credential. Its only permission is `contents: read`, the
checkout does not keep its token, and the lookup is anonymous. It does
not log in to a registry and writes to none.
- Time limits: 8 minutes for the job, 5 for the step that builds, 2 for
the lookup.
- It is the first job in the file. Existing tests cut the other jobs out
of the file by name, up to the next job or to the end of the file, so no
other place leaves those tests unchanged.
- `.github/workflows/docker-app-check.yml` (new): on a pull request,
builds the `build` and `runner-provider-pack` targets with
`PAPERCLIP_BUILD_COMMIT` set, for `linux/amd64` only.
- `permissions: {}` at the top, `contents: read` for the job, no
credential, no `pull_request_target`.
- It exports nothing (`--output type=cacheonly`) and pushes nothing. It
reads the public build cache that `docker.yml` writes and never writes
it.
- It refreshes the lockfile in the build context first, as `docker.yml`
does.
  - Time limits: 60 minutes for the job, 45 for the two builds.
  - Trigger paths, and why:

    | Path | Reason |
    | --- | --- |
    | `.github/workflows/docker-app-check.yml` | the check itself |
| `Dockerfile`, `.dockerignore` | the recipe, and the files it receives
|
| `server/scripts/prepare-runner-vendor.mjs` | the step of `build` that
reuses or rebuilds the Runner |
| `packages/paperclip-runner/scripts/build-inputs.mjs` | the build stamp
that step compares with the files the image build received |

- `server/src/__tests__/docker-runner-workflows.test.ts` (new, 90
lines): only security properties. The report job has no credential and
one read permission, nothing can read it, and its two steps have time
limits. The check has no credential, cannot publish, and exports
nothing. In both, every action is pinned by commit and no shell script
holds an expression.
- `doc/runner-image-contract.md` and `doc/DOCKER.md` name the job and
the check.

## Verification

macOS (Node 26.12.0):

- In `server/`: `docker-runner-workflows`, `runner-image-workflow`,
`docker-build-stamp`, `container-init-reaping`,
`cloud-image-bundled-plugins`, `docker-image-layer-reuse` and
`cloud-image-sentry` tests: 52 tests pass in 7 files.
- `tests/runner-e2e` workflow tests (`workflow-security`,
`private-trust-workflow-security`, `workflow-image-only`,
`paid-runner-routing`): 27 tests pass in 4 files.
- `node --test .github/scripts/tests/*.test.mjs
scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: 962 of 962 pass.
- Both workflow files parse as YAML.
- `docker buildx build --target runner-artifacts-context --cache-from
type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-amd64
--output type=cacheonly .` succeeds, so both flags are accepted and the
cache is readable.

Linux (`node:24.22.3-bookworm`, with `zstd` and `jq` added): the same
three sets pass: 52, 27 and 962 of 962.

The two commands the report job runs were run by hand, with no
credential, on clean exports of the tree. For the inputs `0e6662ad…d1ca`
the answer was `published`, with the two index digests that `master`
published. For a commit whose image was not yet published the answer was
`inline`, `absent`. Part 1 has the output.

With the job at the end of `docker.yml`, two existing tests failed. They
are the reason for its place: `docker-canary-promotion.test.mjs` reads
the last job to the end of the file, and
`release-verify-workflow.test.mjs` allows no BuildKit configuration
after `merge-and-push`.

Never run:

- Either workflow. Only GitHub can run them. This pull request starts
the check by its own path; the report job first runs on `master`.
- The two builds of the check (`--target build`, `--target
runner-provider-pack`) on a local host. Their time and disk use on a
hosted runner are not known.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The check builds the whole `build` stage, with the UI and server
builds. Its duration is not measured. It runs only when one of five
paths changes, and its first real run is on this pull request.
- `docker.yml` holds one concurrency slot for each ref until all its
jobs end. The report job normally ends long before the build legs. A
stalled job can hold the slot for at most 8 minutes when the legs end
sooner.
- `scripts/watch-release-source.mjs` warns about every job of the image
run that waited more than 60 seconds for a runner. The report job is one
more job that it can name.
- `scripts/merge-group-image.mjs` refuses to promote a merge-queue image
for a push that changes `docker.yml`. The push that lands this change is
such a push. The merge queue is not enabled.
- The lookup has 2 minutes. The registry client retries a failing
request for longer than that, so a registry outage ends the step by its
time limit, and the job then reports nothing. The run stays green.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:29:33 -08:00
DottaandClaude Opus 5.5 bcadbde449 feat(runner): record the OpenRouter host that served each OpenCode model call (#21298)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agents can use models through OpenRouter, and the Runner
proxies those model calls
> - OpenRouter routes one model across many hosts, and these hosts use
different quantizations (fp4, mxfp4, fp8)
> - Chat evaluations with identical prompts behave very differently
between runs, but the run log does not record which host served a call
> - The run usage shows `provider: "deepseek"` and `biller:
"openrouter"`, and `providerRequestId` is always `null`
> - This pull request records the serving host and the OpenRouter
generation ID for each model call in the run log
> - The benefit is that we can compare evaluation cohorts by host, and
the cost event names a real OpenRouter generation

## Linked Issues or Issue Description

**Subsystem affected**
Runner OpenCode provider proxy, runnerd usage normalization, and native
run usage accounting.

**Problem or motivation**
OpenRouter's default routing sends a request to one of many hosts. Each
host can use a different quantization. Chat evaluation results change a
lot between runs with identical prompts. Paperclip does not record which
host served a call, so we cannot compare results by host.
`usageJson.providerRequestId` is always `null` for OpenCode runs.

**Proposed solution**
Every OpenRouter completion response has a top-level `provider` field
and an `id` (the generation ID). Each streamed chunk has them too. The
Runner proxy reads them from the first chunk after it forwards the
bytes. The Runner attaches them to the usage of that call. The values go
to the run log, and the run's first generation ID goes to
`usageJson.providerRequestId`.

**Alternatives considered**
A new PRP event type such as `provider.upstream.served` would need a new
event enum value. An older control plane rejects an unknown event type.
A `harness.diagnostic` event shows in the task chat as a diagnostic. The
usage payload already flows to every consumer, and older consumers
ignore extra fields. Thus this change uses the usage payload.

**Additional context**
Run-log change only (AGENTS.md section 7). The data stays in
`heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch
`packages/shared/src/telemetry/` or OpenTelemetry.

## What Changed

- New `createUpstreamServedTap`
(`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`).
It is a pass-through stream. It pushes each chunk downstream first, then
it reads `provider` and `id` from the first SSE `data:` event (or from a
JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of
a JSON body. It skips encoded bodies and unknown content types. Any
parse problem or recorder error stops the scan and leaves the response
unchanged.
- `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The
OpenCode session keeps the most recent served call. Each
`item.completed` usage payload and `session.usage()` now carry
`upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint
does not include these fields, so the number of usage events does not
change.
- The OpenCode app-server proxy adds the two fields to
`thread/tokenUsage/updated`. Rust `usage.reported` now fills its
existing `providerRequestId` field and adds an optional
`upstreamProvider`. The TypeScript runnerd transport keeps both fields
when it rehydrates the usage notification. `usage.schema.json` documents
`upstreamProvider` (the protocol manifest and schema bundle are
regenerated).
- Server native turn accounting keeps the first generation ID the run
reports. The usage checkpoint and the native adapter result carry it as
`providerRequestId`, so `usageJson.providerRequestId` and the final cost
event name it.
- `doc/run-log-events.md` documents the fields in a new "OpenRouter
serving host" section.

## Verification

- Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's
stream through unchanged and records the host that served it on the
turn's usage". The test sends a streamed fixture with a comment line, a
host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks
that the bytes are identical and that the turn's usage names `Novita`
and the generation ID. The test fails without the proxy wiring.
- `upstream-served.test.ts`: streamed fixture, chunk split inside a
multibyte character, non-streamed JSON, malformed chunks, a truncated
JSON body, an encoded body, the 64 KiB limit, and a recorder that
throws.
- `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts`
(rehydration), Rust
`provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`,
and server `native-usage-accounting.test.ts` (first generation ID stays
stable across turns).
- Local: OpenCode driver, OpenCode proxy, and runnerd transport suites
(31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes
(one timing test failed once under load and passed alone). Runner and
server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks
pass.
- Latency: a local benchmark proxied a 400-chunk SSE stream 200 times
with the tap and 200 times without it. The median time to first byte was
292 µs with the tap and 259 µs without it (+33 µs, which is the extra
stream stage). The total stream time changed from 491.4 ms to 492.5 ms.
The tap spends about 5.5 µs of CPU to inspect the first chunk, after it
has forwarded that chunk. OpenRouter time to first token is hundreds of
milliseconds.
- Paid evidence: one single-case run of
`chat-features.runner-opencode.local.skills.fork-read-only` on this
branch. The result is in a comment on this pull request.

## Risks

- Low risk. All new fields are optional and additive. An older control
plane ignores the extra usage fields. `usage.schema.json` is not bound
by the event schema, and no validator enforces it on `usage.reported`.
- Saved state from the previous release: this change adds no persisted
run, session, checkpoint, or launch state. A saved session resumes
unchanged. A run that is in flight during a deploy reports usage without
the new fields until its Runner is new. `usageJson.providerRequestId`
then stays `null`, as before.
- `providerRequestId` on the run and its cost event is the run's first
generation ID, not each call's ID. The value is stable, so a late usage
receipt still matches the posted charge. Per-call IDs and hosts are in
the run log.
- If OpenCode starts a new model call before it reports the usage of the
previous call, a usage event can name the newer call. OpenCode reports a
message's usage when its step ends, so this is not expected.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code as an agent with tool use (shell, file edits, test runs). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:15:53 -06:00
Nicky LeachandClaude Fable 5.2 f79e2c5d84 feat(ci): add the Runner image resolver and a no-rebuild mode for the Runner vendor step (#21306)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - `master` now publishes the Runner alone as the image
`ghcr.io/paperclipai/paperclip-runner`, one time for each set of Runner
inputs (#21204, #21229, #21230)
> - The application image still compiles its own Runner on every build.
The next steps make it copy the published Runner instead
> - Before a build can do that, it must answer one question the same way
as every other build: "does a published Runner exist for exactly this
tree?" It must also fail clearly when the Runner it was given does not
match the tree
> - This pull request adds the script that answers the question, and a
mode of the Runner vendor step that fails instead of rebuilding
> - The benefit is that the later change to the image build is small,
and that a wrong or unreachable answer can only mean "build the Runner
yourself"

## Linked Issues or Issue Description

Refs #21204, #21229 and #21230 (the Runner image). No issue exists for
this change.

**Problem or motivation**

Three builders make images that contain the Runner: `docker.yml`, the
merge-queue build and the release preview build. Each one would have to
repeat three steps to find the published Runner of its tree: check the
lockfile, compute the inputs digest in the build, and look up
`inputs-<digest>`. Three copies can decide differently. A build that
copies a Runner built elsewhere also has no Rust toolchain. If the
copied Runner does not match the tree, `prepare-runner-vendor.mjs`
starts a rebuild, and the build fails late at `cargo` with a message
that hides the cause.

**Proposed solution**

Add `scripts/runner-image-resolve.mjs`, the one place that decides. Add
`PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` to
`prepare-runner-vendor.mjs`. Nothing calls the resolver and nothing sets
the variable in this pull request, so no workflow job changes.

**Alternatives considered**

- One command that builds and looks up. Two commands keep a registry
credential, when a caller has one, out of the step that builds.
- A wait for the publication of the commit. The decision whether the
application build waits is open, so the resolver has no wait mode. It
reports `absent` and stops.
- A build argument that names the expected inputs. No `Dockerfile`
declares one yet. The change that makes the application image copy the
Runner will add both ends together.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 1 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode (this pull request)
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build

## What Changed

- `scripts/runner-image-resolve.mjs` (new, 185 lines), with two commands
that share a directory:
- `inputs --dir <dir>` exports the `runner-artifacts-context` target,
runs the lockfile check on that export, and builds the
`runner-artifacts-identity` target when the lockfile is current. It
reads no credential. A lockfile that is behind stops there: no image is
published for such a tree.
- `resolve --dir <dir>` looks up `inputs-<digest>` and prints `mode`,
`reason`, `inputs_digest`, `image`, `slim_image`, `build_contexts` and
`labels`. It also writes them to `$GITHUB_OUTPUT` and a short report to
`$GITHUB_STEP_SUMMARY`.
- `mode` is `published` only for a complete publication whose full index
names these inputs and both platforms. Both images are pinned by index
digest. The slim digest is the one the full index names in
`io.paperclip.runner.slim-digest`; the slim tag is never read on its
own.
- Every other answer is `inline`, with the last four values empty:

  | `reason` | Meaning |
  | --- | --- |
  | `lock-behind` | `pnpm-lock.yaml` is behind the Runner's manifests |
  | `absent` | no tag exists |
| `incomplete`, `broken` | a publication stopped, or its two tags do not
belong together |
| `mismatch` | the full index names other inputs, or lacks a platform |
| `unknown` | the registry refused, failed or could not be reached, or
the lookup threw |

- The resolver reuses `runnerLockState`, `readRunnerImageManifest`,
`runnerPublicationState` and `runnerImageTags` from
`scripts/runner-image-publish.mjs`. That file now exports
`environmentCredentials`; it has no other change.
- `server/scripts/prepare-runner-vendor.mjs`: with
`PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1`, the step reuses a current
build as before. When the build is not current, it prints the reason,
starts no rebuild, leaves the stamp in place and exits with status 1. It
also fails when `PAPERCLIP_RUNNER_VENDOR_REBUILD=1` is set at the same
time. Without the variable the step is the same as before.
- `doc/runner-image-contract.md` has a new section, "Choosing the Runner
for an image build". `doc/DEVELOPING.md` describes the new variable.
- The resolver is not an input of the Runner image, so the inputs digest
does not change.

## Verification

macOS (Node 26.12.0), in `server/`:

- `node node_modules/vitest/vitest.mjs run
scripts/prepare-runner-vendor.test.mjs
src/__tests__/runner-image-resolve.test.ts
src/__tests__/runner-image-publish.test.ts
src/__tests__/runner-image-identity.test.ts
src/__tests__/runner-image-workflow.test.ts
src/__tests__/server-package-build-script.test.ts`: 179 tests pass in 6
files. 27 of them are new resolver tests and 3 are new tests of the
vendor step, whose end-to-end test also has a new case.
- `tsc` on the new test file, with the settings of
`server/src/__tests__/tsconfig.json`: no error.

Linux (`node:24.22.3-bookworm`, GNU tar 1.34, the worktree mounted
read-only): the same files, 179 tests pass.

Real runs, on clean exports of the tree (`git archive`), with no
credential, on a `linux/arm64` Docker host:

- On `master` at `62d673cc3f`: `inputs` gave `lock=current` and
`inputs_digest=0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`.
`resolve` gave `mode=published`,
`image=ghcr.io/paperclipai/paperclip-runner@sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162`
and
`slim_image=ghcr.io/paperclipai/paperclip-runner@sha256:f63ea43666ed708f1b0e5d2ade8bdc002885bf04252b9ed6cee21f7920e54f3c`.
These are the two indexes that `master` published for those inputs.
- The same tree with one line added to a Runner source file: another
digest, `mode=inline`, `reason=absent`.
- On `master` at `bc71c23e0a`, which changed the Runner's inputs, before
its image was published: digest
`88fe2929321b4706c77f11a098572ce990fec4449d505544c62ef3a84c28308a`,
`mode=inline`, `reason=absent`.
- Four deliberate faults in the resolver each make a test fail: no index
check, only `absent` is inline, the identity is built for a lockfile
that is behind, and the slim image is taken by tag.

Never run:

- A lookup with a registry credential. The package is public, so no
caller needs one now.
- Any workflow. No workflow calls the resolver in this pull request.
- `linux/amd64`.
- The full test suite, the full typecheck and `pnpm build`. `pnpm verify
--explain` reports exit 2, "broader verification required", so CI must
supply them.

## Risks

- Low for existing behaviour: nothing calls the resolver, and the vendor
step is the same unless the new variable is set.
- The resolver does not verify the attestation of the image. The
workflow that consumes the image must do that from the digest the
resolver reports.
- The resolver reports both build contexts, `runner-artifacts` and
`runner-artifacts-slim`. A build that reads only one stage gets one
context it does not use.
- A `broken` publication is `inline`, not a failure. The Runner image
workflow fails for that state.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:03:15 -08:00
Nicky LeachandPaperclip 731b52453c fix(environments): verify workspaces on run leases and release settled tasks (#21289)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core owns run leases and delegates remote execution to environment
plugins.
> - A retry must verify its old checkout on the placement that will run
the task.
> - A completed task must still release its provider resources.
> - This change uses the existing lease for workspace verification and
always calls the release hook after settlement.
> - Saved settlement proof lets cleanup retry without repeating
execution work.

## Linked Issues or Issue Description

Refs #21237. That PR records stop proof before lease release. This PR
keeps provider cleanup after settlement and changes workspace
verification sequencing. Both touch task completion; preserve both
changes if the merge order requires conflict resolution.

**What happened?**

Workspace verification ran before a portable task had a lease, so a
provider could not verify the checkout on its selected placement. Core
also skipped the provider release hook after successful task completion.
This could leave resources allocated after the run ended.

**Expected behavior**

Verify the old checkout through the selected run lease. A confirmed
refusal can use the authorized fresh-workspace fallback. A transport
failure must remain retryable. Call the provider release hook after
settlement and retry failed cleanup without changing the run result.

**Steps to reproduce**

1. Retry a task whose prior checkout uses plugin-managed storage and
portable task placement.
2. Verify that the plugin receives the same task lease for verification
and preparation.
3. Complete a task, fail the provider release once, and reconstruct the
cleanup controller.
4. Confirm that cleanup calls only the release hook and retains the
completed run outcome.

## What Changed

- Acquire the logical run lease before provider workspace verification
and reuse it for preparation and execution.
- Preserve the existing session and workspace choices. Recompute
workspace-dependent inputs only after a definite refusal; acquire
credentials and managed runtime inputs once.
- Save the exact refusal and lease before cleanup. Use a
workspace-scoped creation key only for the resulting fresh attempt.
- Replay interrupted verification on the same unlaunched lease. Keep
transport errors retryable and reject replay after execution starts.
- Always call the provider release hook after task settlement. Pass
refreshed settlement proof and route failed releases through the
existing cleanup reaper.
- Add database regressions for call order, restart, retained writers,
refusal, transport errors, and settlement-only cleanup replay.

`plugin-workspaces.ts` grows from 997 to 1,001 lines. The four added
lines extend the existing workspace transaction and lease callbacks.
Keeping the checks there preserves the ownership assertion, provider
result validation, and binding update in one operation. Extracting only
the verification branch would split that transaction without removing a
separate responsibility.

## Verification

- Server TypeScript check passed: `tsc -p server/tsconfig.json
--noEmit`. The complete `server/scripts/typecheck-layers.mjs` check also
passed, including all test-tree layers.
- The first CI typecheck caught a readonly array in a test fixture that
the narrower source check did not include. Commit 8d9ec19e7a corrects
that fixture; all layered checks pass afterward.
- Added a public heartbeat refusal regression in 7c598a4da3. All ten
session-preparation tests pass; independent execution of the refusal
case also passes. It checks successful completion, the saved fresh
workspace, a new conversation, and one credential acquisition.
- Nine database/service suites passed 335 tests: plugin warm tasks,
plugin runner tasks, startup cleanup recovery, credential writeback,
plugin workspaces, session preparation, plugin environments, retry
workspace continuity, and heartbeat workspace preparation.
- After the final restart change, both targeted refusal/restart tests
passed. The primary agent independently reran those cases and the final
server typecheck. After integrating current master, the primary agent's
independent rerun passed all 237 tests across the warm-task,
plugin-workspace, session-preparation, and heartbeat-workspace suites.
- `pnpm verify` returned exit 2 and requested broad CI coverage because
the plugin execution contract documentation changed. No full local
Core-suite pass is claimed.
- Previous-revision CI was green on 8d9ec19e7a: 59 checks passed and 3
were skipped. The first run failed the unchanged legacy-continuation
browser assertion and credential-cleanup server shard. Each passed on
one targeted retry with no code changes. The credential-cleanup
integration suite also passed all 21 tests locally on Node 22. The
browser failure concerned an existing continuation metadata marker; the
server failure cascaded after a cleanup error left a test spy installed.
The successful retries do not establish the cause of the first failures.
- Local database fixtures use simulated provider workers. No live
provider or deployed environment was changed.

- Additional verification for 59c7a1dae8: the layered server typecheck
passed. Independent execution passed all 265 tests across durable task
retries, workspace preparation, and public heartbeat preparation. The
new shutdown regression preserves the fresh Retry choice, waits for
parent cleanup, and prevents the successor from inheriting the parent
lease reference. Same-run recovery retains its receipt. Required CI is
green on 59c7a1dae8: 58 checks passed and 3 were skipped. Standard
review scored 5/5 with no unresolved comments. The separate Superagent
Security Scan is neutral/inconclusive because its workflow failed; a
direct rerun request returned HTTP 404. That scan is not claimed as
passed.

## Risks

- Providers must use the supplied task lease for verification and return
a definite `binding_mismatch` only when the checkout is unconfirmed.
Transport errors retain the retry decision.
- A refused checkout waits for confirmed release before a fresh
allocation. Failed deletion remains owned by the existing cleanup
reaper.
- Task settlement and physical resource release stay separate. Retained
handoff leases do not enter this release path.
- No migration or SDK schema change is required.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) implemented the changes and tests.
The primary Codex agent independently reviewed the connected Core and
plugin paths and ran checks. A separate GPT-6.1 Sol agent reviewed
lifecycle and restart boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue using the bug template labels
- [x] I have not referenced internal or private project details
- [x] My branch name describes the change
- [x] I have added relevant tests and documentation
- [x] I have considered and documented risks
- [x] All required CI gates are green
- [x] Greptile is 5/5 with no unresolved comments
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 21:56:02 -08:00
DottaandClaude Fable 5.2 db3d7223e0 fix(verify): report stale SDK public input pins locally, before CI (#21283)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite proves the plugin SDK public types against
pinned source. The pins are in
`tests/runner-e2e/source-sdk-public-inputs.json`.
> - The pins cover every file in `packages/shared/src` and
`packages/plugins/sdk/src`, and the SDK build inputs (for example
`pnpm-lock.yaml`).
> - A PR that edits a pinned file and does not refresh the registry
fails three files in the "Runner support suite" lane.
> - Between 2026-11-12 10:27Z and 2026-11-13 01:14Z, this failure
occurred in 14 CI runs on 12 PRs.
> - The local check, `pnpm verify`, returned exit 2 ("let CI cover it")
for these changes. It did not mention the pins. Authors pushed, and CI
failed.
> - This pull request makes `pnpm verify` find stale pins before the
push. It also makes the opaque fixture error name the file and the fix.
> - The benefit is that authors find and fix the problem in seconds,
locally, instead of after a 6-minute CI lane.

## Linked Issues or Issue Description

**What happened?**
Three files in the "Runner support suite" lane failed together on many
unrelated PRs: `source-sdk-evidence.test.ts`,
`source-sdk-public-inputs.test.mjs` and
`source-sdk-public-type.test.ts`. Two of the three failed with the
opaque error `Reviewed SDK reference source changed`.

**Root cause**
The pins on master are not stale. The tests pass on master (58/58). Each
failing PR edited a pinned input and did not refresh the registry. I
compared the PR heads with their merge base:

- All 14 failing heads (12 PRs) changed a pinned input. None of them
changed `source-sdk-public-inputs.json`.
- All 19 heads in the same window that changed a pinned input and passed
this lane also refreshed the registry. This includes later heads of
three PRs that failed first (#21119, #21121, #21156).
- Two of the 12 PRs are Dependabot PRs that change only `pnpm-lock.yaml`
(#19593, #19594). The lockfile refresh bot already resyncs the pins for
lock changes on master.

**Steps to reproduce**
1. Start from `origin/master`.
2. Add one line to `packages/shared/src/types/heartbeat.ts` and commit
it.
3. Run `node node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts source-sdk-evidence.test.ts
source-sdk-public-inputs.test.mjs source-sdk-public-type.test.ts`.
4. The result is the same as CI: 3 files fail, with the same 4 failures.
5. Run `pnpm verify --explain`. It returns exit 2 and does not mention
the pins.

**Expected behavior**
`pnpm verify` reports the stale pins and the refresh command before the
author pushes.

## What Changed

- `tests/runner-e2e/source-sdk-public-inputs.mjs`: add
`staleSdkPublicInputsInWorkingTree(root)`. It applies the existing drift
guard to the working tree (tracked and untracked files that git does not
ignore). Add `isSdkPublicInput(file)` and the shared failure message
`staleSdkPublicInputsMessage(stale)`. The committed-tree guard keeps its
behavior and now uses a shared comparison.
- `scripts/local-verify.mjs`: when the diff touches a pinned input, the
registry or `report-lock.json`, `pnpm verify` checks the pins first. If
they are stale, it exits 1 and prints the files and the refresh command.
The check reads files only. It does not write the registry.
- `tests/runner-e2e/source-sdk-reviewed-fixture.ts`: the error `Reviewed
SDK reference source changed` now names the file and gives the refresh
command.
- `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: new test for the
working-tree guard (uncommitted edit, untracked file, ignored file). The
CI drift test uses the shared message.
- `doc/DEVELOPING.md`: document the new check under "Incremental local
verification".

No assertion is weakened. No pin is changed. No file under `.github/` is
changed.

## Verification

- Master, without this change: the three files pass (58/58).
- Red: on master plus a one-line edit to
`packages/shared/src/types/heartbeat.ts`, the three files fail with the
CI signature. `pnpm verify --explain` returns exit 2 and does not
mention the pins.
- With this change on the same red commit: `pnpm verify` exits 1 in
about 15 s and prints `tests/runner-e2e/source-sdk-public-inputs.json is
stale for 1 pinned input(s): packages/shared/src/types/heartbeat.ts
(changed). Commit your change, run ... --write, and commit the updated
registry.`
- Green: I followed the printed command (`node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`, then
commit). The three files then pass (59/59, with the new test).
- `node --test scripts/__tests__/local-verify.test.mjs`: 14/14 pass.
- `pnpm verify` on this branch returns exit 2 because
`scripts/local-verify.mjs` is a global CI input. Required CI supplies
the broad coverage.

## Risks

- Low risk. The new check only reads files. It runs only when the diff
touches a pinned input.
- `pnpm verify` now fails (exit 1) for a change that also fails in CI.
Before, it returned exit 2.
- The check needs a working `git ls-files` in the repository. `pnpm
verify` already needs git.
- Dependabot lock PRs still fail this lane until the lockfile bot
refreshes the pins. This PR does not change that path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent with tool use (shell, git,
GitHub CLI). Commits are attributed to Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:03:39 -06:00
DottaandClaude Opus 5.5 8a0f95849d perf(native-runtime): start the first turn without re-copying launch files, and never fail a run on a slow copy (#21287)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode in a remote sandbox must have a durable,
verified copy of the provider launch files before their first turn (the
rule from #19622)
> - Today each new run copies these files out of the sandbox, uploads
them and verifies them again, even when the company already has a ready
artifact with the same files
> - The whole copy runs under one five-minute clock. When eight runs
start together on small sandboxes, the copies compete, and slow copies
fail the run before its first turn
> - This makes cold starts slow (p50 273 s from run start to turn
submitted) and makes half of the first runs fail in a burst
> - This pull request reuses a ready artifact with the same digests,
shares one in-flight copy per company and manifest, and bounds each
network step instead of the whole copy
> - The benefit is that first turns start in about one minute on a cold
burst and in about 20 s when the artifact is already ready, and a slow
copy no longer fails the run

## Linked Issues or Issue Description

Refs #19622

**What happened?**
When 8 OpenCode runs started together on 1 vCPU Daytona sandboxes, 4 of
8 first runs failed before their first turn. Each run copied, uploaded
and verified the same provider launch files. The single 300 s clock over
the whole copy expired. Run start to turn submitted was p50 273 s and
p95 314 s.

**Expected behavior**
A run starts its first turn without a new copy when a ready, verified
artifact with the same files exists. A slow copy does not fail the run.

**Steps to reproduce**
1. Use a company with remote OpenCode agents on Daytona sandboxes with
durable storage.
2. Start 8 runs at the same time on new sandboxes.
3. See the time before the first turn and the failed runs.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- `retainReadyNativeProviderArtifacts` (`native-provider-artifacts.ts`):
reuse a ready, verified company artifact with the same paths and SHA-256
digests through a lookup only. No file is copied, uploaded or verified
again. The lookup (`readyNativeProviderArtifactCandidates`) matches
target, paths and digests in Postgres with jsonb containment on the
company's ready rows (existing partial index), newest first, limit 4. It
does not load or hash other retained versions. No migration.
- `shareNativeProviderArtifactCapture`
(`native-provider-artifact-capture.ts`): one in-flight capture per
company and manifest digest per server. Other runs wait for it, then
reuse its result.
- The single 300 s clock over the whole capture is replaced by a bound
for each network step (upload, verifying download): max(120 s, size at
256 KiB/s). A timeout has the typed cause
`native_provider_artifacts_transfer_timeout`. The capture makes up to 3
attempts from the local copy.
- Stop now interrupts the capture and the wait for a shared capture
(`ActiveNativeSession.admissionAbort`).
- `doc/opencode-cold-recovery.md` describes the new behavior.
- Tests: new `native-provider-artifact-gate.test.ts` (7 tests), 5 new
tests in `native-harness-backup-storage.integration.test.ts`, and new
executor variants `opencode-new-durable-slow-capture` and
`opencode-new-durable-ready` in `native-session-executor.test.ts`.

## Verification

Local (this branch, on master 9b613562b7):

- `npx vitest run
src/services/native-runtime/native-provider-artifact-gate.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
3 files, 1511 passed, 7 skipped.
- `npx tsc --noEmit -p server`: pass.
- Lookup scale: the integration test "loads only the ready artifact
whose launch files match, however many other versions the company keeps"
adds 200 ready non-matching versions. It returns only the matching
artifact. Without the SQL filter it fails (4 rows returned).
- Red/green: with master's `native-session-executor.ts` and the new
tests, `opencode-new-durable-slow-capture` fails with `TimeoutError: The
operation was aborted due to timeout`, and `opencode-new-durable-ready`
fails because the ready artifact is not looked up. With this change,
both pass.

Daytona (same commit content, cherry-picked onto the baseline commit
84a1ee3e11 with #21262; image `sha256:9461cc75…`; 1 vCPU sandboxes;
OpenCode):

| Wave | Shape | First runs OK | Run start → turn submitted | Other |
|---|---|---|---|---|
| Baseline `w-20261113T011027-s1702` | 8 runs started together | 4/8 (4
failed) | p50 273 s / p95 314 s | "only a getter" TypeError ×8,
`bootstrap_not_proven_incomplete` ×8 |
| Fix `w-20261113T031045-s1702` | same seed and shape | 8/8 | p50 66.0 s
/ p95 85.7 s | 8/8 tasks hands-off, 0 Retry banners; first agent event
p50 72.9 s / p95 90.0 s; exactly 1 upload, 7 runs waited on it (~65 s
shared capture); all failure codes 0 |
| Fix `w-20261113T034033-s1702` | 16 tasks, cap 8 | 16/16 | p50 19.1 s /
p95 24.3 s | second batch (artifact already ready, cold sandboxes) gate
time 24, 25, 26, 27, 31, 31, 45, 119 ms; first agent event p50 25.9 s |

## Risks

- The rule from #19622 still holds: a durable, verified artifact exists
before the first turn. Reuse only accepts a ready, verified artifact of
the same company with the same paths and digests.
- The first run per company and manifest per server still pays the full
transfer.
- Two servers can still upload duplicate artifacts for the same company
and manifest. Sharing is per server.
- No test drives Stop through the executor during a capture. Signal
handling is unit-tested only.
- A capture that fails all 3 attempts still ends blocked with a working
Retry. A follow-up PR (pre-turn failures continue automatically) removes
that.
- The documented five-minute deadline per capture is replaced by bounds
per network step. A very large artifact on a slow link can take longer
than five minutes in total, but each step still has a bound.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Default context
window; extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:48:55 -06:00
DottaandClaude Fable 5.2 c4f639ed1b fix(ci): re-run cancelled runs that lost a runner, and let runner-loss evidence win (#21291)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - PR CI runs about 50 jobs per run on AWS spot instances. AWS can
reclaim a machine mid-job, and the job dies.
> - `ci-runner-loss-rerun.yml` re-runs such a run once, automatically,
when every failed job lost its runner.
> - In a 17-hour sample, 42 runs lost a runner and were still the PR's
current head. The automation rescued only 14.
> - Two gaps caused most of the misses. A spot reclaim on RunsOn v3.2.3
usually ends the job, and the run, as `cancelled`; the workflow and
script only look at `failure`. Also, the Runner support suite always
carries `gateway_indexing_incomplete` error annotations from a passing
test, so a lost runner there reads as a real failure.
> - On-demand fallback for the fleet was turned off on 2026-11-13, so
reclaims will happen more often. About 20% of runs are expected to lose
at least one job.
> - This pull request makes the re-run act on cancelled runs and jobs,
but only with positive runner-loss evidence, and lets that evidence win
over other error annotations.
> - The benefit is fewer manual "Re-run failed jobs" clicks: an
estimated 34 of 42 instead of 14 of 42 in the sample.

## Linked Issues or Issue Description

No public issue exists.

**What happened?**
Run
[43285064435](https://github.com/paperclipai/paperclip/actions/runs/43285064435)
lost the runner of `ci / General tests (server (8/12))`. The job
concluded `cancelled` (its log has `##[error]The runner has received a
shutdown signal`). The run concluded `cancelled`, so the re-run
workflow's `if:` skipped it.

Run
[43325855069](https://github.com/paperclipai/paperclip/actions/runs/43325855069)
lost the runners of `ci / Runner support suite` and `ci / General tests
(server (2/12))`. Both logs have the shutdown signal. The re-run bot
([decision
log](https://github.com/paperclipai/paperclip/actions/runs/43327873358))
said "1 of 1 failed jobs did not lose their runner", with evidence
`gateway_indexing_incomplete`. It ignored the cancelled job. It read the
annotation as a real failure.

**Expected behavior**
Both runs get one automatic re-run of their failed jobs.

**Steps to reproduce**
The new tests replay trimmed snapshots of both runs
(`.github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json`
and `run-43325855069.json`) through the script.

In a 17-hour sample on 2026-11-12, 42 runs lost a runner and were still
the PR's current head. Only 14 were re-run automatically. 16 stayed red
because the run concluded `cancelled`. 4 stayed red because an error
annotation in the lost job was read as a test failure.

**Additional context**
The `gateway_indexing_incomplete` annotation does not come from the
shutdown. `scheduled-gateway-indexing.test.ts` prints `::error` lines
while it passes, 84 seconds before the shutdown signal in this run.
Attempt 2 of the same run passed with the same two failure annotations.
So, before this change, a lost runner in the Runner support suite could
never be re-run.

## What Changed

- `ci-runner-loss-rerun.yml`: the job also runs when the PR run
concluded `cancelled`. The trigger type, `permissions`, the repository,
path, event and attempt-1 conditions do not change.
- `runner-loss-rerun.mjs`, `eligibleRun`: accepts `completed/cancelled`
as well as `completed/failure`.
- `runner-loss-rerun.mjs`, `decide()`: classifies jobs that concluded
`cancelled` as well as `failure`. It still skips the `ci / verify` and
`ci / e2e` aggregates, and a failed or cancelled routing gate still
blocks the re-run.
- `decide()` now runs `staleReason` (open PR, current head, no newer
run) before it reads any annotation or log, and again just before the
re-run, as on master. A run cancelled by a new push has about 40
cancelled jobs; it now costs two API listings and no log reads. The
final check closes the window where a push lands during the log reads: a
re-run of the old run would join the PR's `cancel-in-progress`
concurrency group (`pr-<number>` in `pr-trusted.yml`) and cancel the new
commit's CI.
- Following CONTRIBUTING.md ("Do not submit a PR that includes code
comments"), this PR adds no new code comments. The reasons for each rule
are in this description and in `doc/DEVELOPING.md`.
- `classifyJob`: runner-loss evidence wins. A lost-communication
annotation means lost. A cancellation annotation plus a shutdown signal
in the log means lost, even if the job also has other error annotations.
A cancelled job without this evidence (for example, a person cancelled
the run) counts as a real failure and blocks the re-run.
- Fixtures: two new trimmed snapshots from the example runs (jobs with
`id`, `name`, `status`, `conclusion`; failure annotations; `##[error]`
log lines only). The two older fixtures get the real annotations and log
error lines of their cancelled jobs. Those jobs were spot-reclaimed too
(each log has the shutdown signal), but the old code ignored them.
- `doc/DEVELOPING.md`: the runner-loss paragraph describes the new
rules.

## Verification

- Red first. Against the unchanged script, the new tests fail for the
reasons above: 43285064435 gives `run is completed/cancelled, not
completed/failure`; 43325855069 gives `1 of 1 failed jobs did not lose
their runner` with evidence `gateway_indexing_incomplete`, the same
decision the bot logged.
- Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`:
25 of 25 pass.
- Push during log reads: the PR head moves after the first log read.
With the final freshness check removed, this test fails (the script
re-runs); with it, there is no re-run and the reason is `no open pull
request still has this run's head commit`.
- Guard cases, all with no re-run and no POST:
- Superseded run: 52 jobs cancelled, the PR head moved. The test asserts
zero annotation and log reads.
- Person cancelled the run: cancelled jobs with only "The operation was
canceled." in the annotation and log, and some never-started jobs with
no annotation.
- Mixed: one cancelled lost job and one real test failure (`1 of 2
failed jobs did not lose their runner`).
- Once only: a cancelled run on attempt 2 is refused after one API read.
The workflow condition test still requires `run_attempt == 1`.
  - Cancelled routing gate: left for a person.
- Policy job command: `node --test '.github/scripts/tests/*.test.mjs'`:
806 of 806 pass.
- `pnpm verify`: exit 2 ("Global CI input changed:
.github/scripts/runner-loss-rerun.mjs"). It selects no local Vitest
files and asks for required CI coverage. I did not run `pnpm verify
--full`; required CI on this PR supplies that coverage.
- This workflow runs on `workflow_run` from the default branch. This
PR's own CI cannot exercise the new behavior. The fixture tests are the
evidence. I did not dry-run the script against live runs.

## Risks

- A test that really fails, and then a spot reclaim of the same job a
minute later, now gets one wasted re-run. Attempt 2 then fails for real.
It cannot loop: the workflow `if:` and `eligibleRun` both accept attempt
1 only.
- A run a person cancels on the current head now starts the re-run job.
The job reads the annotations, and the log of each job with a
cancellation annotation, and then does not re-run. This costs some API
calls on a GitHub-hosted runner.
- A concurrency cancel by a label event that does not request CI (the
newer run is `skipped`) passes the head check. Its cancelled jobs have
no runner-loss evidence, so it is not re-run.
- Trust: no change to `permissions`, the trigger, the accepted
repository, event, workflow path, attempt, or the routing-gate rule. A
cancelled run reuses its attempt-1 gate output, the same as a failed run
does today.
- Estimated effect, from a 17-hour sample: 14 of 42 rescued before; 16
cancelled runs and 4 runs with error-annotation noise become eligible
(about 34 of 42). The other 8 stay skipped on purpose (a real test
failure in the same run, a person re-ran first, or the PR closed).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent. Model ID claude-opus-5-5, with
tool use (shell, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:31:50 -06:00
Nicky LeachandClaude Fable 5.2 25e9a9c2e9 feat(ci): publish both variants of the Runner image from master (#21230)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Parts 2 and 3 of this stack build the Runner alone as an image with
two variants, and check it on pull requests. Nothing publishes it
> - Downstream sandbox and VM images can copy the Runner only from a
registry, and they must be able to verify where it came from
> - A published image must be exactly the bytes that were checked, and
its name must never point at something else later
> - This pull request publishes both variants from `master` as
`ghcr.io/paperclipai/paperclip-runner`, one time for each distinct set
of inputs, and attests them
> - Nothing consumes the image in this pull request. The application
image build does not change
> - The benefit is one verified, attested Runner that later changes can
copy by digest

## Linked Issues or Issue Description

Refs #21204 (part 2 of this stack) and #21096 (the merge-queue image
build). No issue exists for this change.

**Problem or motivation**

The Runner image exists only inside a pull-request check. A consumer
cannot pull it. There is no rule for which commit publishes an image, no
protection for its tags, and no way to verify a copied Runner.

**Proposed solution**

Publish from a `master` push in this repository, and from nothing else.
Build only when no image has the digest of the commit's inputs. Let the
build legs hold no write access. Push the bytes that the legs checked by
digest, attest each multi-platform index, and write the tags last. Write
the full tag after the slim tag, so that the full tag means "both
exist".

**Alternatives considered**

- Publish from `docker.yml`. That workflow holds one concurrency slot
for each ref, and the Runner image needs its own signer identity, so
that its attestation cannot stand in for the application image's.
- Resolve the lockfile in the workflow before a `master` build, as
`docker.yml` does. Then the published Runner depends on the state of the
npm registry at run time, and the digest cannot be computed again from a
commit.
- Hand each variant to the publish job as its own OCI archive. The two
archives hold the two shared layers two times: 1.27 GB for each
architecture in place of 0.85 GB.
- Write the full tag first. Then a consumer can see a full image whose
slim image does not exist yet.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 4 of 4. Each part is on top of the one before it:

1. #21228 (merged): an optional field in the provider pack manifest
2. #21204: build and check the Runner image, with no publication
3. #21229: layers by change rate, and the slim variant
4. #21230: publish both variants from `master` (this pull request)

## What Changed

- `.github/workflows/runner-image.yml`: adds the trigger `push` to
`master` and the job `publish`.
- `identity` checks the committed lockfile against the Runner's
manifests, computes the digest in the build context, and asks the
registry whether these inputs are published.
- `build` is the job of part 3. It also records what it built, on every
event. On a canonical `master` push it uploads that. It cannot write to
the registry on any event. The only token in the job can read packages,
and only the lookup step receives it.
- `publish` runs only for a `master` push in this repository. It builds
nothing and runs no dependency code.
- Lockfile rule. A `master` push builds from the committed
`pnpm-lock.yaml` and resolves nothing, so the digest is a function of
the commit. When the committed lockfile is behind the manifests that the
Runner build reads, the run writes a notice, publishes nothing, and
stays green. The commit that merges the lockfile refresh publishes the
image. Pull requests and manual runs resolve the lockfile first, always
build, and never publish.
- Hand-off. Each leg joins its two OCI archives into one OCI layout that
names the images `slim` and `full`. The layout stores the two shared
layers one time. `record` then reads the layout back, each image by its
name, and checks every blob against its digest. The leg's record binds
the layout to both image digests, the architecture, the inputs digest,
and the commit. Those digests are the digests of the archives that part
3 unpacks and checks, so the tree that is checked and the image that is
pushed are the same bytes. The publish job accepts only records for its
own inputs and commit, and only a layout that still has the recorded
bytes.
- `record` runs on pull requests too. Only the upload is for a `master`
push. A pull request therefore runs the hand-off on real build output.
- Publication order. The publish job pushes the four images by digest
(`skopeo copy --preserve-digests`). It makes a candidate index for the
slim variant, then one for the full variant that names the slim index in
the annotation `io.paperclip.runner.slim-digest`. It verifies that each
index names exactly the recorded images, attests both, and then writes
`inputs-<digest>-slim` and, last, `inputs-<digest>`.
- The full tag is the completion marker. Every "is it published" lookup
reads the full tag. `publication-state` reports one of:

  | State | Registry | What a run does |
  | --- | --- | --- |
  | `absent` | Neither tag | Builds and publishes |
| `incomplete` | The slim tag alone | Builds and publishes. It replaces
the slim tag with its own image, then writes the full tag |
| `published` | The full tag, and the slim tag at the digest that the
full index names | Nothing |
| broken | The full tag, and a slim tag that is missing or points at
another image | Fails, with the steps for an operator |
| `unknown` | No clear answer | Builds. The publish job fails before it
writes a final tag |

- Recovery rule. A run that stops between the two tags leaves the state
`incomplete`. It does not block those inputs. The next run never keeps
the earlier slim image: it writes its own slim tag immediately before
its full tag, verifies both against its own index digests, and then
reads the pair back as every later run will see it. The broken state
cannot be repaired by a run, because no run moves a full tag. An
operator must delete the two package versions and re-run the workflow
run of a `master` push with those inputs.
- Pairing rule for consumers, in the contract: resolve the full tag
first, take the slim digest from the annotation of the full index, and
pull the slim image by that digest.
- `verify-index` accepts `linux/arm64` with the variant `v8`, which some
BuildKit versions record, and no other variant.
- `scripts/runner-image-publish.mjs` (533 lines): the commands
`lock-state`, `record`, `verify-legs`, `publication-state` and
`verify-index`. It reads archives with the reader of
`scripts/runner-image-oci.mjs` (607 lines here), which can now read one
named image of a layout. `lock-state` refuses a checkout, because pnpm
can rewrite the lockfile of the tree that it checks. The workflow gives
it an export that no build reads.
- The workflow test pins the build arguments of the image build. Build
arguments are not part of the image identity.
- `doc/runner-image-contract.md` (197 lines): the section "Publication":
the lockfile rule, the order, the pairing rule, the recovery rule, the
create-only tags, the internal `candidate-` tags, the command that
verifies provenance, and the step to make the package public.
`doc/DOCKER.md` says where the image is published.

## Verification

Tests, and the build jobs of the workflow on GitHub. Nothing in this
pull request ran against a registry, and no publish job has run.

- Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and
zstd): `runner-image-publish.test.ts` (75),
`runner-image-workflow.test.ts` (12), the three other Runner image test
files, the provider pack archive tests and the existing Dockerfile
tests: 372 tests pass in 10 files. The same files pass on macOS.
- The tests of the leg record failed on Linux in the first runs of this
pull request: their fixtures named the archive members `./index.json`,
which bsdtar finds and GNU tar does not. The fixtures now name the
members as buildx does, and the publish script uses the one reader of
part 3.
- Leg records: both variants are accepted for this run's inputs and
commit; other inputs, another commit, a changed layout, a record without
a variant, an archive that lacks a blob, a full image that does not
start with the slim image's layers, and a build that is not a
publication build are refused.
- Publication state: all five states, and the half-finished publication
end to end. A slim tag alone lets the next run publish. A full image
that is left with another run's slim tag is reported as broken.
- Index: exactly the recorded images of one variant; the full index must
name its slim index, and the slim index must name none; `v8` is accepted
on arm64 only.
- Registry lookups: a missing tag only when the registry says so; every
refused, failed, or unreachable request is `unknown`.
- Workflow: job permissions; the publish job and the upload are behind
the canonical `master` push; the build steps receive no credential; the
full tag is written last, after a strict lookup and both attestations;
one export and a time limit for each build command; the two build
arguments; actions pinned by commit; no expression inside a shell
script.
- `record` on the two OCI archives of a real `linux/arm64` build from
part 3: the joined layout is 850,149,376 bytes, and both images read
back by name with every blob under its digest. The two archives together
are 1,273,646,592 bytes. The image digests in the record are the digests
that `unpack` reports for the trees that were checked.
- `tests/runner-e2e` workflow tests: 39 pass. `node --test
.github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image
contract scripts: 33 pass. `docker buildx build --check` reports
nothing. The image identity is the same as in part 3
(`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`),
because this pull request changes no input of the image.

GitHub, at this head:

- The `Runner image` workflow passes with these changes: `Build and
check (amd64)` in 18m30s and `Build and check (arm64)` in 7m48s. Each
leg builds both variants, checks them, and writes the leg record.
`Publish and attest` is skipped, as it must be on a pull request.

Never run:

- The publish job. It runs only for a `master` push.
- The publish path: the registry lookups with a workflow token, `skopeo
copy` from `oci-archive:<layout>:<name>` by digest, `docker buildx
imagetools create` with index annotations, and the tag writes.
- The attestations, and the `gh attestation verify` command in the
document.
- `actionlint`. It is not installed on the local host.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- **Manual step.** GitHub creates the `paperclip-runner` package private
on the first publication. A maintainer must make it public one time.
Until then, consumers cannot pull the image without credentials.
- The first `master` runs of the workflow are the first test of the
publish path. A failure before the tags publishes nothing. A failure
between the two tags leaves the `incomplete` state, and the next run
with those inputs repairs it.
- A slim tag can move while its publication is incomplete. A consumer
must resolve the full tag first and take the slim digest from the
annotation of the full index. It must never resolve
`inputs-<digest>-slim` by tag on its own. The contract says so.
- `candidate-` tags are written before the attestations and can name an
index that is not attested. They are internal.
- The broken state needs an operator, and it fails every `master` run
with those inputs until then. It can arise only when someone changes or
deletes a tag outside the workflow.
- **Known residual.** Two `master` runs that both start before a new
digest is published can each build the image. Each build leg and the
publish job look again before they work, and the publish job runs one at
a time for each digest. The cost is runner time, not correctness.
- A commit whose lockfile is behind the Runner's manifests has no
published Runner image until the lockfile refresh merges. A lockfile
that the refresh cannot fix leaves every `master` run green with only a
notice, for as long as it lasts.
- The tags are create-only because the workflow never moves a tag of a
complete publication. The registry does not enforce that. Consumers must
pin the digest and verify the attestation.
- `payload.runnerSourceRevision` in a pack from this image is the commit
that published the image. Code that reads it as the application's commit
must change before it consumes the image.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 20:27:36 -08:00
Nicky LeachandClaude Opus 5.5 717aae060a fix(environments): let same-run recovery create a replacement once its sandbox is proven stopped (#21223)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute on environment leases, and `createOwnedSandbox`
records each sandbox creation so that a lost reply never creates a
second sandbox
> - The record key is company, environment, run and request key. The
plugin request key is the environment fingerprint, so it is the same for
every attempt of one run
> - A same-run recovery can need a new environment after the first lease
is closed with successful cleanup and a termination receipt
> - The old record is still `attached` to the closed lease, so the
recovery fails with "Sandbox creation has already transferred ownership"
before the provider is called
> - This pull request lets the creation fence accept the same stop proof
that recovery already requires. A creation record whose sandbox is
proven stopped becomes resolved, and a resolved record no longer holds
its key
> - The benefit is that same-run recovery gets one replacement sandbox,
while unconfirmed cleanup still blocks a second sandbox

## Linked Issues or Issue Description

Fixes #21214

Refs #20755. That pull request fixes the same failure with a different
rule. It retries under a new key generation (`<key>#2`, `#3`, ...) when
the recorded lease has ended, and it accepts any ended status, including
`failed`, without a termination receipt. A lease whose cleanup is not
confirmed can then produce a second sandbox while the first one can
still run. This pull request keeps the original key and requires the
same stop proof that Core uses elsewhere
(`hasRemoteTerminationReceipt`). It does not add a key generation.

## What Changed

- `server/src/services/sandbox-creation-operations.ts`: when
`createOwnedSandbox` finds the key held by an `attached` operation whose
lease is not active, it checks the same stop proof that same-run
recovery requires before it acquires:
`hasRemoteTerminationReceipt(resolveRemoteTerminationLease(lease))`, for
the sandbox that this operation created (`creationProviderLeaseId`, or
the lease's `providerLeaseId`).
- With that proof, it runs the same admission transaction one more time.
That transaction first marks the old operation `resolved` (`WHERE id AND
state = 'attached' AND lease_id`), and then inserts the new operation.
  - Without that proof, it throws exactly as before.
- The check covers every source of a stop receipt: `releaseLease`, the
Runner `complete` operation (which writes receipts with direct SQL for
reusable leases), stop-and-retain settlement, and the receipt that
`resolveRemoteTerminationLease` derives for an expired reusable lease.
- A task replacement can leave an earlier operation attached to a lease
that now holds another sandbox. That operation is not proven and keeps
its key.
- Concurrent recoveries serialize on the row update. The second one
updates nothing, finds the first one's unresolved operation through the
partial index, and replays or refuses as before. There is one provider
call.
- Migration `0475_sandbox_creation_owner_unresolved.sql` recreates
`sandbox_creation_operations_owner_idx` under the same name as a partial
unique index (`state <> 'resolved'`). The drop and the create are in one
migration transaction. The Drizzle schema matches.
- The conflict lookup in `createOwnedSandbox` ignores resolved rows. An
active attached lease is still replayed. Pending and
`needs_reconciliation` rows still refuse. `releaseLease` does not
change.
- `doc/DATABASE.md` and `doc/idle-environment-cleanup.md` describe the
operation lifecycle.

## Verification

- `pnpm --filter @paperclipai/db generate`
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/server typecheck`
- `vitest run src/migration-snapshot-drift.test.ts` (db)
- `vitest run src/__tests__/sandbox-creation-operations.test.ts`: 48/48.
With the service change reverted, the 5 replacement tests fail.
- The 9 other server test files that read `sandbox_creation_operations`,
plus `environment-runtime.test.ts`: 564/564
- `pnpm check:migrations` passes

New tests in `sandbox-creation-operations.test.ts`:

- A stop receipt from `releaseLease` (sandbox driver, and the plugin
driver through `acquireRunLease` in the #21214 sequence), from the
`complete` operation's direct write on a retained reusable lease, and
derived only from the destroyed successor of an expired reusable lease
each let the next creation resolve the old operation and create exactly
one replacement. A retry replays that replacement. These tests fail
before the fix.
- Failed cleanup, success without a receipt, a receipt for another lease
or another provider task, a retained lease, and pending cleanup each
still throw "transferred ownership" with no provider call.
- An earlier task's operation that a task replacement left attached
keeps its key.
- Two concurrent recoveries make one provider call. This test fails
before the fix.
- Reapplying migration 0418 keeps the partial index.

I did not run `pnpm verify` or the full suite locally. CI runs them.

## Risks

- Migration: the drop and the rebuild of the index take an exclusive
lock on `sandbox_creation_operations`. The table has one small row for
each sandbox creation, so the lock is short.
- Rolling deploy: the migration is marked `online_compatible`. The
previous release uses an untargeted `ON CONFLICT DO NOTHING`, which also
matches a partial unique index. Its conflict lookup has no state filter,
so it can find a resolved row and throw. That fails closed and never
creates a second sandbox.
- A legacy runner-task operation without `creationProviderLeaseId`
matches any lease it is attached to. Runner-task operations record this
field before they attach, so this is only a theoretical gap.
- An operation stays `attached` until the next creation with the same
key checks it. This is the same as before when no recovery comes.
- Stop-and-retain settlement is not tested directly. The shared receipt
check covers it.
- The fence is unchanged for every case without a stop receipt.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents) and extended thinking. One sub-agent
wrote the change. A second session reviewed it before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:24:34 -08:00
DottaandClaude Opus 5.5 4dc5492c2d fix(recovery): retry a failed Retry run's sandbox resume failure when the sweep reaches it first (#21124)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on its next run
> - When the resume fails before provider work
(`reusable_sandbox_resume_failed`), the run gets a bounded automatic
retry
> - On a hosted staging server, 8 of 14 such failures in 10 hours did
not get that retry. Each task ended `blocked` and needed a person to
press Retry
> - In all 8, the stranded-task sweep blocked the task between the
failed write and the run's own retry scheduling. The block cleared the
claim that the retry needed
> - This pull request makes the sweep give such a run to the same
bounded retry scheduler, not escalate it
> - The benefit is that a transient sandbox resume failure no longer
costs the task, also when settlement is slow or the server stops during
an upgrade

## Linked Issues or Issue Description

Refs #21108, #21113, #21078

**What happened?**
A person pressed Retry on a task. The run tried to resume the task's
saved Daytona sandbox before provider work. The resume failed
(`cause=transport_timeout`, or `provider_unavailable status=503`), and
the run was written `failed` with `reusable_sandbox_resume_failed`. Then
the stranded-task sweep moved the task to `blocked` and cleared its
claim. This happened 1 to 43 seconds after the failed write. The run's
executor scheduled its bounded retry 2 to 81 seconds after the failed
write, and the gate refused it ("The automatic retry could not
revalidate the original user continuation."). Read-only audit of 14
failures (all legacy runtime, `executionRecovery: bootstrap,
providerWorkStarted: false`): 6 got their retry, and the 5 most recent
ones recovered. 8 were refused this way and wait for a person. In one
example the run was written failed at 14:23:31, the task was blocked at
14:24:14, and the retry was refused at 14:24:52. The server was being
replaced at that time.

**Expected behavior**
A transient preparation failure before provider work continues with its
bounded retry. The sweep must not take the claim away from that retry.
When the retry budget is spent, or the continuation is refused, the task
escalates once, as before.

**Steps to reproduce**
1. Press Retry on a task whose reusable sandbox resume fails before
provider work.
2. Let the run's executor be slow to settle the failure (more than the
remaining controller lease, at most 60 seconds), or stop its server
after the failed write.
3. Let the stranded-task sweep run on another server. The task becomes
`blocked`, and no retry is scheduled.

**Root cause**
`classifyContinuationFailure` lists `reusable_sandbox_resume_failed` and
the other transient preparation codes in
`NON_RETRYABLE_CONTINUATION_ERROR_CODES`. The sweep assumes that the
executor already scheduled the dedicated bounded retry, so it escalates
at once and never asks the scheduler. #21108 added a skip while the
executor still settles the run. That skip uses a live local execution or
an unexpired controller lease. The lease stops renewing when the run is
written terminal (renewal requires `status = 'running'`). So the skip
covers at most about 60 seconds after the last renewal, and it covers
nothing when the executor's server stops during settlement.

**Paperclip version or commit**
Staging at `a65644d1e1`. Current `master` has the same code path.

**Deployment mode**
Hosted (cloud), two servers alive during each upgrade.

**Agent adapter(s) involved**
`codex_local` on a Daytona sandbox environment. The change is
adapter-independent.

## What Changed

- `server/src/services/heartbeat/retries.ts`: add
`failedPreparationOwnsBoundedRetry(run)`. It is the persisted form of
the run-completion decision: a failed legacy run with no native session,
bootstrap evidence that provider work did not start, and a transient
preparation code. A restore failure counts only when its owner marked it
retryable. Codes that an accepted interaction retries itself are
excluded.
- `server/src/services/recovery/service.ts`: when the latest run is such
a run, the sweep does not escalate it as non-retryable. It goes to
`scheduleRecoveryRetry`, which is the run's own bounded scheduler. The
scheduler reuses a successor that already exists, so the executor and
the sweep create one successor. The task escalates when the scheduler
reports exhaustion or a refused continuation. When the scheduler
declines the retry for another reason (for example a permanent refusal),
the task escalates once as a non-retryable failure, as before. The
#21108 settling skip still runs first.
- `server/src/services/heartbeat.ts`: pass the predicate to the recovery
service.
- `doc/execution-semantics.md`: document the rule.

## Verification

- New tests in `server/src/services/task-retry.test.ts`, in the staging
shape: a Retry-admitted run failed with
`reusable_sandbox_resume_failed`, bootstrap evidence, and an expired
controller lease from another server.
- "retries a failed Retry run's transient preparation failure when the
sweep reaches it first" (`slow_executor`, `executor_gone`). Red before
the fix: the task became `blocked`, as on staging. Green after the fix:
the task stays `in_progress` with one `scheduled_retry` successor
(`transient_failure`, attempt 1), and dispatch accepts it. For
`slow_executor`, the executor's own scheduling then reuses the same
successor, and there is still one successor.
- "still escalates a failed Retry run's transient preparation failure
once its bounded retries are spent": the task becomes `blocked` with no
successor.
- "escalates a failed Retry run's transient preparation failure once
when the scheduler declines it permanently" (review finding): red before
the follow-up commit (the task stayed `in_progress` with no run), green
after (the task is `blocked` once, and a second sweep adds no notice).
- `pnpm exec vitest run src/services/task-retry.test.ts`: 114/114 pass.
- `pnpm exec vitest run src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/heartbeat-process-recovery.test.ts` (with task-retry): 631
of 633 passed. The 2 failures were 15-second timeouts under parallel
load in native Board-wait tests that this change does not touch. They
passed when run alone.
- `tsc -p server/tsconfig.json --noEmit` passes.
- Not run: a live staging reproduction. The staging evidence is
read-only (run events, task activity, run records).

## Risks

- The sweep can now schedule the bounded retry before the executor does.
The scheduler locks the task and the run and reuses an existing
successor, so there is never a second successor. All continuation gate
checks still apply.
- The sweep acts only after the #21108 settling skip lapses, so a
healthy run is not affected. No wait is added before a normal turn
start.
- No schema change. Rows from older releases follow the same rule
because the predicate reads only persisted fields.
- Out of scope: the provider-side causes (Daytona 503 at 14:21Z, resume
calls that time out). #21113 handles the in-run wait for idle transport
timeouts.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, read-only API reads, Vitest on embedded PostgreSQL) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:20:55 -06:00
Nicky LeachandClaude Opus 5.5 afe86e9ded fix(runtime): record a plugin task's stop receipt when its stop is proven (#21237)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin environment runs each Paperclip Runner turn as a provider
task, and a follow-up run can continue the same Claude/ACPX conversation
from the saved checkpoint
> - Core authorizes that continuation only when the predecessor run's
remote execution has stopped (`remoteExecutionHasStopped`), so two
Runners never write one checkpoint
> - The predecessor's own settlement already proves the stop: the
provider reports `executionStopped`, and Core then completes the task
> - On a per-turn (ephemeral) lease, Core records the stop receipt only
when it releases the lease, and that release can be held for a long time
(for example by a pending workspace export)
> - In that window, an ordinary follow-up fails with `adapter_failed:
Plugin Runner retained session is not authorized`, even though the stop
is already proven
> - This pull request records the stop receipt when the stop is proven,
and the stop check accepts it before the release
> - The benefit is that follow-ups continue as soon as the predecessor
is proven stopped, and the release gates keep their meaning

## Linked Issues or Issue Description

Fixes #21211

Refs #21214 and #21223. Those cover the creation fence for same-run
recovery, which still requires a released lease.

Production evidence for this change: in both observed failures, the
predecessor was a per-turn plugin run. Its `task.settle` completed with
all provider calls successful. Core then did not release its lease for
30 minutes or more, and no release call was made in that time. The
follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint`
→ `resolvePluginTaskResume`) within seconds of admission.

## What Changed

- `server/src/services/environment-task-runtime.ts`
- After a recorded provider stop, `complete` writes the bound stop
receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the
current lease for every lease policy. Before, it did this only for
`reuse_by_environment` leases. Reusable tasks still confirm every owner
and still move retained owners to released.
- New work on a lease (`submit`, `copilot_metadata`, first portable
resume) removes the stop receipt in the same write that clears
`runnerTaskExecutionStopped`. A replacement physical task does not copy
the old receipt.
- `server/src/services/remote-execution-termination.ts`
- New `hasBoundRemoteStopReceipt` checks only the receipt binding
(company, run, lease, provider and physical task).
`hasRemoteTerminationReceipt` uses it and keeps its released-only
semantics. Resource cleanup gates do not change.
- `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a
lease that is released with its receipt, or that is still `active` (not
released) with a bound stop receipt. A lease whose release started and
did not settle does not count. After release starts, the release owns
the receipt.
  - New `remoteExecutionHasReleased` gives the release-based answer.
- `server/src/services/environment-run-orchestrator.ts`: same-run plugin
recovery uses `remoteExecutionHasReleased`. The same-run resume check
after it requires released receipts, so its behavior does not change.
- `resolvePluginTaskResume` keeps its checks and their order. Its
refusal now names the first failing check: `Plugin Runner retained
session is not authorized (reason: <code>)`, for example
`predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The
old message stays as the prefix, and no ids go into it. The
stale-availability recheck in `environment-task-runtime.ts` names its
refusal the same way. Production telemetry for #21211 could not tell
which check fired. The next occurrence will show it.
- `doc/execution-semantics.md` and
`doc/testing-native-remote-continuity.md` describe the stop receipt and
when it is recorded.

Caller audit for the changed meaning of `remoteExecutionHasStopped`:

| Caller | Needs | Decision |
| --- | --- | --- |
| `agent-instruction-working-copies.ts` | stop | Accepts. Directory
cleanup still requires a released "destroyed" receipt. |
| `environment-run-orchestrator.ts` | release | Moved to
`remoteExecutionHasReleased`. |
| `plugin-task-resume.ts` | stop | Accepts. This is the fix. |
| `native-runtime/native-runner-ownership.ts` | stop | Accepts.
`hasPendingRunCleanup` still counts the active lease as pending. |
| `native-runtime/preserved-native-result-finalization.ts` | stop |
Accepts. Workspace recovery has its own gate. |
| `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts.
`hasPendingRunCleanup` still holds active leases. |
| `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop
acknowledgement and quarantined ownership retirement. Holds and blockers
still apply on the normal path. |
| `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors
still fail it. |

## Verification

From `server/`, after `node scripts/agent-preflight.mjs`:

- `npx vitest run
src/__tests__/environment-task-authority.integration.test.ts`: 10/10
pass. The 2 new tests fail without the fix.
- The plugin task and environment runtime suites
(`environment-task-authority.integration`,
`plugin-runner-task.integration`, `plugin-warm-tasks.integration`,
`environment-task-runtime`, `plugin-task-source-recovery.integration`,
`environment-runtime`): 6 files, 366 pass.
- The audited caller suites (`agent-instruction-working-copies`, case
handoff legacy and native, `environment-run-orchestrator`,
`heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`,
`native-remote-retry-admission`, `preserved-native-response-wait`): 9
files, 806 pass.
- `pnpm --filter @paperclipai/server typecheck`: pass.

The new integration tests cover:

- `complete` writes a bound stop receipt on a still-active ephemeral
lease. `remoteExecutionHasStopped` is true and
`hasRemoteTerminationReceipt` is still false.
- `resolvePluginTaskResume` (inspect and launch) authorizes a
follow-up's new task, and the real host `read_state` checkpoint read
succeeds, while the predecessor's release is held.
- Refusals: no provider stop, a receipt for another provider task, an
unstopped sibling lease, and a release in progress.
- New work on the lease removes the receipt. Release overwrites the
receipt with its own receipt. Warm owners keep their behavior.

I did not add an executor-level test. `native-session-executor.test.ts`
mocks `resolvePluginTaskResume`, so the integration test runs the real
checkpoint read path instead. I did not run the full suite locally. CI
runs it.

## Risks

- `remoteExecutionHasStopped` now returns true earlier for a run whose
leases are still held. Each caller is audited above. Callers that need
released resources use the release-based predicates.
- The stop receipt can now be on an `active` lease. The SQL readers of
`remoteExecutionTermination` (cost accounting, execution blockers,
idle-work proof, orphan settlement) also require a released status.
`pinFailedObservedWorkspace` treats an active ephemeral lease with a
receipt as not pinnable. That path applies only to the remote-workspace
flow and falls back to `preserveNativeFailedWorkspace`.
- Leases that settled before this change have no early receipt. They
continue to wait for their release.
- Nothing compares the refusal message exactly. Matchers use substrings,
and run failure classification does not use the message.
- A follow-up still fails closed if the predecessor's stop was never
proven. This change does not add a recoverable wait.
- The reason the release was held in production is not addressed here.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents, Honeycomb trace queries) and extended
thinking. One sub-agent wrote the change. A second session reviewed it
before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:20:45 -08:00
DottaandClaude Opus 5.5 497ccba8f1 fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner that the server
controls over the durable PRP channel
> - When a run is interrupted, a same-run retry reattaches the live
Runner and reads its session (`session.snapshot`) before it continues
> - On staging, the reattached Runner authenticated and restored its
session, and then did not answer the read
> - The recovery kept that Runner, so each later attempt adopted the
same Runner and failed the same way until the attempt budget ended the
run
> - This pull request makes the next attempt replace that Runner with a
fresh Runner over its saved journal, and makes a failed Codex history
read say what failed
> - The benefit is that an interrupted Codex run continues without a
manual Retry, and its provider turn is not sent again

## Linked Issues or Issue Description

Refs #21161, Refs #21053, Refs #20853

**What happened?**

A Codex agent on Daytona ran recovery runs (wake
`issue_recovery_action_restored`). In one hour, 11 of about 35 finished
runs failed. All failed the same way:

- `native_session_recovery_failed: HarnessCapabilityUnavailableError:
read is unavailable: Error: runnerd exited while waiting for
session.snapshot` (8 runs)
- `native_session_recovery_failed: HarnessReconciliationError:
codex_history_read_failed: thread/turns/list requires supported
paginated history; Error: provider_transport_failed: PRP command
session.snapshot timed out (runner connected; pending ahead 0; last
runner event seq 338 76s ago)` (3 runs)

The run log of each run shows one sequence:

1. Attempt 1 fails with `runnerd did not report the provider turn
identity`. The close leaves the Runner alive and unsettled for the
same-run reattachment.
2. Attempt 2 adopts that Runner (same PID). The Runner authenticates.
The log shows `runnerd restored its durable PRP session and provider
thread`. Then the driver's recovery read (`thread/read` or
`thread/turns/list`, which both send `session.snapshot`) gets no answer.
The Runner still answers WebSocket pings, but it sends no command result
and almost no events (`pending ahead 0`, last event 76 s to 391 s
earlier). The recovery treats this as transient and detaches. The Runner
stays alive.
3. Attempt 3 adopts the same Runner again and fails the same way. In
some runs a protocol fault (`thread_binding_mismatch`) closes the
transport during the read, so the message is `runnerd exited while
waiting for session.snapshot`.
4. The provider attempt budget ends the run.

#21161 did not help. Its wait extension applies only while a queued
answer is being committed. Here the Runner sent no answer at all.

The `paginated history` text was also wrong. `codex-history.ts` called
every failed history read unsupported, also a transport timeout.

**Expected behavior**

A Runner that left a recovery read unanswered is not adopted again. The
next attempt starts a fresh Runner from the saved journal, resumes the
provider session from the saved checkpoint, and does not send the
provider turn again. A failed history read reports the real failure. A
Codex build that refuses paginated history falls back to one complete
thread read.

**Steps to reproduce**

1. Start a Codex run in a remote (Daytona) Runner.
2. Let the first attempt fail at turn start, so that the close keeps the
Runner alive.
3. Let the reattached Runner stop answering commands after it
authenticates (for example, stop the Runner process after its first
answer).
4. See `PRP command session.snapshot timed out (runner connected;
pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3,
with the same Runner PID.

**Paperclip version or commit**

Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on
`master` after `53608bd1d6`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `nativeRecoveryCommandUnanswered` (executor) finds a
`native_session_recovery_failed` whose Runner did not answer a recovery
command (`PRP command <type> timed out` or `runnerd exited while waiting
for <type>`). Identity, authority, integrity and incompatibility
refusals do not match.
- When such a failure ends a `reattach_remote_runner` attempt on a
sandbox target, the failure detail records `adoptedRunnerUnanswered:
true`.
- `claimNativeRestartRecoveries` copies it to the next claim as
`replaceUnansweredRunner: true`, in the same way as
`exitedBeforeSessionOpen`.
- `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps
all identity checks. When the old Runner is still alive, it returns no
adopted process, and the attempt launches a fresh Runner from the saved
journal. The existing launch fence stops the old `paperclip-runnerd` on
that state folder and its whole lineage, and fails the launch closed
when it cannot prove them gone. A Runner that already exited mid-turn
keeps the existing fail-closed path, because its provider or tool
processes can survive where the fence cannot find them. The task then
continues in a fresh session from the saved work. The attempt budget is
unchanged.
- `codex-history.ts`: a history read that fails without a provider
answer is now `codex_history_read_failed: <method> read failed;
<error>`. A provider refusal of the method (`-32601` or `-32600`) on the
first page falls back to one `thread/read` with `includeTurns: true`.
That fallback fails closed when the thread differs, or when turns, items
or turn identities are missing. A refusal on a later page still fails
closed.
- `doc/execution-semantics.md` describes both rules. The launch-fence
code comment now names the replacement case.

### Changed existing test

`codex-history.test.ts` "does not turn an interrupted page or
unsupported method into idle history" used a plain `Error("method not
found")` on page 2 and expected the `supported paginated history` text.
A plain error is now a failed read, not an unsupported method. The test
now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2
and still expects a fail-closed rejection. A new test covers the
failed-read message.

## Verification

- Red before the fix, green after:
- `server/src/services/native-runtime/native-session-executor.test.ts` ›
"a same-run reattachment whose adopted Runner never answered its
recovery read": the replacement and staging-message tests failed on the
old code (the Runner was adopted again, the three staging messages were
not recognized). Two further tests ("never resumes beside an exited
unanswered Runner's lineage") failed on the first revision of this PR,
which relaunched beside an exited mid-turn Runner, and pass now. All 15
pass.
- `native-runner-restart-recovery.integration.test.ts` › "asks the
same-run reattachment to replace a Runner that left its recovery read
unanswered" (embedded Postgres): the claim had no
`replaceUnansweredRunner` on the old code. Now passes.
- `codex-history.test.ts`: 7 new tests failed on the old code. All 16
pass.
- Real `paperclip-runnerd` and fake Codex:
`runnerd-codex-transport-remote-retention.test.ts` › "continues the same
provider thread from a fresh Runner over the journal of a retained
Runner that stopped answering, without replaying its turn". It runs the
staging sequence: turn started, close keeps the stopped Runner, a second
controller adopts it, the Runner stops answering and the controller
detaches, the Runner and provider are killed (the launch fence), and a
fresh Runner over the same journal returns the same thread. The fake
Codex call log shows one `thread/start` and one `turn/start`.
- Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus
the remote-retention, adoption and cold-restore transport suites (442
tests). Two suites timed out under parallel load and pass alone. Server:
`native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation` (1714 passed, 21 skipped).
- `pnpm typecheck` in `server` (all layers) and `tsc` for
`paperclip-runner` source and tests pass.
- Not run locally: the container-gated remote Runner suites
(`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the
full suite.

## Risks

- Medium. The replacement relaunches a Runner over a journal that is not
settled, but only while the old Runner is alive. Safety comes from three
facts: the launch fence stops the old Runner and its lineage with proof
or fails the launch; the Runner restores its own durable journal (as
local `resume_dead_runner` does); and the real-runnerd test shows that
the turn is not sent again. An exited Runner is never relaunched beside
its possible survivors.
- If the previous close left a `runner.suspend` pending, a fresh Runner
can execute it and exit. The attempt then fails as it does today. The
usual detach path after an unanswered read sends no suspend.
- The replacement uses the existing attempt budget. It does not add
attempts.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events
and logs), extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:06:24 -06:00
Nicky LeachandClaude Fable 5.2 9b613562b7 feat(docker): split the Runner image into stable layers and add a slim variant (#21229)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Part 2 of this stack builds the Runner alone as the image
`runner-artifacts`, in one layer of about 850 MB
> - Most of those bytes are provider binaries and Node. They change only
with a provider version or the Node pin, but one layer means that every
Runner source change makes all 850 MB new
> - About half of the image is the pack's transport archives. A consumer
that copies the pack as files does not need them
> - This pull request splits the image into layers by how often their
content changes, and adds a slim variant without the transport archives
> - The benefit is that a new image shares about 93% of its bytes with
the one before it, and a consumer that needs no transport archives pulls
half as much

## Linked Issues or Issue Description

Refs #21204 (part 2 of this stack) and #21167 (provider pack layers). No
issue exists for this change.

**Problem or motivation**

The image from part 2 is one layer. A registry and a builder store and
move layers, not files. A change to one Runner source file therefore
makes a new 850 MB layer, although 790 MB of its content is the same as
before. The image also holds the provider pack two times: as files, and
as transport archives of those files. Some consumers need only the
files.

**Proposed solution**

Write the image as four layers: the pack's large dependency files,
everything else, the transport archives of those large files, and the
other transport content. Make the two large layers byte-identical from
build to build. Add the target `runner-artifacts-slim`, which is the
first two layers. The full image is the slim image plus the two
transport layers, as the same layer blobs. Check all of this on the
built images.

**Alternatives considered**

- Two unrelated images. Then the registry stores the shared content two
times unless the layers happen to match, and nothing ties the two
together.
- Split by path only, with all files of 1 MiB or more in the first
layer. Six of those files are the Runner's own bundles under `dist`.
They change with the Runner's source, so the first layer would change
with it too.
- Leave the transport archives as they were written. Their bytes then
depended on timing (see "What Changed"), and the archive layer never
repeated.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 3 of 4. Each part is on top of the one before it:

1. #21228 (merged): an optional field in the provider pack manifest
2. #21204: build and check the Runner image, with no publication
3. #21229: layers by change rate, and the slim variant (this pull
request)
4. #21230: publish both variants from `master`

## What Changed

- `Dockerfile`: two final stages. `runner-artifacts-slim` is two `COPY
--link` layers. `runner-artifacts` starts from it and adds two more. The
build stage sorts its output into the four layers. Every path is where
it was in part 2: consumers copy by path and see no difference.

  | Layer | Content | Changes with |
  | --- | --- | --- |
| 1 | The pack's dependency files of 1 MiB or more: provider binaries
and Node | A provider version or the Node pin |
| 2 | Everything else, with `runner-image.json` | Every Runner source
change |
| 3 (full only) | The transport archive of each file in layer 1 | The
same as layer 1 |
| 4 (full only) | The other transport archives, the overlay archive and
the receipt | The same as layer 2 |

- Stable bytes. The files of layers 1 and 3 get timestamp 0, and the
large dependency files get it before their transport archives are
written. A file under the pack's `dist` is the Runner's own output, so
it and its archive are in layers 2 and 4.
- `scripts/create-provider-pack-archive.mjs`, two separate commits.
**This script also writes the application image's archives.**
- The same file now gives the same archive every time. The script piped
tar into Node's zstd compressor. When the pipe ended while the
compressor still had its last chunk queued, Node finished the zstd
stream two times, and the archive got an empty frame of nine bytes at
its end. That depended on timing. In the pinned Linux base, 37 of 60
runs of the old code had the frame. The compressor is now ended only
after it has taken every chunk.
- Archive N holds file N of the receipt, and the receipt listed the
files in path order. `dist` sorts before `node_modules`, so one more
large file in the Runner's own output gave every dependency file another
archive name, and layer 3 became a new blob with the same contents. The
receipt now lists the files outside `dist` first. The readers
(`provider-pack-layers.ts`, `retained-provider-pack-source.ts`) pair
file N with `N.tar.zst` and do not use the path order, so they need no
change. For the application image, the archive numbers change one time,
and the bytes of each archive do not change.
- `runner-image.json` is the same file in both variants. Its new field
`fullImageOnly` names the two paths that only the full image has.
`paths` did not change.
- `scripts/runner-image-layout.mjs check` takes `--variant slim|full`.
It is required and has no default. A slim tree that holds a transport
path fails. A full tree without them fails.
- `scripts/runner-image-oci.mjs` (new, 585 lines). It is not an input of
the image, because the image build does not run it.
- `unpack <oci archive> --out <tree>` writes the tree of an image from
the layers of its archive, in order. It checks every blob against its
digest. It refuses a whiteout, an absolute path, a path with `..`, a
path that is written two times, an entry behind a link, and every entry
that is not a file, a directory or a link.
- `check-layers --slim <oci archive> --full <oci archive>` checks the
table above on the built images. It reads the receipt to decide which
archive belongs in which layer. It fails when the full image does not
start with the slim image's layer blobs in the same order, when a path
that is not a directory is in two layers, and when a shared layer holds
transport content under any name or directory.
- The script reads the archive and its layers itself. It does not use
the system's `tar`, so every platform reads them the same way. It finds
a member only under the name that buildx gives it.
- `.github/workflows/runner-image.yml`: each leg builds each variant one
time, as an OCI archive and nothing else. It unpacks both archives, runs
`check` on each tree with its variant, runs `check-layers`, and runs the
smoke matrix on the slim tree. The tree that is checked is therefore the
bytes of the image, not a second export of the same build. Every step
that builds has a time limit. The workflow still has no registry access.
- Why one export: in the first runs of this pull request, the slim build
stalled on all four legs. Each build command had two exports, a tree and
an archive. The slim command, with every stage from the cache, wrote its
archive and then made no progress on the tree for 50 minutes. A test now
holds the rule of one export and a time limit for each build command.
- The smoke matrix runs on the slim tree. The daemon is in a layer that
both variants share, so one run covers both.
- `server/src/__tests__/runner-image-identity.test.ts`: the test that
holds the rules for the Runner stages now finds a stage wherever
BuildKit starts one. It fails for `FROM x as name`, a flag on `FROM`, a
lowercase or indented `COPY`, an `ADD` or `ONBUILD`, a `COPY` with
another flag, and a `COPY` that continues on the next line. Nine samples
show each case.
- Two comments are corrected. The comment on
`PAPERCLIP_RUNNER_INPUTS_DIGEST` in `build-provider-pack.mjs` said that
no check reads the field. The `Dockerfile` comment on the publication
argument named a workflow "that builds the image for publication".
- `doc/runner-image-contract.md` (159 lines): the two variants, the
section "Layers", how the checks get their tree, and the new checks.
`doc/DOCKER.md` names the slim target.

Script sizes: `runner-image-layout.mjs` 969 lines,
`runner-image-oci.mjs` 585, `runner-image-identity.mjs` 166,
`create-provider-pack-archive.mjs` 114.

## Verification

Image identity at this commit:
`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`.

Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and
zstd, the worktree mounted read-only):

- `runner-image-identity.test.ts` (41), `runner-image-layout.test.ts`
(100), `runner-image-oci.test.ts` (50), `runner-image-workflow.test.ts`
(8), `provider-pack-archive.test.ts` (47, none skipped),
`retained-provider-pack-source.test.ts` (19),
`remote-provider-pack-verification.test.ts`,
`docker-image-layer-reuse.test.ts`, `docker-build-stamp.test.ts`: 293
tests pass in 9 files.
- The same files pass on macOS. The tests of `check-layers` failed on
Linux in the first runs of this pull request: their fixtures named the
archive members `./index.json`, which bsdtar finds and GNU tar does not.
The fixtures now name the members as buildx does.

`linux/arm64`, on a local Docker host, with the new export path:

- Each variant builds as an OCI archive only: the full target in 156
seconds without a cache, the slim target in 5 seconds from the cache.
- `unpack` writes the full image (10,519 entries that are not
directories) in 4 seconds, and the slim image in 4 seconds.
- `check <tree> --variant <variant> --expect-inputs-digest <digest>
--expect-source-revision <sha> --expect-architecture arm64` passes on
each unpacked tree. `runner-image.json` is the same file in both.
- `check-layers --slim <slim> --full <full>` passes.
- The smoke matrix passes on all five base images, from the unpacked
slim tree.

| Layer | Digest | Bytes | In |
| --- | --- | --- | --- |
| 1 | `sha256:3f21b15b3a99…70ac1c` | 379,609,589 | slim and full |
| 2 | `sha256:d5567acc977f…b4f12a` | 43,906,956 | slim and full |
| 3 | `sha256:6a340088ca2c…6ba5b5` | 410,938,259 | full |
| 4 | `sha256:789e97f5db88…3eab44` | 15,656,249 | full |

- The slim image is 423,516,545 bytes in two layers. The full image is
850,111,053 bytes in four.

Stability, with three builds that share no cache for the build step:

| Build | Source revision | Runner build output | Layer 1 | Layer 2 |
Layer 3 | Layer 4 |
| --- | --- | --- | --- | --- | --- | --- |
| A | `aaaa…` | this commit: 6 files of 1 MiB or more under `dist` |
`3f21b15b…` | `d5567acc…` | `6a340088…` | `789e97f5…` |
| B | `bbbb…` | this commit | `3f21b15b…` | `a2f8a4fe…` | `6a340088…` |
`4e6c492d…` |
| D | `dddd…` | one new source file that adds two large files under
`dist` | `3f21b15b…` | `33b0bdaf…` | `6a340088…` | `0ea322bf…` |

- Layers 1 and 3 are byte-identical in all three: 790,547,848 of
850,111,053 bytes, 93%. Build D has two more large files under `dist`,
eight in place of six, and another image identity. Its dependency files
keep the archive numbers 0 to 20. With the earlier numbering, those
numbers moved by two, and layer 3 was a new blob with the same contents.
- Layer 1 has the same digest as before this round. Layer 3 has a new
digest one time, because the archives have new numbers.

Other checks:

- `tests/runner-e2e` workflow tests: 39 pass. `node --test
.github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image
contract scripts: 33 pass. `check-docker-deps-stage.mjs`,
`check-node-version-policy.mjs` and `docker buildx build --check` report
nothing.

GitHub, at this head:

- The `Runner image` workflow passes on both architectures with the new
export path: `Build and check (amd64)` in 14m41s and `Build and check
(arm64)` in 8m41s. Each leg builds both variants, checks both layouts
and the layer rule, and runs the smoke matrix. The first runs of this
pull request stalled in the slim export before any check.

Never run:

- `native-session-executor.test.ts`. It does not load on this host, with
or without this change, because `@paperclipai/plugin-sdk` is not built
here. It imports the reader of the retained pack and no archive number.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The two changes to `create-provider-pack-archive.mjs` change the
application image's transport archives: about half of them lose nine
bytes, and all of them get new numbers one time. An archive's content
does not change. A reader pairs file N of a receipt with archive N of
the same build.
- The large dependency files in the pack, and in their transport
archives, now have the modification time 0 (1970). Before, they had the
build time. Nothing in this repository reads those times.
- `unpack` is a reader of tar and gzip that this repository owns. It
reads the layers that BuildKit, GNU tar and bsdtar write (tested with
each), and it fails for a header it does not know.
- The check on the built images, not the recipe, is what proves the
layer rule; it runs in each leg of the workflow.
- `check` now needs `--variant`. A caller that does not pass it fails
with a clear message. No caller exists outside this repository yet.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 19:52:02 -08:00
DottaandClaude Opus 5.5 db802d1230 fix(heartbeat): deliver a queued comment again unless the reading run cites it as handled (#21243)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can add a comment to a task while the agent's run on that
task is still working
> - #21175 made a `get_task_history` read of a queued comment's current
revision count as delivery
> - A read is not handling. In `agent-chat.multiple-handoffs`, Morgan's
running turn read the "Launch sooner" comment, set it aside, and saved
"Launch later". No continuation ran, so the instruction was lost.
> - This pull request counts a read as delivery only when the run's
accepted result cites the comment as handled. Otherwise one continuation
still delivers it, and that continuation is told the comment may already
be applied.
> - The benefit is that a person's comment is never silently lost, and
the #21175 protection against repeated completed work stays in place

## Linked Issue or Issue Description

Refs #21175
Refs #21203
Refs #21210

**What happened?**
Paid run 43321773136 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`), case
`agent-chat.multiple-handoffs`. Morgan's running turn called
`get_task_history` and received the CEO's queued "Launch sooner" comment
at its current revision (`task_history.read` at seq 75). The model
judged the comment as an unauthenticated agent instruction and saved
"Launch later" from the brief. Its accepted result did not mention the
comment. `deliveredQueuedCommentIds` counted the read as delivery, the
release drain dropped the held comment, and no continuation ran.

**Expected behavior**
A read counts as delivery only when the run records the comment as
handled. A comment that was read but not recorded as handled is
delivered once in a new turn. The new turn's prompt says that an earlier
turn already read the comment and may have applied it.

**Steps to reproduce**
1. Start a native run on a task.
2. Queue a comment behind the run.
3. Let the run read `get_task_history` and finish without applying the
comment.
4. Before this change, no continuation runs and the comment is lost.

## What Changed

- `server/src/services/queued-comment-delivery.ts`: the history-read
delivery rule now also requires the reading run's `run.result.accepted`
event to cite `comment:<id>` (top-level `evidence[].ref` or a
criterion's `evidenceRefs`, prefix match so an annotated ref counts).
The read revision, succeeded-run and finished-after-revision rules are
unchanged. All consumers (release drain, stale-continuation save,
ownership reconciler, first-turn queued input, remote-Stop resume) use
this one rule.
- New `readByEarlierTurnCommentIds`, `commentsArrivedDuringRun` and
`taskHistoryHandlingGuidance`.
- `get_task_history` (both tool authorities) returns
`arrivedDuringThisTurn` (the `comment:<id>` refs of comments other
authors posted or edited after the run started) and a `handling`
instruction to cite each applied comment in `paperclip_finish` /
`paperclip_block` evidence. The comment list itself is unchanged. A
failure to compute the list returns no guidance, so the comment stays
owed.
- `buildPaperclipWakePayload` marks a wake comment `readByEarlierTurn`
when another finished run of the agent read its current revision.
`renderPaperclipWakePrompt` adds one line after that comment: check the
current task state first, and do not repeat work that is already done.
- `doc/run-log-events.md`: the delivery rule and the new fields.
- Evals: `agent-chat.multiple-handoffs` (oracle
`runtime-multiple-handoffs.v6`) and `steering.edit-queued-message`
(oracle `edit-queued-message.structural.v6`) accept in-turn consumption
only with a current-revision read and a citation. A read without a
citation must get exactly one continuation whose saved wake marks
`readByEarlierTurn`, and the final saved revision must hold the change
("Launch sooner"; blue). `runtime-queue.ts` reads citations from the
run's `run.result.accepted` event. No scenario YAML changed.

## Verification

- RED on master `b1ef1f3022`,
`server/src/__tests__/heartbeat-held-comment-wake.test.ts`: 10 of 41
fail. The key one: "settles a held message as soon as its run closes the
task (read mid-turn: true, cited: false)" delivered 0, expected 1.
- GREEN: 41/41. The three required shapes:
- read, not cited: one continuation, and its wake marks
`readByEarlierTurn` (`...read mid-turn: true, cited: false` and the
stale-queued-run variant).
  - read and cited: no continuation (`...cited: true`).
  - never read: one continuation, as before.
  - Edited after the read, or a failed reader: still owed.
- `server/src/services/queued-comment-history-read.test.ts` (remote-Stop
resume): succeeded+cited not owed; succeeded+uncited owed; failed+cited
owed. 4/4.
- `packages/adapter-utils/src/server-utils.test.ts`: the prompt line
renders only for `readByEarlierTurn`.
- Neighbouring suites: `muse-control`, `dot-runner`,
`paperclip-runner-tool-authority`, `paperclip-control-plane-port`,
`muse-native-follow-up`, `runner-prp-coordinator`,
`native-session-handoff`, `execution-continuation-history-read`,
`heartbeat-comment-wake-batching`, `ownership-reconciliation-report`,
`run-preparation`: all pass. Server and adapter-utils `tsc --noEmit`:
clean.
- Eval graders: `npx vitest run --config
tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog
tests/runner-e2e/chat-catalog-domains`: 283 files pass. New calibration:
the observed lost-instruction case (read, not cited, no continuation)
fails; read-not-cited with one warned continuation passes; an unwarned
continuation, a missing final change, and a cited read plus a
continuation fail.
- `pnpm test:e2e:runner:typecheck`: pass. `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10.
`node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
`changed: false`.
- Paid run: Runner Full-Stack E2E 43338695850 on throwaway branch
`chat-eval/qa-mr-paid` (this change plus the model-retry fix on master
19f8ed1ea2). Result to follow in a comment.

## Risks

- A run that applies a queued comment but does not cite it gets one more
turn on that task. That turn is told the comment may already be applied.
In the restart-chaos t005 shape (#21175) this can reopen a done task
when the agent does not cite the comment. This is the chosen tradeoff: a
lost comment is worse than one extra checked turn.
- Each `get_task_history` call adds one small indexed read of the run's
`started_at`.
- Upgrade-safe: runs saved by older builds have no citation, so a
read-only delivery from before this change stays owed and is delivered
once more. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:07:08 -06:00
Nicky LeachandClaude Fable 5.2 5d3fa712aa feat(docker): build and check the Runner alone as the runner-artifacts target (#21204)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Today the application image build compiles the Runner. Other images
that need the Runner build it again, or pull the whole application image
to copy it out
> - Each of those builds can produce a slightly different Runner, and
each one costs time
> - The Runner should be built one time for each change to its inputs,
and every image should copy the same bytes
> - This pull request adds the build target `runner-artifacts`, a
contract for its layout, and a workflow that builds and checks it. It
publishes nothing
> - Nothing consumes the target in this pull request. The application
image build does not change
> - The benefit is one checked Runner build that later changes can
publish and copy by digest

## Linked Issues or Issue Description

Refs #21096 (the merge-queue image build) and #21167 (provider pack
layers). Those are earlier work on the same build. No issue exists for
this change.

**Problem or motivation**

The Runner is compiled inside the application image build. The daemon
comes from the `runner-build` stage. The provider pack is built on top
of the full server and UI build. Downstream sandbox and VM images need
the same Runner. They rebuild it from source, or they pull the
application image and copy paths out of it. There is no contract for
those paths and no single identity for a Runner build.

**Proposed solution**

Build the Runner alone, from its own inputs, as a plain file tree. Give
each build an identity: a digest of its inputs. Document the layout as a
contract, and check every built tree against it. Keep every check that
proves the Runner is good in this one build.

**Alternatives considered**

- Keep the current builds and only document the paths in the application
image. This does not remove the duplicate builds, and consumers still
need the full application image.
- Build the Runner in `docker.yml`. That workflow has no pull-request
trigger, so a change to the Runner build would have no check before it
merges.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 2 of 4. Each part is on top of the one before it:

1. `runner/pack-inputs-digest`: an optional field in the provider pack
manifest
2. `runner/runner-image-build` (#21204): build and check the Runner
image, with no publication (this pull request)
3. `runner/runner-image-layers`: layers by change rate, and the slim
variant
4. `runner/runner-image-publish`: publish both variants from `master`

This pull request had one larger form before. Publication, the image
layers and the second variant are now parts 3 and 4.

## What Changed

- `Dockerfile`: eight new stages that end in the target
`runner-artifacts` (`FROM scratch`). They install with a filtered,
frozen lockfile from a context that holds only the Runner's inputs. The
base is a Node image on Debian bookworm, pinned by digest. No existing
stage changed. `cloud` is still the default target.
- The tree holds: the stripped daemon, the Runner package's complete
`dist` with the build stamp that its own build records, the provider
pack with its transport archives, the Grok provisioning inputs, the
transport contract source and a bundled transport module, the lockfile,
and `runner-image.json`.
- `scripts/runner-image-identity.mjs`: computes the image identity. It
hashes every file under the paths that the build copies from the build
context (path, executable bit, content; symbolic links by target), the
whole `Dockerfile`, and the two platforms. The build runs it on the
files that the build context supplies after `.dockerignore`.
- `scripts/runner-image-layout.mjs`: the layout contract. `write`
assembles the tree in the build. `check` verifies a built tree: every
documented path, the build stamp, every pack artifact and tree digest in
`provider-pack.json`, and every transport archive. It also extracts the
transport archives the way a sandbox does and compares the result with
the pack. No checked path may resolve outside the tree.
- Publication form. The build argument
`PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1` is off by default. Only with it
does the pack manifest carry `payload.runnerInputsDigest` (part 1), and
`runner-image.json` records `publication: true`. Another image build
that runs the stage inline builds the same Runner, but its pack names no
image. `check` requires the field in a publication build and refuses it
in any other. `--expect-inputs-digest` passes only for a publication
build.
- `docker/runner-image/Dockerfile.smoke`: runs `paperclip-runnerd
--build-metadata` on each base image that a consumer uses. Each base is
pinned by digest.
- `.github/workflows/runner-image.yml`: two jobs, on pull requests that
touch an input and on manual dispatch. `identity` resolves the lockfile
and computes the digest in the build context. `build` builds each
architecture on a native runner, checks the layout, and runs the smoke
matrix. The workflow has `contents: read` only. It holds no credential,
reads no registry, and publishes nothing. It builds the publication
form, so the check can bind the tree to the digest that the identity job
resolved. The tree stays on the runner.
- `packages/paperclip-runner/scripts/provider-pack-runtime-tree.mjs`:
the tree digest function moves here from `build-provider-pack.mjs`
without a change in behavior, so the layout check can use it. A test
pins its value.
- `.github/workflows/pr-merge-review.yml`: lists the new workflow, as
`doc/PR-MERGE-REVIEW-WEBHOOK.md` requires.
- `doc/runner-image-contract.md` (127 lines): the contract for
consumers. `doc/DOCKER.md` links to it.

Changes from the earlier form of this pull request:

- The identity hashes the whole `Dockerfile`. The script no longer
parses stages, and it no longer refuses recipes.
`scripts/runner-image-identity.mjs` went from 397 lines to 166. A short
test keeps the three rules that make the hash enough: the Runner stages
start only from pinned images, only `runner-artifacts-context` reads the
build context, and it copies only the paths that the identity hashes.
- The contract document went from 384 lines to 127.
- The workflow tests keep only the security properties: job permissions,
no credential and no registry access, triggers, actions pinned by
commit, and no expression inside a shell script. They no longer pin step
names, order, or wording.

## Verification

Verified on `linux/arm64` only, on a local Docker host:

- `docker buildx build --target runner-artifacts --platform linux/arm64
--build-arg PAPERCLIP_RUNNER_SOURCE_REVISION=<sha> --build-arg
PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1 --output type=local,dest=<tree> .`
builds in 166 seconds without a cache. The image is one layer of
850,121,093 bytes. The tree is 1,755,345,443 bytes in 9,969 files.
- `node scripts/runner-image-layout.mjs check <tree>
--expect-inputs-digest <digest> --expect-source-revision <sha>
--expect-architecture arm64` passes on the exported tree.
- `docker buildx build --file docker/runner-image/Dockerfile.smoke
--build-context runner-artifacts=<tree> docker/runner-image` passes on
all five base images.
- The digest from `--target runner-artifacts-identity` equals the digest
in the built `runner-image.json`:
`d05f51f075cb8fec687a1d27f574ba7cb41cd8ebbe9927c5cbb2cfc13b147880`.
- The same target with `PAPERCLIP_BUILD_COMMIT=<sha>` and no publication
argument builds. Its pack has `runnerSourceRevision` and no
`runnerInputsDigest`. `check` passes on it, and `check
--expect-inputs-digest` fails with "this is not a publication build".
- An edit to `ui/package.json` and to `server/src` leaves the digest
unchanged. A comment added to an application stage of the `Dockerfile`
changes it.
- The two points above ran one commit state earlier. The only difference
to this commit is one type annotation in a comment of
`scripts/runner-image-layout.mjs`.
- A named build context replaces the `runner-artifacts` stage, and no
Runner stage runs. A build with the application build's view of the
context accepts the tree's `dist` and build stamp as current
(`assessRunnerBuild`).
- The daemon has the same bytes in the five cold builds of this stack
that were compared.
- Tests: `cd server && node node_modules/vitest/vitest.mjs run` on
`runner-image-identity.test.ts`, `runner-image-layout.test.ts` and
`runner-image-workflow.test.ts` (132 tests), with
`remote-provider-pack-verification.test.ts` and the existing Dockerfile
tests: 186 tests pass. `tests/runner-e2e` workflow tests: 39 pass. `node
--test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the
image contract scripts and `provider-pack-runtime-tree.test.mjs`: 33
pass. `check-docker-deps-stage.mjs`, `check-node-version-policy.mjs` and
`docker buildx build --check` report nothing.

Never run:

- The `linux/amd64` build. It is the leg that downloads the Cursor and
Copilot assets for the pack.
- The workflow. No job has run on GitHub.
- The transport archive round trip with GNU tar outside the build. The
build runs it with GNU tar. The local host ran it with bsdtar.
- `actionlint` and `hadolint`. They are not installed on the local host.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- Low risk to the application image. The change adds stages and files.
No existing target reaches the new stages.
- The first run of the workflow is the first test of the amd64 build.
This pull request triggers it.
- The `Dockerfile` is one file for all images. Any edit to it, also to
an application stage, changes the Runner image identity. That is the
price of not parsing it.
- The Dockerfile frontend (`# syntax=docker/dockerfile:1.20`) is a tag,
not a digest. A new frontend release under that tag does not change the
identity.
- The image does not run the drift checks of the Runner's `build`
script. They need files outside the Runner's inputs, and repository CI
runs them. A test fails when the `build` script gains a step.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 17:30:51 -08:00
DottaandClaude Opus 5.5 2d1c2db64a feat(mcp): host the MCP Skills extension for connected apps (#21219)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use connected apps through MCP. Paperclip already hosts MCP
Apps views and speaks stateless MCP 2026-07-28.
> - The MCP Skills extension (SEP-2640, Final) lets an app publish
skills (instructions and files) next to its tools. Hugging Face already
serves 25.
> - Paperclip did not read them, so agents missed each app's own
workflow guidance.
> - Skills are a prompt-injection surface. The spec makes manifest
binding, verification, origin tagging and approval revocation host
requirements.
> - This pull request makes Paperclip a Skills host: discover, review,
verify, and deliver app skills to the agents that use the app.
> - The benefit is that connected apps can teach agents their workflows,
under board control.

## Linked Issues or Issue Description

**Problem**
Connected MCP servers can publish skills with the
`io.modelcontextprotocol/skills` extension. Paperclip ignored them. A
board could not see, review or approve them, and agents never received
them.

**Proposed solution**
Implement the host side of SEP-2640. List skills during catalog refresh,
record them per connection and company, bind board approval to the full
file manifest, and load approved skills into agent runs with verified,
lazily fetched content.

**Why it matters**
Apps ship workflow guidance with their tools. Without the extension,
each team must copy that guidance into company skills by hand, and it
goes stale.

Refs #21119, Refs #21140 (the MCP Apps and stateless MCP client work
this builds on).

## What Changed

- **Client capability.** Every stateless request declares
`io.modelcontextprotocol/skills`. Legacy handshake sessions do not. The
capabilities that `server/discover` advertises are remembered beside the
protocol era.
- **Discovery.** Catalog refresh calls `skills/list` (paged) only when
the server declared the extension. A listing inside its `ttlMs` is
reused. A skills failure never fails the tool refresh. Nothing runs
while a run starts.
- **Schema.** New `tool_mcp_skills` (company, connection, skill URI;
frontmatter, sorted manifest, manifest digest, review state) and
`tool_mcp_skill_files` (verified bytes keyed by connection, URI and
digest). Both cascade with the connection.
- **Governance.** New skills wait for review. Approval binds to the
order-free digest of every `{uri, digest, size}`; a request for any
other digest gets 409. A changed manifest or frontmatter revokes
approval and logs `tool_mcp_skill.approval_revoked`. Dynamic, oversized
and invalid skills are declined with a reason. Approve and decline use
the company skill policy (`skills.install`, `external_package`,
`mcp-connection:<id>`).
- **Run time.** Approved skills of connections the agent can use load in
parallel with other skill preparation, inside a 3 s budget. `SKILL.md`
is read with the agent's grant and checked for size, SHA-256 and
frontmatter. A failure calls `skills/get` and takes the skill out of
use. Each skill becomes a per-run bundle on the connector skill path,
named in its app's namespace, tagged with its origin. Paperclip writes
the frontmatter: `allowed-tools` only if the board approved it.
- **Supporting files.** `GET
/api/agents/me/mcp-skills/:skillId/files?path=` serves manifest-listed
files to an agent in an active run, through the skill's own connection.
The sandbox bridge allows the route.
- **Board API and UI.** `GET/POST /api/companies/:companyId/mcp-skills…`
(list, read instructions, approve, decline). The Skills page shows
"Skills from connected apps" with app, state, size, requested tools and
plain-language notes. It renders nothing when no app publishes skills.
- **Docs.** `doc/connections/MCP-SKILLS.md`, linked from
`doc/connections/README.md`.
- **Shared frontmatter parser.** It now decodes YAML single-quoted
scalars. Before, 5 of Hugging Face's 25 skills kept their quotes and
would fail the frontmatter check; now all 25 match. The generated runner
copy is regenerated.

File size notes: `server/src/services/tool-gateway.ts` grows by about
200 lines. The run-time reads must use its grant resolution and
`callAssignedConnectionProtocol`, like MCP Apps view reads. Splitting
the gateway is a separate refactor. `CompanySkills.tsx` and
`CompanySkills.production.tsx` gain 2–4 lines each; the section lives in
its own component.

## Verification

- `server/src/__tests__/mcp-skills.test.ts` (15 tests) uses a fake
stateless skills server and embedded Postgres. It covers: discovery only
after the server declares the extension; the capability in `_meta`;
`ttlMs` reuse; lazy fetch (no reads at listing or approval; `SKILL.md`
at load; supporting files on read); a stale-digest approval refused; a
reorder that keeps approval; a changed or added file that revokes it;
dynamic skills declined; a same-size tampered `SKILL.md` rejected,
refreshed with `skills/get`, and taken out of use; mismatched
frontmatter rejected; reads outside the manifest refused; the same URI
and name from two servers kept apart; company scoping; a slow app
skipped within the budget and cached later; HTTP authorization (board
only, agent needs an active run, run header must match) and activity
logging.
- Red/green: with the digest check disabled, the tamper test fails; with
the manifest comparison disabled, the revocation test fails. Both pass
with the code.
- `server/src/__tests__/mcp-protocol-versions.test.ts`,
`mcp-apps.test.ts`, `connection-instructions.test.ts`,
`tool-gateway-discovery-http.test.ts`: 47 pass (two capability
assertions updated for the new extension).
- `packages/shared/src/frontmatter.test.ts` adds a single-quoted scalar
case; the runner contract test passes (23 tests).
- `ui/src/components/skills/ConnectedAppSkills.test.tsx` (3) and
`ui/src/pages/CompanySkills.test.tsx` (44) pass.
`packages/adapter-utils/src/sandbox-callback-bridge.test.ts` (89)
passes.
- `pnpm --filter @paperclipai/server typecheck` passes. UI `tsc` passes
except the known missing generated Storybook catalog files.
- `pnpm verify` returned exit 2 (doc change is a global CI input);
required CI supplies broad coverage.
- Live check against `https://huggingface.co/mcp` without credentials,
through Paperclip's own services and the real egress guard: 25 skills
discovered as pending; three approved; cold load 135 ms, cached load 13
ms; a supporting file read through the manifest; the bundle tagged with
its origin.
- Storybook: `Agents and reusable capabilities/Skills and Skill
Studio/Connected app skills` (needs review, all states, tool request,
name collision, reading instructions).

## Risks

- **Migration:** two new tables only. No changes to existing tables.
Safe with the previous release running.
- **Run start:** companies without an approved app skill pay one indexed
read. With approved skills the wait is capped at 3 s; late or failing
skills are left out with a `skills.mcp_unavailable` run warning and
never fail the run.
- **Agent harness enforcement:** Paperclip cannot gate each
code-execution tool call an agent makes while following an app's skill.
Approval is explicit and per skill, the approval note says agents may
run commands the skill describes, and `allowed-tools` is never granted
without a separate tick.
- **Shared-home adapters** receive app skills like company skills. A
revoked skill leaves on the next skill sync.
- Not in this PR: per-connection opt-in for dynamic skills; serving
Paperclip's own skills from its public MCP endpoint;
`resources/directory/read`; app skills on an agent's skills tab.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs, live HTTP checks).
Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:18:06 -06:00
DottaandClaude Fable 5.2 826d41e646 feat(ui): redesign Settings → Secrets around what people manage (#21225)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents need API keys and tokens. People manage them in Settings →
Secrets.
> - The page showed every credential in one table. Tokens that Paperclip
saved for connections, chat channels and model sign-ins sat beside the
few secrets a person added, many with record IDs as names.
> - Each row repeated an always-true status, a version and a key. New
secret opened on two stacked mode switches and two storage pickers.
> - A person could not see which agents use a secret without opening it,
and had to open it again after creation to give an agent access.
> - This pull request redesigns the page, the New secret dialog, the
details panel, the Vaults tab, the import dialog, the Proposals tab and
My secrets. It keeps the existing data and API calls.
> - The benefit is a list that shows what a person manages, a create
flow with two fields, and a secret that an agent can use as soon as it
is saved.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The Settings → Secrets page, its New secret dialog, the secret details
panel, the Vaults tab, the import-from-vault dialog, the Proposals tab
and the My secrets tab.

**Subsystem affected**

UI: `ui/src/pages/Secrets.tsx` and `ui/src/pages/secrets/*`.

**Current behavior**

- One table lists all secrets. On a busy organization most rows are
credentials that Paperclip saved itself, and many have record IDs as
names.
- Each row shows Active, a version, a chip and the key. The row does not
show which agents use the secret.
- A three-line help banner is always visible.
- New secret starts with two mode switches (Organization / Each user,
Managed value / External reference), then a provider picker and a vault
picker. The Create button is disabled without a reason.
- The details panel has Details, Usage and Access events tabs.
- The Vaults tab has one section per provider, with empty placeholder
boxes and a status picker in the form.

**Proposed behavior**

- The list shows the secrets that the team added. Each row shows who
provides the value (Organization, or Each user with a count) and which
agents use it.
- Credentials that Paperclip saved collapse into "Managed by Paperclip".
They are grouped by origin and have readable names.
- One Filter menu covers provided-by, storage and status. Active filters
show as chips. Archived secrets stay hidden until a person asks for
them.
- New secret has a name, one explicit Organization / Each user choice, a
masked value, an optional list of agents, and "Stored in" only when more
than one storage place exists. Create stays enabled and says what is
missing. A `NAME=value` pasted into Name fills both fields.
- The details panel is one scrolling view: agents, other places the
secret is set, details, and recent activity.
- The Vaults tab is one flat list with health and secret counts. The
connect dialog has two required fields. Paperclip checks a new vault
immediately after it is saved.

**Reason and benefit**

A person can find their own secrets, see if a secret is used, and create
a secret that an agent can use in one step.

**Breaking changes**

None. The API gains one optional query parameter (`order` on the board
proposals list). The default order is unchanged. See Risks for the UI
behaviors that changed.

## What Changed

- **List.** Rewrote the Secrets tab in `Secrets.tsx`. Added
`secret-list-parts.tsx` for the Provided by and Used by cells, the help
popover and the managed groups. The Folders / Flat toggle is removed.
Folders are always on and search covers all folders.
- **Managed grouping.** Added `secret-origin.ts`. It infers the origin
of a secret from the name and key conventions of the server modules that
create secrets (`tool_app.*`, `tool_oauth.*`, `slack_manager.*`,
`chat.*`, `ai_*`, `routine-*`, `agent-observability.*`). It is used for
presentation only.
- **Agent usage.** Added `agent-secret-access.ts`. It holds the grant
and revoke logic that was inside the details panel, and builds one usage
index from the agents list for all rows.
- **New secret.** Rewrote the dialog. Added `buildCreateVaultOptions`
and `parseEnvAssignment`. Agents selected in the dialog get an
environment variable binding after the secret is saved. A failed grant
shows an error and keeps the secret.
- **Details.** Replaced the tabs with sections. "Also set in" lists the
bindings that are not agents. Activity shows agent names and the last
five entries, with "Show all activity".
- **Vaults.** Flat list with a built-in Paperclip row, a Default badge
that follows `getSelectableProviderConfig`, Check again, Enable, and an
Add vault menu. The dialog has Name and AWS region, with naming and tags
in a disclosure. The Status picker is removed.
- **Import, Proposals, My secrets.** Restyled to the same design.
Proposals adds a Recently decided list. The board list endpoint `GET
/api/companies/{companyId}/secret-proposals` takes a new optional
`order=resolved` to sort by decision time.
- **Copy.** "Provider vault" is "vault". "Local encrypted" is
"Paperclip" or "Paperclip-encrypted". Toasts and block reasons use plain
words.
- **Stories.** `secrets.stories.tsx` has new stories for the managed
section, a folder, New secret, the details panel and Vaults. The
Storybook usage fixture now returns the `target` field that the API
returns.
- **Tests.** Updated the page, import, proposals and My secrets tests.
Added `secret-origin.test.ts`. Updated the runner E2E vault-reference
journey and the Storybook visual render-state copy for the new labels.
- **Docs.** `Secrets → Provider vaults` is now `Secrets → Vaults` in
three documents.

`Secrets.tsx` was 4,855 lines and is 4,951 lines. The redesign edits the
page in place to keep its queries, mutations and deep links. Splitting
the file is a separate change.

## Verification

- `cd ui && node ../scripts/run-tsc.mjs -b`: no errors.
- `cd ui && pnpm vitest run src/pages/secrets src/pages/Secrets.test.ts
src/pages/Secrets.render.test.tsx
src/components/AgentSecretAccessEditor.test.tsx
src/components/AgentConfigForm src/components/IssueThreadInteractionCard
storybook/map-tooling.test.ts`: 14 files, 467 tests passed.
- `cd server && pnpm vitest run
src/__tests__/secret-proposals-routes.test.ts -t "decision time"`:
passed.
- `pnpm vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog-domains/external-vault-reference
chat-catalog-domains/external-personal-secret-menu
chat-catalog-domains/external-personal-secret-entry-dialog
chat-catalog-domains/external-personal-secret.test`: 32 passed, 2
skipped.
- `node scripts/check-token-gates.mjs`, `node scripts/check-maps.mjs`,
`node scripts/check-module-boundaries.mjs`, `node
scripts/storybook-catalog.mjs`: clean.
- `pnpm verify`: exit 2. It asks for broader coverage because a runner
E2E source file changed. Required CI supplies that coverage.
- All 22 Secrets stories render and their play functions pass in
headless Chromium against a local Storybook.
- End to end, on an isolated local instance (`paperclipai worktree init
--empty`, one company, two agents), with a headless Chromium script. 16
of 16 steps passed, with no page errors:
  - Empty state. Create explains missing fields.
- Create an organization secret and give an agent access. The agent's
saved config has the `secret_ref` binding.
- Paste `NAME=value`. Create an each-user secret. Add a value in My
secrets. The list shows "1 of 1 added".
  - Create a folder, create a secret in it, search across folders.
- In details: add an agent, replace the value, remove the agent. The
agent's saved config changes each time.
  - Disable, archive, and the Archived filter.
  - Managed secrets leave the team list and show under their groups.
- Add a Paperclip-encrypted vault and an AWS vault. New secret then
shows "Stored in".
- The import dialog explains a denied vault. The Proposals tab shows its
empty state. The list at 390 px width.
- Not run: the paid runner E2E journeys. I updated
`external-vault-reference.ts` for the new labels and could not execute
it. A live AWS import, and a proposal from a real agent run, were not
exercised.

## Risks

- **Origin grouping is a heuristic.** It reads naming conventions that
live in server modules. If a module changes its naming, its secrets show
in the team list again. Nothing is hidden or blocked by it. A recorded
origin on each secret is the durable fix.
- **Removed UI.** The Folders / Flat toggle, the "View references"
dialog, the vault Status picker, and creating draft GCP or HashiCorp
vault entries from the Add vault menu. Existing draft vaults still show
and can be edited. A disabled vault now has an Enable action.
- **Default status filter.** The list now shows active and disabled
secrets. Before, it showed active secrets only.
- **Built-in storage in "Stored in".** A provider's built-in storage is
not offered once that provider has a usable vault. The form already
replaced that choice with the vault before this change.
- **Agent access at creation.** Access is a second request for each
agent after the secret is saved. If one fails, the secret exists without
that access and an error says so.
- **Automation contracts.** Labels that agents and the runner E2E
journeys depend on are kept: the key in each My secrets row, "Set
value", "Set my value", "Update my value", "Save value", and the import
dialog test IDs. The import entry point moved into the "More secret
actions" menu and the journey is updated for it.
- **Value masking** uses `-webkit-text-security`. A browser without it
shows the value as typed.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) through Claude Code, with tool
use, code execution and browser automation. Three delegated agents on
the same model restyled the import dialog, the Proposals tab and My
secrets.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 19:17:41 -06:00
DottaandClaude Opus 5.5 b0233b5b33 fix(native): count a parent run's read of a finished child as delivery of its result (#21227)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task gets a wake when a child task completes. If the parent
is running, that wake waits behind the run.
> - At finalization, a parent's done claim stays open
(`native_child_completion_pending`) while such a wake is still owed, so
the parent never finishes without the child result.
> - The server had no record of what a running parent actually read. A
parent that read its finished children mid-turn and then finished the
task still kept the wake owed.
> - The task stayed `in_progress`, the wake started a second parent
turn, and the agent wrote a second completion summary.
> - This pull request records a run's read of a finished child and
counts that read as delivery of the child result.
> - The benefit is one completion turn when the parent already read its
children, while unread child results still get a fresh turn.

## Linked Issues or Issue Description

Refs #21189 (child results saved before the run started). Refs #21175
(the same pattern for comments: `task_history.read`). Refs #15372 (the
pending child-completion guard).

**What happened?**

Restart-chaos wave `w-20261112T223803-s7101`, on a master build that
includes #21189, task t006 (OpenCode, native runner, staging-mix parent
with two subtasks):

- Parent run `bfe9941c` started at 22:55:56.
- Its children CHA-11 and CHA-12 finished at 23:07:53 and 23:11:38.
- The child-completion intent and its deferred dispatch row were saved
at 23:11:38, after the run started. So the #21189 rule counted them as
new input.
- At 23:26:41 the run's reasoning says "Both subtask ledgers exist …
Both subtasks are done". The run then saved the CHILD hashes and posted
DONE at 23:33:39.
- At finalization, the done claim was committed as
`native_child_completion_pending`. The deferred wake then started run
`40928116`, which wrote a second completion summary.

Root cause: `hasPendingNativeChildCompletion()`
(`server/src/services/native-runtime/native-child-completion-delivery.ts`)
proved delivery only by wake binding and timestamps. No durable record
showed that the finishing run had read the completed child.

**Expected behavior**

A finishing run's recorded read of a completed child counts as that
child result being delivered. A read while the child was open, a read by
another run, or a failed event write does not count. An unread child
result still keeps the task open and gets one fresh parent turn.

**Steps to reproduce**

1. Start a native parent run.
2. Complete a child while the run is active. The child-completion wake
waits behind the run.
3. Let the run read the child (`get_task_context`, or `GET
/api/issues/:childId`), then claim `done`.
4. Finalize and release the run. On `master`, the decision is
`native_child_completion_pending` and the wake becomes a second run.

The same wave's t001 grade ("duplicate completion") is not this bug.
That run only said it *will* post the DONE marker later, and the grader
counts any mention. Its `subtasks_once` failure comes from the
restart-recovery continuation. This PR does not change either one.

## What Changed

- `native-child-completion-delivery.ts`:
- New `recordChildTaskRead()` writes a `child_task.read` run-log event.
The payload is `{ parentIssueId, children: { id: { status, updatedAt } }
}`, and it lists only children in `done`.
- `hasPendingNativeChildCompletion()` no longer counts a
child-completion wake as pending when the finishing run recorded a read
of that wake's completed child in `done` at or after the child's
completion decision (`status_decisions.created_at`, or the wake's
`createdAt` when the decision row is missing).
  - The #21189 start-time rule is unchanged.
- `paperclip-runner-tool-authority.ts`: `get_task_context` records the
finished child tasks it returns. A failed write is logged and leaves the
result owed.
- `routes/issues.ts`: `GET /issues/:id`, `GET /issues/:id/documents` and
`GET /issues/:id/documents/:key` record the read when the request has
all of these:
  - an agent actor with a run token
  - a child issue in `done`

This runs after the existing access and company checks. So a board user,
another company's agent, or a read of an open task writes nothing. A
qualifying read adds one statement: the existing
`appendHeartbeatRunEvent` fast path, a single `update … returning` /
`insert` CTE bound to the run's id, company and agent. Every other read
adds no query.
- `doc/run-log-events.md`: documents `child_task.read` and how delivery
uses it.
- Tests: a finalization-to-release regression in
`heartbeat-process-recovery.test.ts` and a new
`native-child-task-read-routes.test.ts`.

## Verification

- RED on `master`: in `heartbeat-process-recovery.test.ts` "counts a
child result as read by the finishing parent run (saved after the run
started, read: after the child finished)", the run reads the child
through the real `get_task_context`, then finalization and the
wake-queue release run.
- On master: `native_child_completion_pending`, task `in_progress`, and
the wake is promoted into a second run, the same as t006.
- With this change: `done`, the wake is cancelled, and there is one run.
- The other cases in the same test behave the same before and after this
change:
  - saved before the run started: done
  - not read: pending, fresh turn with the child results
  - read while the child was open: pending
- `native-child-task-read-routes.test.ts` (5 tests):
- An agent read of a done child through each of the three routes records
the read and clears the pending result.
  - An open child records nothing.
  - A board read records nothing.
  - Another company's agent gets no 200 and records nothing.
- Neighbouring suites after merging `origin/master`:
`heartbeat-process-recovery`, `muse-control`,
`heartbeat-held-comment-wake`, `native-status-arbiter-corpus`,
`heartbeat-dependency-scheduling`, `paperclip-runner-tool-authority`,
`agent-task-hierarchy-depth-routes`, and the two new test files: 8
files, 982/982 tests pass.
- `pnpm --filter @paperclipai/server typecheck` passes.

**Not verified**

- No chaos wave has run on this change yet.
- In t006, the parent's tool calls fell in the 4,220 run events that the
evidence omitted. Its reasoning text proves the read, but the exact tool
(`get_task_context`, `get_task`, `read_task_document` or `call_api`) is
not visible.
- A read through `search_tasks`, or through a shell command that calls
the API without the run token, is not recorded. Such a result stays owed
and gets one fresh turn, which is the old behavior.

## Risks

- A recorded read of a done child now lets the parent finish without
another turn. If the agent read the child but ignored it, no second turn
reminds it. A read while the child was still open never counts.
- Each qualifying agent read of a done child appends one run-log row.
Board reads and reads of open tasks write nothing.
- No migration. Wakes saved by older builds work unchanged. Without a
recorded read, the old behavior applies.
- This is a run-log change only (instance database). It adds no
Telemetry or OpenTelemetry export.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:07:31 -06:00
Devin FoleyandPaperclip ad320aeb71 fix(native): wait for a dated provider limit reset and keep it out of Sentry (#19775)
Keep native quota cooldowns tied to valid dated reset messages. Parse explicit years and strict month names while preserving already-expired dates. Keep typed native quota and rate-limit failures on their tasks in unexpected Sentry reporting mode. Preserve setup, process, malformed and all-mode reporting.

Verified current-head CI, Greptile 5/5 at Medium impact, local parser/report red-green tests and PostgreSQL integration coverage.

Co-Authored-By: Devin Foley <devin@devinfoley.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-12 17:06:22 -08:00
DottaandClaude Opus 5.5 334b2ae066 fix(recovery): continue a task blocked behind its workspace's ended writer once the hold releases (#21221)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote task workspaces have one writer run at a time, and an ended
writer keeps the workspace until its files are saved or set aside
> - When a task's next runs fail setup on that hold ("The workspace
still has an unfinished writer or save operation."), stranded-work
recovery blocks the task with a board escalation
> - The retained-file recovery from #21136 later released such a hold on
staging (it continued from the last save), but nothing settled the board
escalation, so the task stayed Blocked until a person pressed Retry
> - Recovery must work without a person: the block belonged to the hold,
and the hold is gone
> - This pull request adds a pass to the active recovery-action sweep
that settles that escalation once the workspace's writer hold is free
> - The benefit is that such tasks continue by themselves with one plain
notice and one run, with no Retry

## Linked Issues or Issue Description

Refs #21136
Refs #21143

**What happened?**
On staging, an ended run held its task's own remote workspace after its
superseded workspace export failed. The task's queued runs failed setup
with the unfinished-writer error, and stranded-work recovery blocked the
task with a board-owned escalation. After #21136 was deployed, its
retained-file recovery recorded the hold 77 seconds after server start
and, 19 minutes later, continued the task from its last save and
released the workspace. The task still stayed Blocked behind the board
escalation, and no run was queued. A board user had to press Retry.

**What did you expect to happen?**
Once the hold that caused the setup failures is released, the task
returns to `todo` and its assignee continues it, with no Retry.

**Steps to reproduce**
1. Give a task a remote workspace whose writer is an ended run with
unsaved files.
2. Let the task's next run fail setup with the unfinished-writer error,
and let stranded-work recovery block the task.
3. Release the hold (save the files, or continue from the last save).
4. Run the active recovery-action sweep. Before this change the task
stays Blocked with an active board escalation.

**Why the merged fix did not unblock the task**
The retained-file fallback only releases a Blocked status that its own
hold set (`holdOwnsBlockedStatus`). Here stranded-work recovery set the
block with a different, board-owned escalation, and a late status
decision 1.4 seconds later kept that block. So the fallback correctly
did not take the status, and no code settled the board escalation after
the hold was gone.

## What Changed

- Add `continueTasksAfterWorkspaceWriterReleased`
(`server/src/services/recovery/released-writer-escalation.ts`). It
selects board-owned `stranded_assigned_issue` escalations whose latest
run failed setup with the unfinished-writer error before any provider
work, by the current assignee. In one transaction it checks that the
block belongs to the escalation, that no newer run exists for the task,
that dependencies allow work, and that the workspace's writer hold is
now free. Then it resolves the escalation with a named outcome, returns
the task to `todo`, records one plain activity notice, and records a
pending wake.
- A later status decision that only kept the Blocked status it found
(from `blocked` to `blocked`) does not own the block. A decision that
chose the block still keeps the task blocked.
- Run the pass in `reconcileActiveRecoveryActions`, next to the legacy
rehome pass, and include its next wake retry time in the pass deadline.
- Generalize the legacy rehome wake delivery to
`deliverPendingRecoveryWakes`, keyed by the evidence key, so both passes
share one crash-safe, backed-off wake delivery. The rehome wake keeps
its fresh session. The new wake resumes normally.
- Keep the company selected by the scheduled sweep while a
released-writer wake is owed (`recoveryActionSweepCondition`).
- Document the rule in `doc/execution-semantics.md`.

## Verification

- `cd server && npx vitest run
src/__tests__/released-writer-escalation.test.ts
src/__tests__/legacy-workspace-rehome.test.ts
src/__tests__/issue-blocker-stall-wakeups.test.ts
src/__tests__/issue-blocker-stall-waiter-statement.test.ts
--maxWorkers=2`: all pass.
- Red/green: with the sweep hook removed, "continues the task once the
hold is released" fails (the task stays `blocked`). With the hook, it
passes.
- The new test seeds the staging row shape: a native writer run ended
with `native_finalization_superseded` still writing the task's own dirty
workspace, three legacy setup failures with the unfinished-writer error
and `{ kind: "bootstrap", providerWorkStarted: false }`, a board
escalation with `latestRunErrorCode: "setup_failed"`, the block 4.4
seconds after the last failure, and a later `blocked` to `blocked`
status decision.
- The tests also cover: no change while the writer still holds the
workspace, one wake only, a pending wake after a crash, a refused wake
that backs off, a later decision that chose the block, a dependency that
forbids work, a newer run, and another setup error.
- `cd server && npx tsc --noEmit -p tsconfig.json`: passes.

## Risks

- Low risk. The pass only acts on board escalations whose evidence is
exactly the unfinished-writer setup failure, and only when workspace
reservation would now admit a new writer. It never changes the
workspace, never applies a result, and never releases a writer fence.
- The task gets one normal wake through the issue-state guard (`todo`,
same assignee), so it cannot start twice.
- The pass adds one bounded query per sweep of a company with active
recovery actions.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only browser API reads of staging) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:58:06 -06:00
DottaandClaude Fable 5.2 9a9cd20d91 feat(ci): send one in ten CODEOWNER PR runs and their outside checks to Honeycomb (#21192)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Contributors need fast, predictable CI feedback.
> - PR #16378 added a Honeycomb trace of the main PR workflow, but only
for PRs with the `profile-ci` label.
> - Ordinary CI time was therefore never recorded. All of the last 300
`CI profile` runs were skipped.
> - Reviews and scans outside the `PR` workflow, such as Greptile, were
not in the trace.
> - This pull request exports one CODEOWNER PR run in ten without a
label, and adds outside checks to every trace.
> - The benefit is a continuous record of total CI time and of which job
or check was slowest. The Honeycomb key stays out of reach of outside
contributors.

## Linked Issues or Issue Description

Refs #16378

**What existing behavior does this improve?**

The Honeycomb CI trace that the `CI profile` workflow exports.

**Current behavior**

- The exporter runs only for a PR with the `profile-ci` label. The label
also forces the full suite.
- The trace has the jobs of the `PR` workflow only. Greptile, security
scans, commit statuses and other workflows are absent.
- The only total is the `PR` workflow time.

**Proposed behavior**

- One `PR` run in ten is exported without a label. The run number must
end in 0. The trace has every job and step. Each span has a sample rate
of 10.
- Each trace has one span for each outside check and each other workflow
on the head commit.
- The root span has `ci.workflow_ms`, `ci.all_checks_ms`,
`ci.last_to_finish` and `ci.slowest_check`.

**Reason and benefit**

Maintainers can see how long CI takes and what is slow, for example a
long Greptile review, without a label on each PR. Sampling keeps the
volume near 1.7 million events each month. Export of every run is near
17 million.

## What Changed

- `scripts/ci-profile/authorize.mjs`: add `exportMode` (label or
one-in-ten sample) and `associatedOwnerPr` (the CODEOWNER checks without
the label).
- `scripts/ci-profile/trace.mjs`: add `normalizeChecks` and spans for
outside checks. Add the sample rate and the new root fields. Sampled
runs do not warn about absent test artifacts.
- `scripts/ci-profile/export.mjs`: select the mode. Read check runs,
commit statuses and workflow runs for the head commit. Wait a maximum of
15 minutes for outside apps that are not complete. Sampled runs read no
artifacts.
- `.github/workflows/ci-profile.yml`: no new job. The job condition now
requires a branch of this repository, an actor and rerun actor on the
`AWS_CI_ROUTING_USER_IDS` roster, and a labelled or sampled run. Add
`checks: read` and `statuses: read`. Increase the timeout to 30 minutes
for the bounded wait.
- `doc/ci-profiling.md` and `doc/DEVELOPING.md`: describe sampled runs,
outside checks, the totals and the job condition.

## Verification

- `node --test .github/scripts/tests/ci-profile.test.mjs
.github/scripts/tests/ci-routing.test.mjs`: 163 pass, 0 fail.
- `actionlint .github/workflows/ci-profile.yml`: no findings.
- Dry runs against real runs with the real CODEOWNERS check (`node
scripts/ci-profile/export.mjs --run-id=ID --attempt=1 --dry-run`):
- A sampled CODEOWNER run (run number 64780): 1,144 spans, sample rate
10, 5 outside checks and 1 other workflow.
- A sampled run from an outside contributor (run number 64800): "No
CODEOWNERS-authorized PR at this revision; no data exported."
- A CODEOWNER run that is not in the sample: "This attempt was not
selected for profiling; no data exported."
- New tests cover: the sample rule, fork and outsider PRs, a bot rerun
actor, a stale head, outside-check spans and totals, the bounded wait,
and the job condition.
- Not run locally: `scripts/ci-profile/reporters.test.mjs`. It needs
installed dependencies, and this change does not touch the reporters. CI
runs it.
- No data was sent to Honeycomb before merge. The workflow runs from
`master`, so the first real export occurs after merge.

## Risks

- The label path now also needs the `AWS_CI_ROUTING_USER_IDS` roster. If
that variable is empty, nothing is exported. All four CI CODEOWNERS are
on the roster today.
- The roster is only a first filter. The exporter still checks the PR
author and both actors against CODEOWNERS on `master` before it reads or
sends data. The key stays in the `ci-profiling` environment, which
permits only `master`.
- The export job can now run for a maximum of 30 minutes on a
GitHub-hosted runner while it waits for an outside check.
- GitHub gives no completion time for a workflow run. The span for
another workflow ends at the last update of that run.
- Honeycomb volume increases by about 1.7 million events each month.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with
tool use, code execution and repository inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 16:57:23 -06:00
DottaandClaude Opus 5.5 0904a65d82 fix(server): local board boundary identifies agent processes on Linux it cannot inspect (#21209)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In `local_trusted` mode, the operator's own local processes get
implicit board access without credentials
> - #21181 added a local board boundary. It refuses implicit board
access to requests that come from an agent run's process tree
> - On Linux CI hosts, the boundary sometimes did not decide a loopback
request. It logged a warning and kept the earlier permissive behavior
for that request
> - The cause was not IPv6. IPv4 and IPv6 loopback lookups worked. The
gap was a connection that belonged to a same-user process that the
server cannot inspect
> - This pull request makes the Linux boundary decide those connections
from the process ancestry that the server can read, and refuse them when
that ancestry is not clearly the operator's
> - The benefit is that the local board boundary now holds on Linux for
IPv4 and IPv6 loopback, and the operator's browser and CLI keep working

## Linked Issues or Issue Description

Refs #21181

**What happened?**
On Linux CI hosts, the local board boundary sometimes could not
attribute a loopback request to a process. It then logged that it could
not identify the local process and kept the earlier permissive behavior
for that request. In one chat eval cohort, 37 of 191 server logs
contained this warning. The warning is logged once per server process.

**Expected behavior**
The boundary decides every uncredentialed loopback request on Linux. A
request from an agent run gets no implicit board access. Requests from
the operator's browser and CLI keep implicit board access.

**Actual behavior**
Some requests on Linux were not decided by the boundary, and they got
implicit board access.

## What Changed

- The Linux inspector now records which same-user processes it cannot
inspect. Before, it only recorded that one existed.
- The Linux TCP table lookup now also returns the socket's owner UID. It
covers `/proc/net/tcp` and `/proc/net/tcp6`, including IPv4-mapped
addresses.
- If no inspectable process owns the connection, the server reads the
TCP tables again. If the connection is gone, has changed, or belongs to
a different user, the request is refused.
- If the connection is still open and same-user, the server checks every
process it cannot inspect. Implicit board access stays only if each one
has a readable parent chain that does not belong to an agent run.
Otherwise the request is refused.
- The "cannot identify" fallback now applies only when the platform has
no inspector, or when Linux cannot read the TCP tables. The server keeps
logging a warning in that case.
- `doc/DEPLOYMENT-MODES.md` describes the new Linux behavior.

**How the operator is recognized**
- Browser and CLI processes are normal same-user processes. The server
reads their open sockets and environment, so their requests are
classified as before.
- A browser helper process that the server cannot inspect is still the
operator when its readable parent chain leads to the browser and has no
agent-run markers.
- The server's own requests and requests from the server's ancestors
stay the operator, as before.
- Connections owned by other users were refused before, and they are
still refused.

## Verification

- `pnpm --dir server exec vitest run
src/__tests__/local-agent-process-boundary.test.ts` passes on macOS (15
passed, 10 Linux-only skipped).
- The same file ran on Linux (Docker, `node:24-bookworm`, non-root user,
IPv6 loopback on). Result: 25 passed.
- Red check on Linux with the previous boundary and the new tests: the
agent-run cases for processes the server cannot inspect failed for both
`127.0.0.1` and `::1`. The operator cases passed.
- New Linux-only tests run real processes over `127.0.0.1` and `::1`:
  - an operator process keeps board access
  - an agent-run process is refused
  - an operator process the server cannot inspect keeps board access
  - an agent-run process the server cannot inspect is refused
- New platform-independent tests cover the classification rules and the
`/proc/net/tcp6` fixture parsing.
- A paid chat-feature run on Linux CI, with temporary diagnostic logging
on a separate branch, checks the server logs for the warning and for
refused operator requests. The results will be in a PR comment.

## Risks

- A rare operator tool that the server cannot inspect, and whose parent
chain the server cannot read, now loses implicit board access. Such a
tool can use a board API key.
- The extra TCP table read happens only when a connection has no
inspectable owner. Normal requests do no extra work.
- Platforms without process inspection keep the previous behavior.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, Docker on Linux) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:52:06 -06:00
DottaandClaude Opus 5.5 6ac7211cb9 fix(heartbeat): stop re-delivering a remote-stop follow-up that a succeeded run already read (#21203)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person's follow-up comment must reach the agent once
> - #21175 made a succeeded run's mid-turn read of a comment (through
`get_task_history`) count as delivery for the queued-comment paths
> - The remote-Stop resume path decides delivery with
`undeliveredLegacyUserCommentIds`, which ignores those reads
> - So a follow-up that a succeeded run already read was sent again, and
the agent posted a second DONE on a finished task
> - This pull request applies the same rule on that path and adds a read
helper for a later change
> - The benefit is one answer per follow-up on every resume path, with
no dropped messages

## Linked Issues or Issue Description

Refs #21175

**What happened?**
A saved follow-up comment was resumed after a remote Stop (wake reason
`issue_commented`, idempotency key `remote-stop-comment:…`). A succeeded
run had already read this comment mid-turn. The resumed wake started a
new turn, and the agent posted a second DONE on a finished task. The
cause: `undeliveredLegacyUserCommentIds`
(explicit-native-continuation.ts) checks only wake input and accepted
first turns. It does not check the `task_history.read` records that
#21175 added.

**Expected behavior**
A follow-up that a succeeded run read at its current revision is
delivered. The remote-Stop resume path skips it. A read by a run that
did not succeed does not count, so that follow-up is still delivered
once.

**Steps to reproduce**
1. A native run reads the task history after a person posts a follow-up.
2. The run succeeds. The follow-up stays saved as a deferred wake.
3. The remote-Stop resume path runs for that wake.
4. Before this change, the follow-up is delivered again.

**Paperclip version or commit**
`master` at 549816e6a0.

## What Changed

- `server/src/services/queued-comment-delivery.ts`:
`historyReadDeliveredCommentIds` exposes the #21175 rule for a list of
comment IDs. `taskHistoryReadsForRun({ companyId, runId })` returns each
comment one run read, with the read time and the comment revision that
the read returned. It returns these whatever the run's outcome. It reads
only that run's events through `heartbeat_run_events_company_run_idx`.
No delivery decision uses it.
- `server/src/services/explicit-native-continuation.ts`:
`undeliveredLegacyUserCommentIds` also removes comments that a succeeded
run read at their current revision. Its two callers are the remote-Stop
resume path and the queued explicit-continuation check, and both now use
the #21175 rule.
- Deliberate asymmetry: a read by a run that did not succeed never
counts. A dropped person's message is worse than a duplicate answer.
- `doc/run-log-events.md`: documents both points.
- New test file
`server/src/services/queued-comment-history-read.test.ts`.

## Verification

- RED on `549816e6a0` (`npx vitest run
src/services/queued-comment-history-read.test.ts`): 2 failed, 1 passed.
  - The follow-up that a succeeded run read was still returned as owed.
  - `taskHistoryReadsForRun` did not exist.
- The case where a failed run read the follow-up (still owed) passed, as
intended.
- GREEN: 3/3 pass.
- Neighbouring suites (`first-turn-queued-input`,
`explicit-native-continuation`, `heartbeat-agent-directory-busy`,
`heartbeat-held-comment-wake`, `execution-control-work`): 517/517 pass.
Server `tsc` is clean.

## Risks

- Low risk. The function and its callers keep their contract. It returns
fewer comment IDs only when a succeeded run of the assignee read that
exact revision.
- Comments read through the REST API by non-native adapters are not
recorded, so their behavior does not change.
- No migration. The added lookup uses the time-bounded, index-backed
query from #21175.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:23:34 -06:00
DottaandClaude Opus 5.5 de675d924a fix(heartbeat): stop a comment the closing run already read from reopening its finished task (#21175)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can add a comment to a task while the agent's run on that
task is still working
> - The comment must reach the agent once: in the running turn, or in
one later turn
> - In a restart-chaos wave, a comment that the running turn already
read reopened the finished task, and the agent did the completion turn a
second time 15 minutes later
> - This pull request makes a read by the running turn count as
delivery, and holds the comment behind the running run so the run's own
release settles it
> - The benefit is no reopened done task, no repeated completion turn,
and an unread comment that is delivered when its run finishes, not after
a slot wait and a stale cancellation

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261112T181948-s7101`, task t005 (OpenCode,
native Runner), during a two-server overlap deploy (K15).
- 18:26:56 run A started on the old server. 18:26:55 traffic switched to
the new server.
- 18:27:03.953 a board user posted a follow-up comment. The new server
handled it. Run A was not in that process's `activeRunExecutions`, so
`filterZombieCoalesceTarget` treated it as a zombie. Admission returned
`proceed`, and the comment got its own queued run B (18:27:04.252). The
queued run is not part of the release drain.
- 18:33:32 run A called `get_task_history`, received the comment,
applied it, and finished the task (`done`, 18:34:20).
- Run B waited for an agent slot until 18:41:49 and was cancelled as
`continuation_task_ownership_changed`.
- `deferUndeliveredCommentsOfStaleRun` then saved the comment again. Its
"unread" test is only "a run started before the comment and finished
after it". `deliveredQueuedCommentIds` had no record of run A's read.
The release drain reopened the task (done to todo, 18:41:51), and run C
(18:49:22) posted a second "Task complete" summary. The harness grade
`done_comment_once` failed.

**Expected behavior**
- A comment that the running turn received is consumed. No queued work
outlives that run, and the task stays done.
- A comment that the running turn did not receive is delivered once,
through the normal comment-on-done-task path, when that run finishes.
- A stale continuation is never re-issued for a comment that a completed
run already read.

**Steps to reproduce**
1. Start a native run on a task.
2. Post a person's comment while another server instance executes the
run (or seed a queued comment run behind it).
3. Let the run read `get_task_history`, then finish the task.
4. Before this change, the queued comment run is cancelled as stale, the
task reopens, and the agent runs a second completion turn.

**Paperclip version or commit**
`master` at ef6d347f79.

## What Changed

- `server/src/services/queued-comment-delivery.ts`: new run-log event
`task_history.read` and `recordTaskHistoryRead`.
`deliveredQueuedCommentIds` also counts a comment as delivered when a
succeeded run of the assignee on that task read the comment's current
revision (`historyReadDeliveredSql`). All existing consumers use this
one check: the release drain liveness check, the stale-continuation save
(`saveUndeliveredUserComments`), the ownership reconciler, and
first-turn queued input.
- `paperclip-runner-tool-authority.ts` and
`runner-semantic-authority.ts`: `get_task_history` records the returned
comment IDs and `updatedAt` values. The tool output does not change. A
failed record is logged and leaves the comment owed, so it is delivered
again.
- `server/src/modules/wake-queue/application/use-cases.ts`: a comment
follow-up for the same agent's `running` run is held as a deferred wake
even when this process does not track that run. Before, a run executed
by another server instance was treated as a zombie, and the comment got
its own queued run that bypassed the release drain. Coalescing into an
untracked run is still refused.
- `doc/run-log-events.md`: documents `task_history.read`.
- Regression tests in
`server/src/__tests__/heartbeat-held-comment-wake.test.ts`.

Intended behaviour chosen: in t005 run A received the comment mid-turn
through its own `get_task_history` call (run events at 18:33:32 show the
comment ID in the tool result, and the result says it added the
follow-up line), so (a) applied. The fix makes that read the durable
proof of delivery. A comment that no run read still follows the existing
outbox: it is held behind the run and delivered once at the run's
release, reopening the done task as a person's comment does. Envelope
task history of another message's turn still does not count (unchanged
doctrine and test).

## Verification

- RED on master (`ef6d347f79`), `npx vitest run
src/__tests__/heartbeat-held-comment-wake.test.ts -t "mid-turn|another
server|held message"`: 3 failed, 1 passed.
- "consumes a message the closing run read mid-turn, so its stale queued
run reopens nothing": delivered 1, expected 0.
- "holds a message behind a run another server is executing instead of
queueing a run of its own": a queued run was created.
- "settles a held message as soon as its run closes the task (read
mid-turn: true)": delivered 1, expected 0.
- The unread variant (delivered once, no cancelled stale run) passes on
both, as intended.
- GREEN with this change: all 4 pass.
- Review follow-up (f43e79c3): a history read counts as delivery only
for a succeeded run on the read revision. Tests cover an edit after the
read (the stale queued run delivers the comment once) and reads by an
edited or failed run (the comment stays owed). Full file: 37/37 pass.
- Security-scan follow-up (239f0597): the delivery check probes each run
binding (`native_issue_id`, `context_snapshot->>'issueId'`) through its
own existing index with `offset 0`. It reads only the assignee's
succeeded runs of the task with `finished_at >= ` the comment's current
revision. It does not use `started_at`, because the t005 reader started
before the comment. It stops at the first matching `task_history.read`
event of each run through `heartbeat_run_events_company_run_idx`. No
migration is needed.
- Seeded plan (local PostgreSQL: 200 tasks, 4,000 runs, 1.2M run events,
one comment read by one run), `EXPLAIN (ANALYZE, BUFFERS)`:
- Before: 1.202 ms, 6,097 shared buffers. It scanned every event of all
21 runs of the task.
- After: 0.030 ms, 23 shared buffers. It reads one index-matched run and
one event. The full `deliveredQueuedCommentIds` call takes 1.55 ms.
- New case: a succeeded run's read is not delivery when the comment was
edited after that run finished. Full held-comment file: 38/38 pass.
- `muse-control.test.ts` compared the whole run row after a
`get_task_history` call. It now allows that read's own run-log append
(`nextEventSeq` +1, `updatedAt`), and every other column must be
unchanged. That CI failure (General tests server 2/12) came from this
PR. "e2e shard (5/8)" failed because the self-hosted runner lost
communication; that is not related to this PR.
- Runner and Muse suites (`dot-runner`, `paperclip-control-plane-port`,
`runner-prp-coordinator`, `historical-task-questions`,
`paperclip-runner-tool-authority`, `muse-native-follow-up`,
`muse-control`): 509/509 pass. Server `tsc` is clean.
- Neighbouring suites: `heartbeat-held-comment-wake`,
`src/modules/wake-queue/**`, `execution-continuation`,
`paperclip-runner-tool-authority`, `first-turn-queued-input`,
`heartbeat-comment-wake-batching`, `ownership-reconciliation-report`,
`heartbeat-zombie-guard`: 555/556 passed in one loaded run. The 2
timeouts (one batching test, one reconciler `afterAll` hook) pass when
run alone (1/1, 44/44).
- `server` `tsc --noEmit`: clean.
- The restart-chaos harness already grades this class
(`done_comment_once` in staging-mix). No new cell is necessary.

## Risks

- Product choice: a comment counts as delivered when a succeeded run
read it through `get_task_history`, even if the model ignored it. This
is the same trust that applies to a comment delivered as wake input. The
alternative fails worse: it reopens a done task and repeats a completed
turn. An edit after the read, or a reader run that fails, leaves it
owed.
- Legacy (non-native) adapters that read comments through the REST API
do not write the record. Their behavior does not change.
- A comment follow-up behind a truly dead running run now waits as a
deferred wake until recovery releases that run. Before, its queued run
also could not start until the lock was released.
- Each `get_task_history` call appends one small run-log row. This is
not in the path before `turn.start`.
- Upgrade-safe: continuations queued by older builds still go through
the dispatch stale check, and that path now reads the same delivery
record.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:31:11 -06:00
DottaandClaude Opus 5.5 f92f91f0fb fix(native): finish a parent claim when its child results arrived before the run started (#21189)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task gets a wake when one of its child tasks completes. The
wake carries the child results.
> - When the parent already has a run, the child result wake waits
(`deferred_issue_execution`) behind that run.
> - At finalization, a parent completion claim stays open
(`native_child_completion_pending`) while such a wake is still queued,
claimed or deferred. This guard protects child results that arrive while
the parent works.
> - The guard did not check when the child result arrived. A child
result saved before the parent run started was already visible to that
run, but it still blocked the claim.
> - So the task stayed `in_progress`, the deferred wake started a second
run, and the agent repeated its completion summary.
> - This pull request counts a child result as undelivered only when it
was saved after the finishing run started.
> - The benefit is that a parent that already read its child results
finishes in one turn, and child results that arrive during the run still
get a fresh turn.

## Linked Issues or Issue Description

Refs #15372 (added the pending child-completion guard). Refs #20935
(same family: a self-wake after a settled run).

**What happened?**

Restart-chaos wave `w-20261112T181948-s7101`, task t001 (OpenCode,
native runner, staging-mix parent with two subtasks). Both subtasks were
done before parent run `8006a205` started at 18:45:29. The run read
them, posted its final comment and claimed `done` at 18:46:15.860. A
child-completion wake for the parent had been deferred since about 18:42
behind that run. The finalizer found that wake and committed
`native_child_completion_pending`, so the task stayed `in_progress`. 63
ms later the release promoted the deferred wake into run `9282de34`
(`invocationSource: automation`, `wakeCommentId: null`). Its
`paperclip.native-continuation.v1` turn carried the subtask results that
the first run had already used, plus the agent's own two comments
(resume delta since the base run). That run posted a second "complete"
summary and only then moved the task to `done`.

Root cause: `hasPendingNativeChildCompletion()`
(`server/src/services/native-runtime/native-child-completion-delivery.ts`)
treated every queued, claimed or deferred child-completion wake for the
issue as unread, except a wake bound to the finishing run itself. It did
not compare the wake's save time with the run's start.

**Expected behavior**

A child result that was saved before the finishing run started is not
new input for that run. The completion claim completes the task. The
deferred wake is cancelled at release as a wake on a terminal task. A
child result saved during the run still keeps the task open and gets a
fresh parent turn.

**Steps to reproduce**

1. Create a parent task with a done child. Save a deferred
`issue_children_completed` wake for the parent.
2. Start a native parent run after that wake was saved. Let it claim
`done`.
3. Finalize the run. On `master` the decision is
`native_child_completion_pending` and the task stays `in_progress`; the
deferred wake then starts a second run.

## What Changed

- `native-child-completion-delivery.ts`:
`hasPendingNativeChildCompletion()` counts a child-completion wake only
when its `createdAt` is at or after the finishing run's `startedAt`. A
run without a recorded start keeps the old behavior: every such wake
stays pending. The finalizer, the status-commit recheck and status
reassessment all use this function.
- `heartbeat-process-recovery.test.ts`: a regression test finalizes a
real native done claim with a deferred child-completion wake saved
before (task becomes `done`) or after (task stays `in_progress` as
`native_child_completion_pending`) the run started. It then releases the
issue execution through the wake queue. In the `before` case the
deferred wake is cancelled and there is one run total. In the `after`
case the wake is promoted into a fresh run that carries the child
results.
- `doc/architecture/native-status-arbitration.md`: describes when a
child result counts as undelivered.

## Verification

- RED on `master` (`ef6d347f79`): `heartbeat-process-recovery.test.ts`
"treats a child result saved before the parent run started…" fails with
decision `reasonCode: "native_child_completion_pending"`, `toStatus:
"in_progress"`. This is the same decision as the wave. The "after" case
passes on `master`.
- GREEN: both cases pass with the fix.
- Neighbouring suites pass: `native-status-arbiter-corpus` (includes the
queued / claimed / deferred / consumed child-delivery cases from
#15372), `heartbeat-dependency-scheduling` (revised child completion
delivered after a running parent), `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, and `src/services/native-runtime`.
- `pnpm --filter @paperclipai/server typecheck` passes.
- The regression test runs finalization and release end to end. The
cancel-at-release step itself is existing behavior, also covered by
`modules/wake-queue/application/use-cases.test.ts` ("cancels stale
assignee continuations before claiming promotion on %s tasks", `done`
case).

## Risks

- A child result saved before the run started no longer blocks the
claim. The child status was already `done` when the run started, so the
run could read it. If the clocks of two server instances differ, a child
result saved during the run could look older than the run start; the
window is the clock skew.
- A re-dispatched native status wake gets a new row with a later
`createdAt`. Such a row stays pending, which is the old behavior.
- No migration. Saved wakes from older builds work unchanged, because
the check reads only existing columns.
- Scope: this change does not alter the comment-wake promotion path or
the resume-delta builder.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:19:32 -06:00
DottaandClaude Opus 5.5 f95bbe081e fix(auth): agent shells can only reach the local API with their own run credentials (#21181)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In `local_trusted` mode, the server gives board access to local
requests that carry no credentials, so the operator can use the app
without a login
> - Agent runs execute on the same machine, so their tools can also send
local requests
> - An agent run must act only with its own run credentials and
permissions
> - This pull request makes implicit board access belong only to the
operator's own local processes
> - The benefit is that agent shells and tools can only reach the API
with their own run credentials

## Linked Issues or Issue Description

**What happened?**
In `local_trusted` mode, a process inside an agent run could receive the
board identity for an API request that had no credentials. A permission
denial for the agent then did not stop the change.

**Expected behavior**
A request from inside an agent run uses the run's own credentials and
permissions. Implicit board access stays available to the operator's
browser and terminal tools.

**Deployment mode**
`local_trusted` (private, loopback). Authenticated deployments do not
use implicit board access and do not change.

**Additional context**
Found by the chat feature evals. The bundled board skill also described
implicit board access in words that an agent could follow.

## What Changed

- Add `server/src/middleware/local-agent-process-boundary.ts`. For a
request without credentials, it identifies the local process behind the
TCP connection (Linux: `/proc`; macOS: `netstat -anv` and `ps`). If that
process or one of its ancestors belongs to an agent run, the request
gets no implicit board access. An agent run is a descendant of
`paperclip-runnerd`, or a process that carries a run identity variable
(`PAPERCLIP_RUN_ID`, `PAPERCLIP_AGENT_ID`, `PAPERCLIP_AGENT_KEY_ID`,
`PAPERCLIP_RUNNER_INSTANCE_ID`) with a value that is different from the
server's own value.
- The walk stops at the server process and its ancestors. Requests from
the server process itself, and from an operator whose own development
server runs inside the same agent execution, keep board access.
- The result is cached for each connection. Only one process lookup runs
at a time. Connections that arrive during a lookup wait for the next
lookup, which handles all of them together (one `netstat` and two `ps`
calls on macOS, one `/proc` walk on Linux). All lookups are
asynchronous, so they do not block other work in the server.
- `actorMiddleware` and the live events WebSocket upgrade apply the
check in `local_trusted` mode. A refused request is unauthenticated
(`actor.type = "none"`), the same as an anonymous request in
authenticated mode.
- Platforms where the server cannot identify the process (for example
Windows) keep the previous behavior and log one warning.
- Rewrite the `paperclip-board` skill authentication section. It no
longer describes implicit board access to agents, and it says that a 401
or 403 inside an agent run is final.
- Document the behavior in `doc/DEPLOYMENT-MODES.md`.

## Verification

- `cd server && npx vitest run
src/__tests__/local-agent-process-boundary.test.ts`. The test starts a
real `local_trusted` server and real child processes:
- an operator process without credentials gets board access (green
before and after);
  - an in-process request gets board access;
- a process with a run identity gets no board access (red before this
change, green after);
- an agent process that starts a tool with an empty environment gets no
board access (red before, green after);
- an operator inside the same execution as its own server keeps board
access.
- A burst test with a slow inspector: 12 parallel connections and 2
connections that close early. It checks that only one lookup is active
at a time, that the burst needs at most two lookups, that the event loop
stays responsive, and that later requests still work.
- Parser tests for the Linux TCP table and the macOS `netstat` format,
and a test that the runner daemon itself is not treated as an agent
process.
- Ran `agent-auth-middleware`, `agent-identity`, `live-events-ws`,
`live-events-toast`, `live-events-run-read` and
`issue-create-deduplication-routes` tests locally. `npx tsc --noEmit -p
server/tsconfig.json` passes.
- Paid runner E2E evidence for the affected chat cases is linked in a
comment.

## Risks

- Medium. The check runs once for each new local connection without
credentials. On macOS it starts `netstat` and `ps` (about 30 to 80 ms).
On Linux it reads `/proc`. Bearer requests do not run it.
- If a process can not be identified on a supported platform, the
request gets no implicit board access. On Linux, when a same-user
process hides its sockets, the check keeps the previous behavior so that
the operator's browser continues to work.
- This is a boundary for the documented path, not a sandbox. Agent runs
use the operator's OS account. A process that intentionally leaves its
run's process tree, or reads the instance files directly, is outside
this check. The broader option is a product decision: bind implicit
board access to a capability that only the operator's browser and CLI
hold, or run agents under an OS sandbox that blocks the server port.
This pull request does not change `local_trusted` semantics for the
operator.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:05:15 -06:00
Nicky LeachandPaperclip 2168f50741 Forward provider credentials from verified image catalogs (#21186)
## Thinking Path

> - Paperclip manages AI agents for work.
> - Application images can ship plugins that provide task environments.
> - Paperclip starts plugin workers with a restricted process
environment.
> - An image author can know which provider credentials an image-bundled
worker needs.
> - The image catalog already identifies selected bundles and verifies
their content.
> - This change adds a way for the image catalog to declare the provider
keys that a verified worker receives.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Image-bundled environment drivers can use the worker credential delivery
already available to built-in providers.

**Current behavior**

The host provides fixed credential forwarding for certain built-in
providers.

**Proposed behavior**

A verified image bundle can receive provider API keys declared on its
own catalog entry.

**Reason and benefit**

Image authors can scope deployment credentials to the verified worker
that needs them.

## What Changed

- Add an optional provider credential declaration to the image-owned
distribution catalog.
- Scope forwarding to the verified worker for the selected bundle.
- Publish distribution catalog support for image builders.

## Verification

- The new loader regression failed before the change because the old
catalog rejected the credential declaration.
- Ran six focused Vitest suites for bundled activation, package
identity, distribution catalog validation, worker environment, and
managed configuration. 166 tests passed and one existing test was
skipped.
- Ran `pnpm --filter @paperclipai/server typecheck`. It passed.
- Reviewed the diff for secret values and personal information. Test
keys are synthetic.
- A downstream image build fixture exercised the catalog reader, bundle
verification, and worker environment together. The supported build
passed only the declared provider key to the verified worker. The build
for an older target omitted the declaration.
- Local verification used embedded PostgreSQL and prebuilt fixtures.

## Risks

- Deploy images with catalog credential declarations only to Core
versions that support the field. Older hosts reject unknown catalog
fields.
- The image author controls which verified bundle receives each provider
API key.

## Model Used

- Implementation: GPT-6.1 Sol (`gpt-6.1-sol`), delegated through Codex.
Independent Codex review and verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs
- [x] I have described the issue in the PR
- [x] I have used public references
- [x] My branch name describes the change
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address reviewer comments before requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 20:53:08 +00:00
DottaandClaude Opus 5.5 267983356c feat(mcp): speak stateless MCP 2026-07-28 to provider servers, keep older servers working (#21140)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through Paperclip's MCP client. The client calls
each provider's remote (Streamable HTTP) or local (stdio) MCP server.
> - The client spoke only protocol 2025-06-18. It always did the
`initialize` handshake and it sent client capabilities only in that
handshake.
> - MCP 2026-07-28 removes the handshake and sessions. Every request
carries its version, capabilities and client identity in `_meta`.
Servers that only speak 2026-07-28 reject the old requests, and they
never see that Paperclip renders MCP Apps views.
> - Public servers already speak it: DeepWiki and Context7 answer
`server/discover` with 2026-07-28 today.
> - This pull request makes the client negotiate the version per server.
It speaks stateless MCP to 2026-07-28 servers and keeps the handshake
for 2025-11-25 and older servers.
> - The benefit is that new and old providers both work, and a tool call
to a stateless server is one HTTP request with no handshake.

## Linked Issues or Issue Description

Refs #21119 (MCP Apps host). This PR sends its MCP Apps capability to
stateless servers too.

**Subsystem affected**
Server: the MCP client in `server/src/services/mcp-http.ts`, catalog
discovery, the tool gateway (`tools/call`, context reads, stdio), and
the remote MCP OAuth flow.

**Problem or motivation**
The client used protocol 2025-06-18 only. MCP 2026-07-28 makes MCP
stateless: no `initialize`, no `Mcp-Session-Id`, per-request `_meta`,
`server/discover`, `resultType` with multi round-trip requests (MRTR),
cacheable lists, required `Mcp-Method`/`Mcp-Name` headers, and RFC 9207
`iss` validation. Servers that speak only 2026-07-28 cannot be used.
Paperclip advertised MCP Apps support only in `initialize`, so stateless
servers never saw it.

**Proposed solution**
Probe each server once with `server/discover`, pick the newest mutually
supported version, and save it on the connection. Send stateless
requests to modern servers and keep the handshake for legacy servers.
Tool calls use the saved version and never probe. The saved version is
checked again only when a request built from it is refused.

**Alternatives considered**
Inline negotiation (send each request in modern shape first and fall
back on `400`) adds a failed request to every call to a legacy session
server. Probing on every tool call adds a round trip before each tool
runs. Saving the negotiated version keeps tool calls at one request.

**Roadmap alignment**
MCP protocol currency for the Apps and connections substrate. It follows
the MCP Apps host work in #21119.

**Additional context**
Spec:
https://modelcontextprotocol.io/specification/2026-07-28/changelog. Out
of scope (separate work): the Tasks extension, Skills over MCP,
`subscriptions/listen`, progress and cancellation, icons,
enterprise-managed authorization.

## What Changed

- `mcp-http.ts`: `negotiateMcpProtocolVersion` probes with
`server/discover`. A `DiscoverResult` or a modern error
(`-32020`/`-32021`/`-32022`) means a modern server. Anything else means
a legacy server, which gets `initialize`. A `401`/`403` keeps the
challenge for OAuth discovery. The era is remembered per connection and
endpoint.
- `mcpHttpRequestInit` builds every request. It always sends
`Mcp-Method` and `Mcp-Name` (Base64 sentinel when unsafe). For modern
versions it adds `_meta` (`protocolVersion`, `clientCapabilities` with
`extensions["io.modelcontextprotocol/ui"]` and URL elicitation,
`clientInfo`) and `Mcp-Param-*` headers from `x-mcp-header`.
- Legacy `initialize` now requests 2025-11-25 and advertises the MCP
Apps extension.
- Catalog refresh saves `config.mcpProtocolVersion`. It drops tools with
invalid `x-mcp-header` annotations. It reuses a fresh `tools/list`
result (`ttlMs`, `cacheScope`). It probes again when the saved version
is refused, and it negotiates again after a credential refresh.
- `tools/call`: a saved modern version is called directly (no probe, no
handshake). A modern rejection of a legacy-shaped call, or `-32022`,
gets one retry with a negotiated version. The server rejected the first
request before it ran, so the retry cannot repeat a side effect. A
remembered legacy server that refuses `initialize` is probed again.
- MRTR: `input_required` with URL requests becomes the existing provider
link flow (`provider_interaction_required`). The next identical call
sends `requestState` and `inputResponses`. Form, sampling and roots
requests stop with `provider_input_required` and a plain message. A
missing `resultType` means `complete`.
- Context reads (`resources/*`, `prompts/*`) use the saved version,
cache fresh results, and stop on `input_required`. The error code is
kept, so resource not found (`-32602` or `-32002`) is visible.
- stdio: probe with `server/discover` (2 s limit). Restart a server that
exits on the probe. Send `_meta` to modern servers.
- OAuth: refuse a callback without `iss` when the server advertises
`authorization_response_iss_parameter_supported`. Check `iss` before
acting on a provider error. Send `application_type: "native"` for
localhost callbacks and `web` otherwise. Issuer-bound clients and the
CIMD-first order already existed.
- Docs: `doc/connections/MCP-PROTOCOL-VERSIONS.md`, linked from
`doc/connections/README.md`.

## Verification

- New suite `server/src/__tests__/mcp-protocol-versions.test.ts` (19
tests) uses a fake 2026-07-28 server and a fake legacy session server.
It covers discovery, `_meta`, headers, version fallback, `-32022`,
`resultType`, MRTR link and retry, form stop, cache TTL, legacy flow,
and recovery when a saved legacy server moves to 2026-07-28. Embedded
Postgres runs the end-to-end cases.
- New OAuth tests in `generic-mcp-connection.test.ts`: `iss` required
when advertised, `iss` checked on provider errors, `native` for
localhost, and a stateless server listed again after a revoked token.
- Red/green: with the master service code, all 21 new protocol and OAuth
tests fail. With this change they pass. The token-refresh regression
test fails without its fix and passes with it.
- Existing suites: 12 touched files, 1088 tests pass. `tsc --noEmit`
passes for `server`. Updated expectations count the one-time
`server/discover` probe, and the Calendly DCR test now expects `native`
for its localhost callback.
- `pnpm verify` exits 2 (a doc is a global CI input), so CI supplies the
broad coverage.
- Live check with the changed client, without credentials:
- DeepWiki (`mcp.deepwiki.com/mcp`): discover 200 → 2026-07-28, no
session, `tools/list` 200 (3 tools).
- Context7 (`mcp.context7.com/mcp`): discover 200 → 2026-07-28, no
session, `tools/list` 200 (2 tools).
- GitMCP (`gitmcp.io/docs`): discover 400 → `initialize` (2025-03-26) →
session → `tools/list` 200 (5 tools).
- Circleback (`circleback.ai/api/mcp`): discover 401 with its
`WWW-Authenticate` challenge kept for sign-in.

## Risks

- A legacy server that is not saved as session-required gets one extra
request (`server/discover`) the first time it is refreshed. Then the
version is saved. Saved session-required connections are not probed.
- DCR from a localhost callback now registers a `native` client. This is
what the spec asks for. Existing registrations are reused.
- An authorization server that advertises `iss` support but does not
send `iss` now fails sign-in with a clear message. The spec requires
this.
- MRTR answers are kept in memory for 30 minutes. After a restart the
server asks again, and the user sees the link again.
- No schema change. `config.mcpProtocolVersion` is a new key in the
existing JSON config.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits, web fetch of the MCP spec), extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 14:34:26 -06:00
DottaandClaude Fable 5.2 31c3412bbb perf(db): build tx.query on first read instead of for every transaction (#21148)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Every server module reaches PostgreSQL through one client,
`createDb` in `packages/db`. The server has about 1,260 `transaction(`
call sites. Run events, runner checkpoints and the run queue use a
transaction for each write.
> - Drizzle builds one relational query builder for every schema table
when it creates a database object. It creates a database object for each
transaction and for each savepoint.
> - The Paperclip schema has 321 tables. One `BEGIN` therefore allocates
about 100 KB and uses about 80 microseconds of CPU before the first
statement. One `SAVEPOINT` costs the same again. Five places read the
relational API inside a transaction.
> - On a hosted instance the server main thread was near one full core,
and garbage collection used a quarter of it. A fixed 100 KB for each
transaction is garbage that no caller asked for, and it grows with every
new table.
> - This pull request makes `createDb` build `tx.query` the first time a
transaction reads it.
> - The benefit is 82% less allocation and 7 times fewer garbage
collections for an empty transaction, and 45% less allocation for one
native run event. Transaction behavior does not change.

## Linked Issues or Issue Description

Related: #21131 (one transaction of two statements for a native run
event). That pull request reported this cost as a follow-up. No issue
exists for it.

**What existing behavior does this improve?**
The cost of `db.transaction()` and of a nested `tx.transaction()`
(savepoint) on the client that `createDb` returns.

**Subsystem affected**
Database client (`packages/db`).

**Current behavior**
`createDb` calls `drizzle(sql, { schema })`. Drizzle's `PgDatabase`
constructor then runs this loop for the root client, for every
transaction and for every savepoint:

```js
for (const [tableName, columns] of Object.entries(this._.schema)) {
  this.query[tableName] = new RelationalQueryBuilder(...);
}
```

A transaction is a `PgTransaction`, which extends `PgDatabase`
(`drizzle-orm/pg-core/session.js`). `PostgresJsSession.transaction` and
`PostgresJsTransaction.transaction` construct one for each `BEGIN` and
each `SAVEPOINT` (`drizzle-orm/postgres-js/session.js`).

Measured on `master` at 4e932fd178, Node 26, local PostgreSQL, 3,000
iterations, median of 5 passes:

| Scenario | Allocated | CPU | Collections per 1,000 |
| --- | --- | --- | --- |
| Statement outside a transaction | 11.5 KB | 27 µs | 0.3 |
| Empty transaction | 125.8 KB | 131 µs | 5.0 |
| Transaction with one statement | 136.6 KB | 153 µs | 5.3 |
| Transaction with an empty savepoint | 231.6 KB | 235 µs | 9.7 |
| Empty transaction, Drizzle without a schema | 18.5 KB | 53 µs | 0.7 |

The allocation profile of an empty transaction attributes 96.8 KB of
115.6 KB sampled to the `PgDatabase` constructor (`pg-core/db.js:17`),
reached from `PgTransaction` and `postgres-js/session.js:109`. About 50
KB is the builders and the `query` object. About 46 KB is
`Object.entries` of the schema.

The relational API is in use, always as a single-table `findFirst({
where[, columns] })`. It has 22 call sites. Five of them run on a
transaction: `services/announcements.ts:19` and `:25`,
`services/decision-training.ts:331`, and
`services/resource-memberships.ts:361` and `:369` (the service is built
on a transaction in `updateAgent` and in
`services/primary-agent.ts:60`). The other 17 run on the root client.

**Proposed behavior**
A transaction or savepoint that does not read `tx.query` builds no
relational query builder. A transaction that reads `tx.query` gets the
same builders as before, bound to that transaction.

**Reason and benefit**
Less garbage and less CPU on the server main thread for every write
path. Hosted statement statistics showed 13.4 native run events per
second and 5.3 checkpoint inserts per second. Each is one transaction,
so those two paths alone discarded about 2 MB per second. The total
transaction rate of the hosted instance was not measured.

**Breaking changes**
None. No schema change, no API change, and the `Db` type is identical.

## What Changed

- `packages/db/src/lazy-transaction-queries.ts` (new):
`drizzleWithLazyTransactionQueries(client, schema)`.
- It creates the root client with the schema, as before. `db.query` and
`db._` do not change.
- It creates a second Drizzle client on the same postgres.js client,
without the schema, and `db.transaction` opens transactions through it.
`BEGIN`, `SAVEPOINT`, commit, rollback and connection reservation are
still Drizzle's and the driver's own code. No Drizzle internals are
patched.
- Each transaction and savepoint gets a `query` accessor. The first read
builds one `RelationalQueryBuilder` for each table from the root
client's schema description and the transaction's session, then stores
the result as a plain property. `tx._.schema`, `tx._.fullSchema` and
`tx._.tableNamesMap` keep their values.
- `packages/db/src/client.ts`: `createDbClient` calls the new function.
The work-signal and response-capture wrappers install on top of it in
the same order as before.
- `packages/db/scripts/benchmark-transaction-allocation.ts` (new): the
benchmark in this description.
- `doc/DATABASE.md`: new section "Relational queries inside
transactions".

## Verification

Benchmark, same script before and after (`createDb` client):

```sh
cd packages/db
TSX_DISABLE_CACHE=1 node --expose-gc --import tsx scripts/benchmark-transaction-allocation.ts 3000 5
```

| Scenario | Allocated before | Allocated after | CPU before | CPU after
| Collections per 1,000 before | after |
| --- | --- | --- | --- | --- | --- | --- |
| Statement outside a transaction | 11.5 KB | 11.3 KB | 27 µs | 31 µs |
0.3 | 0.3 |
| Empty transaction | 125.8 KB | 22.4 KB | 131 µs | 54 µs | 5.0 | 0.7 |
| Transaction with one statement | 136.6 KB | 35.6 KB | 153 µs | 82 µs |
5.3 | 1.0 |
| Transaction with an empty savepoint | 231.6 KB | 35.6 KB | 235 µs |
111 µs | 9.7 | 1.3 |
| Transaction with a savepoint of one statement | 242.7 KB | 44.8 KB |
331 µs | 217 µs | 10.7 | 2.0 |

After the change, no Drizzle frame is in the allocation profile of an
empty transaction (19.8 KB sampled). The script also runs Drizzle with
the schema and Drizzle without a schema in the same process, so the
comparison can be repeated on any later revision.

The machine was shared and under load (load average 6 to 35 on 18 CPUs).
Allocation and collection counts were stable across four runs. CPU
values moved by a factor of two between runs. The CPU columns come from
the two runs with the lowest load.

One native run event, through the real append path:

```sh
TSX_DISABLE_CACHE=1 EVENTS=1500 PASSES=3 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-run-event-append.ts
```

`native-stream`: 220.4 -> 120.9 KB allocated for each event, 12.0 -> 5.3
collection pauses per 1,000 events. `native-burst`: 221.1 -> 121.0 KB,
10.6 -> 5.3 pauses. Statements for each event do not change (4.6).

Tests, all on real PostgreSQL
(`packages/db/src/lazy-transaction-queries.test.ts`, 15 tests):

- Red and green for the mechanism. On `master` these fail, and with the
change they pass:
- `builds none for a transaction or savepoint that never reads
tx.query`: 321 builders for the transaction and 321 for the savepoint on
`master`, 0 with the change.
- `allocates the same for an empty transaction as a client without the
schema`: 101 KB more than a schema-less client on `master`, below the 16
KB limit with the change. It reads V8's allocation counter, not a timer,
and it skips on a Node.js without that counter.
- Semantics. These pass on `master` and with the change, which shows the
behavior is the same:
  - `commits and returns the callback's value`
- `rolls back and rejects with the thrown error` (thrown error,
`tx.rollback()`, SQL error)
- `keeps the outer transaction usable after a savepoint rolls back`
(three levels)
- `discards a released savepoint when the outer transaction rolls back`
  - `applies the isolation level and access mode`
- `runs concurrent transactions on separate reserved connections and
releases them` (12 transactions on a pool of 3)
- `returns the only connection to the pool after a commit, a rollback
and a failed savepoint` (pool of 1)
  - `behaves the same with prepared statements enabled`
- Instrumentation. These pass on `master` and with the change:
- `registers each transaction and savepoint with the client's work
signals`: the transaction and the savepoint have the client's
work-signal scope, the registration statement runs on the transaction's
own connection, listeners get `intent` at once and `settled` only after
the outer commit, and a rollback reconciles. When transactions bypass
the work-signal wrapper, this test and five tests in
`work-signals.test.ts` fail.
- `response capture counts every statement of a transaction, a savepoint
and tx.query`: six statements, counted for each operation and table,
including two relational reads.
- Every `tx.query` pattern in the codebase:
- `reads tx.query through the transaction, as the announcement and
decision-training services do`
  - `reads tx.query inside a savepoint and follows its rollback`
- `gives tx.query one builder per schema table, the same object on every
read`

Commands run locally on the final revision:

- `pnpm exec vitest run src/lazy-transaction-queries.test.ts
src/work-signals.test.ts src/response-capture-postgres.test.ts` in
`packages/db` on the final revision: 43 passed.
`prepared-queries-postgres.test.ts` passed on the first commit.
- Before the rebase onto current `master`, also
`response-capture.test.ts`, `response-capture-bundle.test.ts`,
`client-options.test.ts`, `client-teardown-registry.test.ts`,
`checkpoint-pool.test.ts`, `client-disconnect-replay.test.ts`: 111
passed in 10 files.
- On the first commit, server callers of `tx.query` and `db.query`, and
hot transaction paths: `announcements-routes`, `decision-training`,
`heartbeat-run-event-atomic-append`, `heartbeat-run-event-sequencing`,
`primary-agent`, `resource-memberships-routes`, `workspace-preferences`,
`sidebar-preferences-routes`, `paperclip-control-plane-port`,
`native-runner-checkpoint-authority.integration`, `dispatch-work`,
`decision-target-work`, `issue-thread-interactions-service`: all passed.
`workspace-preferences` timed out once in its database start hook while
eight suites ran in parallel, and passed alone.
- `pnpm exec tsc --noEmit` in `packages/db`: passed. A type probe showed
that `Db` is the same type as before.
- `pnpm check:module-boundaries`: passed.
- `pnpm verify` exits 2: `packages/db` is a broad input. CI supplies the
full suites. I did not run the full local suite, repo-wide typecheck or
build.

Not verified: the hosted transaction rate, and the effect on a hosted
instance. CI uses Node 24, which has no V8 allocation counter. On CI the
allocation test skips itself, and the builder-count test covers the
mechanism. The benchmark needs Node.js 25 or later and stops with that
message on Node 24. I did not run that error path, because this machine
has Node 26 only.

## Risks

- Every module uses this client. The mitigation is that the transaction
path is still Drizzle's own `transaction()` on the same postgres.js
client. Only the schema argument is absent. The semantics tests pass
unchanged on `master` and on this branch.
- The change relies on Drizzle surface that is not documented API. The
lazy `tx.query` calls the `RelationalQueryBuilder` constructor from
`drizzle-orm/pg-core/query-builders/query`. That subpath is in Drizzle's
export map and the constructor is typed, but `drizzle-orm/pg-core` does
not re-export it. The change also writes the three schema fields of
`tx._`, which the types mark `readonly`. In `drizzle-orm@1.0.0-beta.22`
the same path exports a different class. An upgrade that changes the
constructor fails the typecheck and the `tx.query` tests. An upgrade
that renames `db._.schema` makes the function return the plain eager
client: behavior stays correct, the saving is lost, and the
builder-count test fails. The repository owner decides whether this
dependency on Drizzle is acceptable. A version that uses only documented
API removes the 22 relational reads from the server and creates the
client without the schema.
- `tx.query` is now an accessor until its first read. Code that reads
the property descriptor of `query` sees the difference. No code in the
repository does that.
- The second client and the builders use Drizzle's default dialect
options, as the root client does. If `createDb` ever passes `casing`,
`logger` or `cache` to Drizzle, the same option must go to all three.
The source comment says so.
- A transaction that reads `tx.query` pays the old cost once, on first
read.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code, with tool use
and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 14:31:21 -06:00
Nicky LeachandPaperclip 7f69ef8b44 Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path

> - Paperclip lets plugins own managed Git workspaces.
> - A portable Runner task can move to a different physical allocation.
> - The checkout must be present on the allocation that launches the
Runner.
> - Workspace preparation and finalization need exact placement when
they retry.
> - This change checks the placement at Runner admission and records it
before finalization.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Workspace preparation and finalization for portable Runner tasks.

**Current behavior**
A task replacement can change the allocation after Git preparation. A
lost finalization reply can leave retry without the original placement.

**Proposed behavior**
Prepare the checkout on the actual allocation before admission. Record
finalization placement so an exact retry can use it.

Portable ownership and controller recovery are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066).

## What Changed

- Reserve the managed workspace before acquiring the portable task
lease.
- Pass the task placement to the workspace provider. Reuse the described
task while placement stays the same and prepare again before admitting a
replacement.
- Record the initial preparation's run, agent, generation, and optional
task placement before dispatch. Inspection and retries on the same
Launcher replay the same request; VM and sandbox preparation use the
same recovery path.
- Keep provider request identity separate from Git credential authority:
reservation reconciliation uses the existing project-consent path in a
host-only scope bound to the plugin and workspace. Foreground
preparation still requires an active run; consent, repository, and
revocation checks are unchanged.
- Acquire the current Launcher before retrying preparation. If placement
changes, use ordinary preparation on the replacement so the provider can
settle the prior writer and recover the retained checkout. No additional
provider discovery call is added.
- Select finalization placement before publishing the operation record,
then dispatch and recover using that saved choice. Retain the Launcher
during bounded finalization retries.
- Preserve interrupted pre-upgrade finalization requests without
placement, including their original operation IDs and provider routing.
No migration or new durable mode flag is required.

## Verification

- Final revision `74ea3e6712`: server typecheck and exact SDK source-pin
verification (538 inputs) passed.
- All 185 tests passed across the full plugin worker manager, project
Git authorization, and plugin workspace integration suites (including 56
workspace tests). Independent verification reran the three lost-reply
cases: inspection, same-Launcher retry, and replacement after deleting
the original Launcher. The tests assert the correct request identity,
durable intent, and clearing that intent after success.
- Before the final retry-accounting fix, all 323 tests passed across
workspace admission, runtime execution, native workspace finalization,
plugin workspace integration, and terminal cleanup retry. Regressions
exercise crashes after operation insertion but before dispatch in both
finalization paths, exact replay after lease release, and retry after
ambiguous placement clears without publishing an incomplete operation.
Pre-upgrade interrupted requests retain their operation IDs despite an
available portable lease.
- Final retry-accounting fix: all 84 native workspace finalizer,
failure, and supersession tests passed. A real missing-plugin regression
first failed on the prior commit, then verified healthy-run progress,
three-attempt exhaustion, terminal cleanup, and absence of a misleading
workspace operation. Independent verification also passed the legacy and
crash-before-dispatch cases.
- Worker-process tests cover both preparation RPC methods: the provider
still receives the original run and agent, maintenance receives
project-scoped credential authority, and foreground or mismatched-scope
requests remain denied. Database tests confirm a stopped run cannot
authorize foreground Git access while valid project consent can
authorize maintenance.
- Existing metadata protection coverage proves callers cannot overwrite
the private preparation intent or read it through workspace responses.
- Two existing Launcher Git tests passed: retry preserves dirty files
and the original base commit without fetching, and retry resumes the
saved checkout intent while rejecting a changed pending branch.
- Implementation was delegated to GPT-6.1 Sol and independently
reviewed. The GitHub stack's automatic rebase preserved the original
workspace-placement patch.

- [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706)
passed on `74ea3e6712`, including server and workspace tests, Runner,
end-to-end tests, typecheck, build, and canary dry run. Ordinary
Greptile and
[Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388)
both scored `74ea3e6712` 5/5, with no unresolved findings.

## Risks

- Live Daytona end-to-end qualification is pending. The checkout record
persists on the mounted volume; replacement preparation relies on the
provider confirming the previous writer is settled before using that
retained checkout. Local tests cover Core request routing and Launcher's
durable checkout recovery separately.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 12:12:41 -08:00
DottaandClaude Fable 5.2 eaef46b60b feat(ui): browser page-load tracing to Honeycomb for hosted instances (#21156)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI is the main surface. Its load speed is a product
requirement.
> - The team wants to track page-load times over time and give each
metric an owner and an SLO in Honeycomb.
> - Server traces show only the server part of a page load. They do not
show JavaScript start-up, lazy route code, rendering, or client-side
navigation.
> - Paperclip is also self-hosted by others. Browser tracing must stay
off there, and it must not slow any page.
> - This pull request adds real-user browser tracing for hosted
Paperclip Cloud instances only. It loads lazily and sends a closed set
of attributes.
> - The benefit is one `page.load` measurement per page (named by route
pattern), plus Web Vitals and API spans that join the server trace,
ready for Honeycomb queries and SLOs.

## Linked Issues or Issue Description

No public issue exists. Feature description:

**Subsystem affected**

Board UI (`ui/`), the authenticated session route
(`server/src/routes/auth.ts`), and the shared session schema.

**Problem or motivation**

We cannot measure how long the all-tasks page or a task page takes to
load for real users. Server traces stop at the API response. Most
navigation is client side, and the standard OpenTelemetry document-load
instrumentation does not cover it. Without a per-page browser
measurement, no agent can own a page-load SLO.

**Proposed solution**

The server sends a `browserTracing` config in the authenticated session
only when the instance is cloud managed and
`PAPERCLIP_BROWSER_TRACES_ENDPOINT` is set. The browser then measures
every hard load and client navigation in one shared recorder tied to the
router's own route tree. After the page settles, it lazily loads an
OpenTelemetry chunk that exports `page.load`, `web_vital.*`, and
same-origin `/api/` fetch spans to Honeycomb. An allowlist removes every
other attribute before export.

**Alternatives considered**

`@honeycombio/opentelemetry-web` was the first choice. Its defaults add
the full page URL, query string, hash, referrer, and user agent to every
span and to the resource, and it bundles log and metric pipelines. We
would have to replace its exporter and remove most of its defaults.
Plain OpenTelemetry web packages plus `web-vitals` are smaller and send
only what we choose. Honeycomb needs only OTLP/HTTP and its ingest-key
header.

**Roadmap alignment**

This is internal hosted-instance observability. It changes no product
behavior and no self-hosted behavior.

## What Changed

- `server/src/browser-tracing-config.ts`: resolves the browser tracing
config. It returns `null` unless `isCloudManagedInstance()` is true and
`PAPERCLIP_BROWSER_TRACES_ENDPOINT` is set. It accepts only a Honeycomb
ingest key (`hcaik_` prefix), requires `https` (except loopback), and
fails closed with one value-free warning on any invalid setting.
Optional: `PAPERCLIP_BROWSER_TRACES_INGEST_KEY`, `_SAMPLE_RATE` (default
1), `_SERVICE_NAME` (default `paperclip-web`), `_ENVIRONMENT` (default
`SENTRY_ENVIRONMENT`).
- `GET /api/auth/get-session` adds `browserTracing` (config or `null`),
the same delivery path as the front-end Sentry DSN. The shared
`authSessionSchema` has the new optional field
`browserTracingConfigSchema`.
- `ui/src/lib/browser-tracing/page-load-recorder.ts` and `controller.ts`
(main bundle, no OpenTelemetry imports): one `page.load` measurement per
hard load (from navigation start) and per client navigation (from the
history change). A measurement ends at the first painted frame where no
full-page loading state is mounted and either every page-declared
content signal is ready or no query without data is fetching. Redirects
continue the measurement. A new navigation interrupts it. 30 s times
out. The sample decision is made once per page session before any
download.
- `ui/src/components/PageLoadTracker.tsx`: `App` passes the same
`<Route>` tree it renders, so every page is reported by route pattern
(for example `/:companyPrefix/issues/:issueId`) with no per-page code.
- `usePageContentReady` hook and blocking gates: `PaperclipLoading`
(also the route `Suspense` fallback) blocks readiness. The task page
uses the existing `issue-detail:navigate→content-paint` point. The plain
all-tasks list signals when its first rows render.
- `ui/src/lib/browser-tracing/sdk.ts` (lazy chunk, loaded after the page
settles): OpenTelemetry web tracer, OTLP/HTTP exporter, fetch
instrumentation for this origin's `/api/` only with `traceparent` sent
only to this origin, `web-vitals` LCP/INP/CLS, and a context manager
that makes the open `page.load` span the parent of its API calls.
`span-sanitizer.ts` applies the closed attribute allowlist (no URLs,
ids, query strings, status messages, or event attributes). Uncaught
errors are not sent because Sentry already reports them.
- New `ui` dev dependencies, pinned exactly. Each is needed, and the
OpenTelemetry ones are the same versions the server already resolves:
- `@opentelemetry/api@1.9.1` (already in the workspace): tracer and
context API.
- `@opentelemetry/sdk-trace-web@2.10.0`: the browser tracer provider,
batch processor, and stack context manager.
- `@opentelemetry/instrumentation-fetch@0.221.0`: fetch spans and
`traceparent` injection.
- `@opentelemetry/resources@2.10.0` (already a server dependency): the
resource.
- `@opentelemetry/otlp-transformer@0.221.0` (already a server
dependency): OTLP JSON serialization for a 60-line `OtlpFetchExporter`.
This replaces `@opentelemetry/exporter-trace-otlp-http`, which is a
server optional peer: installing it in the workspace made the server's
exact-version gate test see it as present.
  - `web-vitals@6.2.3`: LCP, INP, and CLS.
All of them are in the lazy chunk only. The lockfile is not committed:
`pr-trusted` rejects lockfile edits in pull requests, CI installs with
`--no-frozen-lockfile` after the frozen attempt reports
`ERR_PNPM_OUTDATED_LOCKFILE`, and the lockfile bot refreshes it after
merge.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the three shared files (the session schema is part of the pinned public
SDK surface; the contract must live in `packages/shared`).
- Docs: new `doc/browser-performance-tracing.md` (enablement, config,
key handling, span and attribute list, page-load definitions, example
Honeycomb queries and an SLO SLI). Pointers in `doc/observability.md`
and the Honeycomb tracing skill.
- Two App routing tests mock `PageLoadTracker` because they render `App`
without a `QueryClientProvider`.

## Verification

- `pnpm verify` returns exit 2 (broader verification required) because
docs are global CI inputs. Required CI supplies the broad coverage.
- Focused local tests, all pass:
- `ui`: `vitest run src/lib/browser-tracing
src/components/PageLoadTracker.test.tsx
src/components/SentryGate.test.tsx src/api/auth.test.ts
src/lib/issue-detail-performance src/pages/IssueDetail src/pages/Issues
src/App src/components/CloudAccessGate.test.tsx
src/components/IssuesList`: 17 files, 484 tests.
- `server`: `browser-tracing-config.test.ts`, `auth-routes.test.ts`,
`sentry.test.ts`: 63 passed, 5 skipped.
  - `packages/shared`: `validators/access.test.ts`: 19 passed.
- New tests prove: the SDK module is never imported when the session
config is `null`, absent, or signed out, or when the session is not
sampled; route templates contain no ids, slugs, or tokens; a client
navigation produces a `client` measurement under
`/:companyPrefix/issues/:issueId`; `traceparent` goes only to this
origin's `/api/` and not to a third-party or non-API URL; exported spans
carry only allowlisted attributes. A mutation check (tracker forced on)
turned the "never imports" tests red.
- Typecheck: `ui` (`pnpm typecheck`), `server` (`pnpm --filter
@paperclipai/server typecheck`), and `packages/shared` pass.
- Real-browser run: a local dev server with a local OTLP collector and
headless Chromium. The self-hosted session returned `browserTracing:
null` with the endpoint set, and the page made 0 SDK and 0 collector
requests. With a cloud session config injected into the session
response, the collector received a hard `page.load` (route
`/:companyPrefix/issues`, `ready_signal=idle`), a client `page.load` to
`/:companyPrefix/issues/:issueId` (`ready_signal=content`), a back
navigation, LCP/INP/CLS spans, and `/api/` fetch spans parented to the
open `page.load` span. No task identifier, company id, task title, host,
or URL appeared in the exported payload. No third-party request carried
`traceparent`.
- Bundle size (`ui` production build, initial JavaScript closure of
`index.html`): before 3,813,986 B raw / 1,153,991 B gzip; after
3,820,743 B / 1,156,590 B (+6.8 KB raw, +2.6 KB gzip, +0.2%). The lazy
SDK chunk is 84 KB raw / 27 KB gzip and is not fetched unless tracing is
on and sampled.
- Review fixes (each red on the old code, green on the new): the SDK now
loads by a 35 s deadline even if the page never settles, so a timed-out
hard load is exported (`controller.test.ts`); an agent chat with no
saved task uses the idle rule instead of a content gate that never fires
(`usePageContentReady.test.tsx`); the sample rate is rounded to exactly
1/N so Honeycomb's `SampleRate` weight matches
(`browser-tracing-config.test.ts`). The real-browser run was repeated
after the exporter change with the same result.

## Risks

- Low risk for self-hosted installs: the server never sends a config, so
no tracing code beyond the small recorder runs, and the SDK chunk is
never fetched.
- On hosted instances the feature is off until an operator sets
`PAPERCLIP_BROWSER_TRACES_ENDPOINT` and an ingest key. When on, the
controller wraps `history.pushState`/`replaceState` to time navigation
starts and restores them on sign-out.
- Every signed-in browser can read the ingest key, like the Sentry DSN.
Only Honeycomb ingest keys (send-only) are accepted. A collector
endpoint can keep the key server-side.
- Hard-load API calls happen before the SDK loads, so they have no fetch
spans. The hard-load `page.load` span is still recorded.
- The browser-to-server trace join needs server tracing. The hosted
image does not install the OpenTelemetry server peers today
(`CLOUD_BUNDLED_SERVER_DEPS` is `@sentry/node` only). Browser spans work
without it.
- `ui/src/pages/IssueDetail.tsx` was already over 1,000 lines. This
change adds 5 lines there for the task page content signal, next to the
existing content-paint measure.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use
(shell, file edits, local browser automation). Reasoning effort: low.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 13:48:35 -06:00
DottaandClaude Fable 5.2 8990e94005 fix(ci): keep community PR runs from delaying CODEOWNERS jobs in the hosted queue (#21166)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Its pull request CI runs on two pools: an AWS fleet for CODEOWNERS,
and GitHub-hosted runners for every other contributor
> - Some short jobs for CODEOWNERS pull requests also run on
GitHub-hosted runners: the commitperclip review and the merge review
notifier
> - GitHub starts about 550 hosted jobs each hour for this repository,
and all hosted jobs wait in one first-come queue
> - A community pull request run puts 52 jobs in that queue at one time,
so the short CODEOWNERS jobs wait behind them
> - This pull request limits how many jobs one hosted run can queue, and
lets the short CODEOWNERS jobs use the reserved AWS metadata fleet
> - The benefit is that community pull requests do not delay CODEOWNERS
work

## Linked Issues or Issue Description

**What happened?**

On 2026-11-12 the repository had about 348 queued workflow runs. Jobs on
the AWS fleet started in a median of 29 seconds. Jobs on `ubuntu-latest`
did not:

| Job | Median wait | Job length |
| --- | --- | --- |
| PR merge review notification | 42 min | 9 s |
| commitperclip PR Review | 36 min | 15 s |
| Community PR test jobs | 57 min (p90 76 min) | 7 min |

Between 12:00Z and 18:00Z, hosted job starts stayed between 492 and 567
each hour while the number of running hosted jobs changed from 63 to
134. The start rate is the limit. Community PR runs supplied 3,462 of
the 3,744 hosted jobs in that window (92%).

**Expected behavior**

A short job for a CODEOWNERS pull request starts in about one minute,
whatever the community pull request load is.

**Steps to reproduce**

1. Open several pull requests from forks in a short time.
2. Open a pull request as a CODEOWNER.
3. Look at the start time of `commitperclip PR Review` and `PR merge
review notification` for the CODEOWNER pull request.

## What Changed

- `pr-trusted.yml`: each of the four test matrices has `max-parallel: 4`
when the run uses `ubuntu-latest`. The run has the same jobs. The other
jobs wait in the run, not in the shared queue. AWS runs have no limit
(256).
- `commitperclip-review.yml` and `pr-merge-review.yml`: the job uses the
reserved `paperclip-ci-routing-x64` fleet when
`AWS_CI_SHORT_JOBS_ENABLED` is `true`, the pull request branch is in
this repository, and the author and the sender (or the run actor and the
triggering actor) are in the AWS admission list. The admission variables
are the same as in `ci-routing.yml`. All other events use
`ubuntu-latest`.
- Check-run and commit-status notifications stay on `ubuntu-latest`.
Those events do not identify the author.
- `ci-routing.test.mjs`: 24 placement cases for the two jobs, and a
contract test for the matrix limit.
- `doc/DEVELOPING.md`: the two settings and the runner group step.

## Verification

- `node --test .github/scripts/tests/ci-routing.test.mjs
.github/scripts/tests/notify-pr-merge-review.test.mjs
.github/scripts/tests/ci-plan.test.mjs
.github/scripts/tests/ci-profile.test.mjs
.github/scripts/tests/run-quality-gates.test.mjs`: 200 tests pass.
- `actionlint` accepts the three workflows, including the `max-parallel`
expression.
- Not verified before merge: `pr.yml` calls `pr-trusted.yml@master`, and
the two short workflows run from master. The effect on queue time can
only be measured after merge.

## Risks

- A community pull request run takes longer when the hosted queue is
empty: about 45 minutes for the server matrix, compared with about 13
minutes. Today these runs take 150 to 200 minutes because of the queue.
- The AWS routing does nothing until a maintainer sets
`AWS_CI_SHORT_JOBS_ENABLED`. Before that, the `paperclip-ci-routing`
runner group must allow the two workflow references in
`doc/DEVELOPING.md`. If the variable is set first, the two jobs for
CODEOWNERS pull requests stay queued.
- The two jobs run only scripts from master and no pull request code.
The secrets they read do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 13:48:25 -06:00
DottaandClaude Opus 5.5 8ff45c0fa9 feat(connections): render MCP Apps views in the task chat, starting with Circleback (#21119)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents reach external apps through MCP connections. Paperclip
records each call and shows it as a tool row in the task chat.
> - MCP Apps (SEP-1865, extension `io.modelcontextprotocol/ui`, spec
2026-01-26) lets an MCP server attach an interactive `ui://` view to a
tool. Claude, Claude Code and VS Code render these views in chat.
Circleback uses them to show meeting cards.
> - Paperclip dropped the tool's `_meta.ui` at discovery and the
result's `_meta` at call time. Its gateway reported MCP App UI as
"unsupported". Circleback was reachable only through Composio, whose
toolkit has no `Display*` tools.
> - This pull request makes Paperclip an MCP Apps host. It also adds
Circleback as a native catalog connection.
> - The benefit is that a Circleback recap in a task shows the same
meeting cards as in Claude, and any other MCP server with views works
the same way.

## Linked Issues or Issue Description

**Subsystem affected**
Connections and apps (remote MCP gateway) and the task chat.

**Problem or motivation**
An agent that uses Circleback in Paperclip can only return text. The
same tools in Claude Code show interactive meeting cards
(`DisplayMeetings`). Paperclip discards the MCP Apps metadata, so it
cannot show them.

**Proposed solution**
Paperclip implements the host side of MCP Apps end to end. It keeps the
tool's view declaration and records each call that has a view. It
renders the view under the tool row in a sandboxed iframe that uses the
MCP Apps postMessage protocol. Circleback becomes a native catalog
connection, so its `Display*` tools are available.

**Alternatives considered**
Composio's `circleback_mcp` toolkit has 11 read tools and no `Display*`
tools, and its meta-tools return plain JSON, so it cannot carry views.
The spec's double-iframe sandbox proxy on a separate origin needs a
second origin that Paperclip does not have. A single opaque-origin
`srcdoc` frame with an injected CSP gives the same isolation.

**Roadmap alignment**
This extends the existing Apps catalog and MCP gateway. It adds no new
integration substrate (decision D4 in doc/connections/README.md).

**Additional context**
The standard, the design and the testing plan are in
`doc/connections/MCP-APPS.md`.

## What Changed

- **Shared:** `packages/shared/src/mcp-apps.ts` adds MCP Apps types,
`_meta.ui` and resource parsing (current and legacy keys), and a CSP
builder that keeps only well-formed `https` origins.
- **DB:** migration `0471` (`online_compatible`) adds
`tool_catalog_entries.ui_meta` and a new `tool_mcp_app_views` table.
- **Server, catalog:** discovery keeps each tool's `_meta.ui`. It is not
part of the version hash, so a view never quarantines a reviewed tool.
- **Server, gateway:**
- Paperclip advertises the MCP Apps client capability when it
initializes a session.
  - App-only tools (`visibility: ["app"]`) are hidden from agents.
- The result `_meta` is kept for the view and is not sent to the agent.
- Each successful call to a tool with a view records a view row.
Recording is best effort.
- The gateway's `initialize` now reports `paperclip/mcp-app-ui:
host_rendered`.
- The context protocol (`resources/read`) now uses the injected test
transport, like the other remote calls.
- **Server, routes:** `server/src/routes/mcp-apps.ts`
- `GET /api/issues/:issueId/mcp-app-views` lists a task's views. It
needs task read access.
- `GET /api/issues/:issueId/mcp-app-views/:viewId` returns the view
document, read live through the agent's connection.
- `POST /api/issues/:issueId/mcp-app-views/:viewId/tool-calls` runs the
view's `tools/call`. It needs `tools:use` and reaches only tools of the
same connection that allow `app`. It uses the connection test-call path,
so policy, Ask first and audit apply.
- **UI:** `ui/src/components/mcp-app/`
- The host protocol: `ui/initialize` with theme, style variables and
size limits; tool-input; tool-result; size-changed; `tools/call`;
`ui/open-link` (http and https only); teardown.
- An opaque-origin iframe (`allow-scripts`, never `allow-same-origin`)
with the CSP placed before the view's markup.
- The views of a run render with that run's reply, in the live turn and
in settled history. They are keyed by the reply's run, so history needs
no transcript.
- **Circleback:** a native catalog app (`mcp-oauth`, DCR, `user
user:write`) with:
  - the official logo;
- a reviewed risk classifier (Share* tools are writes, Delete* tools are
destructive, Display* tools are reads);
  - warnings in the sign-in step;
  - `doc/connections/CIRCLEBACK.md`.
  
Search now prefers a native app over an aggregator listing named "X
MCP".
- **Storybook:** `Tasks/MCP App views` shows the tool row with its view,
the ready state (Refresh calls a tool), loading, view unavailable, and a
tool call that needs approval.

## Verification

- `pnpm exec vitest run packages/shared/src/mcp-apps.test.ts
ui/src/components/mcp-app/`: 19 passed.
- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts`: 3
passed, on embedded Postgres. The test connects Circleback through the
catalog with DCR OAuth against a fake Circleback MCP server. It checks:
  - `ui_meta` is stored;
  - an app-only tool is hidden from the agent;
- an agent call records one view with arguments, structured content and
`_meta`;
  - a call without a view records nothing;
  - the routes serve the view's HTML and CSP;
  - the view's tool call works and a model-only tool is refused;
  - another task, another company and an agent actor are refused.
- Circleback connector: 13 focused files, 645 tests passed. These are
the app definitions, search, aggregator fallback, connection intents,
tool policy, `AppsConnect` and `Browse`.
- Typecheck passed for `@paperclipai/shared`, `@paperclipai/server` and
`@paperclipai/ui`. `node scripts/storybook-catalog.mjs` and the
migration compatibility check also passed.
- Storybook check in the browser:
- In both themes, the view runs the handshake, renders the meeting cards
from `structuredContent`, picks up the theme variables, and resizes to
its content.
  - Refresh sends `tools/call` and re-renders the cards.
- **Live Circleback end to end** on a local instance, with a real
Circleback account:
- Connected through the catalog entry with Circleback's dynamic client
registration and OAuth.
  - Discovery stored the real view declarations:
    - `DisplayMeetings` uses `ui://circleback/meetings.v2.html`.
    - `DisplayActionItems` and `DisplayCalendarEvents` also have views.
- `ListAssigneeOptions` and `SetCalendarEventRecording` are app-only
helpers with no view. They are hidden from agents.
  - A `claude_local` agent called `DisplayMeetings` on two tasks.
- The real 3.5 MB Circleback view rendered the meeting cards with the
agent's reply.
- Clicking a card opened that meeting in Circleback through
`ui/open-link`.
- The live run found two bugs, both fixed in this PR, each with a
regression test:
    - The 1 MB tool-result limit blocked the view document.
    - Views were hidden inside the folded tool history.

## Risks

- **New migration:** a new table and a nullable column. It is
`online_compatible`.
- **Existing connections:** tools get `ui_meta` on the next catalog
refresh. Until then, no view shows and the tool row is unchanged.
- **Untrusted provider HTML:** the view runs as provider HTML in an
opaque-origin sandbox. It cannot reach Paperclip's session, DOM or API.
Its network access is limited to the origins its server declared. A
view's tool calls use the viewer's permissions and the normal policy and
Ask-first path.
- **Agent tool lists:** app-only tools are removed from them. No current
provider marks tools app-only, so no listed tool disappears.
- **Search:** the change that prefers a native app over an "X MCP"
aggregator listing affects every app.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
subagents (one subagent authored the Circleback catalog connector).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:28:05 -06:00
DottaandClaude Opus 5.5 ef6d347f79 fix(runner): keep reattached runners answering after a restart (#21161)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute in a remote Runner that the server controls over
the durable PRP channel
> - After a server restart, the next server reattaches to the live
Runner and sends recovery barriers (`runner.drain`, `session.snapshot`)
> - These barriers timed out while the Runner stayed connected, and each
same-run retry failed the same way
> - After three attempts the run stopped with
`native_session_retry_exhausted` and needed a manual Retry
> - This pull request finds the two causes and fixes both in the
controller and the transport
> - The benefit is that restart recovery finishes on its own, without a
manual Retry

## Linked Issues or Issue Description

**What happened?**

After a server restart, a run reattached to its live remote Runner. The
Runner authenticated. Then a recovery command got no recorded answer for
30 s or more, although the Runner stayed connected. Errors on staging:

- `PRP command session.snapshot timed out (runner connected; pending
ahead 0; last runner event seq 869 55s ago)`
- `PRP command runner.drain timed out (runner connected; pending ahead 1
[semantic_tool.result] oldest 170s ...)`
- `runnerd exited while waiting for runner.drain`

Each reattach failed the same way. After three attempts the run ended
with "Workspace repair required" (`native_session_retry_exhausted`).

Root causes, proven with the real Runner and the fake Codex harness:

1. The Runner answered in about 40 ms. At reconnect it also sends the
event backlog that it collected while the server was down. The
controller handles inbound frames of a connection in order, and it
commits each event (accounting write and checkpoint flush) before the
next frame. So the answer waited behind the whole backlog. With 400
events at 100 ms each, the answer was handled after 44 s, and the 30 s
barrier had already failed. The `semantic_tool.result` case is the same:
its answer waited behind the backlog, and the next command is sent only
after that answer is handled. The Runner code was not the cause.
2. When a Runner did not answer during close, the close path queued
`runner.drain` and `runner.suspend`, kept the Runner alive for
reattachment, and saved them in the journal. The next attempt delivered
that old `runner.suspend` to the reattached Runner, and the Runner
suspended and exited.

**Expected behavior**

A reattach to a live Runner completes recovery. A Runner answer that is
queued behind earlier frames is not a timeout. A reattach never delivers
a terminal command from an earlier attempt.

**Steps to reproduce**

1. Start a Codex run in a remote Runner with a turn that streams output.
2. Restart the server. Let the Runner collect a few hundred events while
it is disconnected.
3. Let the next server reattach while event commits are slow, as on a
loaded database.
4. See `session.snapshot timed out (runner connected; pending ahead 0)`.
5. Alternatively, let a close path leave `runner.drain` and
`runner.suspend` pending, then reattach. See `runnerd exited while
waiting for runner.drain`.

**Paperclip version or commit**

`master` at 3fb35fcf13.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`).

## What Changed

- `DurablePrpControlPlane` records each Runner `command_result` frame
when it arrives, before the ordered frame chain handles it.
`queuedCommandResultProgressAt(commandId)` returns the latest time that
the chain made progress while that answer is queued.
- `waitForRunnerCommand` keeps waiting past its deadline while the
answer to the command, or to a pending command ahead of it, is queued
and the chain advanced in the last 30 s. When the chain stops for 30 s,
the barrier fails as before. A wait that starts after its deadline still
rejects a late ACK. Frame order does not change.
- When a barrier completes after such an extension, it credits the time
past the deadline (plus the 5 s post-stop grace) to the admission clock
(`RunnerStallLedger.credit`). Later barriers that share a cold admission
deadline (Pi, OpenCode, ACPX) keep their time.
`#waitForProviderIdentityWork` checks the ready state once before it
rejects an expired deadline.
- `DurablePrpControlPlane.withdrawUndeliveredTerminalCommands()` runs
only when the transport adopts a live Runner. The controller sends only
the first pending command, so no command after it reached the Runner,
and a live, serving Runner cannot have executed a suspend. The method
removes every pending `runner.suspend` and `runner.shutdown` from that
pending suffix. It keeps every other pending command, for example a
`semantic_tool.result` that a still-running tool callback queued after
the close, and renumbers the kept commands so the Runner's command
sequence stays contiguous. Pending drains stay, because they have no
effect.
- A generated command id now skips an id that a renumbered command
already holds.
- `doc/execution-semantics.md` describes both rules.

### Changed existing test

`runnerd-codex-transport-remote-retention.test.ts` asserted that the
reattaching controller delivers the failed close's `runner.suspend`, so
the adopted Runner suspends and exits, and a later attempt relaunches it
(#20923). On staging that path failed the attempt with `runnerd exited
while waiting for runner.drain` (run 3e28c2b0) and did not recover. This
PR requires that a reattach does not inherit that suspend. The test now
asserts that the adopted Runner stays alive, answers `thread/read`,
keeps a `ready` journal, gets no suspend, and no duplicate Runner is
launched. The retention part of the test (the close keeps the frozen
Runner alive) is unchanged.

## Verification

- Red before the fix, green after (real `paperclip-runnerd` and fake
Codex):
- `... past an event backlog ahead of its recovery snapshot result`
failed with the staging error `session.snapshot timed out (runner
connected; pending ahead 0; ...)`. Debug timing showed the Runner
answered in 40 ms and the controller handled the answer 44 s later.
- `... past a runner.suspend left by an earlier attempt's close path`
failed with the staging error `runnerd exited while waiting for
runner.drain`.
- `... past a runner.suspend with a tool result queued after it` failed
the same way with tail-only withdrawal (the Greptile case), and passes
now. The tool result is delivered once and all sequence numbers stay
contiguous.
- Unit tests for `waitForRunnerCommand`: it waits while a queued answer
advances, it times out when progress stops, and a later barrier that
shares the admission deadline still completes after an extended one.
- `... past a tool result its predecessor queued after the restart`
guards delivery of a `semantic_tool.result` queued between controller
generations.
- The existing `shares Pi's absolute cold deadline across recovery
barriers and rejects a late ACK` passes unchanged.
- `cd packages/paperclip-runner && pnpm exec vitest run
src/live/runnerd-codex-transport.test.ts
src/live/runnerd-codex-transport-remote-retention.test.ts
src/live/runnerd-codex-transport-reattach-carried-commands.test.ts`
passes (320 tests).
- Related suites pass: adoption, carried suspend, stale controller
journal, stall, deadlines,
`src/backends/harness-driver-backend.test.ts`, and all
`src/control-plane` tests (330 tests).
- `tsc --noEmit -p tsconfig.test.json` passes in
`packages/paperclip-runner`.
- `pnpm verify` asked for broader coverage (exit 2). Required CI
supplies it.

## Risks

- A barrier can now wait longer than its deadline when the Runner has
answered and the controller still commits a large backlog. The wait ends
30 s after progress stops. This trades a slower recovery for no failed
run.
- The withdrawal applies only to a live adopted Runner. If such a Runner
had somehow journaled a withdrawn suspend, the next command gets a
`controller_sequence_mismatch` rejection, and the Runner executes
nothing.
- Renumbering changes the `controllerSeq` of pending commands that never
reached the Runner. Their command ids do not change.
- Not in this PR: a fallback that stops a connected Runner that never
answers and resumes it from its journal. With these fixes, the observed
staging failures no longer reach that state.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool
use (shell, file editing, browser for read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:50:43 -06:00
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00
DottaandClaude Opus 5.5 90ae3c4337 fix(native): process a superseded export's late stop receipt and stop its sandbox when none arrives (#21143)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run can be retired as superseded while its workspace export
fails. Its files stay unsaved, and the run stays the writer of the
task's workspace.
> - #21136 records those files for the automatic workspace recovery.
That record needs the sandbox's stop-and-retain receipt.
> - The native run projection worker checked for that receipt for one
hour after the run ended. After that, it dropped the run. Nothing woke
it when the receipt arrived later.
> - If the receipt came late or never, the run kept the task's workspace
fenced. Every queued run on the task waited and then failed setup.
Nobody owned the problem.
> - This pull request makes the worker process a late receipt, stop and
keep the sandbox itself when no receipt arrives, and never drop the run
silently.
> - The benefit is that the task's queue moves again by itself, and two
writers never use one workspace.

## Linked Issues or Issue Description

Refs #21136 (review thread on
`native-superseded-workspace-export.ts:84`)

**What happened?**
`preserveSupersededWorkspaceExport` returned `recheckAt: null` when the
sandbox stop receipt had not arrived within `finishedAt + 1h`. The
projection worker then deleted its entry. A later stop receipt
(`settleStopOnlyCleanup`) woke only the workspace recovery queue, not
the projection queue. `preserveNativeFailedWorkspace` requires
`hasConfirmedSandboxStopAndRetain`, so no hold was recorded. The run
stayed the dirty writer, and queued runs kept failing setup.

**Expected behavior**
A late receipt is processed at once. When no receipt arrives, Paperclip
stops and keeps the sandbox itself and waits for the provider's
confirmation. The workspace stays held while the sandbox may still run.
When the provider confirms that the sandbox is gone, the task continues
from its last save. When the stop stays unconfirmed for hours, the task
shows one plain notice, and the check continues.

**Steps to reproduce**
1. Retire a native run as superseded after its workspace export fails
(the #21136 shape).
2. Let the run's sandbox stop not settle (for example, the server that
owned the release is replaced).
3. Wait more than one hour after the run ended.
4. See that the projection worker drops the run, and that a later stop
receipt records nothing. The workspace stays held.

## What Changed

- `sandbox-stop-and-retain.ts`: a confirmed stop-only receipt now also
notifies `DELIVERY_QUEUES.nativeRunProjection` (scoped to the company
and run). New `prepareUnconfirmedSourceSandboxStop` puts a
stop-and-retain intent on the source lease from its current status. It
skips a destroyed sandbox, a stop already in progress, and a cleanup in
flight.
- `native-superseded-workspace-export.ts`:
- Within one hour of the run ending, the worker checks every minute
(unchanged).
- After that hour, it requests the stop-and-retain itself. The existing
pending-cleanup sweep dispatches the stop and retries it until the
provider confirms. The worker checks again every 15 minutes, and the
receipt wakes it at once. The workspace stays held.
  - A sandbox that another lease holds is never stopped.
- When the provider confirms that the sandbox was destroyed, the files
cannot be saved. In one transaction (task, run, then workspace locks),
the worker sets aside only that run's workspace epoch, wakes queued-run
dispatch, and records `issue.previous_run_files_lost` on the task.
- When the stop is still unconfirmed six hours after the run ended (or
the sandbox is held by another lease), the task gets one
`issue.previous_run_files_waiting` activity entry. The worker keeps
checking. It never drops the run.
- A stop-only dispatch now settles when the provider confirms that the
exact sandbox is already gone. Before, the runtime accepted only a
`stopped` reply and threw on `destroyed`, so a gone sandbox kept the
stop pending forever.
- `sandbox-stop-and-retain.ts`: new `acceptedStopOnlyReceipt` accepts
`stopped`, or `destroyed` for a stop-and-retain intent (a native
workspace export resume still needs `stopped`). `settleStopOnlyCleanup`
stores a `destroyed` receipt as `released` / `success` /
`workspace_source_unavailable`, removes the stop intent, never writes a
retain receipt, and wakes the projection and workspace recovery queues.
- `environment-runtime.ts`: the plugin `environmentStopLease` and
built-in `stopLease` checks use `acceptedStopOnlyReceipt`.
- `workspace-recovery.ts`: when its own stop settles as `destroyed`, the
source is unavailable (the existing alternative-strategy path) instead
of a binding failure.
- Daytona plugin: stop of a sandbox that Daytona reports as not found
returns `destroyed` (destroy already did this) instead of throwing. boxd
already returns `destroyed`. Other sandbox plugins have no stop-only
method, so nothing changes for them.
- New shared `SANDBOX_ALLOCATION_HOLDER_STATUSES` (`active`, `retained`,
`pending_cleanup`). The stop-only cleanup ownership check and the
worker's "held by another lease" guard both use it, so a retained warm
sandbox is never stopped and the two checks cannot drift.
- `ui/src/lib/activity-format.ts`: plain labels for the two task
activity entries. They do not ask a person to fix anything.
- `doc/execution-semantics.md`: document the rule.
- Not changed: the shared owner prover still refuses to treat a stopped
instance on another host as proof that its child processes stopped.

## Verification

- `server/src/services/superseded-workspace-export-recovery.test.ts` (4
new tests, red/green):
- "checks again every minute while the run's own release may still stop
its sandbox": the recheck branch. Nothing is recorded or stopped, and
the workspace keeps its writer.
- "stops and keeps the sandbox itself when no stop receipt arrived, and
records the files once the late receipt lands": after the hour, the
lease gets a stop-only intent. A second pass changes nothing. The late
receipt wakes the projection queue. The next check records the hold, and
the system recovery frees the workspace. Fails on master (`expected {
…(19) } to match object { status: 'pending_cleanup' }`).
- "names the unconfirmed stop on the task once, and keeps checking, when
the provider never confirms it": the expiry branch. One notice, the
check continues, no hold, and the workspace stays held. Fails on master.
- "continues from the last save when the provider confirms the sandbox
is gone": epoch 1 is set aside, the workspace is free, the task says so,
and the result is not applied. Fails on master.
- Review-round tests (all 5 fail on the first head `a208bfa58f`, pass
now):
- `environment-runtime.test.ts` "settles a plugin stop whose provider
reports the sandbox gone (boxd, Daytona not found) through the cleanup
sweep": provider dispatch through `releaseRunLeases` and
`sweepPendingCleanupLeases`. The lease settles `released` with a
`destroyed` receipt and no retain receipt.
- `environment-runtime.test.ts` "settles a built-in stop whose provider
reports the sandbox gone, without claiming retained files": the built-in
`stopLease` path. No destroy call.
- `superseded-workspace-export-recovery.test.ts` "continues from the
last save when Paperclip's own stop finds the sandbox already gone": the
worker's stop request gets a `destroyed` reply, the projection queue
wakes, and the workspace is freed at epoch 2.
- `superseded-workspace-export-recovery.test.ts` "never stops a sandbox
a retained warm lease still holds, says so once and keeps checking": the
source lease is unchanged, a recheck is returned, and `sandbox_in_use`
is recorded.
- Daytona `plugin.test.ts` "explicit stop confirms a sandbox Daytona no
longer has as destroyed".
- `environment-runtime.test.ts` (207) and Daytona `plugin.test.ts` (421)
pass.
- Related suites pass: `native-run-projection-work`,
`retained-native-session-cleanup-work`, `workspace-recovery-stop-proof`,
`native-failed-workspace-recovery`, `legacy-workspace-restore-recovery`,
`environment-retained-orphan-sandbox`,
`workspace-recovery-retry-copyback`, `workspace-recovery-fallback` (398
tests), and `ui/src/lib/activity-format.test.ts`.
- `tsc --noEmit` for `server`.

## Risks

- Low to medium risk. The new behavior acts only on an ended superseded
run that failed in its workspace export, still holds a dirty workspace,
and has no retained-file hold.
- The worker now stops an ended run's sandbox one hour after the run
ended if no stop receipt arrived. The stop uses the existing stop-only
contract and keeps the sandbox. A sandbox held by another lease is never
stopped.
- A destroyed sandbox's unsaved files are not recoverable. The task
continues from its last save and says so.
- If a provider never confirms a stop, the workspace stays held (by
design: no second writer). The task shows one notice, and Paperclip
keeps retrying.
- Shared cleanup code now treats a confirmed `destroyed` stop-only reply
as settled. Every caller was checked: the cleanup sweep and run release
count it as settled; the workspace recovery treats the source as
unavailable; the fallback treats it as stopped (nothing can run there).
No caller reads it as retained files, because no retain receipt is
written. A native workspace export resume still requires `stopped`.
- Each extra delivery notification per settled stop-only cleanup is
cheap. The projection worker's scoped load finds nothing for runs that
do not match.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:54:47 -06:00
DottaandClaude Opus 5.5 dfc510b887 fix(server): hand new run starts to the replacement server during a deploy overlap (#21127)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud replaces the server process on each release. The old
server keeps serving until the new one passes health checks and takes
the traffic.
> - During that overlap the old server keeps starting new runs right up
to SIGTERM, although the new server can already start them.
> - SIGTERM then cuts off exactly those runs while they still prepare.
In one audited swap the old server started 30 runs after the new server
booted, and 23 of them ended `process_lost`.
> - Admission must never pause, so the fix must hand over who starts new
runs, not stop admission.
> - This pull request makes the new server record when it starts runs.
The old server then leaves new run starts to it, and starts them again
if the new server stops or dies.
> - The benefit is that a deploy no longer starts runs on a server that
is about to stop, and the company always has one server that starts
runs.

## Linked Issues or Issue Description

Refs #20852 (preserves preparing and launching turns that a deploy
interrupts; this PR reduces how many such turns the old server starts).
Related: #21046, #21107 (shared controller-instance owner proof), #20986
(queued-run dispatch signal).

**What existing behavior does this improve?**

The start of new runs while two servers overlap during a Cloud deploy.

**Subsystem affected**

Server run admission (`server/src/services/heartbeat/queue.ts`), the
controller instance table, and startup and shutdown in
`server/src/index.ts`.

**Current behavior**

The old server starts queued runs until it receives SIGTERM. Runs that
it starts in the last minutes of the overlap are interrupted while they
prepare. Nothing tells the old server that its replacement already
starts runs: `controller_instances.epoch` is written at process start,
and that happens minutes before the new server can start runs.

**Proposed behavior**

The new server records `dispatching_at` and a short
`dispatch_lease_expires_at` on its `controller_instances` row right
after its startup queued-run pass. The old server reads this through an
in-memory flag at run admission and leaves new run starts to the new
server. Runs it already started continue and are handed over at SIGTERM
as today. If the new server stops, drains or dies, the old server starts
runs again within seconds.

**Reason and benefit**

Fewer runs are cut off by each swap. There is never a moment without a
server that starts runs.

**Breaking changes**

None. The migration adds two nullable columns. Releases without this
code ignore the columns, and their rows never carry the marker, so a
mixed-version overlap behaves as today.

## What Changed

- `controller_instances`: new nullable columns `dispatching_at` and
`dispatch_lease_expires_at` (migration
`0471_controller_instances_dispatch_marker`, two `ADD COLUMN`
statements, no lock beyond the catalog update).
- New `server/src/services/run-start-handover.ts`. It is one worker on
the shared delivery coordinator (`DELIVERY_QUEUES.runStartHandover`).
Its renewal and recheck are coordinator deadlines, and a drain change or
expiry wakes it.
- Replacement side: `begin()` records the intent and writes the marker
(it creates the boot's row if the row is missing). If the server drains
at that moment, the worker writes the marker when the drain ends. The
worker then publishes a `run-start-handover` hint and renews the 20 s
lease every 5 s. It renews only while an older boot that is not stopped
or fenced may still serve, and for at most 30 min. All marker writes run
in one serialized chain. `release()` at shutdown lands after any renewal
already in flight. A clear that fails is retried, and shutdown waits for
the release. While the server cannot dispatch (drain, quiesce, admission
closed), the worker clears the lease.
- Old-server side: `isYielding()` is an in-memory read. The hint
refreshes it, and so does one read that a dispatch pass waits for (at
most 1 s) when the last read is more than 2 s old. A read that gets no
answer within 1 s never keeps the server yielding. The server admits,
and the yield also ends when the replacement's last seen lease runs out
on the old server's own clock. Reads are a single flight on the
dedicated connection. While it yields, the server rechecks every 3 s,
calls `onResume` when the marker is gone, and sends a coalesced
queued-run-dispatch hint for skipped starts.
- It yields only to a newer epoch with a live marker. It never yields to
an older boot, to itself, or to a row without the marker, and it does
not yield without a row of its own.
- **Database invariant:** every statement of the hand-over runs in a
small transaction on the dedicated one-connection controller-lease pool
(`controllerLeaseDb`) with a 1 s statement timeout. That covers reads,
marker writes, the `run-start-handover` notification (commit-bound, in
the marker write's own transaction) and the skipped-start hint. None of
it touches the shared pool or registers tracked database work. The pool
is shared only with legacy controller-lease renewals, and never with the
liveness pool that native lease renewals use. A hand-over statement
waits at most one legacy batch (10 s), inside the 15 s between a marker
renewal and the marker's expiry. It delays a legacy renewal (60 s lease,
renewed every 10 s) by at most 1 s. I did not piggyback on the
controller-instance lease renewal, because that renewal runs only while
the server holds work and only every 30 s.
- **Import audit of `run-start-handover.ts`:**
  - `drizzle-orm` `sql`: builds statements, no I/O.
- `controllerLeaseDb`: the dedicated connection, used by every statement
here.
- `notifyDatabaseWorkOnCommit` (new in `work-signals.ts`): runs
`pg_notify` only on the transaction it is given and registers no intent,
so the coordinator's `reconcileDatabaseWork` has nothing of ours to
check.
  - `logger`: no I/O to the database.
  - `controller-process-identity`: in-memory boot id and host hash.
- `delivery-work-coordinator`: type only. The worker is registered for
lifecycle and deadlines; its `run` awaits only the serialized marker
chain, and reads are not awaited.
  - `DELIVERY_QUEUES`: constants.
  - `task-admission` drain subscriptions: in-memory events.
- `heartbeat.ts`: run admission is also closed while the server yields.
A `runStartHandover` service option (default: the process's registered
hand-over) lets one process host two assembled services in tests.
`queue.ts`: `startNextQueuedRunForAgent` and
`resumeQueuedRunsAfterDrain` first call `refreshRunAdmission()`. That
call returns `null` (nothing to await) unless the hand-over read is
stale.
- `index.ts`: creates the hand-over only when the instance lease is on,
the deployment notification channel exists, and the scheduler is
enabled. It calls `begin()` after the RunDispatch startup pass and wakes
that queue. It releases the hand-over when run admission closes at
shutdown, and stops it before the lease stops.
- `deployment-notifications.ts`: keeps the LISTEN connection open while
a hand-over involves this server, so skipped starts and lease changes
reach the other server at once.
- `doc/execution-semantics.md`: new section "Run starts during a deploy
overlap". It covers the marker, who yields, failure handling, what
counts as starting a new run, and what a post-deploy audit should show.

**Marker contract for other components (for example the deployer):** a
replacement starts runs when its `controller_instances` row has an
`epoch` greater than the old boot's, and `dispatching_at is not null and
dispatch_lease_expires_at > now() and stopped_at is null and fenced_at
is null`. `dispatching_at` stays set after the lease lapses, as the time
the boot began starting runs.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-start-handover.test.ts`: assembled-app test
with two `heartbeatService`s, each with its own pool, listener,
coordinator and boot. A wake that the old server accepts stays queued
there. The replacement's real queued-run dispatch starts it, and claims
are observed per pool. After a replacement release, the old server
starts the run itself. Red: with the yield removed from
`isRunAdmissionClosed`, the old server claims the run.
- `cd server && npx vitest run
src/__tests__/run-start-handover.test.ts`: 19 tests pass against
embedded PostgreSQL. Each test uses two servers, each with its own pool,
boot id, LISTEN connection and delivery coordinator. The tests cover:
- the old server is ready and the new server starts; the new server sets
the marker; the old server yields; the queued run is claimed only by the
new server, and a skipped start wakes the new server's dispatch
- the old server learns from its own dispatch path without a
notification: a read older than 2 s is refreshed before the start
- the new server dies after it sets the marker: the old server resumes
and runs its queued-run pass
  - the new server shuts down: the old server resumes at once
- the new server drains: it releases the marker and takes it back
afterwards
- a replacement that starts up during a drain takes over when the drain
ends
  - a shutdown release during an in-flight renewal stays released
- a clear that fails at shutdown (statement timeout behind a row lock)
is retried by a later release
- an older server that outlives the lease TTL keeps yielding while the
replacement renews
- a yield that the old server cannot confirm within 1 s (its only
connection is busy) does not stop starts (red/green)
- a yield ends when the replacement's lease runs out, even while every
read waits for a connection (red/green)
- renewals blocked past the TTL hand starts back once, then the
replacement takes them again (bounded flapping)
- a clear that fails when the replacement starts draining is retried
well inside the lease (red/green)
- with both servers' shared pools blocked past the TTL, the marker is
written, renewed and released, and the old server yields and resumes,
all before the block ends (red when hand-over statements use the shared
pool)
  - an older-release row without the marker changes nothing
- a server never yields to an older boot or to itself, and does not
yield without a row of its own
  - a lone server writes once, never renews, and admits from memory
  - without a hand-over (self-hosted) there is nothing to await
- Red: on `origin/master` the test file cannot run, because the module
and the columns do not exist. The old server has no signal and keeps
starting the handed-over run.
- `cd server && npx vitest run
src/__tests__/deployment-notifications.test.ts
src/services/heartbeat/dispatch-work.test.ts
src/__tests__/delivery-work-coordinator.test.ts
src/__tests__/controller-instance-lease.test.ts src/shutdown.test.ts`:
all pass.
- `cd packages/db && pnpm check:migrations`: safety and compatibility
checks pass. `cd server && npx tsc --noEmit -p .`: clean.
- `pnpm verify` asks for broad verification because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

## Risks

- Run starts move from the old server to the new server during the
overlap. If the new server cannot dispatch, the old server resumes on
the lease clear (about 3 s) or the lease lapse (up to about 23 s after a
crash). If a hand-over read fails, the server keeps admitting.
- If the old server was not listening for notifications, it can start
runs for up to about 2 s after the marker (the age of its last read). On
a dispatching server, the pre-dispatch read adds one small bounded query
at most every 2 s.
- An older boot that has been idle for more than 30 min counts as gone.
The new server then does not renew, and the old server admits again as
today.
- The migration adds nullable columns only. The deployer can apply it
while the previous release serves.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:09:53 -06:00