Commit Graph
953 Commits
Author SHA1 Message Date
Devin FoleyandPaperclip cc9ff75cb8 fix(runtime): keep managed GitHub access across a controller restart (#21928)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs give agents managed GitHub access through `gh` and `git`
launchers on the provider's `PATH`
> - A provider process can keep running while the server restarts, and
the next controller adopts it
> - The detaching controller deleted the launcher directory, and the
adopting controller staged launchers in a new random directory
> - The running provider still names the old directory, so every later
`gh` or `git` call ran without credentials
> - This pull request keeps the launcher directory stable for one
provider session and task, and re-arms it after the restart
> - The benefit is that a long run keeps its GitHub identity across a
server deploy or restart

## Linked Issues or Issue Description

No public issue exists. This describes the bug in-PR, with the bug
template labels.

**What happened?**
An agent run on a Daytona sandbox used `gh` with managed GitHub access.
The server restarted during the run (a rolling deploy), and the
controller re-attached to the live provider process. After that, every
`gh` call failed with `You are not logged into any GitHub hosts`. `git
credential fill` returned no token. The agent reported that its token
had "expired" and used the unauthenticated public API for the rest of
the run.

**Steps to reproduce**
1. Start a native run with managed GitHub access on a remote sandbox.
2. Run a `gh` command in the run. It succeeds.
3. Restart the server while the provider process is still running. The
new controller re-attaches to the process.
4. Run a `gh` command again in the same run. It fails with "not logged
in", and no `Paperclip: GitHub ...` diagnostic appears.

**Expected behavior**
After the new controller adopts the provider, `gh` and `git` get
credentials from the new controller, as they did before the restart.

**Root cause**
- `createNativeGitHubAccess` staged the launchers in
`native-session-<random UUID>` and put that path in the provider's
`PATH`, `BASH_ENV`, and `GH_CONFIG_DIR`.
- On a restart detach, `githubAccess.stop()` ran `rm -rf` on that
directory (`detachReleasedWarmNativeSession`, and the run's final
`github_stop` while `nativeRunsDetachingForRestart` is set).
- The adopting controller made a new random directory. The provider
environment cannot change after spawn, so the provider never saw it.
- The shims were gone. `gh` resolved to the real binary with an empty
`GH_CONFIG_DIR`, so no Paperclip diagnostic appeared.

Refs #15354. That PR keeps ACPX sessions across launcher rotation in the
legacy adapter path. This PR covers the native runtime path.

## What Changed

- `createNativeGitHubAccess` takes an optional `launcherKey`. The
launcher directory is `native-github-<sha256(key)>`, so the same
provider session and task always get the same directory.
- The native session executor passes a key built from
`nativeSessionScopeKey` and the issue id.
- `prepareGitHubOperationLaunchers` takes an optional `broker`. It then
stages broker-file launchers and an owner-only `broker.json`. The
launchers already read `broker.json` on each operation in this mode
(`githubLauncherSource({ brokerFile: true })`). A restaged directory
therefore gives the running provider the new endpoint.
- Native access passes `broker` only for remote sandbox targets. A
sandbox provider is the one that outlives a controller restart. Local
launchers live in the shared host temp directory, so they keep the
capability in the provider environment only, as before.
- `stop({ preserveLaunchers: true })` keeps the launcher directory. A
restart detach of an active run and of a released warm session use this
option. A normal stop still removes the directory.
- A final stop does not remove a directory that a newer access in the
same process has already restaged.
- The callback bridge runtime stays in its own per-access directory. A
new bridge never shares a queue directory with an old bridge.
- `syncRemoteTextFiles` takes a `private` flag. It sets `umask 077` for
the whole staging script, so the upload copy, the partial file, and the
final file are never readable by group or other users. Local launcher
files are written to a temporary file and renamed, so a running launcher
never reads a partly written file.

## Verification

- New `native-github-access` test on a sandbox target: a live parent
process gets credentials, the access stops with `preserveLaunchers`, and
a new access with the same key re-arms the directory. The same parent
PID then gets the new controller's credentials, with empty stderr.
`broker.json` has mode `600`. The final stop removes the directory.
- New previous-release cutover test: a provider launched with the old
random-directory layout keeps working without managed GitHub after its
controller removes the directory. The new controller does not resolve
credentials for it. A provider that the new controller launches gets
credentials.
- New local regression: local launcher files do not contain the broker
capability, and there is no `broker.json`.
- New local restart test: a local provider adopted after a restart gets
a Paperclip diagnostic and no credentials. The new controller resolves
nothing for the old endpoint.
- New `sandbox-callback-bridge` test: staging a private file runs with
umask `077` and leaves the file at mode `600`.
- New `native-github-access` test: when a superseded access stops, its
successor's launchers stay usable. A different key gives a different
directory.
- New `native-session-executor` test: a run detached for restart while
active stops its GitHub access with `{ preserveLaunchers: true }`, and
creates it with a `launcherKey`. The existing overlapping-failure test
now asserts `{ preserveLaunchers: false }`.
- The new tests fail on the old behavior. I checked this by forcing
`brokerFile: false`, forcing `preserveLaunchers = false`, removing the
`umask 077` line, and writing `broker.json` for local targets.
- Passed: `native-github-access` (26), `native-session-executor` +
`github-operation-credentials` + `heartbeat-github-launchers` +
`native-stop-accounting` + `heartbeat-project-env` (1640 passed, 7
skipped), `native-runner-restart-recovery.integration` and
`heartbeat-native-restart-suspension`.
- Passed: adapter-utils `execution-target`, `github-launcher`,
`github-launcher-environment`, `sandbox-callback-bridge`; runner
`acpx/github-launcher`, `acpx/environment`,
`codex/codex-security-config`.
- `pnpm --filter @paperclipai/adapter-utils typecheck` and the server
`pnpm typecheck` pass.
- Not run locally: `native-warm-checkpoint-ownership.integration` needs
the compiled `paperclip-runnerd` binary. CI builds it.

## Previous-release story

- No persisted run, session, checkpoint, or database state changes. Only
the in-sandbox launcher layout changes.
- A provider launched by the previous release has a random
`native-session-*` launcher directory in its environment. The previous
release's controller removes that directory at the cutover restart. The
new controller cannot find it, so it does not re-arm it.
- That provider carries forward without managed GitHub until its next
spawn, which is the current behavior. The run is not refused and is not
retried. `gh` and `git` run without managed credentials, as they do
today after a restart. The cutover test covers this.
- A provider that the new release launches gets the stable directory and
survives later restarts.

## Risks

- On sandbox targets, the provider can now read the bridge capability in
`broker.json`, as well as in its environment. The file holds the same
per-access transport token that the environment already holds, not a run
token or a GitHub credential. The file and its staging copies are mode
`600`, and the broker still checks the active run and grants on each
request. Local targets write no `broker.json`.
- Local providers that outlive a restart still lose managed GitHub. This
is not a regression: master deletes the launcher directory at the
detach, so the old local provider also runs without credentials there.
With this PR, the local launchers stay in place and print a `Paperclip:
GitHub ...` diagnostic instead of a silent "not logged in". Re-arming
local providers needs a broker channel that other local runs under the
same user cannot read. That is out of scope here, because `broker.json`
in the shared temp directory was a security finding on this PR.
- On sandbox targets, `git` and `gh` children no longer inherit
`PAPERCLIP_API_URL`, `PAPERCLIP_API_KEY`, or
`PAPERCLIP_GITHUB_BRIDGE_TOKEN`. Broker-file mode removes them, as it
already did in its existing tests.
- After a restart detach, a launcher directory stays in the workspace
runtime directory. If no controller adopts the provider, that directory
stays until a later access with the same key stops, or until the
workspace is removed. It contains no GitHub credential.
- Between the detach and the adoption, `gh` gets a transport error from
the old endpoint. It then prints `broker_transport_unavailable` and
continues without credentials, as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs). Standard context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-16 12:13:04 -08:00
Devin FoleyandPaperclip 0308108531 fix(native): pipeline tool progress events with text deltas (#21533)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs store every relayed runner event in
`heartbeat_run_events`. Each stored event also locks and updates the
run's `heartbeat_runs` row.
> - Text deltas already commit in batches. Every other event, including
each `tool.execution.progressed` event, commits in its own transaction.
> - Tool receipts and usage accounting lock the same run row. Under
parallel tool calls, accounting failed with "other updates to the same
run kept conflicting", and a provider-pack receipt save hit a lock
timeout.
> - #21520 made events that arrive while a batch commits join the next
batch. This pull request adds tool progress events to the pipelined
types, so they use those batches too.
> - The benefit is fewer run-row transactions in tool-heavy turns,
without changing event order.

## Linked Issues or Issue Description

**What happened?**
A routine run started several subagents that made about 190 tool calls
in about 10 minutes. Three tool calls failed with "could not save this
run's budget or usage records because other updates to the same run kept
conflicting", and a provider-pack admission receipt failed with a lock
timeout (55P03). Every tool progress event of the run was a separate
transaction that updated the same run row.

**Expected behavior**
Observational events such as tool progress commit in batches, so they
hold the run row lock fewer times.

**Steps to reproduce**
1. Run a turn that makes many tool calls in parallel.
2. Each `tool.execution.progressed` event commits in its own transaction
and updates `heartbeat_runs`.

**Relevant logs or output**
```
could not save this run's budget or usage records because other updates to the same run kept conflicting
native provider-pack admission receipt could not be saved (cause=55P03)
```

Related: #21531 tried letting the runner relay events faster, which
would make this contention matter more. It is closed.

## What Changed

- `paperclip-control-plane-port.ts`:
- `pipelinedEventTypes` names `item.delta` and
`tool.execution.progressed`. Both go through the batched append path and
its group commit from #21520.
- The text-delta boundary is now tracked when an event is appended: a
pipelined event that is not a delta ends it, and a delta with text
starts it. The first text delta after any other event still commits at
once.
- `contracts/control-plane-port.ts`: optional `pipelinedEventTypes`, and
the `streaming` option documents that it applies to those types.
- `native-session-runtime.ts`: the runtime pipelines the port's
`pipelinedEventTypes`, and only `item.delta` when the port does not name
them.
- Tests:
- `paperclip-control-plane-port.test.ts`: three progress events appended
while a batch commits share one transaction, and a following event
commits after them.
- `native-session-runtime.test.ts`: progress appends start without
waiting only when the port pipelines them.

## Verification

- After rebasing onto current master: `pnpm exec vitest run
src/services/native-runtime/paperclip-control-plane-port.test.ts` (in
`server/`): 108 passed.
- `npx vitest run src/native-session-runtime.test.ts` (in
`packages/paperclip-runner`): 298 passed, 1 failed. `settles a governed
wait through real Runner shutdown after a missing Codex terminal` needs
the built Rust test binaries (`fake-codex-app-server`), which my
checkout does not have. CI builds them.
- `pnpm exec tsc --noEmit -p .` in `server/` (after building its
workspace dependencies) and in `packages/paperclip-runner` pass.
- `npx vitest run src/__tests__/chat-channels.integration.test.ts
src/__tests__/heartbeat-process-recovery.test.ts
src/services/native-runtime/native-plan-wait.test.ts
src/services/native-runtime/paperclip-control-plane-port.test.ts` (in
`server/`): 2,586 passed, 2 failed. `binds Slack tools to the admitted
linked request...` reads a skills checkout that is not next to my
worktree. `materializes direct external-chat finals once...` failed in
the full file run but passes alone, with and without this change.

## Risks

- Progress events now commit up to one transaction later than before.
Their receipts and live UI updates follow their batch.
- A batch that contains progress events still writes rows one by one
inside its transaction, because progress events register chat-progress
wakes. The gain is one transaction and one lock wait per batch, not one
row update per batch.
- If a batch fails, every event in it fails, and the runner resends
them. That is the same as for text deltas today.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-16 11:25:59 -08:00
DottaandClaude Opus 5.5 3452639a81 fix(runner): reject only the overflow call when parallel tool calls exceed the result reserve (#21913)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs call Paperclip tools through the runner's provider
tool bridge
> - The bridge reserves 768 KiB for each pending call against an 8 MiB
budget per turn
> - When an agent sends many tool calls in parallel, the reserve for the
pending calls can go over the budget, although the bytes the turn really
holds are small
> - The bridge then reports the durable receipt limit, and the backend
stops the whole turn. The run ends as cancelled with `needs_review`, and
the task stays blocked with a Retry button
> - This pull request rejects only the overflow call as retryable, and
keeps the turn stop for real held bytes. It also adds a server backstop:
a turn stopped at the receipt limit continues by itself in a fresh
session
> - The benefit is that a parallel burst of tool calls no longer leaves
a task waiting for a person

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos wave, an opencode agent sent 10 parallel `get_task`
calls. The turn already held about 0.7 MB of results. 10 × 768 KiB plus
0.7 MB is more than 8 MiB, so `begin_call` returned
`active_turn_receipt_limit()` for the 10th call. This branch does not
set `durable_run_receipt_limit_reached`, so it is a temporary reserve
overflow. `handle_tool_call` sends every receipt-limit error to
`stop_turn_at_tool_receipt_limit`, which interrupts the turn. The run
ended as cancelled with `needs_review` and a human review request. The
release then blocked the task
(`native_continuation_requires_reconciliation`) and showed Retry. No
continuation was scheduled. The task moved again only because a person
posted a comment ten minutes later.

**Expected behavior**
The overflow call gets a normal, retryable rejection, and the turn
continues. The agent can send the call again after a pending result
arrives. If a turn does stop at the receipt limit, Paperclip continues
the task in a fresh session, as it does for `agent_tool_call_loop`
(#21734). The task does not show Retry.

**Steps to reproduce**
1. Start a native turn that holds about 600 KiB of settled tool results.
2. Send 10 tool calls in parallel.
3. On master, the 10th call stops the turn, and the task becomes blocked
with Retry.

**Paperclip version or commit**
`master` at c5218f5ec4 (wave), rebased on 991f773d7f.

## What Changed

- `provider_bridge.rs`: the two `begin_call` branches that fail only
because of the per-call result reserve now return a new retryable error
(`tool_calls_in_flight`). The provider sees "Too many tool calls are in
flight. This call was not executed. Wait for the pending tool results,
then retry it." The branches that set
`durable_run_receipt_limit_reached` (real held bytes, exact identity
capacity, saturated replay history) still stop the turn.
- `provider_backend.rs`: the invalid tool call result sets `retryable`
from the error, so this rejection is `retryable: true`. Other rejections
stay `false`.
- `native-tool-receipt-limit.ts` (new): reads the persisted,
source-bound `harness.diagnostic` event with code
`semantic_tool_turn_receipt_limit` for the run, and holds the lane
constants and the hint for the next session.
- `status-arbiter.ts`: a run that did not succeed and whose runner
stopped it at the receipt limit preserves the task status and schedules
a retry with cause `semantic_tool_turn_receipt_limit` (and releases run
resources when it was cancelled). This applies only when the task owner
holds the decision, with no review, governance gate, open blocker, pause
hold or prior `blocked` status, and at most 2 times per chain. Other
cases keep their current path.
- `native-provider-capacity-retry.ts`: the capacity retry is now one
lane of a shared fresh-session retry helper. The new receipt-limit lane
creates a `scheduled_retry` run due in 5 seconds with
`forceFreshSession` and a liveness instruction that tells the agent to
make fewer parallel calls.
- `status-decision-committer.ts` and `native-run-finalizer.ts`: wire the
new lane, and require the task owner under the issue lock, as for the
capacity retry.

- `acpx_provider_session.rs` (Claude, Cursor and other ACPX agents)
handles the same retryable error per call: it drops the pending sidecar
call and resolves only that call with the retryable error. Other
`begin_call` errors still fail closed.
- The external provider path (Dot/Muse) is out of scope and unchanged:
it still fails closed on this overflow, as on master. Making a
same-requestId retry safe there needs retry generations across the
executor, the driver command ids and the broker receipts. That is a
follow-up.
- The receipt-limit backstop reads only the diagnostic of the final turn
in the run's session.
- A person's Stop always wins: the finalizer re-reads the saved Stop on
each arbitration attempt, the retry helper refuses under the run lock,
and the #21734 tool-call loop continuation is not queued, or is
cancelled, for a stopped run.

The runner change is runner-image content. The Daytona runner image must
be rebuilt before sandboxes get the runner fix. The server backstop
works with older images.

## Verification

- Red then green, `provider_bridge.rs` unit test
`parallel_calls_over_the_result_reserve_reject_only_the_overflow_call`:
with about 600 KiB settled, 9 calls are admitted. On master the 10th
returns `is_active_turn_receipt_limit()`. Now it is a retryable
rejection, the durable flag stays false, and the same call is admitted
after one pending result settles.
- Negative, `held_result_bytes_over_the_limit_still_stop_the_turn`: real
held bytes over the limit still return the receipt limit with the
durable flag set.
- `cargo test -p paperclip-runner-core -- --skip task_memory::`: all
pass (640 lib tests plus the integration suites). The `task_memory`
monitor tests are timing-based kernel OOM monitor tests that do not run
correctly on macOS (one hangs); this change does not touch them, and CI
runs them on Linux.
- Red then green, `native-tool-receipt-limit.integration.test.ts` (real
Postgres): a run stopped at the receipt limit gets one `scheduled_retry`
successor with `forceFreshSession` and the hint. The release does not
block the task and creates no recovery action. The task stays
`in_progress`. The presentation shows `retry_scheduled` with no Retry,
then `working` after promotion. With the master arbiter, no successor is
created. A negative case (another diagnostic code) keeps
`cancellation_run_only`.
- Red then green: ACPX
`receiver_rejects_only_the_tool_call_that_overflows_the_result_reserve`
(11-call burst through the fake sidecar).
- Red then green, server: an earlier turn's diagnostic gives no
continuation; a Stop saved before or right after the finalizer's first
read gives no continuation and settles the cancellation with no recovery
action; the retry helper throws `NativeStatusRaceError` for a stopped
run; the loop continuation is not queued, and a queued one is cancelled,
for a stopped run.
- `status-arbiter.test.ts`: the new branch, the 2-continuation bound,
and the `blocked` exclusion.
- Focused server tests pass: the new test,
`native-provider-overload.integration.test.ts`,
`status-arbiter.test.ts`, `native-status-arbiter-corpus.test.ts`,
`heartbeat-run-dispatch-work.test.ts`,
`native-run-finalizer-telemetry.test.ts`,
`native-workspace-finalizer.test.ts` (306 tests). `tsc --noEmit` passes
for `server`.
- `pnpm verify` exits 2 (a runner-core change needs broad coverage), so
required CI supplies it.

## Risks

- Dot/Muse agents still lose the turn on this overflow until the
follow-up lands (no change from master).
- An agent that keeps more than about ten large calls in flight now gets
rejections for the extra calls instead of a stopped turn. It can send
them again after results arrive. Real held bytes still stop the turn.
- The receipt-limit continuation starts a fresh session. The in-turn
conversation is lost; the task, comments, documents and saved workspace
stay. It runs at most 2 times per chain, then the current path applies.
- The retry lane shares the failure-retry count with the capacity lane.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21734

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:53:55 -06:00
DottaandClaude Opus 5.5 93111b175b fix(wake-queue): keep the child-completion reason when a person's comment merges into its held wake (#21918)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The wake queue holds new work for a task while an agent run is
active. A later wake for the same task merges into the held wake
> - A parent task must not finish while a finished child's result is
still waiting for a parent turn. The finish gate checks for this pending
child result
> - The gate finds the held child-completion wake by its context wake
reason
> - When a person's comment merged into that held wake, the merge
replaced the wake reason with `issue_commented`. The gate then did not
see the pending child result
> - This pull request keeps every merged reason in the held wake and
makes the gate read that list
> - The benefit is that a merge keeps the meaning of both wakes: the
comment is delivered, and the pending child result still blocks a
premature finish

## Linked Issues or Issue Description

Refs #21914

**What happened?**

A restart test held a child-completion wake behind a running parent run.
A person then posted a comment. The comment wake merged into the held
wake. After the merge, the held wake's
`_paperclipWakeContext.wakeReason` was `issue_commented`, not
`issue_children_completed`. `hasPendingNativeChildCompletion` matches
held wakes by that field only. So the finish gate could no longer see
the pending child result.

**Expected behavior**

The held wake keeps both reasons. The finish gate still sees the pending
child result. The next run still receives the person's comment.

**Steps to reproduce**

1. Start a run for a parent task.
2. Hold a child-completion wake behind the run.
3. Post a person's comment on the parent. The comment wake merges into
the held wake.
4. Call `hasPendingNativeChildCompletion` for the running parent run. It
returns `false`.

**Paperclip version or commit**

`master` at 20b0d4e5a5.

**Relevant logs or output**

The suspected effect on finish timing in the test run was not real. The
two rejected finish calls failed for other reasons: the first had
invalid tool input (a missing `remainingWork` field) and the second had
no accessible deliverable. Both happened before the comment merge. Also,
the run had already read the finished child before the merge, so the
gate was already satisfied for that child. The gap is real for a run
that has not read the child yet. The new test proves it.

## What Changed

- `admitWakeBehindIssueExecution` records `wakeReasons` in the merged
held-wake context when two different reasons merge. `wakeReason` keeps
the latest reason, as before, so comment delivery and person-wake
handling do not change.
- New `withMergedWakeReasons` helper in the wake-queue domain context
module.
- `hasPendingNativeChildCompletion` matches a held wake when
`wakeReason` is `issue_children_completed` or when `wakeReasons`
contains it.
- New embedded-Postgres test in
`heartbeat-comment-wake-batching.test.ts`: a child-completion wake is
held behind a running parent, then a person's comment merges in. The
gate still sees the pending child result, and the next run receives the
comment.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-comment-wake-batching.test.ts
src/__tests__/native-child-task-read-routes.test.ts
src/modules/wake-queue` passes (16 files, 400 tests). This includes both
#21914 reopen tests.
- `cd server && npx vitest run
src/__tests__/native-status-arbiter-corpus.test.ts
src/services/workspace-recovery-admission.test.ts
src/__tests__/heartbeat-held-comment-wake.test.ts` passes (343 tests).
- `cd server && npx tsc --noEmit` passes.
- Red check: with only the source change reverted, the new test fails.
The merged context has no `wakeReasons`, and the gate check after the
merge fails with `expected false to be true`.
- `pnpm verify` exits 2 (an opaque setup dependency). Required CI
supplies broad coverage.

## Risks

- Low risk. The change adds one context key on merged held wakes and
widens one gate query. `wakeReason` does not change.
- A wider gate can hold a parent finish for a child result that a merge
used to hide. That is the intended behavior. The gate still ignores a
child result that the run already read.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 10:37:46 -06:00
DottaandClaude Opus 5.5 77ddf260bf fix(runner-e2e): keep the supervisor off the heartbeat graph and ship the plugin SDK build (#21860)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The paid Runner Full-Stack E2E workflow proves chat features against
a real source server
> - Since master f7a3497e9d, every paid cell fails before any test runs:
the source server cannot find `@paperclipai/plugin-sdk/dist/index.js`
> - The supervisor builds the plugin SDK before it starts Paperclip, but
its own static imports now load the SDK first
> - This pull request removes the heavy import edge, adds a regression
test for that class, and builds and checks the SDK once per campaign
> - The benefit is that paid campaigns run again, and a missing
workspace build fails once in the build job instead of in 199 paid cells

## Linked Issues or Issue Description

Refs #21791

**What happened?**
Chat-eval cohort 17 (master f7a3497e9d) failed in all 199 cells. Each
cell exited in the "Run paid cell" step. The private Playwright log
shows `ERR_MODULE_NOT_FOUND` for
`server/node_modules/@paperclipai/plugin-sdk/dist/index.js`, imported
from `server/src/services/reusable-sandbox-resume-error.ts`, and then
`Process from config.webServer was not able to start`. Cohort 16 (master
819f61f1fd) passed 169/199 with the same workflow steps.

**Root cause**
#21791 (df9914dbd9) added `import { NATIVE_RETAINED_OWNER_WAIT } from
"./native-controller-recovery.js"` to `native-runner-checkpoint.ts`.
`tests/runner-e2e/server.ts` statically imports
`controller-recovery-observer.ts`, which imports the checkpoint reader.
The new edge connects it to the full heartbeat graph:
`native-runner-checkpoint -> native-controller-recovery ->
native-finalization-notifications -> external-agents -> muse ->
heartbeat -> heartbeat/run-completion -> reusable-sandbox-resume-error
-> @paperclipai/plugin-sdk`. The plugin SDK exports only its generated
`dist`. The supervisor builds that `dist` in
`prepareSourceServerDependencies`, but ES static imports load before
that call. The workflow installs with `--ignore-scripts` and never built
the SDK, so the supervisor crashes at load. #21554 is not involved: it
changes only release workflows.

**Expected behavior**
The supervisor loads without generated SDK output. The campaign build
job builds and checks the SDK once, and each cell restores it.

**Steps to reproduce**
1. Remove `packages/plugins/sdk/dist`.
2. Run `node cli/node_modules/tsx/dist/cli.mjs
tests/runner-e2e/server.ts`.
3. Before this change: `ERR_MODULE_NOT_FOUND` for the SDK. After this
change: the normal `PAPERCLIP_RUNNER_E2E_SERVER_LOG is required` error,
which shows that all static imports loaded.

## What Changed

- Moved `NATIVE_RETAINED_OWNER_WAIT` and
`NATIVE_AGENT_DIRECTORY_BUSY_WAIT` to a new leaf module
`native-recovery-wait-codes.ts`. `native-controller-recovery.ts`
re-exports them, so other importers do not change.
`native-runner-checkpoint.ts` imports the leaf. This also removes a
checkpoint/controller-recovery import cycle.
- Added `tests/runner-e2e/server-startup-import-graph.test.ts`. It uses
esbuild to walk the static import graph of `tests/runner-e2e/server.ts`
and fails with the full chain if it reaches `@paperclipai/plugin-sdk`.
- `runner-full-stack-e2e.yml`: the "Build reusable runner campaign
artifacts" job runs `scripts/ensure-plugin-build-deps.mjs` (the same
certified helper the cell uses), checks that `shared` and `plugin-sdk`
dist and their completion markers exist with a clear `::error::`, and
ships both dist trees in the campaign bundle. The cell restore step
checks both entry points with a clear `::error::`. The helper's
fingerprint receipt lets the cell reuse the shipped output instead of a
rebuild.
- `workflow-security.test.ts` pins the build, packaging and guard.

## Verification

- Red/green: `npx vitest run --config tests/runner-e2e/vitest.config.ts
server-startup-import-graph` fails with the old import and passes with
the fix.
- Local reproduction above: fails before, loads after.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
workflow-security server-startup-import-graph native-completion`: 137
passed.
- `node --test tests/runner-e2e/native-completion-git-source.test.mjs`:
36 passed.
- Server `tsc --noEmit`: clean.
- Paid proof of the source fix: one cell
(`chat-features.runner-opencode.local.access.private-resource`),
dispatched with `--ref master -f
target_branch=fix/runner-e2e-plugin-sdk-dist`:
https://github.com/paperclipai/paperclip/actions/runs/43804619166 . That
run uses the master workflow and this branch's source, so it proves the
import-graph fix alone restores the cells.
- The workflow changes (SDK build, bundle and guards) cannot run in a
paid campaign before merge, because paid runs use the default-branch
workflow. `workflow-security.test.ts` pins them. A 1-case paid check
from master follows the merge.
- The failed "Authorize paid campaign" check on this commit is from a
dispatch with a branch `--ref`. The default-branch gate rejected it as
designed. It is not a PR-event check.

## Risks

- Low risk. The constants keep their values and are re-exported from the
old module.
- The campaign bundle grows by the `shared` and `plugin-sdk` dist trees
(small). If the fingerprints do not match in a cell, the helper
rebuilds, as before.

## Model Used

- Claude, `claude-opus-5-5`, via Claude Code with tool use (shell, file
edit, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 10:08:11 -06:00
DottaandClaude Opus 5.5 b4fa1c7fb1 perf(recovery): load changed rows, not every task, for company hints, boots and reconnects (#21872)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The task recovery queues (stranded assignments, assigned-work
invariant, resolved dependencies, stale issue locks) check tasks to find
work that has no live run and no queued wake
> - A plain company hint, or an unscoped `"*"` reload, made each queue
load and check every eligible task of the company, at about 22 read-only
statements per task
> - Many writers send that hint (issue creation, documents, budgets,
work intents, agent status changes), every server boot reloads every
task, and the notification connection publishes a `"*"` reload each time
it opens or reconnects
> - On staging this shows as bursts of about 1,400 to 1,550 checks for
one company roughly every 9 minutes, with almost no real work
> - #21861 removed one writer. This pull request fixes the class: a
company hint, a boot and a reconnect now load only changed rows since
the queue last looked, and the hourly full pass is saved so a replaced
server keeps its schedule
> - The benefit is that settled tasks are checked by the hourly full
pass and by real changes, not by every write, deploy or reconnect

## Linked Issues or Issue Description

Refs #21861, #21831, #21867

**What happened?**
A company hint `{ companyId }` on a task queue loaded every eligible
task of the company with no changed-rows filter. The same happened for a
`"*"` reload. Three sources send these all the time:
1. Each server boot starts every task queue with a `"*"` reload.
2. `reconcileDatabaseWorkNotifications` publishes a `"*"` reload to
every queue each time the deployment notification connection opens or
reconnects. The connection opens at boot and again whenever a native
execution starts after an idle period.
3. Many writers send a bare company hint: issue creation, document lock,
unlock, restore and delete, budget pause and resume, work intent
promotion, agent status changes, watchdog writes, and others.

In a staging-shaped fixture each server replacement caused two full
reloads (boot and connection open).

**Expected behavior**
A company hint, a boot or a reconnect re-checks the tasks whose rows
changed. A change that makes tasks eligible without writing any of their
rows (a company or project resume, a tree hold release) still re-checks
every task of the company. An agent status change re-checks that agent's
tasks. The hourly full pass stays the safety net, also across server
replacements.

**Steps to reproduce**
1. Seed one agent with settled in-review tasks and a live native run.
2. Start the recovery workers with
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:1`.
3. Restart the workers, call `reconcileDatabaseWorkNotifications(db)`,
or create an issue.
4. See one `check` event per settled task with `wake: "reload"` or
`wake: "company"`.

**Paperclip version or commit**
`master` at 278660149c.

## What Changed

- `recovery/work.ts`: the full-pass bookkeeping now decides a row filter
per load.
- A plain company hint and the changed-rows backstop load the rows
changed since the queue last looked at the company
(`changedTaskCondition`, minus the existing 5-minute margin). They load
every task only when no full pass is recent.
- A boot and a reconnect use a new reload scope. One statement loads,
per company, every task when that company has no saved full pass within
the hour, and otherwise the rows changed since that saved pass (or since
this server's last reload).
- An agent scope resolves the tasks the agent is assigned, currently
reviews, or holds a run lock on, and checks only those.
- The graph fallback for a graph too large to list now sends the
explicit `everyTask` hint, so it keeps its old meaning.
- When scheduling is suppressed, the queue forgets its passes and
ignores older saved ones, so the first pass after suppression checks
every task (entries were dropped while suppressed).
- New table `issue_recovery_deferrals` (queue, issue, company, due_at),
in the same migration. A row exists while a task's check has deferred
itself (grace period, back-off, live owner lease). A boot or reconnect
reload also loads these tasks and checks them again, which rebuilds each
deadline. Deferrals cannot be rebuilt from changed rows: several defer
with no limit while none of the task's rows change (see the table
below). The store is built so that any race or lost write costs at most
one extra check, never a missed one:
- A deferring check upserts the row with its deadline when the deadline
differs from the one this server last read or wrote (one statement; none
when unchanged). A failed write keeps the full pass unsaved, so a
successor's full pass rebuilds the deadline, and drops the server's
memory of the row.
- A check that no longer defers deletes the row only if `due_at` still
equals the deadline this server read or wrote (one statement, only when
it knew of a row). A row another server rewrote since stays.
- A reload deletes the row of a task it did not load (the task left the
queue, for any reason) when, in the same statement, `due_at` still
equals the value the reload read and either the task is closed or that
deadline passed more than 10 minutes ago. This bounds the table without
listing each queue's eligibility rules.
  - A check that does not defer and knows of no row adds no statement.
- Two overlapping servers both keep the row current; a crash leaves it
in place.
- New table `issue_recovery_full_passes` (company, queue, started_at),
migration `0488`. A full pass writes it only after every task it loaded
has been checked (a check that fails keeps it unsaved), so a server
replaced mid-pass leaves no record and its replacement runs the full
pass again; a server with no memory of its own reads it. This keeps the
hourly full pass on one schedule for replaced and overlapping servers.
Reads and writes are one statement each and only happen when a full pass
is due or done. A failed read means a full pass; a failed write means
one extra full pass later.
- `issue-recovery-notifications.ts`: new scope fields `everyTask`,
`agentId` and `reload`, `notifyAgentRecovery`, `resolveAgentIssues`, and
the reload hint.
- `run-dispatch-notifications.ts`: `RunDispatchScope.everyTask` and
`notifyAgentStatusDispatch`. The agent lifecycle status-change effect
uses it instead of a company hint.
- `work-signals.ts`: `setDatabaseWorkReconnectHint` lets a topic declare
the hint it gets on reconnect instead of an unscoped reload. The task
queues declare the reload hint. Other topics keep the unscoped reload.
- `scoped-deadline-work.ts`: optional `initialHint` for the first pass
after registration. The task queues start with the reload hint.
- Writers that make tasks eligible without changing their rows now send
`everyTask`: company resume (`companies.ts`), project and company budget
resume (`budgets.ts`), tree hold resume and hold creation
(`issue-tree-control.ts`), conversation reset that releases a tree hold
(`agent-conversations.ts`), work intent promotion for `pause_lifted` and
`tree_hold_released`, task reparenting and task deletion (`issues.ts`),
company import relations (`company-portability.ts`), and company-wide
chat milestones.
- Writers with a known task now send a task or graph hint instead of a
company hint: document lock, unlock, restore and delete
(`documents.ts`), blocker edits and attachment delete (`issues.ts`),
watchdog writes (`task-watchdogs.ts`).
- Native run writers that sent a company hint now send a graph hint for
the run and its task: Runner file handoff
(`native-runner-file-handoff.ts`) and chat attachment reuse
(`chat-attachment-reuse.ts`).
- Agent-scoped hint: only the agent lifecycle status-change effect sends
it, for transitions that go through the lifecycle (pause, resume, hold
and release, terminate, error recovery, environment pause, and budget
holds through `applyAgentBudgetHoldInTransaction`). A run starting or
finishing writes the agent row directly (execution start,
`finalizeAgentStatus`) and sends only run-scoped hints, which re-check
the run's task and its graph. Queued-run dispatch after a run finishes
is unchanged (its own dispatch queues).
- `assigned-work-sweep.ts`: `listSuspects` takes the row filter as SQL.
- Tests: new census `issue-recovery-company-reload-census.test.ts`, new
`issue-recovery-saved-deferrals.test.ts` (real test database, recorded
checks), a `restart` helper in the recovery fixture, and a work-signals
test for the reconnect hint.

## Verification

Census (`issue-recovery-company-reload-census.test.ts`, real workers,
fake clock, 28 simulated minutes, 140 + 20 settled reviews, an agent
with a live native run, no fixture agent can start a run). Steps: server
replacement plus connection open, issue creation, document lock, project
budget hard stop, agent pause, agent resume, second server replacement
plus connection open.

| | Settled-task checks | Stranded checks by wake | Statements |
| --- | --- | --- | --- |
| Before (master code) | 1,440 (320 per replacement, 160 per other step)
| company 830, reload 662, hint 163 | 33,485 |
| After | 0 | hint 168, reconnect 12, backstop 7, agent 5, company 5 |
4,293 |

An earlier run of the same census at staging size (700 + 80 settled
reviews, 35 minutes) gave 8,028 settled checks and 165,882 statements
before, and 240 settled checks (only the paused agent's own tasks) and
9,714 statements after.

Guard tests in the same file:
- A task stranded by a write that sends no hint, right before each step,
is checked within 1 minute (replacements, issue creation, budget stop,
pause, resume) or within the 5-minute backstop plus 2 minutes (document
lock, which now re-checks only its own task). Passes before and after.
- A server replaced after its first full pass loaded every task but
before it checked them all: the replacement checks every unchecked task
within 2 minutes. Fails before the deferred save (112 tasks never
checked), passes after.
- Run start and finish (same census file): an agent with a settled
backlog runs start, finish, start, finish through the run row, the
execution-start agent write, the execution-run hints and
`finalizeAgentStatus` (no process runs). 0 settled-task checks on this
branch and on master; only the run's task is checked. A run that fails
has its task checked within 1 minute of the run ending.
- `issue-recovery-saved-deferrals.test.ts`: a task's check defers 20
minutes, the full pass is saved, the server is replaced 5 minutes later
without shutdown work. The replacement checks the task within 1 minute
of the original deadline. Fails at the previous head (never checked
before the next full pass), passes after.
- Same file: server B saves a deadline; server A boots and reads it; B
clears it and defers again to a new deadline; A's check stops deferring
and clears the row it read; B is replaced. The successor checks at B's
deadline. Fails at the previous head (A deleted B's row by id), passes
after.
- Same file: a saved deferral is removed at the next reload once its
task closed.
- Same file: an open task that leaves the queue (taken over by a user)
while deferred has its row removed by the first reload after its
deadline is 10 minutes past. Fails at the previous head (row kept
forever), passes after.
- Same file: after scheduling suppression, a plain company hint checks
every task again even though a recent full pass is saved.
- A change that no changed-rows clause sees is not checked by either
server replacement, and is checked by the next full pass on the saved
schedule. Fails before (each replacement did a full pass), passes after.

The file takes about 30 s locally. Every wait has a 20 s real-time limit
and each test a 2-minute limit, as in #21867.

Focused suites (all `issue-recovery-*` suites and censuses,
assigned-work, scoped deadline work, deployment notifications, budgets,
companies, run dispatch notifications, and related idle tests) on the
final head (34 files with the new deferral test): 288 passed, 1 skipped.
The #21867 agent-row census still reports 0 settled checks.
`packages/db` `work-signals.test.ts`: 34 passed. Server `tsc --noEmit`:
passed. `pnpm --filter @paperclipai/db check:migrations`: passed. `pnpm
verify` returned exit 2 (broad input: the new migration), so required CI
supplies broad coverage.

## Risks

Time bound that changes:
- A task stranded by a change that no changed-rows rule sees was often
found sooner by an unrelated company hint, boot or reconnect. It is now
found by the next full pass: at the next backstop (needs activity in the
company), boot or reconnect after one hour from the last saved full
pass. This was already the documented worst case.

Row changes the changed-rows rules see (`recovery/changed-tasks.ts`),
for auditing completeness. A task is loaded when, since the window
start:
- its own `issues.updated_at` moved;
- a `heartbeat_runs` row whose `context_snapshot.issueId` is the task
has `updated_at` in the window;
- an `issue_recovery_actions` row with the task as `source_issue_id` has
`updated_at` in the window;
- an `agent_wakeup_requests` row whose `payload.issueId` is the task has
`updated_at` in the window;
- an `issue_thread_interactions` row of the task has `updated_at` in the
window;
- an `issue_comments` row of the task was created in the window;
- its assignee agent row has `updated_at` in the window, is not
`running`, and has no heartbeat since that update (pause, resume, hold,
termination, configuration).
Not seen: company, project, tree-hold and budget rows (their resumes
send `everyTask`), the activity log, other agents' rows, lease renewals
that do not move `updated_at`, and time passing. Deferrals are covered
separately by the saved deferral rows.

Rebuilt across a replacement (no change): every pending deferral, by the
saved deferral rows. Every way the four checks defer, from tracing each
`onDeferredDeadline` call:

| Queue | Deferral | Deadline | Bounded after a seen row change? |
| --- | --- | --- | --- |
| stranded | terminal run still settling | run lease, or now + 30 s
while the run executes in memory | no (repeats while it executes) |
| stranded | chat agent daily limit | next 00:00 UTC | < 24 h |
| stranded | accepted interaction, no run since | interaction resolved +
2 min | yes |
| stranded | transient continuation back-off | run finished + up to 16
min | yes |
| dependencies | refused or failed stall wake | last attempt + up to 20
min | yes |
| dependencies | waiter's workspace hold | now + 1 h | no (repeats while
held) |
| dependencies | waiter's agent takes no wakes | now + 10 min | no
(another agent's row) |
| locks | live legacy controller lease | lease expiry | no (renewal does
not move `updated_at`) |
| locks | lapsed lease, owner proven alive | lease + up to 5 min | yes |
| locks | run updated less than 5 min ago | run updated + 5 min | yes |
| locks | adopted or lost child process still alive | now + 30 s | no
(repeats while it lives) |
| locks | prior-boot setup | about run updated + 5 min | yes |
| assigned work | settling, saved-work due, reattach | row time + up to
10 min, or now + 2 s during reattach | yes |
| assigned work | repair budget, company run brake, stalled queued run,
saved work busy | now + 60 s to 1 h | no (repeats while the condition
lasts) |
| assigned work | `waiting_unchanged` (same refusal state) | last
refusal + up to 6 h (activity log) | no |
| assigned work | continuation cap, spacing, fallback back-off | first
attempt + up to 1 h | no (activity log) |

`unchanged` (assigned work only) changes queue priority, not whether a
deferral is pending.
- Unchanged: the 5-minute changed-rows backstop, the 1-minute bound for
paused, resumed, held and terminated agents (covered by the
#21861/#21867 guards and the new agent scope), and whole-company
re-checks for company and project resumes and tree hold releases.

Other risks:
- During the deploy that ships this change, the old build still sends a
bare company hint for a company or budget resume. The new build reads
that as changed rows only. The old build re-checks every task for its
own hint while its workers run; otherwise the next full pass does. After
this deploy both builds send `everyTask`.
- If another server's `everyTask` hint is lost while this server's
notification connection is closed, the reconnect no longer re-checks
every task. The writing server checks its own hint; otherwise the next
full pass does.
- The first boot after this deploy has no saved full passes, so it runs
one full pass per company (as every boot did before).
- Nothing is added before a run's first turn or on the run-start path.
- Schema change: two new company-scoped tables with cascade foreign
keys, no change to existing tables. The old build ignores them. Neither
is a timer: no deadline is stored or scheduled from them, they only tell
a boot which tasks to check again.
- Extra statements: per deferring check, one upsert when its deadline
changed (a deferral to `now + 30 s` changes each time, so about two
writes a minute per such task) and none otherwise; one delete when a
check stops deferring; one read of the queue's deferral rows and at most
one prune per reload; one write per completed full pass.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 05:52:57 -06:00
DottaandClaude Opus 5.5 d149e30323 fix(native): stop a finished warm owner's retained Runner when suspend is refused, and release a suspended Runner whose process left (#21853)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes an idle session after 5 minutes, and closes it
again for an environment delete, idle sleep, or a restart.
> - In a zero-downtime release-swap test, two warm closes failed and did
not recover.
> - In the first case, the Runner of a finished run refused
`runner.suspend`. The server kept the Runner alive and the session
registered. An environment delete returned 409 thirty-five times, and
only an outside sandbox destroy released the session. A live Runner and
a paid sandbox were held, and idle sleep was blocked.
> - In the second case, the Runner durably suspended and exited, but its
process check returned exit 44 (`process_identity_changed`) on every
attempt. Exit 44 also covers "a live process without `--runner-id`", so
no proof could release the owner.
> - Also, the bounded idle retries from #21782 did not run for the first
case.
> - This pull request stops the retained Runner of a finished owner with
the verified port-holder retirement, releases the owner only when a
fresh check reports `process_not_running`, splits exit 44, accepts a
durably suspended Runner with no live process that carries its runner
id, and keeps the idle retries alive after a failed sweep.
> - The benefit is that a finished warm session can no longer hold a
live Runner and a sandbox until someone destroys the sandbox.

## Linked Issues or Issue Description

Refs #21782
Refs #21835

**What happened?**
Release-swap gate wave, two warm-session close failures:

1. A finished run's idle close failed with
`native_session_close_unrecoverable`: `runner.suspend` failed,
`runnerRetained: true`, Runner lifecycle `ready`. Only close attempt 1
was logged. An environment delete with `forceStopUnsettled` could not
discard the session, because `closeWarmNativeSession` discards only with
a Launcher task owner or a suspended Runner. The delete got 409
thirty-five times. The session was released only after an outside
sandbox destroy.
2. A Runner's `runner-state.json` showed lifecycle `suspended`, with
drain and suspend `completed`. The process check exited 44 on every
attempt, and the Runner process was gone later while the sandbox stayed
started.

**Expected behavior**
A finished owner whose Runner refused suspend is stopped and released,
and a failed stop keeps it held. A durably suspended, settled Runner
with no live process that carries its runner id releases its owner. A
bare `process_identity_changed` never releases an owner.

**Steps to reproduce**
1. Keep a warm sandbox session idle after its run succeeds.
2. Make its close fail with `NativeSessionCloseUnrecoverableError` with
`runnerRetained: true`.
3. Call `closeWarmNativeSessionsForEnvironment({ forceStopUnsettled:
true })`. On master the result is `failed: 1`, every time.

**Why the retries did not run**
- The transport memoizes `close()`. A second close returns the same
rejected promise and does no sandbox work. The 27 environment-delete
failures have identical timings (`totalMs: 6055`), and the other
Runner's daemon log shows no sandbox commands during its three retries.
So a retry can only succeed through a fresh proof, not through the
close.
- `closeIdleWarmNativeSessions` (environment delete, idle sleep,
restart) clears the entry's idle timer, which is also the pending retry
timer, and never re-arms it after a failed close. The test "keeps
retrying a failed idle warm close after a failed environment sweep
cleared its retry timer" is red on master for this reason.
- The log shows one more `close()` request on the same transport in the
same second as attempt 1, and no attempt 2. Every path that closes an
idle owner from outside the timer clears that timer. The log does not
show which caller it was, so this part is not proven.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`:
- A warm entry keeps its sandbox runner and remote directory when the
run used a sandbox that the server reaches directly (no Launcher task).
  - `warmCloseFailureProvesOwnerGone` adds two proofs:
- The close failed with `NativeSessionCloseUnrecoverableError`,
`runnerRetained: true` and not suspended. The server stops the session's
own Runner with `retireSandboxRunnerPortHolders` (owner run must be
`succeeded` or `cancelled`, same agent, idle state; TERM, then KILL). It
releases the owner without a checkpoint only if a fresh process check
then exits 45 (`process_not_running`). A stop that cannot be confirmed
keeps the owner.
- The close failed with the new `process_runner_id_missing`. A fresh
sandbox inspection must show lifecycle `suspended`, settled state (empty
outbox, no pending terminal delivery, provider cleanup, or warm
transition), matching run, session and runner ids, and no process that
carries `--runner-id <id>`.
- Both the idle-timer failure path and `closeIdleWarmNativeSessions`
(environment delete with `forceStopUnsettled`, idle sleep, restart) use
these proofs.
- The first Runner-check or refused-suspend close failure for a sandbox
owner logs one evidence line: stage, reason, Runner lifecycle, suspend
command status, proof window, and an inspection of the sandbox (durable
lifecycle, settled, identity match, marker pid, and pid/state of each
`paperclip-runnerd` or runner-id process). It logs no nonce and no
command line.
- A sweep that fails to close an owner whose idle close already failed
re-arms the bounded idle retry.
-
`server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts`:
`REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT` exits 48
(`process_runner_id_missing`) for a live process whose command line
lacks the runner id. Exit 44 now means only an identity file mismatch.
- `server/src/services/native-runtime/sandbox-runner-port-holders.ts`:
`retireSandboxRunnerPortHolders` takes `only: { sessionDirectory, runId
}` to retire one session's Runner. New `sandboxSessionRunnerExited` and
`inspectSandboxSessionRunner` (a read-only `node` script).

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts
src/services/native-runtime/sandbox-runner-port-holders.test.ts
src/services/native-runtime/sandbox-runner-port-holders.integration.test.ts
src/services/runner-lost-mid-turn-continuation.test.ts
src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts`
- Red on master (master sources with these tests): "stops the retained
Runner of a finished warm owner that refused suspend ..." returns
`failed: 1`. "keeps retrying a failed idle warm close after a failed
environment sweep ..." sees 2 closes, not 3. "releases at controller
restart a warm owner whose Runner durably suspended ..." returns
`failed`.
- Negative tests: a Runner that survives TERM and KILL stays `failed: 1`
with no exit check. A run that is still `running` is never signalled. A
live process that still carries the runner id, and a bare
`process_identity_changed`, keep the owner.
- Script tests: the process check exits 48 for a live process without
`--runner-id`, 0 with it, and 44 for a changed marker. The inspection
script reports state, marker pid and runner processes, never the nonce
or a command line, and fails closed without `/proc` or through a
symlink. The port-holder integration test stops only the named session's
finished Runner (a real local process) and then proves it exited.
- `pnpm verify` exited 2 (opaque setup dependency), so focused checks
and required CI give the coverage. `npx tsc --noEmit -p server`: no
errors.

## Risks

- A finished owner's retained Runner is now stopped when its suspend was
refused. The session is released without a new checkpoint. The last
saved checkpoint stays the recovery source, which is the same as an
environment delete today.
- The stop uses the same rules as a successor's port-holder retirement:
finished owner run, same agent, idle Runner state, and a verified
process identity. Any doubt keeps the owner.
- The stop can take up to about 20 s (TERM and KILL waits) inside a
sweep, while the entry stays fenced.
- The first close failure of a sandbox owner runs one read-only
inspection command (5 s limit).
- Previous-release story: no persisted state or launch identity changes.
The scripts are sent with each command, so a server always runs its own
exit-code table. Exit 44 from the previous release maps to
`process_identity_changed`, which still never releases an owner.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 20:54:47 -06:00
DottaandClaude Opus 5.5 03309f1650 fix(native): release an idle warm owner whose sandbox Runner port a later run took over (#21835)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes idle sessions, and closes them again before it
exits.
> - A sandbox has one Runner port. Before a new run launches its Runner
on a reused sandbox, it retires every other Runner on that port.
> - In a zero-downtime release-swap test, a run finished on the outgoing
server and stayed warm. 77 ms later, a follow-up run started on the
incoming server. It reused the same sandbox, started a fresh provider
session, retired the old Runner, and launched its own Runner on the same
port.
> - The outgoing server's idle close could not suspend the old Runner.
Its checkpoint was correctly refused with
`native_runner_checkpoint_owner_lost`. The idle timer retried 3 times
and then kept the owner registered. That phantom owner blocked idle
sleep for 4.8 minutes, and the shutdown close failed again.
> - The existing proofs that a warm owner is gone (#21782) do not count
"a later run launched its own Runner on this sandbox".
> - This pull request adds that proof. When it holds, the server forgets
the owner without a checkpoint. The checkpoint fence does not change.
> - The benefit is that a superseded warm owner no longer blocks idle
sleep or fails a clean shutdown.

## Linked Issues or Issue Description

Refs #21627
Refs #21782

**What happened?**
An idle warm owner finished on the outgoing server. A follow-up run on
the incoming server reused the same sandbox and launched its own Runner
on the sandbox's single Runner port. The owner's idle close failed with
`native_runner_checkpoint_owner_lost` 4 times. The owner stayed
registered, held the idle-work flag, and blocked idle sleep. At SIGTERM
the restart sweep logged "idle warm native sessions did not close" with
"runner did not durably suspend before checkpoint". No task was
affected.

**Expected behavior**
A failed idle close releases its owner when a later run proves that the
owner's Runner is gone. The checkpoint stays refused while the successor
is live.

**Steps to reproduce**
1. Finish a warm run on a sandbox and keep it idle.
2. Start a successor run for the same task on another server. Give it a
lease on the same sandbox, its own `native.process_identity_recorded`
event and its own `session.started` event.
3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result
is `closed: 0, failed: 1`, and `readNativeSessionIdleWork()` stays
`present`.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: new
`warmRunnerPortTakenBySuccessor`, used by
`warmCloseFailureProvesOwnerGone` after the existing sandbox-termination
proof. It holds only when another run has all of these:
- an `environment_leases` row with the same environment and
`providerLeaseId` as the owner;
- its own `native.process_identity_recorded` event (`source_event_id is
null`, so an inherited identity does not count);
- a `session.started` event from its own Runner instance
(`source_instance_id` equals the run's `runner_instance_id`);
  - both events created after the owner run's `finishedAt`.
- When the proof holds, the idle-timer path and the controller-restart
path log "warm native session's Runner or sandbox is gone; released its
owner" and forget the owner without a checkpoint. Both paths already
call `warmCloseFailureProvesOwnerGone`.
- `validateNativeRunnerCheckpointOwner` does not change. A checkpoint
still cannot land under a live successor.
- Cost: the proof runs only after a failed close. It does one
primary-key read of the owner run and one query on
`environment_leases.provider_lease_id` (indexed), joined to
`heartbeat_runs` by primary key, with `heartbeat_run_events` read
through `(company_id, run_id)`.
- New real-database test file
`server/src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts`.
- `native-session-executor.test.ts`: one environment-delete test counts
the extra proof read when the first proof does not hold.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts`:
6 passed.
- "releases an idle warm owner whose checkpoint is refused because a
successor on another server launched its own Runner on the same sandbox:
own_runner": the close is refused with
`native_runner_checkpoint_owner_lost`, the result is `closed: 1, failed:
0`, the idle-work flag is `none`, the owner has no checkpoint row, and
`assertNativeRunnerCheckpointOwner` still throws
`native_runner_checkpoint_owner_lost`.
- These stay failed and keep the owner registered: the successor is on
another sandbox; the successor has only
`native.process_start_requested`; the successor has only inherited
events (`source_event_id` set); the successor's events come before the
owner finished.
- "releases a superseded idle warm owner from its idle timer without
writing a checkpoint": the idle timer releases the owner after the first
refused close.
- Red on master (test file only, fix reverted): `own_runner` fails with
`closed: 0, failed: 1`; the idle-timer test fails with the idle-work
flag still `present`. The 4 negative cases pass on master and with the
fix.
- `cd server && npx vitest run
src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts`:
9 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: all
passed.
- `cd server && npx tsc --noEmit -p .`: no errors.
- `pnpm verify`: see the comment below for the result.
- Not run: a restart-chaos wave. The release-swap timing is reproduced
in the database test, not on a live sandbox.

## Risks

- A superseded owner is now released without a checkpoint. Its old
provider conversation cannot be resumed. The successor had already
started a different conversation and holds the task, so no one would
resume the old one.
- The proof needs the successor's own Runner identity and own session
start after the owner finished, on the same sandbox lease. A successor
that reused the owner's Runner, only requested a start, or recorded its
events earlier does not count. In those cases the close fails as before.
- The proof adds two small indexed reads after a failed close only. The
normal close path does not change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 17:49:01 -06:00
DottaandClaude Opus 5.5 f71aa135a0 fix(server): release sandbox leases for runs stopped before runner attach (#21788)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a `sandbox` environment run each task in a paid remote
sandbox that an environment lease tracks
> - When a run stops, the control plane must release that lease, and
terminating an agent waits until every lease of the agent is released
> - In a load test, a run stopped after its sandbox was ready but before
its runner started never released its lease, so the sandbox kept running
and the agent stayed in "terminating" forever
> - In the same test, terminating an agent whose failed run had a
scheduled same-run retry waited about 18 minutes, until the stale retry
time
> - This pull request makes lease release finish for every stop window,
records any release that must wait with a retry time, adds a hint-driven
safety net, and makes termination cancel a pending native retry at once
> - The benefit is that a stopped run never leaves a paid sandbox
running, and agent termination finishes in seconds without manual repair

## Linked Issues or Issue Description

**What happened?**
A load test with ten OpenCode agents in a `sandbox` environment stopped
its agents mid-run (pause, then terminate). Two defects appeared.
1. A run cancelled while it selected its provider pack (after the
sandbox was acquired and the workspace was prepared, before the runner
started) kept its environment lease `active` for more than 40 minutes.
The agent stayed in lifecycle `terminating`. The lifecycle worker
retried every pass and wrote `updatedAt` each time with no recorded
reason. Delete returned 409 "Complete termination before deleting the
agent".
2. A failed native run with a scheduled bootstrap retry kept its lease
until the stale retry time. Agent termination took about 18 minutes. The
run still showed "retry_scheduled" after the agent was terminated.

**Expected behavior**
A stopped run releases its sandbox lease within seconds. If release must
wait, the wait has a recorded reason and retry time, and a background
pass finishes it. Terminating an agent cancels a pending retry, releases
its lease, and reaches `terminated` promptly.

**Steps to reproduce**
1. Start a native run in a `sandbox` environment.
2. Stop it after the sandbox is acquired and the workspace is prepared,
before the runner starts.
3. Pause and then terminate the agent.
4. See that the lease stays `active` and the agent stays `terminating`.

**Paperclip version or commit**
Reproduced on `master` at 62033b9994 (load test build 2026.1106.0+2564).

**Relevant logs or output**
Root causes found in the code:
- `releaseEnvironmentLeasesForRun` returned with no record when
workspace retention refused a cancelled run. The refusal was permanent
when the run's native finalization owner had expired but was not
cleared. The owner is cleared only after `agentExecutionsHaveStopped`
sees every lease released, so the two waited on each other.
- `releaseRunLeases` silently skipped a lease whose prepared native
workspace was still "pending export", even for a run that never started
a runner and so never changed those files.
- The destroy path deferred on any busy warm session in the same
environment, also with no record.
- `stopInvocationsForAgents` returned on the first run that was still
live, so it did not release the leases of the other runs of the agent.
- The agent stop cancelled only `queued`, `running` and
`scheduled_retry` runs. A failed native run with a pending same-run
retry kept its lease until that retry fired.
- The lifecycle worker deferred a pending host step by 2 seconds every
pass, with no reason and no backoff.

## What Changed

- `releaseEnvironmentLeasesForRun` (`run-control.ts`):
- A native run with no runner launch evidence (no
`native.process_start_requested`, no runner event, no process id) skips
workspace retention and warm-session closing. No runner process exists
to fence, and no work ran against the prepared files.
- When retention is refused only because the run's native finalization
owner lease has expired, the release clears that expired owner
(compare-and-set on the exact expired owner of a cancelled run with no
result) and retries retention once. A recorded recovery then stops and
retains the sandbox, so the files stay recoverable.
- Every release that cannot finish now records
`metadata.releaseDeferral` (`reason`, `attempts`, `retryAt`,
disposition) on the active lease and publishes an `orphan-active-leases`
hint. The retry time backs off from 30 seconds to 15 minutes, then
retries hourly without end. The record keeps `failingSince` and the last
reason. A new Stop or terminate operation, or a server restart, triggers
an immediate attempt.
- `nativeWorkspaceExportPendingForLease` treats a cancelled run with no
runner launch evidence as owing no export.
- Safety net: `sweepOrphanedActiveLeases` (the existing hint- and
deadline-driven orphan-lease work) now retries the run's own release for
a due deferral or for a cancelled native run, after the native owner
lease has expired. The orphan-lease loader uses the stored `retryAt` as
the due time. No new timer or poller.
- Agent stop:
- It cancels a failed native run that has a pending same-run retry (or
an interrupted cancellation of one), through the existing
retry-cancellation path. A pause stamps `pauseStop` on those runs too,
so a budget resume can continue them.
- It releases the leases of every terminal run, and it skips a lease
whose deferral is not yet due.
- `agentInvocationStopBlocker` and `agentExecutionStopBlocker` return
why the stop must still wait.
- Lifecycle worker: a pending host or plugin step records
`pendingReason` and `pendingPasses` on the operation, and its retry
backs off from 2 seconds to a 60 second cap. When a release leaves a run
with no unsettled lease, the stopping agent's saved retry time is
cleared and the lifecycle queue is notified, so Pause or Terminate
finishes on the next pass.
- Cancelled-workspace preservation: a stopped run that never started a
runner has its prepared reference moved to
`nativeWorkspaceSyncReleasedUnchanged` when its lease is released, so
the preservation worker no longer selects it. The worker records the
same settlement itself for older runs, and waits on a live owner's exact
lease expiry.

## Verification

- Red on `master`, green on this branch (embedded PostgreSQL, fake
plugin sandbox provider):
-
`server/src/__tests__/heartbeat-lease-release-before-runner-attach.test.ts`:
agent pause then terminate, and board Stop, for each window between
sandbox acquire and runner attach (artifact discovery, pack select, pack
upload, runner launch, transport connect, session open). Each test
checks that the provider gets a stop or destroy call, the lease is
released, the run is cancelled, and the agent reaches `terminated`
within one extra worker pass. It also checks the orphan-lease safety net
for a leaked lease and a recorded deferral that the background pass
finishes when due.
-
`server/src/__tests__/heartbeat-terminate-pending-native-retry.test.ts`:
terminating an agent whose failed run has a bootstrap retry 17 minutes
away cancels the retry, releases the lease and reaches `terminated` at
once. The test covers an open task and a cancelled task.
-
`server/src/modules/agent-lifecycle/__tests__/pending-backoff.test.ts`:
a host step that stays pending records its reason, and its retry delay
grows to a 60 second cap.
- Related suites pass: run-control, recovery, orphaned active lease
sweep, environment runtime, agent lifecycle, native workspace
preservation and recovery, held comment wake.
- `server` typecheck passes.

## Risks

- A cancelled native run with no launch evidence now gets its prepared
sandbox destroyed instead of kept. The proof is the same launch evidence
that restart recovery uses.
- A launched cancelled run whose retention owes nothing is stopped and
retained, as before. A retention that throws (lock or write error, or a
live owner) now keeps the lease active with a recorded reason and retry
time, instead of a silent skip.
- Release now clears an expired native finalization owner of a cancelled
run. An old controller whose lease already expired then fails its next
owner check and stops, which is the normal lease rule.
- On the first start after deploy, the orphan-lease work releases leases
already leaked by cancelled native runs.
- Task state after pause then terminate: the run-scoped stop keeps the
task status (an open task stays open, a cancelled task stays cancelled),
and a run that held prepared files gets the existing workspace recovery
record on its stopped sandbox. No new state is added.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code search, shell, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:54:23 -06:00
DottaandClaude Opus 5.5 c6cdd57f7a perf(native-runtime): reuse a same-content image provider pack across builds (#21784)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote agent run needs the provider pack (Node, OpenCode and the
Runner bridge) inside its sandbox before the provider can start
> - The server reuses a pack that is installed in the sandbox image only
when its full manifest equals the server's own manifest
> - The manifest includes `runnerSourceRevision`, which is the commit
that built the server, so an image pack fails this check on every later
server build, also when every file in the pack is the same
> - The run then uploads the whole pack again, which cost about 15 s per
task in a recent load test
> - This pull request accepts an image pack whose sealed content equals
the expected pack and that differs only in build provenance, and caches
verified archive hashes in the server process
> - The benefit is that a sandbox image stays usable across server
releases that do not change the pack, and the server does less hashing
on its main thread

## Linked Issues or Issue Description

Refs #21781 (merged), which names the failed check on the run spans.

**What existing behavior does this improve?**
Selection of the remote provider pack for a new sandbox
(`provider_pack.select`).

**Subsystem affected**
Server: native runtime, remote provider pack verification and upload.

**Current behavior**
`provider_pack.verify_preinstalled` compares the complete manifest of
the image pack with the server's manifest. Any difference fails,
including `runnerSourceRevision` and `runnerInputsDigest`, which only
record where the pack was built. The run then uploads the full pack. The
server also hashes the same archive or pack directory again for every
task.

**Proposed behavior**
When the run is not bound to a saved pack and the operator did not
select a custom pack, the remote verification accepts an image pack when
(1) its own digest seals its payload, and (2) its schema and payload
equal the expected pack after the two provenance fields are removed.
Every artifact, the `dist` tree, the candidate trees, the package
versions, the Node version, the target platform and the OpenCode version
are still checked in the sandbox, against the same values. The server
checks the reported manifest again and selects it through the pack
authority, as the compatibility path already does. An upload still
happens for different content, a corrupted or partial pack, or no pack.
The server also remembers a verified transport archive (by path, inode,
size and times), so it does not hash it again for the next task.

**Reason and benefit**
In the harness (fake sandbox, 150 ms per command, 300 ms + 10 MiB/s per
transfer, 24 MiB pack), the cross-build case goes from 4.1 s and 24 MiB
uploaded to 1.1 s and 0 bytes. Other cases do not change.

**Breaking changes**
None. A saved or reattached session, and an operator-selected custom
pack, still require the exact pack identity.

## What Changed

- `remote-provider-pack-verification.ts`: new `acceptProvenance` option.
The remote script accepts a sealed pack with the same content and prints
its manifest. The server checks that manifest (`sameContentManifest`)
and returns it. A forged or different report fails with
`installed_manifest_invalid`.
- `native-session-executor.ts` (`prepareRemoteProviderPackOnTarget`):
staged and preinstalled verification use the option when
`acceptProvenanceDifferences` is set. The function returns the selected
manifest. The run caller sets the option only through
`providerPackProvenanceReuseAllowed` (not retained, no explicit custom
pack, selectable authority), and selects a different manifest through
the pack authority.
- `provider-pack-layers.ts`: `verifyArchive` skips the hash for an
archive it already verified with the same file identity.
- `remote-provider-pack.ts`: `providerPackProvenanceReuseAllowed` gate,
with tests for each case.
- `provider-pack-selection-harness.test.ts`: new harness. Uploads go
through the workspace confinement check of the Daytona sync guard. It
runs the real remote verification in local processes, with modelled
sandbox latency and transfer speed, for five cases, before and after.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/provider-pack-selection-harness.test.ts
src/services/native-runtime/provider-pack-archive.test.ts
src/services/native-runtime/remote-provider-pack-verification.test.ts
src/services/native-runtime/remote-provider-pack.test.ts
src/services/native-runtime/retained-provider-pack-source.test.ts
src/__tests__/kimi-connection-probe.test.ts` passes.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-artifact-preparation.test.ts
src/services/native-runtime/remote-provider-pack-authority.test.ts`
passes (1541 passed, 7 skipped).
- Red/green: with the source changes reverted, the harness case
`same_content_other_build` and the archive cache test fail. With the
changes, they pass.
- Harness output (load average about 9 on a shared machine; modelled
time = wall time + modelled sandbox latency):

| Case | Before | After |
| --- | --- | --- |
| Exact image pack | preinstalled, 1.1 s, 0 MiB | preinstalled, 1.0 s, 0
MiB |
| Same content, other build | uploaded, 4.1 s, 24 MiB | preinstalled,
1.1 s, 0 MiB |
| Different content | uploaded, 4.1 s, 24 MiB | uploaded, 4.0 s, 24 MiB
|
| Corrupted image pack | uploaded, 4.0 s, 24 MiB | uploaded, 4.1 s, 24
MiB |
| No image pack | uploaded, 3.7 s, 24 MiB | uploaded, 3.8 s, 24 MiB |

- The pack step runs no database statements.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- This changes which provider pack the server accepts. It accepts a
different manifest digest only when all content fields are equal and
every file is hashed in the sandbox, and only for a run that is not
bound to a saved pack. The selected manifest goes through the existing
pack authority, so launch profiles and receipts bind the installed
pack's own identity.
- The archive hash cache trusts each archive file's identity (inode,
size, times) on the server. The sandbox still checks the uploaded bundle
digest and verifies every uploaded file, so a changed local file still
fails closed.
- This does not help when the image pack has other content (for example
a different contract digest). See the hand-off notes for the image
build.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:07:56 -06:00
DottaandClaude Opus 5.5 df9914dbd9 fix(native): let the outgoing server checkpoint its idle warm session while the incoming follow-up waits for it (#21791)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap. A warm agent session stays
with the outgoing server that ran its last turn.
> - Since #21774, a follow-up that the incoming server claims during the
overlap waits (`native_controller_recovery_waiting_for_owner`) and
checks again every 5 s.
> - About 1 s after the traffic switch (#21627), the outgoing server
tries to save and close its idle warm session. The checkpoint fence
refuses that save with `native_runner_checkpoint_owner_lost`. The task's
current run is the parked follow-up. That run has its own finalization
row under another controller boot, and the fence does not accept that
kind of successor.
> - The close retry from #21782 fails the same way each time. So the
follow-up waits until the outgoing server exits, or until the
predecessor's finish time plus the idle window plus 2 min. In the CHA-5
case that was about 225 s, and the worst case is about 425 s. Speed is a
requirement.
> - This pull request records which owner a parked follow-up waits on,
and lets the fence accept the outgoing owner's final save when the
follow-up is parked on exactly that owner and has not started.
> - The benefit is that the outgoing server saves and closes the session
on its first attempt, and the follow-up continues at its next 5 s check.

## Linked Issues or Issue Description

Refs #21774, #21782, #21697, #21627, #21279

**What happened?**

During a release-swap overlap, a warm follow-up was claimed by the
incoming server while the outgoing server still owned the idle warm
session. The follow-up waited correctly. The outgoing server's idle
close failed with `native_runner_checkpoint_owner_lost` on every
attempt. The wait ended only when the outgoing server exited, about 225
s after the follow-up was created.

**Expected behavior**

The outgoing server saves and closes the session while the follow-up
waits. The follow-up sees the saved session and continues within one 5 s
check.

**Steps to reproduce**

1. Run a warm native session on a remote sandbox and let a run finish on
server A.
2. Start server B with an overlap and switch traffic to it.
3. Post a follow-up comment so that B claims the follow-up while A is
still alive.
4. A's idle close fails with `native_runner_checkpoint_owner_lost`, and
the follow-up waits until A exits.

**Paperclip version or commit**

master `d14a3b88b1`.

## What Changed

- `NativeControllerRecoveryWaitingError` carries the owner that the
follow-up waits on: the predecessor run and its controller boot.
`waitForLiveWarmSessionOwner` and the live-owner wait in
`inspectRemoteWarmSessionRecovery` set it.
- `deferNativeControllerRecovery` stores that owner in the finalization
`failureDetail.waitingForOwner`. The executor passes it through.
- `validateNativeRunnerCheckpointOwner` accepts one more successor kind:
a parked follow-up waiting on this exact owner. All of these must be
true, or the fence still refuses:
- The checkpoint holds an accepted result and has no new work (unchanged
rule).
- The follow-up finalization is `retryable_failure` with
`native_controller_recovery_waiting_for_owner`, `attempt = 0`, no
result, and no lease owner (no claim is running it).
  - `waitingForOwner` names this owner's run and controller boot.
- The follow-up run is a running native run for the same company and
task, on the same native session as the owner's run.
- The follow-up has no process fields, no own process-start events, and
no checkpoint rows.
- The fence locks the follow-up's finalization row (`for share`), so a
resumption claim and the save cannot interleave.
- Tests in `native-controller-recovery.integration.test.ts` (real
Postgres):
- The outgoing server saves and closes its idle warm session while the
incoming follow-up waits, and the follow-up resumes within one recheck.
- Negative cases: the save is still refused when the follow-up has its
own process, its own process-start event, its own checkpoint, or a live
resumption claim, or when it waits on a different owner boot or run.

## Verification

- Red on master: with the `native-runner-checkpoint.ts` change removed,
the scenario test fails with `native_runner_checkpoint_owner_lost` at
the outgoing server's first close attempt. The follow-up keeps waiting.
Green with the change.
- The negative cases pass with and without the change.
- Measured in the test scenario (warm idle window 600 s):
- Before: the follow-up waits until the outgoing server exits, or about
720 s (finish + 600 s idle window + 120 s margin).
- After: the outgoing save and close succeed on the first attempt. The
follow-up is claimable and its wait resolves at `retryAt`, 4998 ms after
it parked.
- `vitest run` of `native-controller-recovery.integration.test.ts`,
`native-runner-restart-recovery.integration.test.ts` and
`native-harness-backup-storage.integration.test.ts`: 349 passed, 14
skipped.
- `vitest run` of `native-session-executor.test.ts`,
`remote-warm-session-recovery.test.ts`,
`native-runner-checkpoint-save.test.ts`,
`native-runner-checkpoint-retirement.test.ts` and
`native-runner-checkpoint-authority.integration.test.ts`: 1566 passed, 7
skipped.
- `tsc --noEmit` for `server` passes.
- `pnpm verify` exits 2 (opaque setup dependency). Required CI supplies
the broad coverage.
- Not run: a release-swap wave. The fix is not yet measured on a live
overlap.

## Risks

- The fence is wider. It accepts only a final save with an accepted
result and no new work, and only while the current run is parked on this
exact owner and has not started. A follow-up that has a claim, a
process, process-start events or checkpoint rows still causes a refusal.
- A follow-up that a server parked before this change has no
`waitingForOwner`. The fence refuses its owner's save as before. At its
next recheck, the follow-up parks again with the owner recorded.
- While a resumption claim runs the follow-up's recheck (a short window
every 5 s), the save is refused. The close retry from #21782 then tries
again.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 11:02:31 -06:00
DottaandClaude Opus 5.5 cec8c2b3f4 perf(recovery): name the run or task in run writers' recovery hints instead of the company (#21765)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues on one Node main thread. The
`stranded-assignments` and `assigned-work-invariant` queues check
assigned tasks that have no live run. One stranded check costs about 19
database statements.
> - #21552 scoped run hints to the run's task graph. #21729 made the
5-minute backstop re-check only changed tasks. A hint with only a
company id still makes the queue re-check every open task of the
company, 25 per pass.
> - Under load on staging (7 native runs), the stranded check ran about
6.7 times per second, and its statement bundle was more than a third of
all statements. An activity fixture shows the cause: ordinary run
writers send a company-only hint once per run, and one company hint
costs about 850 checks. At 1,000 agents this work never stops.
> - This pull request makes those writers name the run (or the task)
they already know. A run that has no task now resolves to no tasks
instead of the whole company. The changed-tasks backstop no longer
counts an agent's own run heartbeat as a change to every task of that
agent.
> - The benefit: with 7 runs a minute plus one taskless run, stranded
checks drop from 18.4 to 1.0 per second and stranded-check statements
from 344 to 7 per second. Real company-wide changes (pause lifted,
budget, company status, tree holds) keep the full company pass.

## Linked Issues or Issue Description

Refs #21729, #21552, #21676, #21724

**What existing behavior does this improve?**

The issue recovery task queues (`stranded-assignments`,
`assigned-work-invariant`, `resolved-dependencies`, `stale-issue-locks`)
and the writers that wake them.

**Subsystem affected**

Server: issue recovery background work and the run lifecycle writers
that publish recovery hints.

**Current behavior**

Many run lifecycle writers know the run but publish a company-only hint,
for example `notifyExecutionRunWork(tx, { companyId })` in the native
finalizer and projection, `settleNativeTerminalWake`, the claimed-wake
settlement, and an environment lease release of a run without a task. A
queued run without a task publishes a company hint. A graph hint for a
run without a task resolves to "whole company". A comment sends a
company hint to `assigned-work-invariant` and `resolved-dependencies`.
Each of these re-checks all open tasks of the company. Also, the
changed-tasks backstop treats any change to the assignee agent's row as
a change to all of that agent's tasks, and every run start and finish
writes that row.

**Proposed behavior**

A writer that knows the run sends the run's graph hint. A writer that
knows the task sends the task hint. A run without a task affects no task
queue entry. An agent row change that is only the agent's own run
heartbeat does not mark its other tasks changed. Company-wide facts keep
the company hint.

**Reason and benefit**

The work is now proportional to activity, not to (activity x history).
See Verification.

## What Changed

- `server/src/services/issue-recovery-notifications.ts`:
- `notifyIssueRecovery` with a task sends the task hint to all four task
queues (before: only to `stranded-assignments` and `stale-issue-locks`;
`assigned-work-invariant` and `resolved-dependencies` got the company
hint). Only comments (`issue-comment-writes.ts`) call it with a task.
- `resolveIssueGraph` returns `[]` when a run-only graph has no task. A
missing task seed still returns `null` (whole company).
- `server/src/services/recovery/work.ts`: a load for an empty graph
returns no rows. The `assigned-work-invariant` queue emits the same
`check` debug event as `stranded-assignments` (with
`PAPERCLIP_DEBUG_WORK_SIGNALS=assigned-work-invariant`), with the load
reason.
- `server/src/services/recovery/changed-tasks.ts`: the assignee-agent
part of the changed-tasks condition skips an agent whose last write was
its run heartbeat (`last_heartbeat_at = updated_at`). A resume, pause or
configuration write (no heartbeat stamp) still counts.
- `server/src/services/heartbeat/queue.ts`: a queued run without a task
gets its id before the insert and publishes a run hint, not a company
hint. The daily-cap comment release publishes the run's graph hint.
- `server/src/services/environments.ts`, `work-intent-outbox.ts`,
`chat-inbound-wake-notice-notifications.ts`: a lease release passes its
run id. `promoteDeferredWorkIntents` scopes `lock_released` with only a
run to that run's graph. `notifyChatInboundWakeNotices` accepts an
optional `runId` for its settlement hint.
- Native and recovery run writers now pass the run id they already have
to `notifyExecutionRunWork`, `writeExecutionControl` or
`notifyExternalConversationSettlement`: `native-run-finalizer.ts` (3),
`native-run-projection.ts` (2), `native-session-executor.ts` (3),
`wake-request-claimed-settlement.ts`, `native-superseded-retry.ts`,
`native-safe-replacement.ts`, `native-restart-recovery.ts` (2),
`native-failed-workspace-recovery.ts`,
`native-recovery-preparation-failure.ts`,
`execution-control-reconciliation.ts`,
`execution-recovery-resolution.ts` (2). These are the native equivalents
of the legacy run paths in `heartbeat/queue.ts`, which already pass the
run id.
- Tests:
- `server/src/__tests__/issue-recovery-activity-hints.test.ts` (new):
staging-shaped company (780 in-review tasks with succeeded native runs,
40 in progress, 30 to-do, 50 agents) on embedded PostgreSQL, the real
recovery, dispatch and retry-cancellation workers, a fake clock, and
real process-adapter runs. Each run is woken by a user comment through
`heartbeat.wakeup`, starts, calls back to post an agent comment and move
the task to `in_review`, and finishes. One (or five) taskless runs a
minute run beside them. It counts checks by queue and reason, task-queue
signals by caller, and statements by owner. It asserts the workload
(runs started and succeeded, agent comments, run events, zero agent
write failures, at most 10% failed runs), then no company check and no
company-only task-queue signal. In CI it runs the 7-runs-a-minute
workload for 4 minutes (about 1 minute of real time). With
`RECOVERY_ACTIVITY_REPORT=<file>` it runs both workloads at full length,
attributes statements by stack, and writes the numbers below to that
file.
- `server/src/__tests__/issue-recovery-deadlines.test.ts`: pickup tests
for each narrowed writer (see Verification).

### Publishers of company-only (or `*`) hints to the recovery task
queues

"Per" is how often the writer fires on a busy instance. "Known" says
whether the run or task is known at the call site.

Narrowed in this pull request:

| Call site | Event | Per | Known | Now |
| --- | --- | --- | --- | --- |
| `heartbeat/queue.ts:5396` `enqueueWakeup` | queue a run without a task
(timer, manual, chat without a task) | each taskless run | run (id
generated) | run graph (no tasks) |
| `environments.ts:1780` `releaseLease` ->
`notifyChatInboundWakeNotices` | environment lease released | each run
without a task | run | run graph |
| `environments.ts:1792` `releaseLease` ->
`promoteDeferredWorkIntents(lock_released)` | lease released | each run
without a task | run | run graph |
| `environments.ts:1934` `releaseLeasesForRun` | leases of a run
released | each run without a task | run | run graph |
| `issue-comment-writes.ts:12` (`assigned-work-invariant`,
`resolved-dependencies` only) | comment | each comment | task | task |
| `native-runtime/native-run-projection.ts:54`
`settleNativeTerminalWake` | native run's wake claim settled | each
native run finish | run | run graph |
| `native-runtime/native-run-projection.ts:177` | native run status
projected | each native run finish | run | run graph |
| `native-runtime/native-run-finalizer.ts:408`, `:629`, `:2080` | native
finalization terminal / failure / projection | each native run finish |
run | run graph |
| `native-runtime/native-session-executor.ts:13018`, `:13085`, `:13284`
| native execution control deadline, cancelled retry, pre-result
settlement | each native run end | run | run graph |
| `recovery/wake-request-claimed-settlement.ts:67` | claimed wake
settled after the run | each run whose claim is reconciled | run | run
graph |
| `heartbeat/queue.ts:590` | daily-cap comment key release | each run
cancelled at the daily cap | run | run graph |
| `native-runtime/native-superseded-retry.ts:42`,
`native-safe-replacement.ts:613`, `native-restart-recovery.ts:1033`,
`:1935`, `native-failed-workspace-recovery.ts:164`,
`native-recovery-preparation-failure.ts:30`,
`execution-control-reconciliation.ts:270`,
`execution-recovery-resolution.ts:830`, `:965` | native recovery
terminal writes | each recovered run | run | run graph |
| `resolveIssueGraph` for any run-only hint of a run without a task (for
example `setWakeupStatus`, `run-lifecycle.ts:1570`) | run lifecycle |
each taskless run event | run | no tasks |

Left with the company hint (company-wide fact, or the graph is not
known):

| Call site | Event | Per | Why kept |
| --- | --- | --- | --- |
| `budgets.ts:371`, `:386`, `:429`, `:437`, `:454` | budget policy
changed, scope paused or resumed | operator or budget event |
eligibility of every task can change |
| `work-intent-outbox.ts:79` with `pause_lifted`, `tree_hold_released`,
`scheduling_resumed`, `reconciler_backstop`, and `accounting_settled`
with an active hard stop | hold release | operator event;
`accounting_settled` once per run only with a hard-stop policy | can
admit any task; guarded by "keeps the company pass for hold releases
that can admit any task" |
| `companies.ts:77`, `:350`, `:353`, `:361`, `:512`, `:522` | company
created, paused, resumed, archived | operator | company-wide |
| `agent-lifecycle.ts:65`, `:72`, `agent-lifecycle-driver.ts:29`,
`agent-configuration.ts:130`, `:159`, `agent-deletion.ts:61` | agent
paused, resumed, configured, deleted | operator | can change many tasks,
and other agents' review paths |
| `access.ts:570`, `:752`, `:912`, `:926`, `:960`, `:975`, `:1175`,
`board-claim.ts:129`, `:142`, `middleware/auth.ts:706` | membership and
authority changes | operator, first sign-in | company authority |
| `chat-channels/endpoints.ts:461`, `identity-links.ts:323`,
`resource-reconciliation.ts:290`, `slack/setup-services.ts:50`,
`chat-github-management.ts:182`, `:352`, `chat-work-notifications.ts:50`
| chat authority changes | operator | company authority |
| `instance-settings.ts:666`, `:669`, `:680` (`*`) | instance setting
changed | operator | instance-wide |
| `issue-tree-control.ts:809`, `:889`, `:1236`, `:1339` | tree hold
created or released | operator | many tasks |
| `company-deletion.ts:83`, `company-portability.ts:6560` | company
delete, import | rare | many tasks |
| `issues.ts:8015` (blockers replaced), `:11165` (task created),
`:12685`, `:12699` (task deleted), `:12991`, `:14618` | task graph
changes | per task create or blocker edit | graph moves; #21552 kept
these |
| `documents.ts:615`, `:673`, `:757`, `:808`, `:860` | document delete
and restore | rare | #21552 kept these |
| `task-watchdogs.ts:854`, `:1841` | watchdog changes | rare | #21552
kept these |
| `system-recovery.ts:17`, `:70`, `task-retry.ts:294`,
`workspace-recovery-capture-policy.ts:59`,
`workspace-recovery-stop-proof.ts:122` | system and workspace recovery |
per recovery | not shown to stay in one graph |
| `agent-conversations.ts:607`,
`chat-channels/telegram/attachment-failure.ts:83`,
`native-runtime/chat-attachment-reuse.ts:1754`,
`native-runner-file-handoff.ts:1317` | conversation and attachment paths
| per failure | not shown to stay in one graph |
| `wake-queue/adapters/postgres.ts:376`, `:410`, `:555`, `:1169`,
`routes/issues.ts:14552`, `issues.ts:12070`, `:12171`,
`heartbeat/run-control.ts:1128`, `:1770`, `projects.ts:100`, `:1051`,
`environments.ts:1617` (`notifyChatInboundWakeNotices` without a task) |
saved notice and wake changes | per manual wake or project change |
company admission state; not reached by run activity in the fixture |
| `heartbeat/run-state.ts:1746`,
`native-harness-backup-expiration.ts:356`, `:526`,
`explicit-native-continuation.ts:705` | session goal clear, backup
expiry, explicit continuation | rare | no run id at the call site |
| `fast-responses.ts:201`, `:559`, `:672`, `:1053`,
`chat-github-response-comments.ts:62`, `:229`, `:440`,
`chat-github-reviews.ts:1063` (`notifyChatResponseAvailability` with a
delivery only) | chat response availability | per chat response | no
task at the call site; not measured here (chat paths are not in the
fixture) |

Per-run-event path: appending run events, saving native checkpoints,
renewing leases and streaming deltas publish no recovery task-queue
hint. `heartbeat-run-events.ts` reaches only live-run recovery queues
(`native.process_*`, `turn.*` events), chat progress, response
presentation, and `notifyNativeStatusReassessment` for control-plane
evidence events only. Checkpoints and lease renewals reach
`notifyLiveRunRecovery`, which does not include the task queues. A run's
start and finish publish its graph hint from several writers (about 20
task-queue signals per legacy run in the fixture, for example
`notifyChatRunMilestones` in `run-identity.ts` activation,
`claimQueuedRun`, `run-completion.ts`, `setWakeupStatus`). They coalesce
into about 3 checks of the run's own task per run. That is not one hint,
but it is bounded by the run's graph. This pull request does not change
it.

## Verification

- Activity fixture (`issue-recovery-activity-hints.test.ts`), checks per
second by queue and reason, and statements per second. Each workload
minute is exactly 60 s of fake-clock time, and the rates use the elapsed
fake-clock time. "Before" is this branch's test against the
`origin/master` service files. Both versions did the same work (7
runs/min: 150 runs started, 143 to 145 succeeded, 133 to 135 agent
comments, about 1,280 run events; 50 runs/min: 629 to 630 started, 620
to 622 succeeded, about 590 agent comments, about 5,550 run events). The
test now asserts this workload and zero agent write failures before it
accepts the counts.

  | | master | this branch |
  | --- | --- | --- |
| 7 runs/min + 1 taskless run/min (600 s), stranded checks/s | 18.30
company + 0.07 hint | 0.91 hint + 0.06 backstop, 0 company |
  | 7 runs/min, statements/s in stranded checks | 343.9 | 7.2 |
  | 7 runs/min, all statements/s | 577.5 | 220.4 |
  | 7 runs/min, company-only task-queue signals | 710 | 0 |
| 50 runs/min + 5 taskless runs/min (360 s), stranded checks/s | 16.86
company + 0.31 hint | 1.05 hint, 0 company |
  | 50 runs/min, statements/s in stranded checks | 314.0 | 7.4 |
  | 50 runs/min, all statements/s | 1,717.6 | 1,387.5 |
  | 50 runs/min, company-only task-queue signals | 2,864 | 0 |

"Statements in stranded checks" counts statements whose stack includes
`reconcileStrandedAssignedIssues`. The other statements are the runs
themselves. Recovery load queries (graph lookups and hint loads) are
12.5 to 17.2/s at 7 runs and 69 to 72/s at 50 runs in both versions.
`stale-issue-locks` saved deadlines check about 1 to 1.6 tasks/s in both
versions.
- Without the taskless runs, master already showed the backstop degrade
under activity: with 7 runs a minute, 2.46 backstop checks/s, because
every run writes its agent's row and marks all of that agent's tasks
changed. With this branch: 0.27/s.
- New tests in `issue-recovery-deadlines.test.ts`, red on
`origin/master` service files and green here:
- "re-checks a run's own tasks for its settlement, lease and dispatch
writers, and nothing for a run without a task": for `notifyRunDispatch`,
`notifyExecutionRunWork`, `notifyExternalConversationSettlement`,
`notifyChatInboundWakeNotices` and
`promoteDeferredWorkIntents(lock_released)` with a run id, the run's
graph (task, parent, dependent) is checked within one pass and none of
300 backlog tasks is. For a run without a task, no task is checked. A
comment checks only its task in `stranded-assignments` and
`assigned-work-invariant`.
- "counts an agent's own run heartbeat as no change to its other tasks,
and a resume as a change".
- "resolves a run without a task to no tasks and a missing task to the
whole company".
- Unchanged and passing: the #21729 pickup tests in
`issue-recovery-idle-backstop.test.ts` (finished run, user comment,
raised budget, resumed agent within a minute; missed hint within the
backstop bound; outside change within the hourly pass), the #21552 tests
in `issue-recovery-deadlines.test.ts` (including "keeps the company pass
for hold releases that can admit any task" and "falls back to the
company pass when a run affects more tasks than a hint names").
- Regression, embedded PostgreSQL: `issue-recovery-deadlines`,
`issue-recovery-work-debug-attribution`, `issue-recovery-signal-fanout`,
`issue-recovery-work-lifecycle` (42 tests);
`native-run-projection-work`, `native-recovery-notice-work`,
`muse-terminal-settlement`, `heartbeat-held-comment-wake`,
`superseded-workspace-export-recovery`,
`native-recovery-preparation-failure`, `upgrade-readiness-post-result`,
`chat-wake-conversation-batch`, `environment-service`,
`retained-native-session-cleanup-work`, `slack-conversation-lifecycle`,
`workspace-recovery-dependency-finalize`,
`native-failed-run-owner-release`, `chat-run-milestone-work` (462
tests).
- `tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- **Recovery time bounds. The owner must see this.** No hint-driven
pickup becomes slower for a task in the run's graph. Two missed-hint
bounds change:
- An agent change that sends no hint and is followed by a run heartbeat
of the same agent inside the 5-minute window (for example a resume whose
hint was lost, after which the agent ran another task) is now found by
the hourly full pass, not the 5-minute changed-tasks pass. A normal
resume sends `pause_lifted` and is checked at once (tested).
- A task outside the run's graph whose recovery depended on the end of a
taskless run, or on the native finish writers above, is now found by its
own hint, the 5-minute changed-tasks pass (its own rows), or the hourly
full pass. Before, any run finish re-checked the whole company. I found
no recovery decision that waits on another run of the agent: concurrency
waits are queued runs that dispatch resumes.
- A run-only hint for a run row that does not exist resolves to no tasks
(it was the whole company). A run row is not deleted while its tasks are
open, except by company deletion, which sends the company hint.
- Native finish writers now send run-scoped hints to the live-run
recovery queues too (`notifyLiveRunRecovery`,
`notifyLiveResourceRecovery`, attention retention). The legacy run paths
already send these with the run id.
- The fixture uses process-adapter runs, not native runs. The native
writers are narrowed by code reading. Their effect on the hosted staging
instance is not verified.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:47:34 -06:00
DottaandClaude Opus 5.5 10ddb26dbb test(server): accept dash's exit code for a sandbox-denied shell write (#21792)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner workspace bridge runs agent commands in a `bwrap` sandbox
that binds only `/usr`, `/bin`, `/lib` and the task workspace
> - A server test checks that a sandboxed command cannot read or write a
file outside the workspace
> - The test only runs where `/usr/bin/bwrap` exists. CI jobs that the
gate sends to GitHub's `ubuntu-latest` now get the `ubuntu-26.04` image,
which has `bwrap`, so the test runs on CI for the first time
> - There it fails on every run, because it expects the write attempt to
exit with 1, but Ubuntu's `/bin/sh` (dash) exits with 2 when a
redirection fails
> - This pull request checks that the write fails and that the file is
unchanged, without depending on the shell's exit code
> - The benefit is that the "General tests (server (7/12))" job passes
again on the new image, and the test still proves the sandbox denial

## Linked Issues or Issue Description

**What happened?**
`server/src/services/native-runtime/runner-bridge-tools.test.ts` ›
`Runner workspace bridge > runs useful commands but denies files outside
the workspace and injected credentials` fails in "ci / General tests
(server (7/12))" on unrelated branches, on the first run and the retry:
- PR #21765, head `98197c87e9`
- branch `feat/run-loop-detector`, run 43732047843, head `b8c6e27429`,
job 132866487949, image `ubuntu-26.04` version 20261111.196.1

```
AssertionError: expected { exitCode: 2, signal: null, …(3) } to match object { exitCode: 1 }
 ❯ src/services/native-runtime/runner-bridge-tools.test.ts:180:19
```

On jobs that ran on the RunsOn `ubuntu24` image there is no
`/usr/bin/bwrap`, so the test is skipped (7 skipped instead of 6).

**Expected behavior**
The test passes wherever the command sandbox is available, and it fails
if the sandboxed command can read or change the outside file.

**Steps to reproduce**
1. In a Debian or Ubuntu container, run `/bin/sh -c 'printf stolen >
"$1"' sh /nonexistent/dir/private.txt`.
2. dash prints `cannot create /nonexistent/dir/private.txt: Directory
nonexistent` and exits with 2. bash exits with 1. `cat` of the same path
exits with 1.
3. In the sandbox, the outside temporary directory is not bound, so the
path does not exist. This is the same case.

**Root cause**
A stale test expectation, exposed by a CI image change. No commit on
`master` changed the bridge, the sandbox or this test today (the last
change to the test is from 2026-11-08). The test expected the shell
write to exit with 1. That is bash behavior. On Ubuntu, `/bin/sh` is
dash, which exits with 2 for a failed redirection. Until today no CI
image had `bwrap`, so the test never ran on CI.

**Production impact**
None. The sandbox denies the access as intended: the `cat` step on the
same run exits with 1 and its output does not contain the secret, and
the write fails with `Directory nonexistent` because the outside
directory is not visible in the sandbox. This is not a sandbox
regression.

## What Changed

- The write step now checks that the outside file still contains
`PRIVATE` first, then that the command was not stopped and exited with a
non-zero code. It no longer requires exit code 1 from the shell.
- The read step and the success step are unchanged.

## Verification

- Linux container (`node:24-bookworm-slim`, `/bin/sh -> dash`): the
write command exits with 2 under dash and 1 under bash, and both are
failures, as the test now accepts.
- `npx vitest run
src/services/native-runtime/runner-bridge-tools.test.ts` in `server/` on
macOS: 11 passed, 6 skipped (the sandbox tests need Linux with `bwrap`).
- I cannot run the sandbox test on this Mac. It runs in this pull
request's CI only when the job lands on an `ubuntu-26.04` runner.

## Risks

- Low risk. Only one test assertion changes. The test still fails if the
write succeeds, because the file content check runs first and the exit
code must be non-zero.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, GitHub CLI, Docker).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:47:19 -06:00
DottaandClaude Opus 5.5 3ee6049f71 perf(native-runtime): name why a preinstalled provider pack was not reused (#21781)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote agent run needs the provider pack (Node, OpenCode and the
Runner bridge) inside its sandbox before the provider can start
> - The server reuses a pack that is already installed in the sandbox
image, or it uploads its own pack
> - In a recent load test every task uploaded the pack, and this step
was about 43% of the time to first token
> - The compatibility check and the preinstalled-pack verification
failed, but the run records did not say which check failed
> - This pull request records the failed check on the run's performance
spans
> - The benefit is that the next run names the reason, so the image or
the check can be fixed without guesswork

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The run performance spans for provider pack selection. Today a failed
`provider_pack.compatibility` or `provider_pack.verify_preinstalled`
span only says `outcome: failed`. The code catches the error and drops
it.

**Subsystem affected**
Server: native runtime, remote provider pack selection
(`server/src/services/native-runtime`).

**Current behavior**
`provider_pack.select` records `reason: cached_packs_unverified` when it
uploads. The compatibility check throws one generic message for about
ten different conditions, and the caller swallows it. The remote
verification script throws a generic `manifest mismatch` without the
fields that differ.

**Proposed behavior**
Each failed check has a short, fixed name (for example
`installation_unprotected`, `contract_digest`, `artifact:nodeCommand`,
`manifest_mismatch:runnerSourceRevision`). The trace records this name
as the `check` attribute on the failed span. `provider_pack.select`
records the last preinstalled rejection as `check` when it uploads. The
compatibility span records `target: image` or `target: workspace`.

**Reason and benefit**
Upload of the provider pack costs about 15 s per task on the test
instance. To remove it, we must know which check rejects the image pack.
The attribute holds only fixed check names and manifest field names,
never paths or values.

**Breaking changes**
None. Error messages keep their existing text. The upload decision does
not change.

## What Changed

- `remote-provider-pack-verification.ts`: the compatibility check names
the failed condition (`failureCheck`). The remote verification script
prints a fixed `paperclip-pack-check=<name>` line, and for a manifest
mismatch it names the differing manifest fields. Missing files now give
a named check instead of a raw stack trace.
- `native-run-trace.ts`: a failed `measure` records the error's
`failureCheck` as the safe `check` attribute.
- `remote-provider-pack.ts`: `provider_pack.select` records the last
preinstalled rejection (or `no_preinstalled_pack`) when it uploads.
- `native-session-executor.ts`: the `provider_pack.compatibility` span
records `target`, the preinstalled-pack selection passes its rejection
callback, and a failed link has the `link_failed` check. No other lines
change.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-run-trace.test.ts
src/services/native-runtime/remote-provider-pack.test.ts
src/services/native-runtime/remote-provider-pack-verification.test.ts
src/__tests__/redaction.test.ts` passes (131 tests).
- Red/green: with the source changes reverted and the new tests kept, 12
tests fail. With the changes, all pass.
- The verification test runs the real remote script in a Node process: a
provenance-only difference gives
`manifest_mismatch:runnerSourceRevision`, a wrong artifact gives
`artifact_digest:nodeCommand`, and a missing manifest gives
`manifest_unreadable`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The change adds diagnostics only. The selection order, the
checks themselves and the upload path do not change.
- The `check` attribute is filtered by the existing safe-attribute list
and a strict character pattern, so it cannot carry paths or secrets.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:46:33 -06:00
DottaandClaude Opus 5.5 d14a3b88b1 fix(native): release an idle warm session whose Runner is proven gone, and retry a failed idle close (#21782)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes idle sessions, and closes them again before it
exits.
> - When an idle close failed, the idle-timer path kept the session
registered and never tried again, unless the sandbox lease was already
inactive.
> - The controller-restart path tried once more, but it accepted no
proof that the session was already gone. It checked a destroyed sandbox
only for a forced environment delete.
> - In a release-swap test, one idle close failed with
`runner_remote_process_identity_unavailable`. The session stayed
registered until shutdown, the shutdown close failed again, and the
sandbox was held until the host stopped it.
> - This pull request treats a failed close as complete when the Runner
is proven gone or the sandbox is stopped or destroyed, retries a failed
idle close a bounded number of times, and logs which Runner check
failed.
> - The benefit is that a dead warm session is released, and the next
failure names its cause.

## Linked Issues or Issue Description

Refs #21627

**What happened?**
In a zero-downtime release-swap test, a warm session's idle close failed
4 s after the server left new run starts. The failure was
`runner_remote_process_identity_unavailable`. The idle-timer path kept
the session registered with no retry. At SIGTERM,
`closeIdleWarmNativeSessionsForRestart` tried again and failed the same
way. The log showed "idle warm native sessions did not close" and the
restart-chaos harness counted a warm-shutdown failure. The log did not
say which Runner check failed, so the cause of the unreadable identity
is not known.

**Expected behavior**
A close whose Runner is proven gone, or whose sandbox is stopped or
destroyed, releases the session. A failed idle close is tried again a
bounded number of times. The failure text names the Runner check stage
and reason.

**Steps to reproduce**
1. Keep a warm remote session idle.
2. Make its close fail with a `RemoteRunnerRecoveryError` (for example,
`process_verification/process_not_running`).
3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result
is `failed: 1`.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`:
- New `warmCloseFailureProvesOwnerGone`. It uses existing proofs only:
the close error's Runner diagnostic is
`process_verification/process_not_running`, or a lease row for the
sandbox has a termination receipt (`hasRemoteTerminationReceipt`) with
state `destroyed`, or the warm run's own lease row has a receipt with
state `stopped`. `pending_cleanup` and a lease without a receipt prove
nothing. A lease read error counts as no proof.
- `closeIdleWarmNativeSessions` (controller restart, environment delete,
idle sleep) uses this proof for every failed close, not only for a
forced environment delete. A released Runner detach failure is
unchanged.
- The idle-timer path uses the same proof. Its earlier status-only rule
from #21504 (any lease status other than `active` or `retained`) is
removed, because `pending_cleanup` is written before the provider stop.
Without proof, it re-arms the timer and tries again, every 5 s, at most
3 times (`WARM_IDLE_CLOSE_RETRY_MS`, `WARM_IDLE_CLOSE_MAX_RETRIES`). The
log line records the attempt number.
- `describeWarmCloseFailure` appends the Runner check as
`[stage/reason/lifecycle]` when the error carries one.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
three tests.
- "an idle warm session whose close fails with
runner_remote_process_identity_unavailable and whose Runner is proven
gone is released at controller restart"
- "names the Runner check in a warm close failure and keeps an unproven
owner registered at controller restart"
  - "retries a failed idle warm close a bounded number of times"
- The #21504 idle-path test is now "forgets a warm owner whose idle
close failed only once its sandbox has a stop or destroy receipt". A
stop or destroy receipt releases the owner. `pending_cleanup`,
`released` without a receipt, and `retained` keep it.
- "reads the warm owner's own lease rows at controller restart" (a stop
receipt for this run releases the owner; `pending_cleanup`, no receipt,
or another run's receipt do not)

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: 1519
passed, 7 skipped.
- Red on master (with only the two exported constants added so the file
compiles): all three new tests fail. The first fails with `"failed": 1`.
- `npx tsc --noEmit -p server`: no errors.

## Risks

- A session is released without a checkpoint only when its Runner is
proven gone or its sandbox is stopped or destroyed. The last saved
checkpoint stays the recovery source, which is the same as the forced
environment delete path today.
- Environment delete and idle sleep now also accept the proof. Before,
they accepted only a destroyed receipt during a forced delete.
`pending_cleanup` is written before the provider stop, so it never
counts as proof here.
- The idle retry adds at most 3 more close attempts per failed session,
5 s apart.
- A failed idle close whose sandbox stop is still pending now keeps its
owner until a receipt lands or a successor retires it. Before, the
status alone released it.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:09:32 -06:00
DottaandClaude Opus 5.5 8f1d39da79 perf(server): cache the instance settings row per process (#21768)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process runs every agent on one Node main thread, and
each database statement costs main-thread time
> - The `default` instance settings row is read on hot paths: live run
event delivery, run log redaction, the stranded-task recovery check,
runner tool calls, adapter routes and run preparation
> - A staging measurement with 7 live runs showed the full-row settings
read at 10.24 statements per second, and 3.5 to 5.4 per second when idle
> - The row changes only when an admin or a boot step writes it, so
almost every read returns the same value
> - This pull request adds a per-process read-through cache for that
row, with in-process invalidation on every write and a 2 second age
bound for changes made by another process
> - The benefit is at most one settings read per 2 seconds per server
process, whatever the load

## Linked Issues or Issue Description

**Is your feature request related to a problem?**
The full `instance_settings` row read (`select id, singleton_key,
default_environment_id, general, experimental ...`) runs at 10.24
statements per second on staging with 7 live runs, and 3.5 to 5.4 per
second at idle. It scales with run events and recovery checks, not with
settings changes.

**Describe the solution you would like**
Cache the row per process. Drop the cached row when a settings write
commits, in this process and, through the existing work-signal
notification, in the other process during a deploy overlap. Bound the
age of a cached row to 2 seconds, so any other change (a direct database
edit, a second process outside an overlap) takes effect within 2
seconds.

**Describe alternatives you have considered**
A database trigger with LISTEN/NOTIFY would also catch direct edits, but
it needs a migration, and work notifications cross processes only during
a deploy overlap today. The existing signal plus a 2 second bound keeps
the change small.

## What Changed

- New `server/src/services/instance-settings-cache.ts`: a read-through
cache keyed by the database client. Concurrent misses share one read. A
cached row is at most `INSTANCE_SETTINGS_CACHE_TTL_MS` (2000 ms) old,
measured from the start of the read. Every caller gets its own copy.
Reads through a transaction handle skip the cache.
- `instanceSettingsService` `get`, `getGeneral` and `getExperimental`
use the cache. Write paths (`update`, `updateGeneral`,
`updateExperimental`, and the feedback vote save) still read the row
fresh.
- Every settings write publishes an `instance-settings-changed` work
signal through the existing work-signal channel
(`@paperclipai/db/work-signals`). Inside a transaction it is the first
signal of the write, so its `pg_notify` payload goes out before the
other wakes of the same commit (for example `status-card-evaluation`).
This process drops the cached row when the write commits. During a
deploy overlap, when work notifications cross processes, the other
process drops its cached row on receipt, before a worker woken by the
same write reads the settings. A reconnect of the notification listener
also drops the row.
- `fillUnsetInstanceDefault` writes through `update({ db: tx })`, so its
signal is sent when its transaction commits.
- A read that starts before an invalidation never stores its result, so
an old value can not come back after a write.
- The test setup clears the cache before each test, because many tests
write the row directly.

Hot callers of the cached row:

- Live run event delivery: `live-run-event-redaction.ts`
(`censorUsernameInLogs`), once per delivery batch.
- Run output and run lifecycle redaction: `heartbeat.ts`
`getCurrentUserRedactionOptions`, used by `run-output.ts`,
`run-lifecycle.ts`, `run-completion.ts`.
- The stranded-task recovery check: `recovery/service.ts` reads
`getExperimental().enableAgentChat` once per checked conversation issue.
- Runner tool calls: `paperclip-runner-tool-authority.ts`
(`enableIsolatedWorkspaces`, `enableExternalObjects`).
- Adapter routes: `app.ts` (`enableNativeRunner`, `enableOpenAiDot`) per
request.
- Run preparation: `heartbeat.ts` (`instance_settings` preparation
step), `queued-run-line.ts`, `queued-run-dispatch-liveness.ts`,
`run-control.ts`.
- Issue and comment paths in `issues.ts` (`censorUsernameInLogs`,
`enableAgentChat`).

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/instance-settings-cache.test.ts` (embedded PostgreSQL):
- 150 hot reads (`getGeneral`, `getExperimental`, `get` from new service
objects) after one warm read: 0 statements. Without the cache the same
test measures 150 statements (red run).
  - 20 concurrent misses: 1 statement.
- `updateExperimental` and `updateGeneral` are visible at once to a new
read in the same process.
- `fillUnsetInstanceDefault` (transaction write of the default
environment) is visible at once after it returns.
- A change written by a second client (another process) becomes visible
within 2000 ms plus a 500 ms test margin.
- A caller that changes the returned object does not change the cached
row.
- A read through a transaction handle sees the transaction's uncommitted
write, two such reads issue 2 statements, and the cache does not keep
the write after rollback.
  - A read that started before an invalidation is not stored.
- A received `instance-settings-changed` notification from another
process drops the cached row (next read issues 1 statement).
- `updateExperimental({ enableStatusCards: true })` sends
`instance-settings-changed` as its first notification, before
`status-card-evaluation`; `updateGeneral` sends it alone.
- The server tests that use instance settings or write the row directly
(230 files): see the run notes in the first comment.

## Risks

- A direct database edit, or a change from a second process outside a
deploy overlap, takes up to 2 seconds to apply in this process. Before
this change it applied at the next read. This includes access flags such
as `enableOwnerInstanceAdmin` (read per cloud-tenant request) and the
default environment.
- A change made through the settings service applies at once in this
process, and during a deploy overlap at once in the other process.
- A new code path that writes the `default` row must call
`announceInstanceSettingsChange`, or its change applies within 2
seconds.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
user-facing setting or command changed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:50:19 -06:00
DottaandClaude Opus 5.5 2b23119ed2 fix(native): keep a warm follow-up that inherited its predecessor's process identity waiting on the same run (#21774)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap. A warm agent session stays
with the outgoing server that ran its last turn.
> - #21697 makes a follow-up wait while that outgoing server is still
alive, instead of starting a second Runner over its session.
> - Runtime selection copies the warm predecessor's process fields
(`processPid`, `processGroupId`, `processStartedAt`) onto the follow-up
and records `inheritedProcessIdentityRunId`.
> - The wait records itself with `deferNativeControllerRecovery`. That
function refused any run with process fields, so it refused every warm
follow-up that the wait stopped. The run failed with
`native_controller_recovery_wait_authority_changed`.
> - This pull request lets the deferral accept an inherited identity,
and clears those inherited fields in the same transaction so the usual
resumption claim can pick the run up.
> - The benefit is that a warm follow-up in a deploy overlap stays on
the same run and does not fail.

## Linked Issues or Issue Description

Refs #21697

**What happened?**
In a zero-downtime release-swap test, a follow-up comment arrived 0.25 s
after its predecessor finished on the outgoing server. The follow-up run
started on the incoming server 1.09 s later.
`waitForLiveWarmSessionOwner` found the outgoing server alive and asked
to wait. `deferNativeControllerRecovery` then refused with
`native_controller_recovery_wait_authority_changed`, because the run
carried the process fields that runtime selection copied from its warm
predecessor. The run failed after 2.7 s with
`native_continuation_requires_reconciliation`. Held recovery started a
fresh run. The task finished, but the run history shows one failed run.

**Expected behavior**
A warm follow-up whose only process identity is inherited from its
predecessor waits on the same run. The resumption owner claims it at the
deadline. A run whose own process has started is still refused.

**Steps to reproduce**
1. Run a warm remote agent turn on server A.
2. Switch new run starts to server B while A still serves.
3. Post a follow-up within about 1 s of the turn's finish.
4. The follow-up fails with
`native_controller_recovery_wait_authority_changed`.

## What Changed

- `server/src/services/native-runtime/native-controller-recovery.ts`
(`deferNativeControllerRecovery`):
- Process fields refuse the deferral only when they are the run's own.
Fields count as inherited when
`runnerProfileJson.inheritedProcessIdentityRunId` is set and the run has
no own `native.process_start_requested` or
`native.process_identity_recorded` event. `run-cost-accounting.ts` uses
the same proof.
- With an inherited identity, the same transaction clears `processPid`,
`processGroupId` and `processStartedAt`. It keeps
`inheritedProcessIdentityRunId`. `claimNativeSessionResumptions`
requires those fields to be null.
-
`server/src/services/native-runtime/native-controller-recovery.integration.test.ts`
(real Postgres):
- New: "keeps a warm follow-up that inherited its predecessor's process
identity on the same run until the outgoing owner's deadline".
- Two new cases in "does not publish an admission wait after authority
changes": `own-process` (process fields, no marker) and
`inherited-then-own-process` (marker plus an own process-start event).
Both stay refused and keep their fields.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts
-t "inherited its predecessor|does not publish an admission
wait|unstarted follow-up"`: 9 passed.
- Red on master: the new test fails with `Error:
native_controller_recovery_wait_authority_changed`. The own-process
cases pass on master and with the change.
- Green: the run is `retryable_failure` with `nextAttemptAt` at the
deadline, its process fields are null, no claim happens before the
deadline, and one claim happens at the deadline.

## Risks

- The follow-up now waits instead of failing. The wait ends when the
outgoing server saves a suspended checkpoint, when it exits, or at the
predecessor's finish plus its idle window plus 2 minutes. In the
observed case, the outgoing server's idle close could not save its
checkpoint (`native_runner_checkpoint_owner_lost`), so the wait would
end only when that server exits. See the follow-up note below.
- An own process is still refused. The proof uses the run's own
process-start events, the same proof as run cost accounting.

**Follow-up note (not in this PR):** The outgoing server's checkpoint
write checks the task's current run. A native successor that is parked
in this wait is not one of the accepted successor kinds, and the idle
close is not retried. So the wait does not end early from a checkpoint.
In the observed timeline, the wait is about 225 s (until the outgoing
server exited). The worst case with the default 5-minute idle window is
about 425 s.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:19:23 -06:00
DottaandPaperclip afca556728 fix(github): clarify configured review task completion (#21748)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub bots publish reviews and report the result of their assigned
task.
> - A GitHub check and a Paperclip task have separate completion rules.
> - The current prompt explains how to wait, but gives little completion
guidance.
> - A bot can keep a complete review in review while it waits for CI or
merge that its assignment does not require.
> - This pull request makes the configured completion rules clear in
GitHub prompts.
> - The benefit is a finite review task with an accurate status and
publication evidence.

## Linked Issues or Issue Description

Related change: Refs #17318 keeps advisory review waits separate from
human attention. This change adds completion guidance in the same GitHub
prompt owner.

**What happened?**

A bot can publish a complete current-head 5/5 review and still wait for
CI or merge. Manual request prompts also lack the task completion
guidance used by automatic reviews.

**Expected behavior**

When the configured assignment allows completion at 5/5, a complete
published review can finish Done. CI and merge are completion conditions
only when the assignment requires them. Explicit tracking rules still
apply.

**Steps to reproduce**

1. Configure an agent to finish after a complete published current-head
5/5 review.
2. Start a GitHub review with CI results unavailable.
3. Inspect the generated prompt. It explains a response wait but does
not explain this completion case or publication safeguards.
4. With this change, automatic, mention, and comment prompts carry the
same explicit guidance.

**Paperclip version or commit**

Base commit: `83541e89ad3b90740ffc61a32c396dddb5834526`.

**Deployment mode**

Built from source. The prompt change applies to GitHub bot tasks.

## What Changed

- Add shared guidance within the GitHub prompt module for configured
completion at 5/5 and finite discussion replies.
- Require a complete assessment, confirmed publication, and the current
head in the 5/5 guidance.
- Preserve explicit CI, merge, and below-5 tracking rules. Keep the
shared status arbiter and check thresholds unchanged.
- Add six prompt cases. State that CI is unavailable in the existing
lifecycle completion case.
- Document the configuration example and its limits.

## Verification

- Six new prompt cases failed before the change.
- `pnpm exec vitest run
server/src/services/chat-github-review-policy.test.ts`: 40 passed.
- `pnpm exec vitest run
server/src/services/native-runtime/external-chat-wait.integration.test.ts
-t 'settles a published'`: 9 passed.
- `pnpm exec vitest run
server/src/services/native-runtime/external-chat-wait.integration.test.ts
-t 'inherits.*review|published.*guard|unknown publication|pending
publication|changed head|inherited.*review'`: 5 passed.
- `git diff --check` passed.
- `pnpm verify` returned exit 2. The connector document requires broader
CI coverage.
- Local preflight installed dependencies and built shared and plugin SDK
packages. It stopped because Cargo is unavailable. Full local typecheck,
build, and test suites were not run.
- [Greptile reviewed this
commit](https://github.com/paperclipai/paperclip/pull/21748#issuecomment-6894002401)
at 5/5 with no actionable findings. Its check passed for
`36dd146cdc00b0f3d7f6fc2971f24166ef0f6bf5`.
-
[CI](https://github.com/paperclipai/paperclip/actions/runs/43711597984)
passed for `36dd146cdc00b0f3d7f6fc2971f24166ef0f6bf5`. Typecheck, build,
Rust/Runner, all server tests, and all eight E2E test groups passed. The
final [CI
gate](https://github.com/paperclipai/paperclip/actions/runs/43711597984/job/132808715297)
passed at 12:17:58 UTC on 2026-11-15.
- At 12:23 UTC, all 61 check records were complete: 57 passed, 1 was
neutral, and 3 were skipped. Snyk also passed. No checks failed or
remained pending.

## Risks

This change gives prompt guidance. It does not enforce a score-to-status
rule. Custom assignments can still require CI, merge, or later review.
The focused tests check prompt content and existing lifecycle behavior;
they do not prove every live agent or webhook journey.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. This session does not expose the exact model variant or
context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in this PR with the bug report fields
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My public branch name describes the change and contains no
internal ticket id
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-15 08:52:01 -06:00
DottaandClaude Opus 5.5 819f61f1fd fix(evals): repair four cohort-15 chat cases and prefix board links in completion feedback (#21682)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat-feature eval catalog runs real chat journeys. It grades
saved state, rendered links and run ledgers.
> - Chat-eval cohort 15 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `36101ebcb9`)
failed four cases where the agent did the right thing or where our own
server text misled it.
> - One failure was a harness race introduced by #21667. Two failures
came from graders that read free-form text with keyword regexes. One
failure came from an unprefixed approval link in the server's completion
feedback, plus a grader that required a literal prefix.
> - This pull request fixes the harness step, replaces the free-form
graders with saved-state and clicked-link checks, and makes the server
emit company-prefixed board links.
> - The benefit is that these cases measure the feature again. The
retained correct outcomes pass, and the relevant broken outcomes still
fail.

## Linked Issues or Issue Description

Refs #21667

**What happened?**
In chat-eval cohort 15, four cases failed:
- `projects.share-project`: the harness followed the agent's ID-based
project link. The app then rewrote the URL to the project slug and
redrew the page. The owner step timed out because Riley's option in the
people picker was "detached from the DOM". #21667 introduced this path.
- `skills.author-skill`: the saved skill was a correct, richer
checklist. Per-sentence keyword regexes rejected it. For example,
"ignore the mouse" tripped the forbidden word "ignore".
- `inbox-search.show-decisions`: the reply linked the right approval and
tasks and said the question was "addressed to a different teammate". A
regex required the word "Riley".
- `agents.hire-needs-approval`: both replies linked `/approvals/<id>`
without the company prefix. The agent copied that link from
`server/src/services/native-runtime/native-completion-feedback.ts`. The
grader required a literal `/<prefix>/approvals/<id>`, although the board
redirects the unprefixed route.

**Expected behavior**
Correct agent behaviour passes. Grading uses saved state, setup literals
and rendered or clicked links, not free-form wording. The server gives
agents canonical company-prefixed board links.

**Steps to reproduce**
1. Use the retained cohort-15 evidence for the four cases.
2. Run the old graders on the retained skill markdown and decisions
reply. Both fail.
3. For share-project, open a project configuration page by ID while the
app rewrites the route after a delay, then use the access picker before
the rewrite completes.
4. For hire-needs-approval, finish a task with a pending linked
approval. The feedback links `/approvals/<id>`.

**Paperclip version or commit**
master `36101ebcb9` (cohort 15); fixed on top of current master.

## What Changed

- `projects.share-project` (`internal-privacy.ts`,
`runtime-support.ts`): after the harness follows the agent's link, it
waits until the URL settles on the project's canonical route key. This
reuses the #21667 approach for agent profiles: `settleAgentProfileRoute`
now calls a shared `settleRouteKey`, and `projectRouteKey` follows the
app's `projectRouteRef`. The owner's picker selection
(`addProjectAccessMember`) now retries as one unit. If the dialog or
picker closes, or an option detaches, it reopens them, searches again
and selects again. It continues only when the picker shows the person
and Add is enabled.
- `skills.author-skill` (`integrator-skills.ts`): removed the
keyword-regex checklist grader, the waiver regexes, and the reply
regexes for "global activation" and "attached to me" claims. The
saved-state checks stay: one agent-created writable skill, its current
revision authored by the responding agent, the same bytes on the skill
and the revision, the Skills/Studio inspection with reload, and
assignment scope from saved assignments and audit rows. The checklist
check now requires the setup literals `375`, `WCAG` and `AA` as whole
tokens in the saved SKILL.md and in the rendered editor. Whether the
skill still asks for keyboard-only navigation and visible labels, and
whether it waives a check, is left to human review. Oracle version
`skill-authoring.v10`.
- `inbox-search.show-decisions` (`internal-decision-search.ts`): removed
all reply regexes, including the `Riley` requirement and the
wording-based separation checks. The grade now uses saved state (pending
decisions unchanged, answered history unchanged, each person's own
attention queue and resolver) and rendered links to the exact approval,
the question's task and the history task. Oracle version
`decision-search.v3`.
- Product (`native-completion-feedback.ts`): the completion feedback now
links `/<prefix>/approvals/<id>` and `/<prefix>/issues/<identifier>`.
This fixes the approval link and the four sibling task links in the same
file.
- `agents.hire-needs-approval` (`runtime-hiring.ts`): the harness now
clicks each reply's rendered link to its approval and records where the
UI lands. The grade accepts a prefixed link, or an unprefixed link that
the board redirect lands on `/<prefix>/approvals/<id>` with the proposed
hire visible. Without such a link, the harness still opens the approval
directly so later phases keep their evidence, and the grade fails.
Oracle version `runtime-hiring.v4`.
- No eval YAML changed, so the TestOps manifest does not change.
`chat-catalog-descriptors.json` records the three oracle version bumps.

## Verification

- Red/green tests (each new test fails on the old code and passes on the
new code):
- `internal-privacy-share-route.test.ts` (real Chromium): the project
page rewrites its route after 0, 400 and 1500 ms, and one case closes
the picker while its first results render. With the old flow, the 400 ms
rewrite and the closed picker both time out on the option click (the
cohort-15 failure). With the fix, Riley is added on the settled route in
all cases.
- `integrator-skills.test.ts`: the retained cohort-15 SKILL.md passes.
The old grader rejects it. A stale skill body, another author, and a
lost 375 width still fail.
- `internal-decision-search.test.ts`: the retained cohort-15 reply and a
reworded reply pass. The old grader rejects the retained reply. A
missing or wrong approval, question-task or history-task link fails.
- `native-completion-feedback.test.ts`: a pending linked approval
produces `[the pending approval](/<prefix>/approvals/<id>)`, and a
pending review stage produces a prefixed task link. The old code
produces `/approvals/<id>`.
- `runtime-hiring.test.ts`: unprefixed links that the redirect lands on
the right approval pass. The old grader rejects them. A link that did
not land, landed on another approval, was not clicked, or is another
href still fails.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog`: all passed (one negative control in `internal.test.ts`
changed from a reply wording to "approved while listing").
- `pnpm test:e2e:runner:typecheck` and `server` `tsc --noEmit`: passed.
- `server` vitest for `native-completion-feedback`,
`native-current-wake-completion-feedback` and
`heartbeat/run-completion`: 201 tests passed.
- Narrow paid run of the four full case ids on this branch: run
43657390762 (results are added in a comment).

## Risks

- Low risk. The product change only adds the company prefix to links in
agent feedback text. It adds one indexed company read, and only on the
branches that return a link.
- Less is graded automatically, by design. `skills.author-skill` no
longer checks that the skill keeps the keyboard-only and visible-label
checks or that it does not waive a check. `inbox-search.show-decisions`
no longer checks how the reply names resolvers or separates the person's
decision from others' and from history. `skills.author-skill` no longer
checks whether the reply describes the assignment scope truthfully. A
person reviews this meaning. Saved state and links still catch the
plausible broken outcomes listed above.
- `agents.hire-needs-approval` now needs a reply link that really opens
the approval. A reply without a link to its approval fails as before.

## Model Used

- Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use
and code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 05:34:56 -06:00
DottaandClaude Opus 5.5 68f641ba0b fix(recovery): continue a native run cut off by a sandbox client timeout (#21736)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime runs an agent's provider turn in a sandbox (for
example Daytona) and records each failure on the run and its native
coordinator
> - A Daytona SDK request can hit its client-side deadline during a
provider turn. The run then fails with `native_session_interrupted` in
an ambiguous state, because the turn started and its outcome is unknown
> - That failure blocks the task under a board-owned watchdog hold. No
recovery owner selected the hold, because its cause is not an
execution-reconciliation cause. The task never ran again, and only a
person's Retry could continue it
> - Recovery from an infrastructure fault must work automatically,
without a person
> - This pull request lets the execution-disposition pass hand that hold
to the existing fresh-continuation fallback
> - The benefit is that the task continues once, in an isolated fresh
run, and the cut-off turn is never replayed

## Linked Issues or Issue Description

**What happened?**
A native run on a Daytona execution environment (native runtime, an
OpenRouter model) started its provider turn. A Daytona request then
failed with `HTTP request timed out after 300000ms waiting for a
response. This is a client-side deadline
(DaytonaConfig.requestTimeoutMs, or the per-call operation/execution
timeout); any operation already started on the server may still be
running.` The run failed with `native_session_interrupted`. The task had
no further run: no automatic retry and no continuation.

**Expected behavior**
The task continues automatically, exactly once, in a fresh run from its
saved work. The cut-off provider turn is not replayed.

**Steps to reproduce**
1. Start a native run for an assigned task. Let its provider turn start
(a `turn.started` event is saved).
2. Make a sandbox request in that turn fail with the Daytona client-side
timeout error.
3. The executor saves the coordinator as `terminal_failure` with
`recoveryMode: ambiguous_state`, blocks the task, and writes an active
board hold with cause `native_session_interrupted`.
4. Run the execution-disposition worker
(`settleUnrecoverableExecutions`). Before this change, nothing happens
and the task stays blocked.

**Paperclip version or commit**
`master` at 9ebc276b42 (observed on a build from 5acf27bc3).

**Deployment mode**
Hosted cloud with a Daytona execution environment.

Causal path:
- The Daytona error is not a named class, so
`nativeSessionFailureSourceCode` returns the generic
`native_session_interrupted`.
- `nativeProviderRecoveryEvidence` returns `ambiguous_state`: provider
events exist, but there is no checkpoint with a provider identity. The
executor then makes the failure terminal on the first attempt (no
same-run resume) and writes a board hold whose cause is
`native_session_interrupted`.
- `unresolvedExecutionDispositions` and `heldExecutionWhere` select only
holds with `EXECUTION_RECONCILIATION_CAUSES` (or a narrow
`resume_native_run` shape). The safe replacement lane accepts
`native_session_interrupted` only for the historical
listener-unavailable case. The assigned-work sweep leaves a task blocked
by an active hold to the hold's owner. So no owner continued the task.

## What Changed

- `server/src/services/execution-recovery-resolution.ts`: the
disposition pass also selects an active watchdog hold with a
restart-class cause (`RESTART_CLASS_FAILURE_CODES`) for a terminal
native run with no retry pending, unless the safe replacement lane owns
that run. It hands the run to `continueFreshInTransaction` and records
the fallback's result on the hold, so a run that does not continue is
checked again only when due.
- `server/src/services/fresh-continuation-fallback.ts`: the fallback
treats an active watchdog hold with a restart-class cause as its own.
When it continues the task, it settles that hold and unblocks the task,
as it does for reconciliation holds.
- New red/green test
`server/src/services/native-runtime/native-client-timeout-continuation.integration.test.ts`.
It drives the real native executor with a provider turn that starts and
then fails with the Daytona client-timeout error.

The cut-off turn is not replayed. The fresh continuation isolates the
old run as it does for every continuation: the old run loses its write
authority, its remote workspace epoch is fenced (or the continuation
gets a new workspace), and the coordinator records `replacementDenied:
fresh_continuation`. The fallback keeps its existing checks:
person-owned causes, exact recovery still under way, and the per-task
hourly cap.

## Verification

- Red on `master`: `npx vitest run
src/services/native-runtime/native-client-timeout-continuation.integration.test.ts`
fails with no continuation recorded.
- Green with this change. The test asserts that the run fails with
`native_session_interrupted` and the Daytona error text in an ambiguous
state, then that two disposition passes record exactly one fresh
continuation, the task goes from `blocked` to `todo`, no active hold
remains, the old run is marked `replacementDenied`, and two
continuation-delivery passes wake the assignee exactly once.
- Focused suites (all tests that use the disposition pass, the
fresh-continuation fallback, or the execution-control schedule, 28
files): 27 files pass. `chat-channels.integration.test.ts` has 9
failures that are unrelated to this change (see the PR comment for the
`master` baseline).
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify --explain` selects 2751 of 3578 files (a near-full run).
I did not run it locally. CI supplies that coverage.

## Risks

- Behavior change: a terminal native interruption hold that before
stayed with the board now continues automatically in a fresh run. This
is the intended product behavior. The fallback still refuses
person-owned causes and still respects the per-task cap.
- Holds with restart-class causes are now settled by the fallback when
it continues a run. Before, they could stay active next to the
continuation record.
- No migration. No API or UI contract change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits) and code execution for tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:15:41 +00:00
DottaandClaude Opus 5.5 01c1e708e9 fix(runner): stop a turn that keeps resending a refused tool call and continue fresh (#21734)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs call Paperclip tools, and
`RepeatedToolFailureGuard` refuses an exact repeat of a call that
already failed validation twice
> - The refusals are not counted and never end the turn, so a model
whose host drops an argument can resend the same refused call until the
run's wall clock
> - In a recorded restart-chaos wave, an opencode/DeepSeek agent sent
the same `PATCH /api/issues/{id}` through `call_api` 431 times in 40
minutes without `body`; 427 were refused. The run held an agent slot and
delayed other tasks
> - The system knows the next step (stop the turn and continue on a
fresh session), so the harness must own it, with no Retry banner
> - This pull request counts refusals, stops the provider turn after a
bounded number with the classified cause `agent_tool_call_loop`, and
queues an automatic fresh-session continuation that tells the model to
use a dedicated tool
> - It also records the declared argument names that each opencode tool
call delivered, so we can tell whether the bridge or the model drops
`body`
> - The benefit is that a looping agent frees its slot in seconds
instead of 40 minutes, and the task continues by itself

## Linked Issues or Issue Description

**What happened?**
An opencode agent on DeepSeek created two subtasks and then tried to set
their descriptions with `call_api` `PATCH /api/issues/{id}`. Every call
arrived with `operationId`, `pathParams`, `query` and `responseText`,
but without `body`. The guard refused each repeat. The model said "I
need to include body" more than 20 times and kept sending the same call:
431 calls, 427 refused, over 40 minutes. The run stayed active and its
agent slot delayed other queued work.

**Expected behavior**
After a small, bounded number of refusals, Paperclip stops the provider
turn and continues the task automatically on a fresh session, with a
hint to use a dedicated Paperclip tool. The board shows no Retry banner
because the system owns this recovery.

**Steps to reproduce**
1. Run a native agent whose model resends one tool call with a missing
required argument.
2. The guard refuses the third and later identical calls.
3. On master, the refusals never end the turn.

**Paperclip version or commit**
`master` at 6400698a8e.

## What Changed

- `repeated-tool-failure-guard.ts`: counts refusals per exact call and
per run. At 6 refusals of one call, or 20 refusals in the run, it throws
`ToolCallLoopError` (`code: "agent_tool_call_loop"`, tool, counts and
argument names, never values). The message tells the model the turn
stops and gives a short hint.
- `paperclip-runner-tool-authority.ts`:
`scheduleToolCallLoopContinuation` writes one idempotent native
status-wake outbox row (key `agent_tool_call_loop:<runId>`) with
`forceFreshSession: true` and the hint as the wake instruction. The
outbox dispatcher admits it with `allowRunCoalescing: false`, so it
never merges into a deferred comment wake or the active run. An assignee
run uses `issue_continuation_needed`. A completion-review run uses
`native_completion_review` with its review assignment, which the
dispatcher revalidates, so the fresh reviewer session keeps the review.
- `native-session-executor.ts`: the session authority epoch queues that
continuation when the guard raises the loop error. The governed-wait
lookup finds the wake on the next tool event and ends the provider turn
as `yielded` with a `same_agent` continuation, the same way the
connection-tools refresh ends a turn. If the wake cannot be queued, the
refusal still reaches the model and the error is logged.
- `status-decision-committer.ts`: a yielded continuation with that key
binds to the queued wake instead of creating a second one.
- `status-arbiter.ts` and `native-run-finalizer.ts`: a pending review
run that ended through this continuation skips the
`native_review_unresolved` resume recovery, because the queued review
wake owns the next reviewer run.
- `mcp-bridge.ts` and `opencode-proxy-finish-diagnostics.ts`: the
opencode tool-delivery diagnostic records `argumentNames` (only names
declared in the tool's input schema, sorted, at most 1,024 characters of
JSON), `argumentNamesTruncated` and `undeclaredArgumentCount`. Values
and undeclared names are never recorded. The forwarder's closed key
allowlist accepts the new fields and validates them, so a record stays
under its 2,048-character limit.

### Threshold choice

- **6 refusals of one exact call.** The guard already runs a call twice
and then refuses with an explicit correction. Six more refusals (eight
identical attempts in total) give a model that reads errors many chances
to change its arguments. In the recorded run the model never changed
them in 427 refusals, so six refusals end that loop within seconds.
- **20 refusals in the run.** A model can alternate between a few
refused variants and keep each one under 6. Refusals happen only for
exact repeats of a certain validation failure, so a healthy run almost
never reaches 20.

### Previous-release story

The guard counters live in memory for one run's tool authority, as
before. A deploy or restart during a run starts the counters again at
zero on the new process; that only delays the stop. The continuation is
an ordinary wake row. If a server on the previous release finalizes a
run that a new server parked, it does not know the key and queues its
own `same_agent` continuation too; the issue execution lock lets one run
and the other coalesces.

## Verification

- Red then green, `repeated-tool-failure-guard.test.ts`: "after N
refusals the guard raises a turn-stopping loop error" and "stops the
turn once refusals across different calls reach the run limit". Both
fail on master.
- Red then green, `native-session-executor.test.ts`: "a turn whose model
repeats a refused call N times ends with agent_tool_call_loop and
schedules a continuation". With the epoch handling disabled it fails (13
calls instead of 8, nothing parked). Also "keeps the refused call's
error when the loop continuation cannot be queued".
- `paperclip-runner-tool-authority.test.ts`: "queues one fresh-session
continuation outbox for the assignee / a completion reviewer stopped by
a tool-call loop" (real Postgres; idempotent).
- `native-tool-call-loop-continuation.test.ts`: with a comment wake
already deferred behind the looping run, the continuation keeps its own
keyed deferred wake, the comment wake is unchanged, and after release a
run starts with `forceFreshSession` and the loop cause.
- `status-arbiter.test.ts`: a pending reviewer stopped by a loop
releases its checkout without a resume recovery.
- `native-status-arbiter-corpus.test.ts`: the existing wake-binding test
now also covers the `agent_tool_call_loop` key.
- `mcp-bridge-delivery.test.ts` and
`opencode-proxy-finish-diagnostics.test.ts`: argument names are
recorded, values and undeclared names are not, invalid records are
dropped, and a boundary test with 32 names of 64 characters stays
forwardable with `argumentNamesTruncated: true`.
- Server: the six touched test files pass (1755 tests). Runner:
`src/drivers/opencode` and `src/cli` pass (569 tests). `tsc --noEmit`
passes for `server` and `packages/paperclip-runner`.
- `pnpm verify` exits 2 (runner package changes need broad coverage), so
required CI supplies the broad run.

## Risks

- A run that hits the limit ends its turn early. The limits are far
above what a model that reads its errors needs, and the task continues
automatically on a fresh session.
- The fresh session loses the in-turn conversation. The task, its
comments, documents and workspace stay, and the continuation names the
cause and the hint.
- The diagnostic now carries declared argument names. These are schema
names, not values, and the forwarder validates the shape before it
forwards a record.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21360

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 01:52:50 -06:00
Nicky LeachandClaude Opus 5.5 a78be8b601 feat(native): send launch preparation and run digests to plugin task prepare (#21717)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Environment plugins can run a native Runner as a plugin task, and a
warm session keeps that Runner between turns
> - On each warm turn Core sends the plugin a `prepare` operation with
the full launch requirements for the new run
> - The requirements' `requirementsDigest` covers the run identity, and
the run identity changes on every turn
> - So a plugin cannot tell that only the run moved, and it must redo
its whole preparation every warm turn
> - This pull request gives the plugin two digests: one that stays the
same while its preparation stays valid, and one that names the run
> - The benefit is that a plugin can transfer only the run identity when
nothing it reconciles changed

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The `prepare` operation that Core sends to an environment plugin for a
plugin Runner task, on cold starts and on warm attaches.

**Subsystem affected**
Native runtime, plugin Runner tasks, the Runner launch contract, and the
plugin SDK environment task contract.

**Current behavior**
A warm attach sends the full requirements. The only digest available,
`requirementsDigest`, includes `runId`, `turnId` and `itemId`, so it
changes on every turn even when the launch is otherwise the same.

**Proposed behavior**
Core computes two digests for every `prepare`:
- `preparationDigest`: the launch requirements an environment reconciles
when it prepares a Runner:
  - the runner, lease and session identity;
  - provider, model and settings;
  - the prompt and instruction text;
- the runtime context's instruction and skill bundles, working copy and
connection instructions.

It leaves out the `runId`, `turnId` and `itemId`, and the runtime
context's MCP assignment and binding (with `aggregateDigest`, which
hashes the assignment). MCP reaches the Runner through Core's tool
authority on every run attachment, never through files the environment
publishes. Core already continues a warm session across an MCP-only
change (`nativeRuntimeContextsMatch` with tool refresh). The binding is
also named per run (`native-mcp:<runId>`). So the digest stays the same
across warm turns of one Runner, lease and session, including when the
agent's MCP access changes.
- `runDigest`: `preparationDigest` plus everything it leaves out:
`runId`, `turnId`, `itemId`, and the MCP assignment and binding. It is
unique to one run's launch.

A plugin that advertises `launchDigestVersion: 1` in its task
description gets both fields on `prepare`.

**Reason and benefit**
A plugin can compare `preparationDigest` with its saved preparation and
transfer only the run identity when they match, instead of a full
preparation.

**Breaking changes**
None. Plugins without `launchDigestVersion` get no new fields.

## What Changed

- `packages/paperclip-runner/src/live/runner-provider-launch.ts`: add
`runnerLaunchPreparationDigest` and `runnerLaunchRunDigest`. The
preparation digest drops the MCP assignment and binding; the run digest
takes the full requirements and includes them. Both use the same
canonical JSON as `runnerLaunchRequirementsDigest`, with their own
schema tags so the three digests never collide. Export them from the
package and the server's vendored runner shim.
- `packages/plugins/sdk/src/environment-tasks.ts`: add optional
`preparationDigest` and `runDigest` to the `prepare` operation, and
optional `launchDigestVersion: 1` to the task description. The
capability gates both digests.
- `server/src/services/native-runtime/plugin-runner-task.ts` and
`native-session-executor.ts`: send both digests on `prepare` when the
description advertises `launchDigestVersion: 1`.
- `server/src/services/environment-task-runtime.ts`: recompute both
digests from the requirements, and refuse a `prepare` whose digests are
missing in part or do not match.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the SDK
public source pin.
- Tests:
- the preparation digest is stable across runs and changes when the
runner, lease, session, instructions, model, approval policy or settings
change;
- with a runtime context, an MCP-only change (the run's binding, no
binding, a changed access set) keeps the preparation digest and changes
the run digest;
- a change to the instruction bundle, instruction entry, a skill bundle,
the skill set, the working copy or the connection instructions changes
the preparation digest;
  - the run digest changes per run;
  - Core sends the digests only when advertised;
- the runtime refuses forged or partial digests before it calls the
plugin.

## Verification

- `npx vitest run src/live/runner-launch-preparation.test.ts` in
`packages/paperclip-runner`: 26 passed.
- `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts
src/__tests__/environment-task-runtime.test.ts` in `server`: 160 passed.
- `npx vitest run source-sdk-public-inputs.test.mjs
source-sdk-evidence.test.ts source-sdk-public-type.test.ts` in
`tests/runner-e2e`: 59 passed.
- `pnpm typecheck` in `server` and in `packages/paperclip-runner`:
clean.
- Not run locally: the full suite. CI covers it.

## Risks

Low risk. The digests are new optional fields behind a plugin capability
version. Core computes them and checks them again before it calls the
plugin, so a plugin can rely on them. They prove only that the launch
requirements are unchanged. A plugin that skips work on a match must
still check its own state, such as files it published into the sandbox.
If the runtime context carries values that change per run, the
preparation digest changes too, so a plugin does its full preparation.
That is safe, but the plugin saves nothing. The MCP binding is the one
such value Core sets per run today, and MCP is excluded entirely. A
plugin that does publish MCP configuration into its sandbox must not
rely on this digest for it. Values outside the requirements are not
covered at all: the account identity and credentials, the GitHub broker,
project IDs, and the plugin's own runtime revision. A plugin must keep
checking those itself. A match lets a plugin skip publishing its launch
configuration, not computing the descriptor: `prepare` must still return
a launch whose `requirementsDigest` matches the current requirements,
which carry the current MCP assignment and binding.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:07:54 -08:00
Nicky LeachandClaude Opus 5.5 8cc26221af perf(native): accept a plugin's in-preparation project-auth check on warm attach (#21710)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Environment plugins can run a native Runner as a plugin task, and a
warm session keeps that Runner between turns
> - On a warm turn Core attaches the kept Runner to the new run: the
plugin prepares the new run, then Core checks managed project
authentication
> - A warm attach has no submit step, so Core sends a separate
`check_project_auth` operation after `prepare`
> - That separate operation is one more plugin dispatch and Launcher
round trip on every warm turn, about 0.12–0.18 s, and it is serial
> - This pull request lets the plugin do the check inside `prepare` and
say so, so Core does not send the second operation
> - The benefit is a faster warm turn with the same proof and the same
failure behavior

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The warm attach of a plugin Runner task (`plugin-runner-task.ts`
`prepareLaunch`). When `launched` is true, Core always calls
`checkProjectAuth` after `prepare`.

**Subsystem affected**
Native runtime, plugin Runner tasks, and the plugin SDK environment task
contract.

**Current behavior**
Cold starts fold the check into submit (`projectAuthCheckAtSubmit`).
Warm attaches send `prepare`, then a separate `check_project_auth`.
Measured on a plugin sandbox, the separate operation costs 0.12–0.18 s
on each warm turn.

**Proposed behavior**
The `prepared` result has a new optional flag, `projectAuthChecked`. A
plugin sets it only when it ran the managed project-authentication check
inside this preparation, for this launch's configuration, provider and
projects, and the check allowed them. Core then records the same lease
proof that an allowed `check_project_auth` records, and skips the
separate check.

**Reason and benefit**
One fewer serial round trip on every warm turn. The plugin already runs
the check for this task; this removes a dispatch, not a check.

**Breaking changes**
None. The flag is optional. A plugin that does not set it, or that saw a
refusal, gets the current behavior. Plugins detect the flag with the SDK
schema before they send it.

## What Changed

- `packages/plugins/sdk/src/environment-tasks.ts`: add optional
`projectAuthChecked: true` to the `prepared` result, with a doc comment
that defines it.
- `server/src/services/environment-task-runtime.ts`: when a `prepared`
result has `projectAuthChecked`, record
`runnerTaskProjectAuthValidation` for the launch's configuration,
provider and projects, without `atSubmit`. This is the proof that
`submit` and `read_file` require.
- `server/src/services/native-runtime/plugin-runner-task.ts`: do not
send `check_project_auth` when the prepared result has
`projectAuthChecked`.
- Tests: a warm attach with the flag makes only `prepare`; without it,
`prepare` then `check_project_auth`. The runtime test covers all four
combinations of `projectAuthCheckAtSubmit` and `projectAuthChecked`.

## Verification

- `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts
src/__tests__/environment-task-runtime.test.ts` in `server/`: 157
passed.
- Not run locally: full typecheck and full suite. CI covers them.

## Risks

Low risk. Core trusts the flag the same way it trusts a plugin's
`check_project_auth` answer and `projectAuthCheckAtSubmit` today. A
refusal still goes through Core's own check, so the conflict error is
unchanged. The plugin side ships separately and sends the flag only when
its SDK accepts it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, tracing queries).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 03:13:48 +00:00
DottaandClaude Opus 5.5 b5ba548112 feat(server): add an instance-admin server metrics endpoint for the scale ramp (#21708)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The scaling plan
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`) has a finish line
that a real-agent ramp harness checks (#21695)
> - Four finish-line criteria need numbers from inside each server
process: main-thread CPU, run event lateness, agent API time before the
handler, and missed lease renewals
> - The server had no way to give them, so every harness run was at best
"incomplete"
> - This pull request adds one instance-admin endpoint that returns
those numbers per process over a short sliding window, off by default,
and connects the harness to it
> - The benefit is that a ramp run can measure every server criterion it
can see, with near-zero cost when on and no cost when off

## Linked Issues or Issue Description

Refs #21695

**Subsystem affected**

Server (instance routes, run event append, native lease renewal),
database client (statement counter), and the scale-ramp harness in
`scripts/scale-ramp/`.

**Problem or motivation**

The scale-ramp harness reads `PAPERCLIP_SCALE_METRICS_URL`, but the
server has no endpoint for it. The criteria `main_thread_cpu`,
`run_event_lateness_p99`, `agent_api_pre_handler_p99` and
`lease_renewals_missed_zero` are therefore always "not measured", and no
run can pass.

**Proposed solution**

Add `GET /api/instance/server-metrics`. It returns a JSON snapshot of
one server process over the last 50 to 60 s. Only an instance admin can
read it. The env flag `PAPERCLIP_SERVER_METRICS` (read once at start,
default off) turns on collection and the route. Off, the path answers
404 and no collector starts.

**Alternatives considered**

- The postgres.js `debug` option for the statement counter: rejected,
because it captures a stack (`new Error()`) for every query.
- Process CPU from `process.cpuUsage()` as `mainThreadCpu`: rejected,
because it includes GC and libuv threads. It is returned as `processCpu`
instead, and `mainThreadCpu` reads the main thread from `/proc` (Linux
only, null elsewhere).
- Default on: the measured cost is negligible (numbers below), but the
event-loop monitor wakes the process every 20 ms when idle, and this is
a new instance-wide surface. It stays off; the runbook says how to turn
it on for a ramp run.

## What Changed

- `server/src/services/server-metrics.ts`: a per-process collector. It
uses `performance.eventLoopUtilization()`, `monitorEventLoopDelay` (20
ms), `process.cpuUsage()`, the main thread's
`/proc/self/task/<pid>/stat` on Linux, and six 10 s buckets of native
`perf_hooks` histograms. One unref'd timer rotates the buckets. Nothing
on the hot path awaits, allocates per call or does I/O.
- `server/src/routes/instance-server-metrics.ts`: `GET
/api/instance/server-metrics`, guarded by the existing
`assertInstanceAdmin` (the check that `/instance/database-backups` and
other instance routes use). It returns numbers, a random per-process
`processId`, `uptimeSec`, `buildRevision` and `windowMs`. It returns no
SQL, names, IDs or hostnames.
- `server/src/app.ts`: when the flag is on, two middlewares time agent
API requests from arrival to the API router, the route is mounted, and
shutdown stops the collector. When the flag is off, nothing is mounted.
- The snapshot also carries `dedicatedConnections` from
`dedicatedDbConnectionStats()` (#21693): `limit`, `inUse`, `waiting` at
request time and `peakInUse`, `queued`, `overLimit`, `nestedOverLimit`
since start. The harness ignores it today; it shows dedicated-connection
queueing in a restart burst.
- `packages/db/src/statement-count.ts` and
`lazy-transaction-queries.ts`: a Drizzle logger that increments one
counter per statement replaces Drizzle's no-op logger. No edit to
`client.ts`.
- `paperclip-control-plane-port.ts`: one call records runner `emittedAt`
to committed row for each committed (not duplicate) runner event.
- `native-execution-lease.ts`: one call counts a lease whose renewal
deadline passed.
- Harness: missed lease renewals are summed per `processId` (several
processes, or a restart, no longer cancel out). A metrics URL on the
target origin gets the board token in the `Authorization` header;
`PAPERCLIP_SCALE_METRICS_TOKEN` still overrides it, and another origin
gets no token.
- Contract: `scripts/scale-ramp/server-metrics.fixture.json` is the
serializer's own output. A server test asserts the serializer produces
exactly this file, and a harness test parses it and checks every field.
- `rules/company-scope-exceptions.mdc`: adds the snapshot as an
instance-wide exception with its reason. The runbook in the plan doc
documents the path, the flag, the credential and each field.

| Field | Provided | Source |
| --- | --- | --- |
| `mainThreadCpu` | yes (Linux; null elsewhere) |
`/proc/self/task/<pid>/stat` utime+stime |
| `eventLoopUtilization` | yes | `performance.eventLoopUtilization()` |
| `eventLoopDelayP99Ms` | yes | `monitorEventLoopDelay`, highest 10 s
p99 in the window |
| `statementsPerSecond` | yes | Drizzle logger counter (not BEGIN/COMMIT
or raw driver calls) |
| `poolWaitP99Ms` | absent | postgres.js has no cheap hook for queue
time |
| `runEventLatenessP99Ms` | yes (null with no events) | runner event
append |
| `agentApiPreHandlerP99Ms` | yes (null with no agent requests) |
arrival and router-entry middlewares |
| `leaseRenewalsMissed` | yes, counter since process start | native
lease deadline timer |

## Verification

- `cd server && npx vitest run
src/__tests__/instance-server-metrics-routes.test.ts
src/__tests__/server-metrics.test.ts`: through `createApp` on the
embedded test database. 200 with the exact field set for an instance
admin; 403 for a company owner who is not an instance admin, for an
agent key, and unauthenticated; 404 and no collector when the flag is
off; values move after agent requests, statements, a late run event and
a missed renewal; the body has no IDs, keys or SQL.
- `cd server && npx vitest run
src/services/native-runtime/native-execution-lease.test.ts
src/services/native-runtime/paperclip-control-plane-port.test.ts
src/__tests__/api-server-timing.test.ts`: the lease and run-event hooks
(red/green checked for the lease hook by removing the call).
- `cd packages/db && npx vitest run src/lazy-transaction-queries.test.ts
src/client-options.test.ts`: the counter counts statements inside
transactions and savepoints.
- `node --test scripts/scale-ramp/scale-ramp.test.ts` and `tsc -p
scripts/scale-ramp/tsconfig.json`: 38 pass, including the fixture
contract and the board-token header.
- `node server/scripts/typecheck-layers.mjs` and `tsc --noEmit` in
`packages/db`: pass.

Overhead, measured on a shared, loaded laptop (load average 8 to 20),
median with min and max:

| Measure | Off | On |
| --- | --- | --- |
| Logger call per statement (5M calls x 7) | 0.28 ns (0.24 to 0.97) |
3.36 ns (3.32 to 6.16) |
| CPU per `select 1` statement, embedded Postgres (3,000 x 10) | 28.7 us
(20.3 to 42.4) | 32.3 us (20.7 to 45.5) |
| CPU per HTTP request, client and server in one process (2,000 x 6) |
67.0 us (40.4 to 107.4) | 64.2 us (40.9 to 83.3) |
| Idle process CPU, 5 s x 3 | 0.01% of a core (0.00 to 2.21) | 0.08%
(0.05 to 0.09) |
| Record one run event (Date.parse + histogram) | - | 148 ns (135 to
188) |
| One snapshot | - | 55 us |

The statement and request differences are inside the run-to-run spread.
The direct cost is about 3 ns per statement, against about 30 us of CPU
per statement.

## Risks

- Low risk. Off by default; off, only the statement counter runs, and it
costs one increment more than Drizzle's no-op logger.
- `mainThreadCpu` assumes Linux USER_HZ of 100 and is null on other
platforms. The harness then uses `eventLoopUtilization`.
- Run event lateness compares the runner's clock with the server's
clock, so clock skew adds to it. Negative values count as zero.
- With several processes, each request reaches one process. The harness
keeps the highest gauge and sums counter growth per process, but a
process it never samples is not seen.
- The statement counter does not count BEGIN/COMMIT or raw postgres.js
calls.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:13:45 -06:00
DottaandClaude Opus 5.5 11b09b7234 refactor: remove doc/execution-semantics.md and its references (#21699)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Contributors and agents read the docs under `doc/` for background on
run execution
> - `doc/execution-semantics.md` grew to 6,310 lines of per-PR notes
> - It was hard to read and it drifted from the code
> - The code and tests are the source of truth for execution behavior
> - This pull request removes the file and every reference to it
> - The benefit is one less stale document that readers can mistake for
a requirement

## Linked Issues or Issue Description

**Issue type**

Outdated or unclear documentation.

**Where is the issue?**

`doc/execution-semantics.md`, and the docs, rules, skills, maps, code
comments and log text that link to it.

**What's wrong?**

The file is 6,310 lines of notes added one PR at a time. Much of it
repeats or contradicts the code. Readers and agents treat it as a
contract, but the code and tests decide the behavior.

**Suggested fix**

Delete the file. Remove each link to it. Keep the documents that link to
it. Point at code where a pointer is still useful. Do not move the
content to another doc.

Refs #10664 (an open docs PR that edits this file; it becomes obsolete).

## What Changed

- Delete `doc/execution-semantics.md`.
- Remove links and pointers in `doc/PRODUCT.md`,
`doc/LOW-TRUST-PRESETS.md`, `doc/TASK-WATCHDOG.md`,
`doc/composer-stop.md`, `doc/native-workspace-finalization-recovery.md`,
`doc/run-log-events.md`, `doc/runner-internals.md` and
`doc/workspace-repositories.md`.
- `doc/TASK-WATCHDOG.md` now points at
`server/src/modules/active-run-watchdog/` for the silent active-run
watchdog.
- Remove the link in `rules/upgrade-safe-run-state.mdc` and
`maps/feature-map/recovery.md`.
- Update the skills `diagnose-why-work-stopped`,
`fix-failed-paperclip-run` and `terminal-bench-loop`. The
`fix-failed-paperclip-run` skill now points at
`server/src/services/heartbeat/` and `server/src/services/recovery/`.
- Remove the file from the source anchors in `.devin/wiki.json`.
- Remove the doc name from four code comments in `server/` and `ui/`.
- The failed-attachment log message no longer ends with a link to the
removed doc. Two tests now check the message text instead of the link.
- Leave two dated plan files under `doc/plans/` unchanged. They are
history.

## Verification

- `git grep -i execution-semantics` finds no hits outside `doc/plans/`.
- Every relative Markdown link in the changed files resolves.
- `node scripts/check-maps.mjs` passes.
- `node --test .github/scripts/tests/*.test.mjs` passes (808 tests).
- `pnpm verify` was not run locally. The worktree has no `node_modules`.
CI runs the changed Vitest files.

## Risks

- Low risk. Most changes are docs and comments.
- One runtime log line is shorter. It no longer has a link. No logic
changes.
- Open PRs that edit the deleted file, such as #10664, will conflict.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
shell access.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:29:25 -06:00
DottaandClaude Opus 5.5 adc3356ad3 fix(native): retry a coordinator claim that loses a deadlock instead of failing the run (#21703)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native (Paperclip Runner) run claims its
`native_run_finalizations` coordinator before it starts any provider
work. The claim is one transaction. It locks the coordinator row, then
the run row.
> - A writer that holds the run row and then waits for the coordinator
row closes a lock cycle. PostgreSQL then aborts the claim with a
deadlock (`40P01`), because the claim waited first.
> - The aborted claim rolls back whole and starts no provider work. But
the raw database error failed the run before its first turn.
> - The failure path then preserved the prepared workspace. That stamped
the unclaimed coordinator `native_continuation_requires_reconciliation`,
and `failRun` used it as the run's error code. The task waited for
reconciliation.
> - A transient database fault must never end a run like this. The
target rate of runs that need Retry is 0%.
> - This pull request retries the claim transaction on transient lock
contention with the shared bounded helper.
> - The benefit is that a fresh run that loses a lock race at its claim
continues to its provider turn by itself.

## Linked Issues or Issue Description

**What happened?**
On a hosted QA stack, a fresh native assignment run (not a retry) failed
after about 50 s. The run status was `failed`. The error code was
`native_continuation_requires_reconciliation`. The error text was
`Database query failed (40P01)`. The stderr was empty. The task needed
reconciliation.

**Steps to reproduce**
1. Prepare a native run with an unclaimed coordinator (`observed`,
attempt 0), as runtime selection leaves it.
2. In another transaction, lock the run row.
3. Start the native session. Its claim locks the coordinator and waits
for the run row.
4. In the other transaction, lock the coordinator row.
5. PostgreSQL aborts the claim with `40P01`, and
`executePaperclipNativeSession` rejects with that error.

**Expected behavior**
The claim retries after the competing transaction commits. The run
reaches its provider turn.

**Paperclip version or commit**
`master` at 958eda6f16.

## What Changed

- `native-session-executor.ts`: the coordinator claim transaction runs
inside `retryTransientLockContention` (`40P01`, `40001`, `55P03`; 5
attempts, 100 ms × attempt backoff). Each attempt re-reads and re-proves
the coordinator, the run, cancellation and lineage, so a retry admits
nothing that the first attempt would refuse.
- A startup context that a rolled-back attempt opened is finished, and
the startup owner's previous context is restored, before the next
attempt.
- New PostgreSQL test
`native-coordinator-claim-deadlock.integration.test.ts`: `retries a
claim that lost a deadlock and reaches the provider`.

## Verification

- RED on `master` (958eda6f16): the new test fails.
`executePaperclipNativeSession` rejects with a `40P01` cause, and the
provider is not called.
- GREEN with this change: `pnpm --filter @paperclipai/server exec vitest
run
src/services/native-runtime/native-coordinator-claim-deadlock.integration.test.ts`
passes. The provider is called once and the coordinator is at attempt 1.
- `tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` for the affected files: see the PR comments.

## Risks

- Low risk. The normal path has no contention and does not retry.
- Under contention, the claim waits at most about 1 s more (backoff
100–400 ms) before provider work. This is before `turn.start`, so time
to first token grows only in the contended case.
- The lock order (coordinator, then run) is unchanged. It matches
admission and the runner checkpoint owner lock. The writer that took the
run row first on the QA stack is not identified, because no server logs
or SQL were available. This change makes the claim survive any such
writer.
- Out of scope: a transient fault in other pre-dispatch statements of a
run with a prepared remote workspace. There, workspace preservation
still stamps the pre-claim code before `failRun`. The pre-claim
fresh-continuation fallback then continues the task.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, embedded PostgreSQL tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20561 (pre-dispatch transient retry), #20682 (provider-pack
admission lock wait).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:23:33 -06:00
DottaandClaude Opus 5.5 958eda6f16 fix(native): wait for a live outgoing server before resuming its warm session in a deploy overlap (#21697)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap. The outgoing server still
serves while the incoming server starts new runs.
> - A warm agent session stays with the server that ran its last turn.
That server closes the session about 1 s after it leaves new run starts
to the replacement (#21627).
> - A follow-up that started within about 1 s of its predecessor's
finish read the session as suspended, while the outgoing server was
still closing it. The follow-up did not wait. It started a second Runner
with the wrong OpenCode launch files, and the Runner refused the launch.
> - The same-run retry then went to the outgoing server, which had
already left run starts. The run failed, and a blocked parent showed a
Retry during its scheduled resume.
> - This pull request makes the follow-up wait for the live outgoing
server, rebuilds the admitted launch files on the new Runner, keeps
same-run retries on the replacement, and shows the scheduled resume for
a blocked parent.
> - The benefit is that a follow-up during a deploy overlap finishes
without a failed run or a Retry banner.

## Linked Issues or Issue Description

Refs #21627

**What happened?**
A zero-downtime release-swap test (Gate B, wave B1 on master 890dcbdd66)
failed 2 of 8 cross-server hand-offs. In both, the follow-up started 1.0
s and 0.6 s after its predecessor finished on the outgoing server. The
other 6 hand-offs started 34 to 217 s later and passed.
- The incoming server took the resume path that skips
`inspectRemoteWarmSessionRecovery`, because the local authority state
read `suspended`. The live-owner wait from #21627 lives only in that
inspection, so it never ran (0 waits recorded).
- The new home had no harness backup, so the run had no provider
artifact receipt. `prepareRemoteRunner` returned early on
`legacyProviderContinuity`, and the launch profile named the provider
pack folder. Runner core refused it: "OpenCode run.prepare launch does
not match the runner-owned qualified profile".
- The same-run retry was claimed by the outgoing server after it left
run starts. Its predecessor retirement reported `not_suspended`. Both
runs failed with `native_continuation_requires_reconciliation`, and
recovery started fresh runs.
- For a parent already blocked on its subtasks, the task showed "failed"
with Retry for 31 s while the same-run resume was scheduled.

**Expected behavior**
The follow-up waits while the outgoing server may still hold the
session. A new Runner gets the same OpenCode launch files as the
session's first launch. Only the replacement claims same-run retries. A
blocked parent shows the scheduled resume and no Retry.

**Steps to reproduce**
1. Run the restart-chaos release-swap preset with fresh homes and a warm
OpenCode staging mix.
2. Let a run finish on the outgoing server after the traffic switch,
with a follow-up comment already queued.
3. See the follow-up start on the incoming server within about 1 s and
fail on `run.attach` with the qualified-profile mismatch.

**Paperclip version or commit**
master 890dcbdd66.

**Deployment mode**
Hosted deploy overlap (release swap) with Daytona sandboxes and S3
storage.

## What Changed

- `remote-warm-session-recovery.ts`: new `waitForLiveWarmSessionOwner`.
It reads the finished run's finalization owner. While that server is
alive inside the warm window, the follow-up defers with the existing
`native_controller_recovery_waiting_for_owner` wait. A harness backup
that holds the suspended state and was completed after the turn proves
that the owner finished its close, so the follow-up continues at once.
- `native-session-executor.ts`: calls that wait before the `suspended`
check, so every warm follow-up on another server's session passes
through it. The backup check runs only when the owner is alive.
- `native-session-executor.ts`: when a resume has a durable session but
no receipt from a harness backup or the admission, it reads the provider
artifact receipt recorded on the session's last run. The new Runner then
gets the admitted launch root, as a cold continuation already does.
- `heartbeat/recovery.ts` and `heartbeat.ts`: a server that yields run
starts to a replacement does not claim same-run retries, from the
recovery sweep or from its own resume timer. It re-checks every 3 s, so
it takes the retries back when the replacement stops.
- `task-interruption-presentation.ts`: a blocked task whose remote run
waits for its `exact_checkpoint_resume` keeps the resume time, so it
shows `retry_scheduled` with no Retry.
- `doc/execution-semantics.md`: documents these rules.

## Verification

Red on master, green with this change:
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "close
window waits for its live lease"`: "a follow-up that starts within the
outgoing server's close window waits for its live lease instead of
relaunching the Runner". Master proceeds and fails with
`runner_state_identity_mismatch`.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "remote
opencode-continuation-prior-receipt"`: a resume relaunch without an
adopted process uses the original OpenCode file paths. Master launches
from the provider pack folder and does not retain the receipt.
- `cd server && npx vitest run src/services/heartbeat/recovery.test.ts
-t "leaves run starts"`: "an outgoing server does not claim a same-run
retry once it leaves run starts". Master claims it.
- `cd server && npx vitest run
src/__tests__/task-interruption-query.test.ts -t "blocked parent whose
run awaits"`: "a blocked parent whose run awaits its exact same-run
resume shows retry_scheduled with no Retry". Master shows `failed` with
Retry.

Also new: "keeps a follow-up waiting for a live outgoing server until
that server has checkpointed the session after the turn" in
`native-controller-recovery.integration.test.ts` (the backup proof and
the owner exit end the wait).

Focused suites pass: native-session-executor,
remote-warm-session-recovery, native-controller-recovery (integration),
heartbeat recovery, task-interruption-query,
task-interruption-presentation, native-recovered-owner-followup,
heartbeat-native-predispatch-retry, run-start-handover. `npx tsc
--noEmit -p server/tsconfig.json` passes.

Not run here: the restart-chaos release-swap wave. The maintainer runs
it after merge.

## Risks

- A follow-up on a session that a live outgoing server ran can now wait
until that server exits or the warm window ends. This applies only when
no harness backup proves that the server finished its close after the
turn. Before, such a follow-up failed and needed recovery.
- The backup proof hashes the restored backup. It runs only when the
previous owner is alive in another process (a deploy overlap), not on a
normal turn.
- A resume without a backup receipt now restores the admitted artifacts
from storage, as a cold continuation does. A session whose last run
recorded no receipt keeps the legacy pack-folder launch.
- A yielding server leaves same-run retries to the replacement. If the
replacement stops, the yield ends with its lease, and the older server
claims them again within seconds.
- Previous-release state: no persisted field is added. Receipts,
finalization owners and `failureDetail.recoveryMode` are already saved
by the previous release.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 19:50:01 -06:00
DottaandClaude Opus 5.5 97a167093d fix(queue): send a message into a starting run without a cancel or recovery errors (#21661)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can queue a message while an agent works, then send it into
the running turn
> - A run that is still starting takes queued messages into its first
turn, and Interrupt hands off to them
> - Both rules only accepted a queue that a user started
> - A user's message can merge into an earlier automatic wake that waits
behind the task, and that wake keeps its system requester
> - Then the starting run did not take the message, Interrupt cancelled
the boot, and the task showed "Couldn't finish this attempt" with Retry
and "Waiting for execution recovery" under the composer until the
cancelled boot settled
> - This pull request makes both rules judge the messages and the
recorded Interrupt intent, not the wake's requester, and shows an
Interrupt hand-off as sent
> - The benefit is that sending a queued message feels immediate and
shows no error

## Linked Issues or Issue Description

**What happened?**
A run started from a resolved blocker. The user queued a message 20
seconds later, while the runner was still starting. The message merged
into an older automatic deferred wake. The queue offered Interrupt
instead of sending it. Interrupt cancelled the starting run. For about 7
seconds the task showed "Couldn't finish this attempt. Your message is
saved." with Retry, and the composer showed "Waiting for execution
recovery. Your message is saved." Then the message was delivered.

**Expected behavior**
The starting run takes the message into its first turn. When an
Interrupt does stop a run, the message shows as sent, with no error
banner, Retry, or recovery text.

**Steps to reproduce**
1. Have an automatic deferred wake for a task (for example from an
earlier blocked execution).
2. Start a run for the task with a Paperclip runner agent.
3. While the runner is still starting, queue a message and click the
send control.

## What Changed

- `deferredWakeJoinsFirstTurn`: a system or unattributed wake qualifies.
Each message must still be the responsible user's own
(`firstTurnEligibleComment`). A wake from another user or an agent keeps
its own delivery.
- Task interruption presentation: find the Interrupt queue by its
recorded `queuedCommentInterrupt` intent, not by `requestedByActorType =
user`. The hand-off now shows as working for a merged queue too.
- Queued-comment read: while a deferred queue carries an Interrupt
intent, report an `interrupt_handoff` wait instead of the recovery
message.
- UI (`TaskChatQueuedMessages`, `IssueDetail`): an `interrupt_handoff`
response counts as sent. The messages stay out of the queue and no
status or error line shows.

## Verification

Red/green tests. Each failed before the fix and passes after it:
- `server/src/__tests__/issue-queued-comments-routes.test.ts`: "gives a
booting run's first turn the user's message merged into an earlier
automatic wake"; the Interrupt fallback case now expects
`interrupt_handoff` on the response and on a poll. 144/144 pass.
- `server/src/__tests__/task-interruption-query.test.ts`: the hand-off
case runs for a user-started and a system-started queue. 32/32 pass.
- `ui/src/components/task-chat/TaskChatQueuedMessages.test.tsx`: "shows
an Interrupt as sent while the stopped turn hands off". 43/43 pass.
- `first-turn-queued-input.test.ts` 8/8, `issue-queued-comment-queue`
7/7. Server and UI typecheck pass.

Root cause from the staging wake diagnostics: the run started at
20:26:54, the message was coalesced at 20:27:17 into a deferred
`issue_execution_deferred` automation wake from the previous day, the
Interrupt at 20:27:20 recorded `execution_recovery`, and the
false-positive hold resolved at 20:27:26.

Not run: local runner E2E steering cells. The OpenCode chat-features
cells require the trusted CI host. The Codex and Claude
`revise-while-running` chat cells failed before steering on a fixture
workspace path (`workspaces/` vs `isolated-workspaces/`), which this
change does not touch. Staging verification follows the deploy.

## Risks

- Low. A wider set of deferred wakes can join a starting run's first
turn. Message eligibility, the claim receipt, and exactly-once delivery
are unchanged.
- An Interrupt intent that never delivers shows "sent" in the composer.
The task notice still shows a failure with Retry after its existing
grace period.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:06:15 -06:00
DottaandClaude Opus 5.5 648b238be1 perf(native-runtime): stop re-finalizing native results whose finalization error is recorded (#21674)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime releases owners of failed runs. When the owner's
controller is in a process space that Paperclip cannot verify, it shows
controller namespace attention on the task instead.
> - On a hosted instance with zero active runs, this writer made about
21% of all work signals. It inserted a new active attention row for the
same task about every 26 seconds.
> - The tasks were already done or cancelled. Recovery resolved each row
as `source_terminal`. The active slot was then free, so the next
owner-release pass inserted a new row and published a graph recovery
signal again. The loop did not stop.
> - A live task had the same defect: after the fresh-continuation
fallback resolved its attention, the next pass made the same incident
again.
> - This pull request stops the writer from surfacing attention for
finished tasks. It also stops the writer from making an incident again
when that run authority was resolved after its lease expired. Those
owners are no longer release candidates and do not set the release
deadline.
> - The benefit is that an idle instance does not write or publish
anything for these runs. Attention still shows once for a live task.

## Linked Issues or Issue Description

**What happened?**
The failed-run owner release (`settleFailedRunOwner`, expired-owner
path) called `surfaceNativeControllerNamespaceAttention` for every
expired foreign owner that had no active attention. It did this even
when the source task was `done` or `cancelled`. Recovery resolved that
attention as `source_terminal`. The next pass found no active attention
and inserted a new board row. Each insert published
`notifyIssueGraphRecovery` and the other execution-control signals. The
stranded-task check then ran again for the task graph. A hosted instance
collected about 45,000 rows on 13 finished tasks in this way.

**Expected behavior**
A pass that changes nothing writes nothing and publishes nothing. A
finished task never gets controller namespace attention. A live task
gets one incident for each exact run authority. A resolved incident
stays resolved until the authority changes.

**Steps to reproduce**
1. Make a native run that failed, with an `observed` coordinator whose
foreign lease has expired and whose controller host is not recorded.
2. Mark the task done or cancelled, or let the fresh-continuation
fallback resolve the attention of a live task.
3. Run the execution-finalization worker with the issue recovery workers
for 30 minutes.
4. On master, each task gets about 40 to 60 new attention rows. The
writer publishes thousands of signals.

## What Changed

- `surfaceNativeControllerNamespaceAttention` (shared helper): before it
creates attention, it checks for a resolved incident for this run that
was resolved at or after the owner's current lease expiry. If one
exists, it writes nothing.
- New `settledNativeControllerNamespaceAttention` SQL predicate. It is
shared by the helper, the release candidate page, and the release
deadline query, so all three use the same rule.
- `settleFailedRunOwner` (expired-owner path) does not surface attention
when the task is `done` or `cancelled`. The restart-recovery claim path
has the same guard.
- `releaseAbandonedNativeOwners` and `nextAbandonedNativeOwnerReleaseAt`
skip foreign owners whose task is finished or whose incident is settled.
A pass that has nothing to do selects nothing, and no immediate deadline
is set again. Local owners keep the normal proof-based release.
- The owner release proofs did not change. A foreign owner is still
never released without proof.
- Tests:
`server/src/__tests__/native-failed-run-owner-release-idle.test.ts`
(real PostgreSQL, real coordinator and workers, fake clock, statement
and signal counters) and three new cases in
`native-failed-run-owner-release.test.ts`.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/native-failed-run-owner-release-idle.test.ts
src/services/native-runtime/native-failed-run-owner-release.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/services/native-runtime/native-restart-recovery-isolation.test.ts`:
80 tests pass.
- Red on master, green with this change. The test seeds one done task
and one cancelled task (each with an earlier resolved `source_terminal`
incident) and one live task, then runs 30 idle minutes:
- master: 126 attention inserts, 249 attention writes, 7,024 signals,
8,025 statements.
- this branch: 1 attention insert (the live task, surfaced once), 2
attention writes (that insert and the fallback's resolution), 36
signals, 350 statements. All of this is startup work. Nothing repeats.
- After the fresh continuation settles a live task's attention, six
unrelated finalization wakes over 30 minutes:
- master: 3 attention inserts, 90 signals, 551 statements, and the
release deadline stays due.
- this branch: 0 attention writes, 0 run writes, 6 signals (the test's
own wakes), 153 statements, and the release deadline is null.
- New unit cases: a done or cancelled task never surfaces, and surfaces
once when it is reopened. A settled incident stays settled over repeated
passes. A renewed authority (a new lease expiry) surfaces one new
incident.
- The three new unit cases fail on master and pass on this branch.
- `tsc --noEmit` for `server` passes.

## Risks

- Low risk. The change only removes writes. It does not weaken a release
proof, an authorization, a budget, a pause, or an approval check.
- An incident that a person resolved or dismissed after the lease
expired is not made again for the same authority. A renewed authority (a
later lease expiry) still makes a new incident.
- A superseded incident (status `cancelled`) still shows again when the
superseding action clears. This keeps the current behavior.
- The historical rows that the loop made are not removed. This change
only stops new rows.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21621, #21616, #21617

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:50:45 -06:00
DottaandClaude Opus 5.5 64c7122dae perf(native-runtime): stop re-surfacing settled controller namespace attention (#21675)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime releases owners of failed runs. When the owner's
controller is in a process space that Paperclip cannot verify, it shows
controller namespace attention on the task instead.
> - On a hosted instance with zero active runs, this writer made about
21% of all work signals. It inserted a new active attention row for the
same task about every 26 seconds.
> - The tasks were already done or cancelled. Recovery resolved each row
as `source_terminal`. The active slot was then free, so the next
owner-release pass inserted a new row and published a graph recovery
signal again. The loop did not stop.
> - A live task had the same defect: after the fresh-continuation
fallback resolved its attention, the next pass made the same incident
again.
> - This pull request stops the writer from surfacing attention for
finished tasks. It also stops the writer from making an incident again
when that run authority was resolved after its lease expired. Those
owners are no longer release candidates and do not set the release
deadline.
> - The benefit is that an idle instance does not write or publish
anything for these runs. Attention still shows once for a live task.

## Linked Issues or Issue Description

**What happened?**
The failed-run owner release (`settleFailedRunOwner`, expired-owner
path) called `surfaceNativeControllerNamespaceAttention` for every
expired foreign owner that had no active attention. It did this even
when the source task was `done` or `cancelled`. Recovery resolved that
attention as `source_terminal`. The next pass found no active attention
and inserted a new board row. Each insert published
`notifyIssueGraphRecovery` and the other execution-control signals. The
stranded-task check then ran again for the task graph. A hosted instance
collected about 45,000 rows on 13 finished tasks in this way.

**Expected behavior**
A pass that changes nothing writes nothing and publishes nothing. A
finished task never gets controller namespace attention. A live task
gets one incident for each exact run authority. A resolved incident
stays resolved until the authority changes.

**Steps to reproduce**
1. Make a native run that failed, with an `observed` coordinator whose
foreign lease has expired and whose controller host is not recorded.
2. Mark the task done or cancelled, or let the fresh-continuation
fallback resolve the attention of a live task.
3. Run the execution-finalization worker with the issue recovery workers
for 30 minutes.
4. On master, each task gets about 40 to 60 new attention rows. The
writer publishes thousands of signals.

## What Changed

- `surfaceNativeControllerNamespaceAttention` (shared helper): before it
creates attention, it checks for a resolved incident for this run that
was resolved at or after the owner's current lease expiry. If one
exists, it writes nothing.
- New `settledNativeControllerNamespaceAttention` SQL predicate. It is
shared by the helper, the release candidate page, and the release
deadline query, so all three use the same rule.
- `settleFailedRunOwner` (expired-owner path) does not surface attention
when the task is `done` or `cancelled`. The restart-recovery claim path
has the same guard.
- `releaseAbandonedNativeOwners` and `nextAbandonedNativeOwnerReleaseAt`
skip foreign owners whose task is finished or whose incident is settled.
A pass that has nothing to do selects nothing, and no immediate deadline
is set again. Local owners keep the normal proof-based release.
- The owner release proofs did not change. A foreign owner is still
never released without proof.
- Tests:
`server/src/__tests__/native-failed-run-owner-release-idle.test.ts`
(real PostgreSQL, real coordinator and workers, fake clock, statement
and signal counters) and three new cases in
`native-failed-run-owner-release.test.ts`.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/native-failed-run-owner-release-idle.test.ts
src/services/native-runtime/native-failed-run-owner-release.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/services/native-runtime/native-restart-recovery-isolation.test.ts`:
80 tests pass.
- Red on master, green with this change. The test seeds one done task
and one cancelled task (each with an earlier resolved `source_terminal`
incident) and one live task, then runs 30 idle minutes:
- master: 126 attention inserts, 249 attention writes, 7,024 signals,
8,025 statements.
- this branch: 1 attention insert (the live task, surfaced once), 2
attention writes (that insert and the fallback's resolution), 36
signals, 350 statements. All of this is startup work. Nothing repeats.
- After the fresh continuation settles a live task's attention, six
unrelated finalization wakes over 30 minutes:
- master: 3 attention inserts, 90 signals, 551 statements, and the
release deadline stays due.
- this branch: 0 attention writes, 0 run writes, 6 signals (the test's
own wakes), 153 statements, and the release deadline is null.
- New unit cases: a done or cancelled task never surfaces, and surfaces
once when it is reopened. A settled incident stays settled over repeated
passes. A renewed authority (a new lease expiry) surfaces one new
incident.
- The three new unit cases fail on master and pass on this branch.
- `tsc --noEmit` for `server` passes.

## Risks

- Low risk. The change only removes writes. It does not weaken a release
proof, an authorization, a budget, a pause, or an approval check.
- An incident that a person resolved or dismissed after the lease
expired is not made again for the same authority. A renewed authority (a
later lease expiry) still makes a new incident.
- A superseded incident (status `cancelled`) still shows again when the
superseding action clears. This keeps the current behavior.
- The historical rows that the loop made are not removed. This change
only stops new rows.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21621, #21616, #21617

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:45:02 -06:00
DottaandClaude Opus 5.5 172c267338 fix(runner): count returned validation rejections and give a copyable call in repeated-failure corrections (#21666)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents change Paperclip records through the `call_api` tool, and the
native runtime runs each tool call through a repeated-identical-failure
guard (#21360, #21398)
> - Some model hosts drop arguments. A model then sends the same failing
call again and again, and each attempt costs a model turn
> - In chat-eval cohort #14 (OpenCode, `deepseek-v4.1-flash`), two cases
sent a body-less `PATCH /api/issues/{id}` 7 and 10 times with one input
hash
> - The guard counted these refusals and stopped running the call after
the second failure. But its correction showed only a `"body": {...}`
placeholder, so the model had no concrete call to copy. Also, the guard
did not count a route's own 400/422 when it came back as an `ok: false`
result
> - This pull request puts a copyable call in each correction and counts
returned 400/422 rejections. It never counts transient failures
> - The benefit is that a model gets the exact call to send on the
second failure, and the third identical validation failure does not run,
whether the rejection is thrown or returned

## Linked Issues or Issue Description

Refs #21360, #21398

**What happened?**

In chat-eval cohort #14, `questions-and-approvals.review-plan` (run
470002dc) and `tasks.private-task` (run bb76a832) sent `call_api PATCH
/api/issues/{id}` with no `body` 7 and 10 times, all with the same input
hash. The pre-dispatch check (`assertBodyPresent`) refused each call and
returned HTTP 400. The guard corrected the second failure and refused
the rest. But the correction said only `Resend with a top-level body
argument: {"operationId": ..., "pathParams": {...}, "body": {...}}`. The
model kept sending the identical call, and each refusal cost another
model turn: 2 min 18 s in review-plan and 24 s in private-task. Also,
when the route itself returns a 400/422 validation response, `call_api`
gives back `{ ok: false, status: 422, ... }` as a result and does not
throw. The guard did not count that, so an identical invalid request
reached the route every time.

**Expected behavior**

On the second identical deterministic failure, the agent gets a
correction with a call it can copy that passes validation. On the third,
Paperclip refuses to run the call and tells the agent to change the
arguments or to stop and report. Transient failures (5xx, timeouts, rate
limits, conflicts) always run again.

**Steps to reproduce**

1. Run a native-runtime agent on a host that drops the `body` argument.
2. Have it update a task with `call_api PATCH /api/issues/{id}`.
3. Look at the tool results. They contain only a `{...}` placeholder. A
route's 422 returned as `ok: false` is never counted.

**Paperclip version or commit**

master d4498ae7ac (cohort #14), reproduced against ecdeed5e7e.

## What Changed

- `runner-api-client.ts`: Add `runnerApiCallExample`. It builds a
copyable call from the operation's catalog schema: the path parameters
the model sent, plus a body with the required fields. If no field is
required, the body has one free-text field, for example
`{"operationId":"PATCH /api/issues/{id}","pathParams":{"id":"<sent
id>"},"body":{"title":"<title>"}}`. The "requires body" correction and
the "body must be a JSON object, not a JSON-encoded string" correction
now include this example in place of the `{...}` placeholder.
- `repeated-tool-failure-guard.ts`: Count a result with `ok: false` and
HTTP 400/422 as a deterministic failure, in the same way as a thrown
400/422. The first returned rejection passes through unchanged. The
second gets a `guidance` note that says the call will not run again. The
third and later identical calls are refused without running, and the
refusal quotes the response.
- Never count transient results: 5xx, 408, 429, 409, 404, and a missing
status (timeouts or unknown outcome).
- A route can reject a request because of the state it found. So the
guard forgets a returned rejection after a different call succeeds. A
refusal before sending depends only on the arguments, and the guard
keeps it for the run, as before.
- Tests: the guard tests cover returned 422 (correct, then refuse), a
changed call that still runs, returned 500/503/429/408/409/404/no-status
and thrown 429/409/timeout that keep running, and a returned rejection
forgotten after a state change. The runner API tests check that the
example in each recorded body-less correction is sent when copied as is,
that the issue-update example passes `updateIssueSchema`, and that every
catalog write with a required body (more than 200) has an example its
request schema accepts.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/services/native-runtime/runner-api.test.ts`: 2190 passed.
- Red/green: with the master guard, the 2 returned-rejection tests fail.
With the master client, 6 example tests fail. With this change, all
pass.
- `node scripts/typecheck-layers.mjs` (server): no errors.
- Per-call cost (median of 5 × 20,000 calls, local M-series): guard on a
successful call 2.13 µs before, 2.19 µs after. Transient-failure path
2.12 µs. Building an example takes 0.67 µs and happens only on a refused
call. Nothing is added before the turn starts. The guard is created with
the run's tool authority, as before.
- Paid narrow run: see the comment below.

## Risks

- Low. The change affects only calls that fail validation twice with
identical arguments.
- A route that returns 400/422 because of state can succeed later with
the same arguments. To cover that, the guard forgets a returned
rejection after any different call succeeds. A model that retries with
no call in between is refused after two attempts and must change the
arguments or report.
- The example uses `<field>` placeholders for free text and for ids that
the model must look up. If a model copies a placeholder verbatim, the
route stores that text. The correction says to send the fields the model
means to change.
- This does not make a model stop resending a refused call. It only
gives better guidance and keeps the refused call from running.

## Model Used

- Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use,
code execution, and extended reasoning).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 15:55:09 -06:00
Nicky LeachandClaude Fable 5.2 0ab73d4e30 feat(native-runtime): ask the Runner for provider-started turns on native ACPX Claude runs (#21456)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner can keep Claude while a background shell is running and
give the cycle of Claude to the controller as a turn that the provider
started. Core can wait for that turn. The transport can ask for it
(#21607), and a controller restart during it is safe (#21612).
> - No run does this yet, because Core does not ask the Runner for it.
> - This pull request makes the native runtime of Core ask, for ACPX
Claude. It is the only part that changes what a production run does.
> - The benefit is the goal of the whole work: a run with a background
command that is still running continues when the command finishes, and
does not fail with `native_finalization_missing`.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request turned the behavior on and had grown too large to
review. It is now three stacked pull requests:

1. #21607 (merged): the transport can ask the Runner for
provider-started turns. No caller asks.
2. #21612: the recovery after a controller restart during a wait or
during a provider-started turn.
3. **This pull request:** Core sets the option for native ACPX Claude
runs.

This is part 3. Its base branch is `fix/provider-turn-restart-recovery`.
Refs #19912 (design).

**What existing behavior does this improve?**

What a native ACPX Claude run does after a turn that ends with no result
while a background command is still running.

**Subsystem affected**

Core, the native runtime:
`server/src/services/native-runtime/native-session-executor.ts` and
`native-provider-turn-wait.ts`. ACPX Claude only.

**Current behavior**

- The run ends at the end of the turn. The Runner restarts Claude, the
background command stops, and the run fails with
`native_finalization_missing`.

**Proposed behavior**

- The native runtime sets `providerStartedTurns: true` in the transport
options for ACPX Claude, at the two places that build those options. One
function, `nativeProviderStartedTurns`, decides this and also decides if
the consumer gets the wait options, so the two cannot be different.
- A turn that ends with no result while a background command is still
running now waits, for a maximum of 10 minutes, for the turn that Claude
starts by itself. That turn is consumed as the next turn of the same
run, and its result settles the run.
- Other providers and other ACPX agents do the same as today.
- Limit of the Runner (see #21421): after a follow-up turn, the Runner
waits again only while a background command is still running. A command
that finishes while Claude reports on an earlier command gets no
follow-up.

**Reason and benefit**

- The opt-in is in the last part, so each earlier part can merge with no
change for a production run.

## What Changed

- `server/src/services/native-runtime/native-provider-turn-wait.ts` (5
lines added, 1 removed): `nativeProviderStartedTurns`.
- `server/src/services/native-runtime/native-session-executor.ts` (4
added, 2 removed): the option at the two places that build the transport
options, and the same function for the wait options.
- `server/src/services/native-runtime/native-session-executor.test.ts`
(4 added, 2 removed): the two native paths set the option for ACPX
Claude and not for other providers.

## State saved by the previous release

- No persisted state changes in this part.
- **A Runner that does not know the field** ignores it, and the run does
the same as today.
- **A deploy during a run:** a turn that started before the deploy has
no opt-in and ends as today. The next turn has the opt-in.
- **Rollback of Core:** the previous Core does not set the option, so
the Runner does not wait.

## Verification

The items below were run on the tree that was verified before the split
into three parts (commit `1b15ded1c4` has that tree, byte for byte). The
head of this branch (`23e2c13ad3`) is that commit on the new head of
#21612; its three server files are the same.

- `pnpm typecheck` and `node
scripts/verify-runner-vendor-dependencies.mjs` in `server`; `npx tsc
--noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`,
`pnpm run typecheck:typescript` and `pnpm run build:typescript` in
`packages/paperclip-runner`: all passed.
- `server/src/services/native-runtime/native-session-executor.test.ts`
and `native-runner-restart-recovery.integration.test.ts`: 1,732 passed,
21 skipped.
- The end-to-end behavior is tested in parts 1 and 2 with the real
`paperclip-runnerd`, where the test sets the option in the way that this
part does.
- Live run with a real Claude account, not in CI, at the adapter level
of the Runner (see #21316): the cycle of Claude was given as a turn with
its own usage, and each Paperclip tool call was admitted.

Not run: a live run through Core, `paperclip-runnerd` and the real
sidecar with a real Claude account (that path needs an API key, not a
local sign-in); a remote sandbox; the full suite. CI supplies the broad
coverage.

## Risks

- **This part turns the behavior on for each native ACPX Claude run.** A
run with no background command is not changed: the Runner waits only
when a shell is still running at the end of a turn that has no result.
- **This part must not merge before #21612.** Without #21612, a
controller restart during a wait or during a provider-started turn fails
the run.
- **A run can now stay open up to 10 minutes longer** when the
background command does not finish or Claude does not start a turn. The
agent computer stays active for that time.
- **A waiting run can show as "suspicious" after 5 minutes of silence**
in the run watchdog. The watchdog only labels a native run; it does not
stop it.
- **Follow-ups that the Runner cannot wait for are lost,** as the limit
above says.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 13:10:20 -08:00
DottaandClaude Opus 5.5 d4498ae7ac fix(runner): settle every recorded tool result and free sessions blocked by a closed turn's pending call (#21490)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call Paperclip tools through the Runner. The Runner records
each call and returns the controller's result to the provider (here
OpenCode)
> - In chat-eval cohort 13, a `list_workflows` result was recorded but
never reached OpenCode. OpenCode waited about 290 s, and the call stayed
pending
> - The next run's `run.attach` then failed with
`durable_pending_tool_calls`, so the person's follow-up question never
got an answer
> - This pull request fixes the result shape that caused the rejection,
makes the Runner settle every recorded result, and lets the next run
settle a call that a closed turn left pending
> - The benefit is that a tool result always reaches the agent, and a
stuck call can no longer block every later turn

## Linked Issues or Issue Description

**What happened?**
Chat-eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`). In
`cases.case-history` and `cases.create-case`, the CEO called
`list_documents` (or `list_projects`) and `list_workflows` (call id 5)
together. The server answered `GET /companies/{id}/workflows` with 200
in 584 ms. The OpenCode MCP bridge diagnostic for call 5 says `outcome:
"unanswered"`, `written: false`, `handlerMs: 289038` (290723 in
create-case). The sibling call was answered in about 1 s. The run ended
with `runner_settlement_incomplete`: `semanticTools.pending` held call 5
and its `semantic_tool.result` delivery command had status `failed`. The
next question started a run whose `run.attach` was refused: `run.attach
requires a settled Codex provider session: durable_pending_tool_calls`.
The run failed in 17 s, and the question was not answered.

Cause: `executeNativeWorkflowTool` returned the Runner API result
unchanged: `{ ok: true, status: 200, operationId: "GET
/api/companies/{companyId}/workflows", ... }`. Every other API-backed
tool renames that field to `apiOperationId`. The Rust tool bridge
(`semantic_response_value`) reads an object with a boolean `ok` and an
`operationId` as a semantic envelope that must name this tool and call.
The REST route did not match, so `apply_result` failed with `semantic
result envelope does not match its provider call`. Nothing was sent to
the provider, and the call stayed pending forever. The red test
reproduces this exact error. The parallel call is not the trigger: the
sibling call settled normally, and the OpenCode MCP bridge answered
other parallel pairs in the same runs. `list_workflows` first appears in
cohort 13, so cohorts 11 and 12 did not hit it. All four workflow tools
had this shape.

**Expected behavior**
Every recorded tool result reaches the provider. When a recorded result
cannot be delivered, the provider gets an error at once and the call
settles. A call that a closed turn left pending does not block the next
run.

**Steps to reproduce**
1. Run an OpenCode agent with Cases enabled on the durable Runner.
2. Ask a question that makes the agent call `list_workflows`.
3. See the call wait about 290 s. Then ask a follow-up question and see
`run.attach` fail with `durable_pending_tool_calls`.

**Paperclip version or commit**
`master` at `df1b12ad8e`.

## What Changed

- `server/src/services/native-runtime/native-workflow-tools.ts`:
workflow tool results name the route `apiOperationId`, like case tools
and `call_api`.
- `provider_bridge.rs`: add `settle_undeliverable_result`. When the
bridge rejects a recorded result for a call it still holds, the call
settles with a bounded, non-retryable runner error. Conflicting or
unknown results still fail as before.
- `provider_backend.rs` (`deliver_semantic_result`): use that
settlement, deliver the error to the provider, and record a
`semantic_tool_result_undeliverable` diagnostic with the call,
operation, reason and the rejected result's digest.
- `provider_bridge.rs` and `provider_backend.rs` (`attach_run`): add
`expire_closed_turn_calls`. When nothing is live (no active or ambiguous
turn, no provider request waiting), `run.attach` settles calls that a
closed turn left pending as an expired failure with an unknown outcome,
and records `semantic_tool_result_expired`. The provider stopped waiting
when its turn ended, and the new run's controller never delivers the old
run's results. A call that a live turn still waits on still blocks
attachment. This follows the existing `tool_call_interrupted_by_restart`
rule: never execute the operation again, and record an unknown outcome
when no result can ever arrive.
- Docs: `doc/run-log-events.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe both
diagnostics.

## Verification

- Red/green, Rust (`cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --test codex_provider --
a_rejected_recorded_result run_attach_expires`):
- Before:
`a_rejected_recorded_result_still_answers_the_provider_and_frees_the_session`
fails with `semantic result envelope does not match its provider call`
(the production error).
`run_attach_expires_a_call_left_pending_by_a_closed_turn` fails with
`run.attach requires a settled Codex provider session:
durable_pending_tool_calls` (the production refusal).
- After: both pass. The fake provider logs `tool-response:failure`, so
it got an answer at once. The next `run.attach` succeeds.
- Parallel calls (`cargo test -p paperclip-runner-core --lib --
provider_bridge`):
`a_rejected_result_in_a_parallel_pair_still_settles_its_own_call` sends
the cohort 13 pair (`list_documents` + `list_workflows`). The rejected
call settles, its sibling keeps its own result, and the next turn
starts. `only_a_closed_turn_expires_its_pending_calls` checks that an
open turn expires nothing.
- Upgrade and mid-run cutover:
`run_attach_expires_a_call_left_pending_by_a_closed_turn` saves state
with a pending call after the turn ends, stops that Runner, and attaches
with a new Runner process from the saved state.
`durable_backend_settles_pending_tools_when_recovery_finds_the_turn_ended`
(unchanged) still passes: a late result in the same run still settles as
`settled_after_turn`.
- Red/green, server (`cd server && npx vitest run
src/services/native-runtime/native-case-tools.integration.test.ts -t
workflows`): before, `expected { ok: true, status: 201, … } to not have
property "operationId"`; after, 2 passed.
- `cargo test -p paperclip-runner-core`: 643 lib tests pass. One lib
test (`fenced_receipt_output_keeps_prefix_when_stdout_never_closes`)
failed once with `EPERM` while signaling a process group under parallel
load. It passes alone 3 of 3 and does not touch this change. The
integration test binaries pass.
- Paid narrow run for `cases.case-history` and `cases.create-case`: see
the comment below.

## Risks

- Previous-release story: no persisted shape changes. State saved by the
previous Runner with a call pending after its turn closed carries
forward. The new Runner settles that call at the next `run.attach`, and
the session continues (tested from saved state with a new process). A
previous-release Runner that receives the corrected `apiOperationId`
result delivers it normally, because the result is no longer read as an
envelope.
- Behavior change: a call left by a closed turn is now settled at the
next run's attachment, not kept forever. Its outcome is recorded as
unknown, and the operation is never run again. A provider turn that is
still live keeps the old refusal (`durable_pending_tool_calls`,
`provider_side_effect_pending`), and the controller still never replaces
the session.
- A recorded result that the bridge rejects now becomes an error for the
agent, not an endless wait. The agent is told the action may have
completed and must check the current state before repeating it.
- Latency: nothing is added before turn start. The attach-time check
walks the in-memory pending calls (usually none) and adds no I/O beyond
the existing state save. The rejected-delivery path now answers in the
same command instead of after the provider's ~290 s client timeout, and
the follow-up run no longer fails after 17 s.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21398 (first release where `list_workflows` was called in this
eval), #19733 (the attach refusal), #20886 (earlier OpenCode delivery
fix).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:21 -06:00
DottaandClaude Opus 5.5 890dcbdd66 fix(native): wait for a live warm session owner during a deploy overlap (#21627)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap: the outgoing server still
serves while the incoming server starts new runs
> - A warm agent session stays open in the server process that ran the
last turn, after that run finished
> - The incoming server took over a warm session while the outgoing
server was alive and still held it, because the finished run's
finalization lease ends at commit
> - The takeover then failed, the task went blocked with a Retry, and in
one case a retry loop ran 27 times on the same sandbox
> - This pull request makes the incoming server wait while the server
that holds the warm session is alive, makes the outgoing server close
idle warm sessions as soon as it leaves new runs to the replacement, and
stops the retry loop when the promised fresh sandbox does not come
> - The benefit is that a follow-up during a deploy finishes without a
person, and no Runner ever gets two controllers

## Linked Issues or Issue Description

**What happened?**
A zero-downtime release-swap test (Gate B, wave B1, master aaec67b891)
found two failures with one cause.
Task 1: run A finished on the outgoing server at 15:13:56. Its warm
session stayed idle in that server. The incoming server claimed the
follow-up at 15:13:57. `inspectRemoteWarmSessionRecovery` asked
`evaluateNativeControllerTakeover`, which allowed the takeover with
`remote_controller_lease_released_or_expired`. That lease covers the
finished run, not the warm session. The outgoing server was alive until
15:18:18. `runner.prp.authenticate` failed after 62.3 s, the run failed
with `remote_warm_session_unproven`, the failure was saved as ambiguous
and owned by the board, and the task stayed blocked with a Retry.
Task 2: the same shape on a later swap started 27 automatic runs in 13
minutes. Fourteen of them said "The next attempt starts on a fresh
sandbox", but each one resumed the same sandbox and got the same refusal
(`journal_unsettled`).

**Expected behavior**
A follow-up claimed during an overlap waits for the warm session owner
to let go, then continues with no Retry banner and no second controller.
If the outgoing server dies with the session open, the takeover
continues after its ownership ends. A message that promises a fresh
sandbox is true, or the retries stop after a bounded number with a true
state.

**Steps to reproduce**
1. Run the restart-chaos release-swap preset with a warm (opencode or
codex) staging mix.
2. Let a run finish on the outgoing server after the traffic switch,
with a follow-up comment already queued.
3. See the follow-up fail with `remote_warm_session_unproven` and the
task go blocked.

**Paperclip version or commit**
master aaec67b891 and 45531b719e.

**Deployment mode**
Hosted deploy overlap (release swap) with Daytona sandboxes.

## What Changed

- `remote-warm-session-recovery.ts`: before live adoption, the run
proves that the server process that ran the finished turn is gone. It
uses the shared owner proof (`proveOwnerGone`): a PID check in the same
process space, or the `controller_instances` lease in another process
space. While that server is alive, the run defers with the existing
`native_controller_recovery_waiting_for_owner` wait and checks again
after 5 s. The wait ends when the server exits or is fenced, or when the
finished turn's warm window (idle timeout plus two minutes) ends.
- `native-session-executor.ts` and `run-start-handover.ts`: a server
that leaves new run starts to a replacement closes its idle warm
sessions after 1 s, through the existing idle-timeout close. A session
that becomes idle after that closes after 1 s too. This is the same
close that shutdown already does, done earlier.
- `predecessor-retirement-fallback.ts`: when a retry meets an unretired
owner on the same sandbox that its source already asked to replace, it
does not ask again. The run fails without another automatic retry, and
the task shows a Retry.
- `predecessor-retirement.ts`: the error no longer says "The next
attempt starts on a fresh sandbox", because the replacement is only a
request.
- `doc/execution-semantics.md`: documents the owner wait, the hand-off
at the swap, and the retry stop.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts`
- New: "waits while the outgoing server that ran the finished turn still
holds its warm session, then takes it over once that server exits" (red
on master: master returned a takeover while the owner was alive).
- New: "waits for a live outgoing server in another process space until
its controller lease ends" (red on master).
- New: "stops waiting for a live outgoing server once the finished
turn's warm window has closed" (bound check).
- `cd server && npx vitest run
src/__tests__/heartbeat-native-predispatch-retry.test.ts`
- New: "stops retrying with a truthful failure when the next attempt met
the same unretired sandbox instead of a fresh one" (red on master:
master scheduled another retry on the same sandbox).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"replacement server starts"`
- New: "closes a warm session promptly when a replacement server starts
this server's runs (before/after the turn ends)" (both red on master).
- Focused suites: native-session-executor, remote-warm-session-recovery,
native-controller-recovery, native-recovered-owner-followup,
native-runner-restart-recovery, heartbeat-native-predispatch-retry,
environment-runtime-reusable-resume-replacement,
task-interruption-presentation, sandbox-runner-port-holders.
- `npx tsc --noEmit -p server/tsconfig.json` passes. `pnpm verify` exits
2 (a doc is a global CI input), so required CI supplies broad coverage.
- Not run here: the restart-chaos release-swap wave. The maintainer runs
it after merge.

## Risks

- A follow-up can now wait up to the outgoing server's remaining overlap
when that server cannot close its idle session. Before, it failed and
needed a Retry. The wait is bounded by the owner's exit or the warm
window.
- The outgoing server closes idle warm sessions earlier during an
overlap. The next turn then resumes from a suspended Runner instead of a
live one, as it already does after a shutdown.
- A retry chain that meets the same unretired sandbox twice now stops
one attempt earlier than the shared cap. The task shows a Retry instead
of looping.
- Rows without a recorded process space keep the existing one-release
local proof, so they behave as before outside the same host.
- Previous-release state: this change adds no persisted field.
Finalization rows saved by the previous release already record the
controller boot, PID, process space and start time, so they carry
forward. A follow-up already deferred by the previous release resumes
through the same wait code.
- Runtime cost: the owner proof adds one `ps` start-time read (same
process space) or one `controller_instances` row read (other process
space) per inspection. It runs once per attempt, at most every 5 s while
a live owner holds the session.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:59:17 -06:00
DottaandClaude Opus 5.5 aaec67b891 fix(native): close idle warm sessions in parallel across agents at shutdown (#21597)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - At shutdown, the server closes its idle warm Runner sessions before
it prepares the hot-restart snapshot
> - It closed them one at a time, and each full close (checkpoint, then
two stopped-copy collections) takes 3-5 s
> - In a zero-downtime swap test, five closes used the whole 20 s
pre-drain budget. Hot-restart preparation timed out, a checkpoint was
cut mid-way, and shutdown logged "hot-restart shutdown preparation
failed"
> - The same test logged 34 "Delivery reconciliation failed" errors, all
from directory cleanup retries that found the agent directory lock busy
> - This pull request closes idle warm sessions in parallel across
agents, one at a time within an agent, and treats a busy directory lock
as a quiet retry
> - The benefit is that shutdown preparation fits its budget with
several warm sessions, and expected lock contention no longer looks like
a failure

## Linked Issues or Issue Description

**What happened?**
`closeIdleWarmNativeSessions` awaited each warm session's full close in
turn. In swap 2 of a release-swap test, five idle sessions closed at
22:28:42, :46, :51, :56 and :59 (one "transport_close_requested"
diagnostic each). The `hot_restart_prepare` step then timed out at 19.9
s. Separately, `recoverDirectoryRelease` threw
`AgentDirectoryOperationBusyError` when another close held the agent
directory lock. The delivery coordinator logged each one at ERROR level.

I first checked if these sessions should have taken the cheap
released-session detach. They could not: `entry.released` is set only
after a successful `session.release()` through a release/resume-capable
plugin Runner task. The Runner state files from all sandboxes in the
test recorded only `run.prepare`, `session.open`, `turn.start`,
`runner.drain` and `runner.suspend` commands, and no `runner.release`.
The Daytona daemon logs show only `/process/execute` and
`/process/session` calls, and no Runner-task API. So no session in that
test was ever released, and a full close was the only correct shutdown
for them.

**Expected behavior**
Idle warm sessions of different agents close at the same time. Two
sessions of the same agent never close together, because they collide on
the agent directory advisory lock. A cleanup retry that finds that lock
busy stays pending until its claimed retry time and logs no error.

**Steps to reproduce**
1. Keep five or more idle warm sessions on remote sandboxes.
2. Send SIGTERM.
3. The closes run one after another and `hot_restart_prepare` times out.

**Paperclip version or commit**
`master` at bbed32450c.

## What Changed

- `closeIdleWarmNativeSessions` groups sessions by company and agent. Up
to four agents close at once, and each agent's sessions close one at a
time. The per-session close logic is unchanged.
- Idle-sleep retirement (`prepareIdleWarmNativeSessionsForSleep`) keeps
a concurrency of one, so it still checks drain authority before each
retirement.
- `recoverDirectoryRelease` catches `AgentDirectoryOperationBusyError`,
keeps the deferred retry time, and returns the pending entry instead of
throwing.

## Verification

- Red, then green (each failed against `master` source and passes with
the fix):
- `closes idle warm sessions of different agents in parallel and one at
a time within an agent` in
`server/src/services/native-runtime/native-session-executor.test.ts`.
Before the fix, the second agent's close was never started while the
first agent's close was pending.
- `keeps a cleanup pending without an error while another server holds
the agent directory lock` in
`server/src/__tests__/agent-directory-working-copies.test.ts`. It holds
the real Postgres advisory lock from a second connection. Before the
fix, the coordinator received an `AggregateError` with
`AgentDirectoryOperationBusyError`.
- `native-session-executor.test.ts` (1502 passed) and
`agent-directory-working-copies.test.ts` pass in full.
- `pnpm typecheck` in `server` passes.

## Risks

- Up to four warm closes now run at once, so shutdown makes more Runner
and database calls at the same time. The cap keeps this small.
- The order of entries in `failures` can change when several closes
fail.
- A busy release now waits for its deferred retry silently. A lock that
stays busy for a long time shows as a stuck pending cleanup, not as
repeated errors.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 04:15:19 -06:00
DottaandClaude Opus 5.5 009c8884a4 perf(attention): read blocker retention state once per page (#21595)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `attention-retention` queue archives old attention items. One
kind is `blocker_attention`: a blocked task whose terminal blocker needs
a person.
> - When the queue loads a page of blocked tasks, it reads `keep` and
`archived_at` from `decision_retention` for each of them, one full-row
`select` per task.
> - A company hint walks every page. On a hosted instance with 2,112
blocker attention items and one active run, these per-task reads ran at
about 16 per second.
> - This pull request reads the retention state for the whole page in
one narrow statement. It also scopes three remaining run-path attention
hints to the task graph.
> - The benefit is one retention read per page instead of one per
blocked task, with the same decisions.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The `blocker_attention` eligibility read in
`server/src/services/attention-retention.ts` (`eligible()`), and three
run-path producers of company-wide attention hints.

**Subsystem affected**

Server background retention (`attention-retention.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-review-participant.ts`,
`native-runtime/native-workspace-reconstruction.ts`).

**Current behavior**

- For each blocked task whose blocker attention is terminal (and for the
owner variant), `eligible()` ran `select <all columns> from
decision_retention where company_id = $1 and source_kind = $2 and
source_id = $3`. It did this during `load`, once per task, on every
company hint.
- The hosted statement is identified by that column prefix: `id,
company_id, source_kind, source_id, source_activity_at, …`.
- I checked every other full-row reader of `decision_retention`. None
fires per run event:
- `syncItems`: attention summary and list requests, which a board calls
once per load; retention `check` and the sweep, which run for due
entries only; Keep and Archive actions.
  - `getState`: user actions only.
  - `archiveReviewedManifest`: proposal acceptance only.
  - `decisions.list`: only when archive proposals exist.
- `native-provider-capacity-retry`, `native-review-participant` and
`native-workspace-reconstruction` sent `{ companyId }` attention hints,
which make the queue walk every page.

**Proposed behavior**

- `pageEvidence` reads `source_id`, `keep` and `archived_at` for every
blocker root and every terminal blocker on the page in one statement
(ids passed as one JSON parameter). `eligible()` uses that map when it
has page evidence.
- A missing row means "not kept, not archived", exactly as the per-task
read returning no row did.
- The per-entry `check` path still reads one row, now only `keep` and
`archived_at`.
- The three producers send `notifyIssueAttentionRetentionWork` with
their task (and run). That is the task, its parents and the tasks it
blocks, with a company fallback past 100.
  - A capacity retry changes the failed-run items of its task.
  - A review participant claim changes the review of its task.
- A workspace reconstruction resolves the recovery action of its task,
which can also change the blocker attention of the tasks it blocks.
- The configuration, lifecycle, import and fallback producers keep their
company hints.

**Reason and benefit**

Background work must not grow with the backlog. A page now costs one
retention read instead of up to 100.

## What Changed

- `server/src/services/attention-retention.ts`: batched and narrowed
blocker retention read; `PageEvidence.blockerRetention`.
-
`server/src/services/native-runtime/native-provider-capacity-retry.ts`,
`native-review-participant.ts`, `native-workspace-reconstruction.ts`:
scoped attention hints.
- Tests in `server/src/__tests__/attention-retention-work.test.ts` and
`server/src/services/native-runtime/native-workspace-reconstruction.test.ts`.

## Verification

- `reads blocker retention state once per page, not once per blocked
task` (I saw it fail first). 160 blocked tasks, each behind its own
unassigned blocker; the blockers' retention rows are kept (40), archived
(40), plain (40) or missing (40). The full-size response capture and the
statement log measure the load of both pages:
- before: one `decision_retention` read per task (160), 26,240 result
bytes;
- after: 2 reads (one per page), 5,480 bytes, none of them selecting
`source_activity_at`.
  - The same tasks are listed: only plain and missing.
  - `check` still drops the kept and archived ones.
- `scopes the attention retention hint of a reconstruction to the task
graph` (I saw it fail first): before, the reconstruction sent `{
companyId }`; after, it sends a hint that names the task.
- `batches 100 blocker graphs with exact terminal timestamps and fresh
terminal visibility` pinned 200 retention reads per page (one per root
and variant). It now expects 1; the rest of that test is unchanged and
passes.
- Not separately tested: the capacity-retry and review-participant hint
changes. Each is one line that passes the task the function already
holds, the same pattern as the reconstruction test.
- `attention-retention-work` (58), `attention-retention-candidates`,
`native-workspace-reconstruction`, `native-review-participant` and
`heartbeat-run-dispatch-work` pass, each file run alone (in `server/`,
against dedicated local `_test` databases).
- `pnpm check:module-boundaries` passes.

## Risks

- The page read happens once, before the per-entry decisions of that
page. Before, each decision read its row just in time. A Keep or Archive
that lands between the page read and an entry's decision is seen by the
next load or by `check`, which re-reads under a row lock, as before.
- A run-path hint that changes more than its task graph would now miss
the wider effect. The 90-day archive delay makes a missed reload cheap.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:05:59 -06:00
DottaandClaude Opus 5.5 c3f9a0c040 fix(retention): evaluate native retention blockers for many sessions in one statement (#21588)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep runner checkpoints and harness backups so that a
session can resume. Retention expires them after the session can no
longer resume.
> - A retention blocker decides whether a session can still resume: a
live or recent run, an open or recently closed task, an unsettled
finalization, or a live lease.
> - The startup checkpoint pass evaluated that blocker in its own
transaction, once for each retained checkpoint, with three reads. A
session that stays blocked (for example, a long-lived open task) was
evaluated again for each of its runs on every server start. On a hosted
instance with frequent deploys and a 40-minute overlap of two servers,
this pass ran almost continuously at about 7 checkpoints/s, about 21
statements/s.
> - Backup reconciliation also used four reads and a transaction for
each run hint that reached a backup.
> - This pull request evaluates the blocker for many sessions in one
fixed-text statement. Only candidates that are not blocked enter the
existing locked transaction.
> - The benefit: a restart costs 2 statements for each page of 200
blocked checkpoints instead of 5 statements and 1 transaction for each
checkpoint, and a backup hint costs 2 statements with no transaction.

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened?**
A hosted instance had only a handful of live agent runs, but its server
main thread was about 95% busy. An analysis of its most-called SQL found
three retention-blocker reads (`company_id, agent_id from
heartbeat_runs`; the session's runs; the session's issues) at exactly
7.0/s each, each in its own transaction. The startup checkpoint pass
sent them once for each retained checkpoint older than 7 days whose
session was still blocked, and it repeated the full pass after every
start.

**Expected behavior**
A checkpoint or backup that is blocked from expiring costs no
transaction and no statements of its own, because one statement
evaluates its session together with the rest of its page. It expires as
soon as its blocker clears: checkpoints on the next startup pass (the
current design), and backups at their exact deadline.

**Steps to reproduce**
1. Create 100 native sessions with 3 finished runs each (finished 10
days ago), settled finalizations, a checkpoint header for each run, and
an open task.
2. Register the startup checkpoint expiration and count statements. Then
register it again, as a restart.
3. On master each start costs 1,502 statements and 300 transactions, and
nothing expires.

**Paperclip version or commit**
`origin/master` at 3c3e6c2d75.

## What Changed

- `SESSION_RETENTION_BLOCKERS_SQL` (new module
`native-runtime/native-session-retention-blockers.ts`) is one fixed-text
statement, sent through `$client.unsafe`. It takes a JSON list of
sessions and returns, for each one, the same blocker reasons as
`sessionRetentionBlocker`, in the same order: missing run, live run,
incomplete native recovery, recent run (checkpoint mode), unknown, open,
executing or recently closed task, pending finalization, live lease. It
also returns the session's resume pointer, run, task and lease
dependencies, and the latest close time.
- `expireCheckpoints` and `expireDurableBackups` evaluate each page of
candidates with this statement. They skip blocked candidates and
candidates whose session still has a resume pointer. A candidate that is
not blocked still enters the existing transaction. That transaction
checks the blocker again under row locks before it retires anything.
- `reconcileBackup` uses the statement for its dependencies and its
deadline. It replaces three dependency reads plus a blocker transaction.
The lease references use one index-backed arm for each reference kind
(run, backup stamp, task, identity lease).
- The unused `lock: false` path of `sessionRetentionBlocker` and the
`backupDependencies` helper are removed.
- The existing index test now checks the plan of the new statement.
Every arm that reads `environment_leases` uses an index in both the
custom and the generic plan.

## Verification

- New suite
`server/src/__tests__/native-retention-blocker-batch.test.ts` (embedded
PostgreSQL, real coordinator registration, statement meter):
- 300 blocked checkpoints: a start costs 4 statements and 0
transactions, and a restart costs the same. On master it is 1,502
statements and 300 transactions for each start. The checkpoints are
kept.
- When a session's task is closed long enough ago, its checkpoints
expire on the next startup pass. The other sessions are not touched.
- A session with a resume pointer keeps its checkpoints for 2 statements
in total.
- Two servers that run the startup pass at the same time expire every
checkpoint exactly once with no failures, because the locked transaction
checks again and the retirement header is compared before deletion.
- 50 run hints that reach 50 blocked backups: at most 2 statements for
each hint (the scope lookup and one blocker statement) and no
transaction. On master it is 9 statements and 1 transaction for each
hint. The backups are kept.
- Existing suites, one at a time: `native-harness-backup-expiration`
(109/109, including the backup expiry flows for task close, reopen and
lease release), `native-harness-backup-expiration-config` (7/7),
`native-harness-backup-storage.integration` (73/73),
`native-harness-backup-publication` (4/4),
`native-retired-backup-source.integration` (38/38).
- `pnpm check:module-boundaries` passes.

## Risks

- The startup pass still runs on every server start. It also runs on
both servers during the deploy overlap. Its new cost is 2 statements for
each page of 200 candidates (the candidate page and one blocker
statement). A page of candidates that are not blocked still uses one
locked transaction for each candidate, as before.
- Two servers can run the pass at the same time. They cannot retire the
same session twice or retire a blocked one: the unlocked statement only
filters, and the existing transaction locks the finalizations, checks
the blocker again, and compares the retirement header before it deletes.
The concurrency test covers this.
- If the SQL blocker and the TypeScript blocker ever disagree in the
"not blocked" direction, the locked transaction still refuses. If they
disagree in the "blocked" direction, retention is only delayed. When
more than one task blocks a session, the reported reason can differ from
the TypeScript loop, which reads the tasks in no fixed order.
- Checkpoints keep their current design: they are considered again on
the next startup, not at a deadline. Backups keep their exact deadline
after a task closes.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:47:01 -06:00
DottaandClaude Opus 5.5 f452b449fd fix(recovery): stop re-checking settled native runs that cannot be released (#21590)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server releases the process identity (`process_pid`,
`process_group_id`, `process_started_at`) of a finished native run when
it can prove that the runner process stopped (#17374)
> - A background queue (`settledNativeProcesses`) finds these runs and
checks them. A check that cannot finish ("skipped" or "pending") writes
nothing, so the run stays in the queue and every recovery hint checks it
again
> - A finished run that a replaced container recorded can never pass the
check, because only the host that recorded the process can prove that it
stopped. Each deploy adds more of these runs
> - On one hosted instance this loop ran about 10 checks per second and
about 55 SQL statements per second, and the server main thread was about
95% busy with only a few live runs
> - This pull request moves each "cannot make progress" reason into the
queue's SQL predicate, adds a growing backoff for "pending" runs, and
makes one check cost one locking statement
> - The benefit is that runs which cannot progress cost nothing until
something that can change the answer happens, and a run that becomes
releasable is still released at once

## Linked Issues or Issue Description

No public issue exists. Bug report:

**What happened?**
The settled native process recovery queue checks the same finished
native runs again on every recovery hint and every 30 seconds, forever.
With 300 such runs and one company-wide hint per second it ran 9 checks
per second and 84 SQL statements per second (9.4 statements per check,
2.2 ms of server CPU per check) on embedded PostgreSQL. A hosted
instance showed the same rates. The runs that never finish are: runs
recorded by a host that is gone (each redeploy starts a new container
with a new hostname), runs whose coordinator phase is not settled, runs
with no checkpoint and no provider evidence, and runs whose local
process is still alive (they defer 30 seconds with no limit). Startup
discovery (`discoverSettledNativeRunProcesses`) also opens one
transaction for every such run on every boot.

**Expected behavior**
A run that cannot make progress is not read again until something that
can change the answer happens. A run that becomes releasable (its phase
settles, provider evidence arrives, its own host checks it, its sandbox
receipt arrives, its process exits) is released promptly.

**Steps to reproduce**
1. Create a failed native run with a recorded local process identity
from another host (or an unsettled coordinator phase, or no checkpoint
and no provider evidence, or a live process).
2. Register the residual recovery work and send company-wide
`notifyLiveRunRecovery` hints.
3. Count the run's `for update` checks: on `master` every hint checks
the run again.

`server/src/__tests__/settled-native-process-rechecks.test.ts` automates
these steps.

**Paperclip version or commit**
`master` at 3c3e6c2d75.

**Deployment mode**
Any. The cost grows with every deploy of a hosted instance.

## What Changed

- `server/src/services/native-local-process-stop.ts`
- New `settledNativeProcessCandidates`: one fixed-text statement that
finds releasable runs. It reads only runs recorded in this server's PID
space (`process_host_hash`), sandbox runs (`remote`, with a non-local
lease), and rows saved before the hash (their latest identity event must
name this host). It excludes runs with an unsettled coordinator phase
and runs with no checkpoint and no provider evidence. Each PID-space
branch reads its own range of the new partial index.
- `releaseSettledNativeRunProcess` now reads the run, phase, evidence,
lease facts and identity match in one `for update` statement (it was 6
statements). It reads the lease rows again only for a sandbox run. The
new `inspectSettledNativeRunProcess` also returns whether a remote stop
acknowledgement can apply.
- Startup discovery uses the same candidate statement, so it opens no
transaction for an excluded run.
- `server/src/services/heartbeat/live-recovery-work.ts` (settled kind
only)
- The load runs the candidate statement and a separate indexed statement
for cancelled `cursor_cloud_api` runs whose Stop is still `requested`.
- The check uses the loaded entry. It no longer runs the load again for
each entry.
- A "pending" run backs off 30 s, 60 s, 120 s, up to 15 min. A
run-scoped hint (identity event, lease cleanup, stop receipt) resets the
backoff.
- A "skipped" run that the predicate cannot exclude (for example a birth
time mismatch on this host) waits for a run-scoped hint or a restart.
- `server/src/services/heartbeat/recovery.ts` (the settled handler
only): returns the outcome. It calls `acknowledgeRemoteStop` only when
the run has a non-local lease or is a cancelled `cursor_cloud_api` run.
For any other run that function reads two rows and writes nothing.
- `packages/db`: migration 0487 builds
`heartbeat_runs_settled_process_identity_idx` and
`heartbeat_runs_pending_api_stop_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. The migration is `online_compatible` and is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. `doc/DATABASE.md` describes both
indexes.

End state for each reason that a check cannot finish:

| Reason | Before | After | What re-admits the run |
| --- | --- | --- | --- |
| Recorded by another PID space | checked on every hint | not read |
only a server in that PID space reads it (a live older server in a
deploy overlap still releases its own runs) |
| Coordinator phase not settled | checked on every hint | not read | the
phase write; the finalizer sends a company hint in the same transaction
|
| No checkpoint and no provider evidence | checked on every hint | not
read | the next hint after a provider or turn event |
| Live local process, unreleased lease, missing sandbox receipt | every
hint and every 30 s | backoff 30 s to 15 min | a run-scoped hint resets
the backoff |
| Other same-host skip | checked on every hint | parked | a run-scoped
hint (an identity event sends one) or a restart |

## Verification

- `pnpm exec vitest run
src/__tests__/settled-native-process-rechecks.test.ts` (new, 12 tests,
real PostgreSQL, real queue and coordinator, fake clock):
- each exclusion: replaced container, unsettled phase, no evidence: 0
checks over 4 hints (red on `master`: 3 to 5 checks);
- a live older server and a stopped older server (different hash): the
new server never reads the run and writes no receipt; this server's own
run is released at start;
- re-admission: phase settles, late provider evidence, legacy rows with
no hash (this host released, another host not read), sandbox run
released from any server with its receipt;
- live process: at most 4 checks over 8 hints (red: 17), released on the
run-scoped hint after the process exits;
- one check is BEGIN, one `for update` statement and COMMIT (3
statements); the next hints do not check it; a run-scoped hint checks it
once more (red: 5 statements and a check on every hint);
- a restart costs one check for each pending run; startup discovery with
50 excluded runs sends 1 statement (red: 302).
  - On `master` 11 of the 12 tests fail.
- Measurement on embedded PostgreSQL, 300 stuck runs, one company hint
per second for 300 s (fake clock), same harness before and after:
- before: 9.0 checks/s, 84.3 statements/s, 9.4 statements per check, 5.9
s CPU (2.2 ms per check);
- after: 0 checks/s, 3.2 statements/s for all seven residual queues
together, 0.16 s CPU.
- Load plan with 12,000 excluded finished runs (10% without a hash) in a
31,000-row `heartbeat_runs`: the candidate statement reads only index
entries for this server's hash, `remote` and `unknown`, and a bitmap
scan of the 1,200 legacy rows: 2.7 ms, 4,416 buffers. Without the index
the same exclusion read all 31,000 rows (31,127 buffers).
- Existing suites, one file at a time: all pass.
`live-recovery-work.test.ts` 51/51 (an earlier run under machine load
hit the `afterAll` hook timeout after all tests passed; the rerun passed
clean, and `native-safe-replacement.test.ts` hits the same hook timeout
on the base commit), `native-settled-process-release.test.ts` 31/31,
`native-local-process-stop.test.ts` 5/5, `heartbeat/recovery.test.ts`
107/107, `native-runner-restart-recovery.integration.test.ts` 231 passed
/ 14 skipped, `native-restart-recovery.test.ts` 49/49,
`native-controller-recovery.integration.test.ts` 30/30,
`retained-native-session-cleanup-work.test.ts` 29/29,
`native-provider-failure.integration.test.ts` 16/16,
`native-safe-replacement.test.ts` 70/70 (with the `afterAll` hook
timeout that also occurs on the base commit),
`native-remote-retry-admission.test.ts` 93/93.
- `pnpm check:module-boundaries`, `pnpm check:migrations` (packages/db),
server typecheck: pass.
- Not run: the full suite and browser suites.

## Risks

- Unresolved by design: a finished run whose server was replaced keeps
its process fields indefinitely and is never released, because only the
host that recorded a process can prove that it stopped, and the shared
prover refuses a controller-lease proof for a run with a process group.
This pull request only stops paying for it. These runs keep no stop
receipt (`native.local_process_stopped`), so what waits for that receipt
still waits: accounting close of the run's usage source, retained native
session cleanup, and instruction or directory working copies that wait
for proof that the run stopped. A follow-up must decide the ownership
rule for a replaced container.
- Rows saved before `process_host_hash` exist are read through the
legacy branch, which checks the latest identity event for each of them
on each load. On an instance with many such rows the load costs one
index lookup per row. The count can be measured with `select count(*)
from heartbeat_runs where process_host_hash is null and process_pid is
not null and status in
('failed','cancelled','timed_out','interrupted')`.
- A run that this server recorded but whose `process_host_hash` changed
(a new PID namespace on the same host, for example a container restart)
is no longer read here. Before, it was read and skipped, so no release
is lost.
- The backoff and parked state live in memory. A restart checks each
pending run once.
- The migration builds two partial indexes concurrently on
`heartbeat_runs`. It holds no long lock.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:29:23 -06:00
DottaandClaude Opus 5.5 e8c79235ca perf(attention): scope run lifecycle retention hints and read only used columns (#21579)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `attention-retention` queue archives old attention items (failed
runs, reviews, questions, recovery actions, blocked tasks). A company
hint makes it reload every candidate of the company, 100 per page, with
eligibility reads for each page.
> - Several run lifecycle producers sent that company hint: every agent
status change after a run, the execution-control writes and legacy run
terminalization when they name only a run, scheduled retries, and the
Runner's task monitor tool.
> - Each page also read whole rows it did not use: full `heartbeat_runs`
rows (recent ones carry tens of KB of `context_snapshot` and
`runner_profile_json`), full `issues` rows (descriptions), and the whole
agent roster with its configuration.
> - This pull request names the task graph or the source kind in those
hints, and reads only the columns each eligibility decision uses.
> - The benefit is fewer company reloads per run event and far fewer
result bytes per reconstruction, with the same decisions.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`. Stacked on #21552 (it builds
on that pull request's change to `execution-control-notifications.ts`);
review that one first.

**What existing behavior does this improve?**

The attention retention queue
(`server/src/services/attention-retention.ts`) and the producers that
wake it on run lifecycle paths.

**Subsystem affected**

Server background retention (`attention-retention.ts`,
`retention-work-notifications.ts`, `execution-control-notifications.ts`,
`heartbeat/agent-status.ts`, `heartbeat/retries.ts`,
`native-runtime/paperclip-runner-tool-authority.ts`).

**Current behavior**

- `heartbeat/agent-status.ts` sends `{ companyId }` on every agent
status change after a run (two places). `heartbeat/retries.ts` sends it
for each scheduled retry. `writeExecutionControl` and
`notifyExecutionRunWork` send it when the scope names only a run. The
Runner's `set_task_monitor` tool sends it for one task.
- On a local fixture with 1,500 open tasks and run churn, the queue
loaded 47 company candidate rows per second from these hints.
- `pageEvidence` read `select *` from `heartbeat_runs`,
`issue_recovery_actions` and (through `getTableColumns`) `issues`;
`reviewEligibility` read `select *` from `issues`;
`interactionEligibility` read the full agent roster; the per-entry
`check` repeated full-row reads.

**Proposed behavior**

- Agent status changes send `{ companyId, sourceKind:
"agent_error_alert" }`. The status moves between `running`, `idle` and
`error`. All three are invokable for blocker attention
(`BLOCKER_ATTENTION_INVOKABLE_AGENT_STATUSES`) and for questions
(`agent-eligibility.ts`). Failed-run attention only excludes terminated
agents, and this path never sets that. So only agent error alerts can
change.
- Run-scoped execution-control writes, legacy run terminalization and
scheduled retries use `notifyIssueAttentionRetentionWork` with the run
id (the run's task, its parents and the tasks it blocks; company
fallback past 100). This is the same helper as #21521. The task monitor
tool scopes to its task.
- `notifyIssueAttentionRetentionWork` matched the run's saved task id
with `issue.id::text = …`. It now validates and casts the id, so the
primary key is used, as #21552 does for recovery.
- Narrow reads, same decisions:
- runs: `id`, `agent_id`, `created_at` and the saved task id computed in
SQL with the same rules as `readRunIssueId` (`issueId` unless it is
absent or JSON null, then `taskId`; only non-empty strings).
  - recovery actions: `id`, `source_issue_id`.
- issues for evidence and `visibleIssue`: the blocker-attention input
columns plus `unblock_descriptor` and `blocked_transition_at`.
- review issues: `id`, `company_id`, `status`, `execution_state`,
`assignee_user_id`.
- agent roster for question addressees: `id`, `company_id`, `name`,
`reports_to`, `status` (the `AgentOrgRow` that
`evaluateAgentInvokability` takes).

**Reason and benefit**

A run event no longer reloads every attention candidate of the company.
A reconstruction no longer transfers whole run, task and agent rows.

## What Changed

- `server/src/services/attention-retention.ts`: column lists for runs,
recovery actions, issues and the roster; the run probe type replaces
whole rows.
- `server/src/services/heartbeat/agent-status.ts`,
`heartbeat/retries.ts`, `execution-control-notifications.ts`,
`native-runtime/paperclip-runner-tool-authority.ts`: scoped attention
hints.
- `server/src/services/retention-work-notifications.ts`: indexed lookup
of a run's saved task.
- `server/src/__tests__/attention-retention-work.test.ts`: two tests.

## Verification

- `reads only the fields its decisions use from large runs, reviews and
agents` (I saw it fail first): 50 failed runs, 50 tasks in review with
large descriptions, 10 questions to a paused agent, about 40 KB of
payload in each run's snapshot, profile and stdout and in the agents'
configuration. It loads the three kinds with the full-size response
capture (`full-database-response-capture.ts`). Before: `heartbeat_runs`
6,317,700 bytes, `agents` 84,336 bytes, `issues` 4,235,810 bytes. After:
under the test bounds of 20,000, 5,000 and 60,000 bytes. The decisions
are the same before and after: 50 failed runs, 40 reviews and 10
questions.
- `scopes run lifecycle attention hints to the run's task graph and
agent status to agent alerts` (I saw it fail first):
`writeExecutionControl` and `notifyExecutionRunWork` with a run id send
the hint for the task and its parent. `finalizeAgentStatus` sends the
agent-alert hint. Before, all three sent `{ companyId }`.
- `attention-retention-work` (57 tests),
`attention-retention-candidates`, `attention-service`,
`execution-control-work` and `heartbeat-agent-status-recovery` pass,
each file run alone (in `server/`, against a dedicated local `_test`
database).
- `pnpm check:module-boundaries` passes.

Not done in this pull request, and why:

- I did not reproduce `syncItems` re-syncing unchanged rows from the
retention `check`. On my fixture (300 stalled blocked tasks, 4,500 old
recovery-action and blocker retention rows, run churn), `syncItems` ran
0 times in a 180-second window and `decision_retention` reads were 0 per
second. `check` calls `syncItems` only for entries that are already due
(older than the 90-day archive delay), and a due entry is archived and
leaves the candidates. So I did not batch `syncItems` or skip checks of
unchanged rows.
- `readIssues` still selects the `issues` columns that blocker attention
needs. I did not narrow `issueService.listBlockerAttentionBatch`.

## Risks

- A producer that changes more than the run's task graph would now miss
the wider effect. Company-level changes (agent configuration, company
status, budgets) keep their company hints. The 90-day archive delay
makes a missed reload cheap: the item is reloaded on the next hint that
covers it.
- The saved task id now comes from SQL. The SQL mirrors `readRunIssueId`
for strings, absent keys and JSON null. A non-string `issueId` value now
yields no task, as before.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:17:21 -06:00
DottaandClaude Opus 5.5 340157d1d6 perf(recovery): scope run lifecycle recovery hints to the run's task graph (#21552)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues in the background. Two of
them, `stranded-assignments` and `assigned-work-invariant`, find
assigned tasks that have no live run and restart them.
> - Every run dispatch and every run state change sent a recovery hint
for the whole company. On each hint these queues reload every eligible
open task of the company and check each one again, at most every 30
seconds per queue.
> - With live runs the hint comes all the time, so the queues never
stop. Their cost is (run activity) x (open-task backlog). On a hosted
instance with about 1,400 open tasks and 5 live runs, the server main
thread stayed at 87-92% of a core.
> - A run event can only change the tasks of its own task graph. This
pull request makes run and task producers name that graph, and keeps one
slow full pass as a backstop.
> - The benefit is that a run event costs a few checks, whatever the
size of the backlog, and a missed hint still delays recovery by at most
five minutes.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The issue recovery queues in `server/src/services/recovery/work.ts`
(`stranded-assignments`, `assigned-work-invariant`,
`resolved-dependencies`, `stale-issue-locks`) and the producers that
wake them on run dispatch, run start, run state change, run finalization
and accounting settlement.

**Subsystem affected**

Server background recovery (`server/src/services/recovery/`,
`server/src/services/scoped-deadline-work.ts`, the notification helpers
in `server/src/services/*-notifications.ts`).

**Current behavior**

`notifyRunDispatch`, `notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` sent `notifyIssueRecovery(tx, {
companyId })`. Most run lifecycle paths call one of them, often two or
three in the same transaction (for example `notifyIssueReviewState`).
Some other run paths sent the company hint directly: the wake status
update, the execution-control deadline writes, legacy run
terminalization, scheduled retries, accounting settlement, task checkout
and task document writes. Each company hint makes each task queue reload
all eligible open tasks of the company, 100 per page, and check each
task again. A company hint is reconstructed at most every 30 seconds per
queue, so one run event per 30 seconds is enough to keep a full pass
running all the time. `issue-recovery-deadlines.test.ts` pinned this: a
company hint re-checks every eligible task.

**Proposed behavior**

Design. For each task queue, a run event of run R on task T can change:

| Queue | What the check reads | What one run event can change |
| --- | --- | --- |
| `stranded-assignments` | T's own runs, wakes, handoffs, questions,
recovery actions, goal, assignee eligibility | T (its runs). The parent
of T and the tasks T blocks when the event also writes T's status. |
| `assigned-work-invariant` | the same plus open blockers and open
children | T, the parent of T (open children), the tasks T blocks (open
blockers) |
| `resolved-dependencies` | blocked tasks whose blockers are all done |
the tasks T blocks |
| `stale-issue-locks` | tasks whose checkout or execution run is R | T |

The assignee's agent status (`running`, `idle`, `error`) does not change
whether the agent is invokable
(`packages/shared/src/agent-eligibility.ts`), and a freed concurrency
slot starts the agent's next queued run through the
`queued-run-dispatch` queue, which keeps its company hint. So the
agent's other tasks are not in the set.

- `notifyIssueGraphRecovery(tx, { companyId, runId?, issueId? })` sends
`{ companyId, graphRunId?, graphIssueId? }` to the four task queues. It
runs no query in the producer transaction.
- When a task queue loads that hint, it resolves the graph with one
bounded, indexed query (`resolveIssueGraph`): the task of the run
(`issue_id`, `native_issue_id`, or the saved `issueId`/`taskId`,
validated and cast so the primary key is used), its parent, and the
tasks it blocks. The four queues share the result for 2 seconds. With no
task or more than 100 tasks, the queue turns the hint into the plain
company hint, so task-less runs coalesce under its 30-second gate
instead of each starting its own full pass. The lookup runs after
commit, so it sees the written state. Graph hints do not wait for the
30-second reconstruction gate (they load at most 100 tasks); hints for
the same run coalesce while dirty, and the lookup is shared across the
four queues, so it runs about once per pass instead of once per producer
call. An earlier round ran the lookup in every producer transaction: 34
lookups/s at 1,500 open tasks and 70/s at 5,000.
- The three company-level queues (`orphan-blockers`,
`active-recovery-actions`, `task-watchdogs`) keep the company hint. They
hold one entry per company or per watchdog, not per open task.
- `notifyRunDispatch`, `notifyScheduledRetryDispatch`,
`notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` use it when the scope names a run
or a task. A scope with only a company keeps the company hint.
- Run paths that knew the run or the task but sent only the company now
pass it. This includes three `writeExecutionControl` deadline writes,
legacy run terminalization (`notifyExecutionRunWork`) and native safe
replacement. Those helpers also forward the scope to
`notifyLiveRunRecovery` and `notifyLiveResourceRecovery`, so their hints
to the live-run queues now carry the run id.
- Recovery action changes (`issue-recovery-actions.ts`: create,
supersede, update, exhaust, resolve) name their source task. A recovery
action is a fact of its source task for these queues. The company-level
`active-recovery-actions` queue still gets the company hint. In the
fixture, failing runs create and resolve recovery actions every few
seconds, and that alone kept a full pass running.
- `writeExecutionControl` and `notifyExecutionRunWork` send the
attention retention hint for a task through
`notifyIssueAttentionRetentionWork` (the task, its parents and the tasks
it blocks), a superset of the single-task hint they sent before. A
recovery action of a blocker can change the blocker attention of its
dependents, so the narrower recovery action scope needs this.
- `promoteDeferredWorkIntents` scopes `lock_released` with a task to
that task, and `accounting_settled` to the run when the company has no
active hard-stop budget policy. Only a reached hard stop can refuse work
because of accounting (`evaluateAdmissionInTransaction`), so without
such a policy the settlement of one run cannot admit another task. The
other hold releases keep the company hint.

Backstop. Each scoped hint also sends `{ companyId, backstop: true }` to
the same four queues. Nothing else arms it. The queue runs it as a full
company pass, but not before 5 minutes after the last full pass of that
queue for that company or of the startup pass. It is a scheduler
deadline (`nextWakeAt` reports it), but it holds no idle intent and does
not mark the queue unreconciled. So:

- Startup still reconstructs every company.
- A failed scoped load retries with the existing back-off (30 seconds to
5 minutes). The backstop is pending in any case.
- A graph with no task or more than 100 tasks loads the whole company at
once.
- A task that a scoped hint missed is checked at most
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes) after the event. While runs
continue, each queue runs at most one full pass per company per 5
minutes. With no activity, nothing is scheduled.

Why arming on activity only cannot leave a stranded task undetected when
the instance then goes quiet. A task becomes stranded only through a
write or through a run that stops without a write:

1. The write names the task's graph (run and task producers). The scoped
pass checks the task at once.
2. The write sends a company hint (pause, budget, agent and company
configuration, tree holds, the producers listed below). The company pass
checks it within the 30-second gate, as before.
3. The write is a run event whose graph does not contain the task. That
same event armed the backstop. The backstop deadline stays scheduled
while the instance is quiet and checks the task at most 5 minutes later.
4. A run stops without any write (process lost, controller gone). The
existing deadline paths find it: `stale-issue-locks` waits for the
stored controller lease, and `active-run-silence` and
`orphan-active-leases` act on their own deadlines. Each of them
terminalizes the run, which is a run write and falls under 1 or 3. The
new test `restarts a task whose run vanished without a final event once
its lease lapses` covers this with the production backstop.
5. The process restarts or a quiet instance is woken by a restart. The
startup pass reconstructs every company.
6. A check that found a task waiting (grace, back-off, monitor) returns
its own deadline. That entry keeps its deadline whatever hints arrive.

Five minutes is the existing upper bound of recovery back-off in this
area: the maximum retry delay of a failed scoped-deadline check
(`scoped-deadline-work.ts`), and `DEFERRED_WAKE_PENDING_BOUND_MS`, the
time `assigned-work-sweep.ts` already trusts a deferred wake before it
calls a task stranded.

Producers that keep the company hint, and why. They can change
eligibility of tasks outside one task graph, or I could not show that
their effect stays in the graph. None of them fires once per run event
in the fixture.

- Eligibility for many tasks: company status (`companies.ts`), budgets
(`budgets.ts`), agent lifecycle and configuration (`agent-lifecycle.ts`,
`agent-configuration.ts`), instance settings (`instance-settings.ts`,
`*`), tree holds (`issue-tree-control.ts`), company deletion and import
(`company-deletion.ts`, `company-portability.ts`), and the hold releases
`pause_lifted`, `tree_hold_released`, `scheduling_resumed` and
`reconciler_backstop`. `accounting_settled` also keeps it when the
company has an active hard-stop budget policy.
- Task writes that can move a task between graphs or change eligibility:
task creation and a task update that changes the parent in `issues.ts`
(other task updates and checkout releases now name the task), document
delete and restore paths in `documents.ts`, task watchdogs
(`task-watchdogs.ts`), system and workspace recovery
(`system-recovery.ts`, `task-retry.ts`,
`workspace-recovery-capture-policy.ts`,
`workspace-recovery-stop-proof.ts`).
- Conversation and chat paths: `agent-conversations.ts`, the daily-cap
release in `heartbeat/queue.ts`, the Telegram attachment failure,
`chat-attachment-reuse.ts`, `native-runner-file-handoff.ts`,
`notifyChatRunMilestones` without a task or run (company authority
change), and `notifyNativeStatusReassessment` from `access.ts`.
- `writeExecutionControl` and `notifyExecutionRunWork` callers that pass
only a company.

**Reason and benefit**

Background work must not grow with the backlog of open tasks. The checks
a run event causes now depend on its task graph, not on the company.

## What Changed

- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery`, `resolveIssueGraph`, the scope matcher and
decoder, `ISSUE_RECOVERY_MAX_ISSUE_IDS` (100) and
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes).
- `server/src/services/recovery/work.ts`: the task queues resolve a
graph hint when they load it (shared for 2 seconds), remember the last
full pass per company, and gate the backstop hint on it.
`registerIssueRecoveryWork` takes an optional `backstopMs`.
- `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects`
accepts `issueIds`.
- `server/src/services/scoped-deadline-work.ts`: an optional
`backstop(scope)` that returns the earliest time of a backstop hint. A
pending backstop hint keeps its deadline but does not count as pending
work or as unreconciled.
- Producers: `issue-recovery-actions.ts`,
`execution-control-notifications.ts`, `run-dispatch-notifications.ts`,
`native-status-reassessment-notifications.ts`,
`external-conversation-settlement-notifications.ts`,
`work-intent-outbox.ts`, `run-cost-accounting.ts`,
`heartbeat/run-lifecycle.ts`, `heartbeat/runtime-execution.ts`,
`heartbeat/retries.ts`, `heartbeat.ts`, `legacy-execution-recovery.ts`,
`issues.ts` (checkout), `documents.ts` (task document writes),
`modules/wake-queue/adapters/postgres.ts` (recovery runs for a task),
`heartbeat/queue.ts` (wake enqueue), `recovery/service.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-safe-replacement.ts`,
`modules/run-dispatch/adapters/workspace-busy-wait.ts` (scheduled
retries).
- `bounds subsidiary scans for100 assignment candidates`: a resolved
recovery action now adds one scoped load per task queue (bound 8 to 11),
and the coalescing check sends one company hint before the burst of 100,
because the first full pass after the resolve is now a backstop pass
with its own key.
- `checks only blocked tasks whose blockers are done`: uses the shared
recording harness and runs out the backstop before its quiet check,
because the blocker's status change is now a graph hint (its 300
dependents overflow to a company load).
- Tests. `issue-recovery.test-fixture.ts` and
`live-recovery-work.test.ts` register the queues with `backstopMs: 0`,
so their quiet checks keep the earlier timing. The new tests use the
production value.

## Verification

New tests. I saw each red test fail before the change.

- `issue-recovery-deadlines.test.ts` `re-checks only the tasks a run
lifecycle event can affect, whatever the backlog` (red before): a run of
the agent on a task with a parent and a dependent, 1,000 backlog tasks
of another agent, and a second open task of the same agent. Seven real
producers (`notifyNativeStatusReassessment`, `notifyExecutionRunWork`,
`notifyRunDispatch`, `promoteDeferredWorkIntents` lock release and
accounting settlement, recovery action create and resolve) check exactly
the 3 tasks of the graph. The recovery action steps were red before the
third commit. With 4,000 more backlog tasks the event issues the same
number of statements. Before the change it checked 1,004 tasks.
- `resolves a run's task graph through index lookups, not the backlog`
(red before the second commit, from a Greptile finding): with 2,000
backlog tasks the graph lookup read 2,005 issue rows
(`pg_stat_xact_user_tables`) because it compared `id::text`. It now
validates and casts the saved id, reads fewer than 50 rows, and returns
exactly the run's graph.
- `finds a stranded task outside the affected set within the backstop
bound` (red before): a task inserted without a hint is not checked by
the run event, and is checked at most 5 minutes after the last full
pass. After 20 quiet minutes, a run event checks a new unrelated task
within 5 seconds, because the last full pass is older than 5 minutes.
Then the queues are quiet.
- `keeps the company pass for hold releases that can admit any task`
(guard, passes before and after): accounting settlement with a hard-stop
policy and a lifted pause still check every task.
- `falls back to the company pass when a run affects more tasks than a
hint names` (guard): a run task that blocks 100 more tasks checks the
whole company.
- `limits an ordinary comment to its issue among one thousand open
assignments`: the old final assertion (a company hint checks all 1,000
tasks) stays, and the test now also runs out the backstop and ends
quiet.
- `issue-recovery-assignments.test.ts`, real heartbeat, real producers,
production backstop (guards; they also pass with the old company hint):
a task whose run failed, a task whose run the system cancelled, a run
that vanished without a final event and is found when its controller
lease lapses, and an agent's queued task that starts when its previous
run ends. Each one recovers within 15 seconds, well before the backstop.
- `scoped-deadline-work.test.ts` `keeps a backstop hint's deadline
without holding the worker busy` (red before).

Suites, each file alone, in `server/` (issue recovery, live recovery,
recovery action and attention suites against dedicated local `_test`
databases): `issue-recovery-deadlines` (13/13),
`issue-recovery-assignments` (7/7), `issue-recovery-work-lifecycle`
(14/14), `scoped-deadline-work` (24/24), `live-recovery-work` (51/51),
`assigned-work-invariant` (32/32), `assigned-work-sweep-round-trips`
(2/2), `saved-work-recovery-overdue` (4/4),
`issue-recovery-dependencies` (5/5), `issue-recovery-conversations`
(3/3), `issue-recovery-actions` (112/112), `attention-retention-work`
(55/55). `recovery-deadline-acceptance` skips without its external
database. `pnpm check:module-boundaries` passes.

Local benchmark. The real server ran against PostgreSQL with
`log_statement=all` and instrumented counters (not committed). Fixture:
1 company, 60 agents, open tasks at 150 / 1,500 / 5,000 (52% in review,
39% blocked behind open blockers, 3% in progress, 2% to do), with
process agents that fail at once so recovery re-dispatches runs all the
time. The tables are analyzed after loading. Each window is 180 s,
starting 150 s after the server is ready, so each "after" window
contains one backstop pass. Base is `master` at 54c3616ad9; after is
this branch at dcd170e29f. The last commit (graph hints without the
30-second gate, task updates and checkout releases scoped) is covered by
the tests below but not re-measured. The machine was shared (load
average 9-16, one base window at 45). Run churn differs between runs, so
the table gives counts per run event (runs created in the window).

| Open tasks | Runs created (/s) base → after | Stranded checks per run
base → after | Assigned-work checks per run base → after |
`agent_task_sessions` reads (/s, 4 shapes, stranded check) | Statements
per run event base → after |
| --- | --- | --- | --- | --- | --- |
| 150 | 0.08 → 0.14 | 45.1 → 6.2 | 2.00 → 0.96 | 3.75 → 1.27 | 2,260 →
1,257 |
| 1,500 | 1.09 → 1.00 | 22.0 → 3.9 | 1.24 → 1.53 | 26.5 → 7.2 | 1,313 →
1,060 |
| 5,000 | 1.12 → 1.99 | 20.4 → 1.6 | 4.10 → 0.91 | 29.3 → 8.1 | 1,400 →
829 |

New statements: the task graph lookup (`with seed(id) ...`) at 0.8 / 9.3
/ 18.5 per second, and scoped loads of the task queues (`select "id",
"company_id" from "issues" ...` grew from 0.7 / 6.6 / 7.8 to 2.9 / 20.6
/ 34.4 per second, each a lookup of at most 100 ids). Stale-lock checks
grew from 0 / 50 / 52 to 22 / 190 / 445 per window: each run event now
checks the locks of its own task at once instead of waiting for the next
full pass. In an earlier round the graph lookup ran in every producer
transaction: 34/s at 1,500 and 70/s at 5,000.

With no activity the server issues no statements before and after this
change (quiet checks in `issue-recovery-deadlines`,
`issue-recovery-assignments`, `live-recovery-work`).

## Risks

- A task outside the graph of the run whose state the event changes is
now found by the backstop, at most 5 minutes later, instead of within 30
seconds. One known case: from #21427, `assigned-work-invariant` counts a
held recovery wake as pending while its agent has queued recovery runs.
When the last one starts, the continuation queue delivers the held wake;
the invariant check of that task waits for the backstop.
- A producer that names a task but changes something wider would now
miss the wider effect. Producers that change eligibility for many tasks
(pause, budget, agent and company configuration, tree holds) still send
the company hint.
- The graph is resolved after commit, from the written state. A parent
change is not a run event and keeps its company hint.
- The live-recovery and issue-recovery suites opt out of the backstop
deadline with `backstopMs: 0`.
- The execution-control deadline writes and legacy run terminalization
now pass the run id. `writeExecutionControl` also forwards that scope to
the live-run queues (`active-run-silence`, `native-process-sessions` and
others), so those hints now name the run instead of the company. Those
queues already handle run-scoped hints from native finalization.
- Test-only option: `registerIssueRecoveryWork(..., { backstopMs })`.
The shared recovery fixtures pass `0` so their quiet checks keep the
earlier timing; the new tests use the production value.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:29:22 -06:00
Nicky LeachandClaude Opus 5.5 e977c3b922 feat(native-runtime): end the wait for the provider's own turn when a person comments (#21455)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner part of this work is complete in #21316 and the pull
requests below it: when a controller sends `providerStartedTurns` on
`turn.start`, the Runner keeps Claude while a background shell is
running, and gives the cycle of Claude to the controller as a turn that
the provider started.
> - Part 2 gave the consumer a wait with a limit and two inputs: a hook
that ends the wait early, and a start time.
> - This pull request gives both from the server for a native ACPX
Claude run. A new comment from a person ends the wait. The start time
comes from an event that is already saved, so a restart of Core does not
make the wait longer.
> - The benefit is that a person who comments does not wait up to 10
minutes for the agent to see it.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one of four parts on the Core side. They are
stacked on #21316:

1. The driver admits a turn that the provider started.
2. The consumer waits for that turn, for a limited time.
3. The server ends the wait when a person comments, and keeps the clock
across a restart.
4. The transport sends the opt-in. Only this part changes what a
production run does.

This is part 3. Its base branch is `feat/core-provider-turn-consumer`.
Refs #19912 (design).

**What existing behavior does this improve?**

Nothing visible yet. What Core gives to the consumer of a native ACPX
Claude run.

**Subsystem affected**

Server: `server/src/services/native-runtime`.

**Current behavior**

- A comment from a person during a run is queued as a deferred wake. The
next run delivers it. This does not change.
- Core has no wait for a provider-started turn.

**Proposed behavior**

- For a native ACPX Claude run, Core gives `executeNativeSession` the
option `providerTurnWait`.
- One delivery worker does the person-comment check for all runs that
wait. The server registers it at startup with the shared delivery work
coordinator, as it registers the cost accounting worker. The worker
listens to the existing notification for deferred wakes. On each notice
it runs one query for each run that waits: a comment from a person on
this issue, made after the start of the run, that a deferred wake of
this agent names. A comment that the query finds ends the wait of that
run. The run then takes the path of today, and the next wake delivers
the comment as today.
- If a database read fails, that run stays owed a check, and the
coordinator runs the worker again after 5 seconds, as for each other
delivery worker. A read that fails for one run does not stop the check
of the other runs. When no run is owed a check, the worker has no
deadline and does no query.
- The start time of the wait is the time of the first saved
`turn.completed` event of the run that has `awaitingProviderTurn`. With
no such event it is empty.
- Each query has the company id in its condition.

**Reason and benefit**

- The comment window starts at the start of the run, not at the start of
the wait. A comment made during the prompt turn must not wait for the
limit.

## What Changed

- `server/src/services/native-runtime/native-provider-turn-wait.ts`
(new, 109 lines) and 4 lines in `native-session-executor.ts`.
- `server/src/index.ts` (4 lines): the registration of the worker, its
stop at server close, and its place in the list of delivery workers that
must be ready. `delivery-work-notifications.ts` (1 line): the queue key
of the worker. The key needs no database change, because each topic uses
the one shared channel.
- `server/src/vendor/paperclip-runner/index.ts`: one type export.
- Tests: `native-provider-turn-wait.test.ts` (new, 252 lines) and one
case in `native-usage-accounting.test.ts` (25 lines).

## State saved by the previous release

- No new persisted state and no schema change. The start time is read
from `heartbeat_run_events`.
- **A run that the previous release started:** it has no waiting
terminal, so the start time is empty.
- **Core restarts during a wait:** the new process reads the start time,
so the limit is the original one. The consumer then continues the wait
(#21454), and the worker checks the comments again from the start of the
run, so a comment that was made before the restart still ends the wait.
- **Rollback of Core during a wait:** the previous Core stops at the
waiting terminal and finalizes the run as `native_finalization_missing`,
as today.

## Verification

At the head of this branch:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json` and `pnpm run build:typescript` in
`packages/paperclip-runner`; `node
scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in
`server`; `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed.
- `native-provider-turn-wait.test.ts`, 10 tests on embedded Postgres
with the real coordinator: a comment from a person in a deferred wake
ends the wait; a comment from an agent, a comment before the run, and a
comment of a different company do not; the start time comes from the
first waiting terminal, and a run of the previous release has none.
- The 6 new tests: a read that fails one time is tried again after 5
seconds and finds a comment that is already saved, with no new notice; a
notice during a read that fails gives a new check; a read that fails for
one run does not hold back a second run, and each run ends only for its
own comment; a run that starts to wait during a check is checked; a wait
that is aborted is removed, nothing checks it again and no deadline
stays; with no run owed a check there is no deadline and no query in one
hour of test time.
- The test helper now registers the wake intent before it writes the
deferred wake, in the same transaction. This is the order of each
production writer (`modules/wake-queue/adapters/postgres.ts` and
`services/heartbeat/queue.ts`). Production code did not change for this.
- Mutation checks: a read that fails clears the owed check (3 tests
fail); an aborted wait is not removed (the abort test fails).
- Also run, unchanged and green: `delivery-work-coordinator.test.ts`,
`budget-enforcement-worker.test.ts`,
`server-startup-feedback-export.test.ts`, `startup-timeline.test.ts`.
The 7 files together: 1,699 passed, 7 skipped.
- `native-usage-accounting.test.ts`: a prompt turn and the
provider-started turn after it are each counted one time, with the
events as the executor receives them. The cost of the provider-started
turn is the session total in the cumulative channel, and it is not added
two times. The accounting code is not changed.

Not run: a run through the executor with a database; the full suite. CI
supplies the broad coverage.

## Risks

- **One more query at each start of a native ACPX Claude run,** on
`heartbeat_run_events` by company and run.
- **The worker is registered in each server process, and it is idle with
no wait.** A notice with no run that waits does no query. No wait starts
before part 4.
- **After an abort during a retry, the 5 second retry of the coordinator
runs one more time.** It finds nothing owed and does no query. The
coordinator has no call to cancel the retry of one worker.
- **A process that does not register the worker does no comment check.**
The server registers it with no condition at startup, and native runs
execute only in that process. Only a unit test can be in this state.
- **A comment ends the wait but does not interrupt a turn that is
running.** That is the rule of part 2.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:55:11 -08:00
DottaandClaude Opus 5.5 3ae7aba643 perf(db): reuse dedicated connections instead of opening one per call (#21567)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server uses `withDedicatedDbConnection` (packages/db) when an
action must hold one database session outside the main pool: advisory
transaction locks, lock keepalive, and `pg_settings` reads for workspace
lifecycle, finalization, source locks, skill caches and status cards
> - Each call built a new postgres.js client, opened a new physical
connection (TCP, TLS and startup through the pooler), built a new
drizzle client with the full schema, and then ended all of it
> - On staging the dedicated application name shows about 21 connection
setups per second, and the main thread is pinned (event-loop utilization
0.965)
> - This pull request keeps a small set of idle single-connection
clients per owner and reuses them
> - The benefit is that a dedicated action costs about 0.2 ms of
main-thread time instead of 4 to 8 ms, with the same exclusive-session
guarantee

## Linked Issues or Issue Description

**What existing behavior does this improve?**

`withDedicatedDbConnection` opens and closes one new database client and
one new physical connection for every call.

**Subsystem affected**

Database client (`packages/db`) and the server services that take
dedicated advisory-lock sessions.

**Current behavior**

Each call builds a new postgres.js client, opens a new connection (TCP,
TLS and startup through the pooler), and builds a new drizzle client
with the full schema. On staging (Neon statement statistics, 326 s
window) `SET application_name='paperclip-workspace-finalization-lock'`
ran 21.4 times per second. Only this factory uses that application name.
In a 30 s staging CPU profile, drizzle `relations.ts` was 2.2%
inclusive, `n:net` 5.8% and postgres.js construction 0.3%. Part of that
is this per-call setup.

**Proposed behavior**

A caller still gets exclusive use of one session for the whole action.
The session and its drizzle client are reused by later actions, so the
per-call cost is the action's own queries, not a full connection setup.

**Reason and benefit**

The server main thread is saturated on staging (event-loop utilization
0.965). Reuse removes about 5 ms of main-thread work per call.

**Breaking changes**

None. The exported function keeps its signature.
`closeDedicatedDbConnections` and `DEDICATED_CONNECTION_IDLE_MAX` are
new exports.

## What Changed

- `packages/db/src/client.ts`: each `createDb` owner now has one set of
dedicated single-connection clients, shared with every private client
derived from that owner (so nested dedicated actions use the same set).
- An action takes an idle client (last in, first out) or opens a new
one. It never waits. Callers hold advisory locks and nest dedicated
actions, so a waiting queue could deadlock.
- On success the client returns to the idle set if fewer than
`DEDICATED_CONNECTION_IDLE_MAX` (4) are idle. Otherwise it is closed.
- When the action throws, the client is closed and never reused. This
covers a failed or cancelled transaction, a broken socket, and an
interrupted lock wait.
- When an action ends the client itself (the source-lock cancel path
calls `$client.end`), the client is never reused.
- Idle clients keep the owner's idle timeout and max lifetime, so
postgres.js still closes idle physical connections after
`DATABASE_IDLE_TIMEOUT_SECONDS`.
- New `closeDedicatedDbConnections(db, timeoutSeconds)` closes the idle
set and stops retention. Ending the owner `$client` does the same. A
later action still works with a connection that closes on release.
- `server/src/index.ts`: shutdown closes the dedicated set next to the
other private pools.
- `doc/DATABASE.md`: documents the idle cap, the "not an admission
limit" rule, and the transaction-scoped session state contract.
- Session state audit: every current caller uses `pg_advisory_xact_lock`
/ `pg_try_advisory_xact_lock` and `set_config(..., true)`. No caller
uses session advisory locks, `SET` without `LOCAL`, `LISTEN`, or
temporary tables. A reset round trip per call is therefore not added.

## Verification

- New `packages/db/src/dedicated-connection-pool.test.ts` (embedded
PostgreSQL, 10 tests).
- Red on `origin/master`: 200 sequential calls used 200 distinct
sessions (`expected 200 to be 1`). All 10 tests failed.
- Green with this change, 3 runs in a row: 200 sequential calls use 1
session; 7 concurrent callers each hold a different session for the
whole transaction, and afterwards at most 4 stay open and later calls
reuse them; a failed transaction's connection is closed; a terminated
socket during the action is not reused; an idle connection terminated by
the server reconnects; a client the action ended is not reused;
transaction-local settings do not leak; shutdown and owner `$client.end`
close every retained connection; nested actions do not block.
- Focused suites pass: packages/db `work-signals`,
`prepared-queries-postgres`, `response-capture-postgres`,
`checkpoint-pool`, `client`, `client-options`,
`client-teardown-registry`, `postgres-connection-recovery`,
`client-disconnect-replay` (132 tests). Server suites that use dedicated
connections: `skill-package-store`, `status-card-evaluation-work`
(updated: it now proves advisory locks are released, since sessions are
reused), `agent-instruction-working-copies`,
`workspace-recovery-retry-copyback`, `native-execution-lease-batch`,
`company-skills-empty-runtime`, `decisions-service`,
`legacy-controller-lease`, `server-startup-feedback-export` (467 tests).
- `pnpm verify --explain` selects 2779 files for a
`packages/db/src/client.ts` change. Per AGENTS.md I ran the focused
checks above and let CI give the broad coverage.
- Benchmark: 200 calls of `withDedicatedDbConnection(db, d =>
d.transaction(tx => tx.execute(select pg_try_advisory_xact_lock(42))))`
against local embedded PostgreSQL 18, 10 warm-up calls, 5 alternating
before/after process pairs per mode. Main-thread time is
`eventLoopUtilization().active`.

| Mode | Variant | Wall ms/call (median, range) | Main-thread ms/call
(median, range) |
| --- | --- | --- | --- |
| TLS | before | 35.55 (32.93–37.52) | 5.18 (4.68–7.95) |
| TLS | after | 0.19 (0.17–0.21) | 0.18 (0.17–0.20) |
| Plain | before | 18.39 (17.91–20.53) | 3.65 (3.58–4.30) |
| Plain | after | 0.17 (0.13–0.18) | 0.13 (0.12–0.14) |

At the staging rate of about 21 calls per second, this removes roughly
100 ms of main-thread work per second (TLS figures; staging CPUs may be
slower).

## Risks

- A retained connection that the server or network drops while idle:
postgres.js reconnects on the next query when it has seen the close
(tested). If the drop and the reuse race, the first query of that action
fails like a main-pool query would. Before this change, a dedicated
action always started on a fresh connection.
- Through a transaction-mode pooler, `SET application_name=...` is a
server-side parameter switch, so its count may not drop as much as the
client-side connection count. Measure the deployed effect with pooler
client connections and main-thread CPU, not only that statement.
- Concurrency above 4 opens extra connections exactly as before; they
close on release. Total retained connections per owner grow by at most 4
idle clients, which still close after the idle timeout.
- An action that sets session-level state would now leak it to the next
action. The contract is documented on `withDedicatedDbConnection` and in
`doc/DATABASE.md`; no current caller does this.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:37:04 -06:00
DottaandClaude Opus 5.5 351b69941c perf(checkpoint): write a routine controller checkpoint save in one statement (#21553)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote runner's controller saves a checkpoint after each runner
event it commits, on a 3-connection pool, before it acknowledges the
event
> - Each save is a transaction of 4 statements (5 where commit
notifications are on): BEGIN, the owner lock with the transaction id,
the write, COMMIT
> - What forces the transaction is the live-run recovery wake that every
save registers. Recovery does not react to a routine save
> - This pull request writes a routine save (new or changed event,
command and cursor records) with one statement that checks the owner and
writes, and keeps today's transaction for every other save
> - The benefit is one statement instead of four for most checkpoint
saves, on the pool that limits one process in a busy minute

## Linked Issues or Issue Description

Refs #21528 (owner lock returns the transaction id). Refs #21549 (load
model at recorded rates; the plan records the checkpoint pool as a
per-process limit).

**What existing behavior does this improve?**
`nativeRunnerCheckpointWriter` in
`server/src/services/native-runtime/native-runner-checkpoint.ts`.

**Subsystem affected**
Controller checkpoint persistence for remote native runs.

**Current behavior**
Every save locks the coordinator row (`native_run_finalizations` `for
update`), checks the owner, calls `notifyLiveRunRecovery` (6 recovery
queues), and upserts the changed parts, in one transaction.

**Proposed behavior**
A routine save is one statement: a CTE locks the coordinator row only
while the lease owner, boot id, generation and an unexpired lease
(`clock_timestamp()`) all match, and the upsert writes only through that
row. It registers no recovery wake. If the owner does not match, it
writes nothing and the save takes today's full path, which runs the
complete owner check (successor rules included) and fails with
`native_runner_checkpoint_owner_lost` when it should.

A save is routine when it has changed parts, no reset, no removed parts,
no header change, no predecessor, and the run is its own authority.
Every other save (the first save of a controller, which is a reset, a
save after a failed save, header or retirement changes, removals,
source-run consumers) is unchanged.

**Reason and benefit**
At recorded rates about a third of one process's statements are
checkpoint saves. In a busy minute the 3-connection pool cannot keep up
at 500 agents (#21549).

**Breaking changes**
None. No schema change.

## What Changed

- `ROUTINE_CHECKPOINT_SAVE_SQL` and the routine branch in
`nativeRunnerCheckpointWriter`.

## Why a routine save needs no recovery wake

I read the six consumers of `notifyLiveRunRecovery`:
- `nativeProcessSessions`, `nativeQuestionCancellations`,
`activeExecutionRecovery`, `settledNativeProcesses`
(`live-recovery-work.ts`): each selects candidates from `heartbeat_runs`
status, runtime mode, `native_run_finalizations` phase, lease and retry
times, the context snapshot and process identity. None reads checkpoint
rows to decide eligibility.
- `strandedAgentStatus` (same file, `agents` rows) and
`activeRunSilence` (`active-run-watchdog/adapters/deadline-work.ts`,
`heartbeat_runs.last_output_at`): neither reads checkpoint rows.
- The checkpoint rows matter to recovery in two places:
`listExpiredNativeRunnerAuthorities` requires that a checkpoint exists,
and the remote bootstrap proof reads its parts. A checkpoint first
appears with a reset save, which keeps its wake. A routine save only
adds event and command parts, which can only take a run further from the
bootstrap proof; recovery reads the current rows when its own deadline
(lease expiry) fires.
- The intent of the wake also fenced a recovery pass against a save in
flight. The routine save is one atomic statement and holds the
coordinator row lock while it writes, so a recovery claim that takes
that row waits for it, and a reader sees the save whole or not at all.

## Verification

- Red/green on embedded PostgreSQL
(`native-runner-checkpoint-save.test.ts`); the new tests failed against
`origin/master` first:
- A routine save is one statement (`with owner as ... for update ...
insert into native_runner_checkpoints`) and registers no recovery wake
(before: 4 statements and a wake).
- A reset save keeps BEGIN, the owner lock with the transaction id, and
COMMIT, and still registers its recovery wake.
- A routine save that waits while another connection takes over the
lease (new owner and generation) falls back to the full path and fails
with `native_runner_checkpoint_owner_lost`; the checkpoint is unchanged.
- The same race where the new owner also resets the checkpoint (deletes
every part and writes its own header): the save fails the same way and
the checkpoint holds only the new owner's header.
- A lease that expires while the routine save waits for the coordinator
row lock (the holder does not change the row) writes nothing: the
statement judges the lease after the lock, as
`lockNativeRunnerCheckpointOwner` does. My first version checked the
lease in the locking CTE itself, and PostgreSQL does not re-check a
condition after a lock wait on an unchanged row; the existing
`native-runner-restart-recovery.integration.test.ts` test "fences
checkpoint delivery when ownership expires during an unchanged row-lock
wait" failed on it, and so does the new test. Both pass now.
- The existing test "refuses a save after the owner's lease expired and
writes nothing" passes through the new fallback.
- Mutation checks: with the owner conditions removed from the statement,
both race tests and the expired-lease test fail; with the lease check
moved back into the locking CTE, the lock-wait expiry test fails.
- Other focused suites, one file at a time:
`native-runner-restart-recovery.integration` (231 passed, 14 skipped),
`native-runner-restart-checkpoint-cost.integration` (1; it greps for
`insert into "native_runner_checkpoints"`, so the statement quotes the
table name), `native-recovered-owner-followup` (97),
`native-controller-recovery.integration` (30),
`native-runner-checkpoint-authority.integration` (1),
`native-restart-recovery-isolation` (1), the port suite with conformance
(105), `heartbeat-native-restart-suspension` (115), `live-recovery-work`
(51 tests pass; on this busy machine its teardown hook timed out, as it
has before on master). `pnpm --filter @paperclipai/server run typecheck`
passes.
- Statements per routine save: 4 → 1 (5 → 1 where commit notifications
are on). I did not rerun the load model for this change: the machine is
too busy for a fair timing comparison today, and the statement count is
exact.

- CPU per operation (a local script, not committed: 600 routine saves of
one event part and the cursor for one run on embedded PostgreSQL, one
connection, three rounds, `origin/master` code against this branch):
main-thread CPU per save 0.38–0.56 ms before and 0.10–0.12 ms after
(encryption included in both); wall time per save 0.45–0.73 ms before
and 0.22–0.30 ms after.

## Risks

- A routine save no longer wakes the six recovery queues. Their own
deadlines and the 30 s reconstruction still run; see the reading above
for why no decision depends on a routine save.
- The owner check of the routine statement accepts only the exact active
owner. Any other state goes to the unchanged full path, so a successor
or a finished result keeps today's rules.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:58:06 -06:00
Devin FoleyandPaperclip b7922570e1 fix(runtime): verify GitHub bridge health before agent startup (#21537)
Verify the inactive GitHub credential endpoint from the execution target
before publishing launcher settings. If HTTP/2 fails during startup,
stop it and verify one file-bridge attempt. Stop an unverified final
bridge before returning isolated anonymous launchers.

Regression tests cover early transport loss, delayed loss reporting,
credential isolation, and failed file-bridge cleanup. The master update
keeps the current MCP approval fixture.

Verification: 96 focused tests and server typecheck pass after conflict
resolution. Full CI passes on 843cd9945e. Greptile is 5/5 with no
unresolved review threads.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 19:41:17 -08:00
DottaandClaude Opus 5.5 f6c5986b61 feat(connections): machine sign-in (MCP OAuth client credentials) for organization connections (#21473)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through connections to remote MCP servers
> - Today an organization connection must sign in as a person (browser
sign-in) or use a static key
> - Service accounts and CI-style integrations have no person who can
sign in
> - The MCP OAuth Client Credentials extension (2026-07-28) defines
machine-to-machine sign-in for this case
> - This pull request adds "Machine sign-in" for organization
connections, with a Paperclip-held signing key (recommended) or a client
secret
> - The benefit is that organizations can connect automation-owned MCP
servers safely, with credentials only in the vault and bound to one
sign-in service

## Linked Issues or Issue Description

**Subsystem affected**
Connections: remote MCP OAuth (server `tool-access/oauth`, tool gateway
token refresh) and the custom MCP server setup flow in the UI.

**Problem or motivation**
Organization-owned automation (service accounts, CI integrations) cannot
connect an MCP server that requires OAuth, because no person can
complete a browser sign-in. The MCP OAuth Client Credentials extension
(`io.modelcontextprotocol/oauth-client-credentials`) solves this with
`grant_type=client_credentials` and RFC 7523 signed client assertions.

**Proposed solution**
Add machine sign-in for organization connections only. Paperclip creates
a keypair per connection, keeps the private key in the company vault,
and shows the public key (PEM and JWK) for the admin to register. A
client secret is also supported. The sign-in service (issuer and token
endpoint) is pinned at setup. Tokens use the existing organization grant
refresh path, so caching, concurrent dedupe and 401 re-request reuse the
existing lease and single flight.

**Alternatives considered**
Reuse the deployment-wide `PAPERCLIP_TOOL_OAUTH_*` client env path that
already sent `client_credentials`. Rejected: it is not per connection,
it keeps secrets in env, and it has no issuer pin. Route machine sign-in
through `connectGalleryApp`. Rejected: its rollback deletes the draft on
a failed test, which would lose a key the admin already registered.

**Roadmap alignment**
Phase 1 of the enterprise MCP plan (no enterprise single sign-on
needed). OIDC login and enterprise-managed authorization (ID-JAG) are
later phases.

## What Changed

- `oauth/client-credentials.ts`: keypair generation (RS256, or ES256
when the service requires it), RFC 7638 key IDs, and RFC 7523 assertions
(`iss` = `sub` = client ID, `aud` = token endpoint, 60 s `exp`, unique
`jti`) with Node `crypto` only.
- `oauth/client.ts`: the token exchange accepts a signed assertion and
then sends only `client_assertion_type` and `client_assertion` (no
`client_id` or secret), as the extension specifies.
- `oauth/client-credentials-service.ts`: discovery of the sign-in
service (protected-resource metadata, then RFC 8414/OIDC metadata with
issuer check, or a token URL entered directly), issuer pinning,
vault-only key and secret handling, draft creation, test-and-activate,
and no-downtime rotation (test the new secret first; stage a new key and
switch only after the provider accepts it).
- `oauth/grant-refresh.ts` and `oauth/connection-refresh.ts`: machine
sign-in requests a new access token with the connection's own
credentials instead of needing a refresh token. It never falls back to
deployment-wide OAuth clients.
- `catalog/discovery.ts`: a rejected token on a machine sign-in
connection never re-runs discovery or rewrites the pinned endpoints.
- `mcp-http.ts` and the tool gateway: declare
`io.modelcontextprotocol/oauth-client-credentials` in client
capabilities only for connections that use it.
- Routes (with OpenAPI and activity logging): `POST
/companies/{companyId}/tools/machine-sign-in`, `GET|POST
/tool-connections/{id}/machine-sign-in[/test|/rotate|/rotate/complete]`.
Removal activity now records `machineSignIn: true`. The runner API
treats these routes as sensitive.
- Shared validators and types for machine sign-in.
- UI: "Machine sign-in" under Advanced authentication in the custom MCP
server flow (organization connections only), `MachineSignInSetup`
(choose Paperclip key or client secret, register the public key, test),
and Storybook stories for each setup state.
- Docs: `doc/connections/MCP-CLIENT-CREDENTIALS.md`, linked from
`doc/connections/README.md` and `GENERIC-REMOTE-MCP.md`.

## Verification

- `server/src/__tests__/tool-access-client-credentials.test.ts`: 16/16
pass. It uses a fake sign-in service that verifies real signatures
against registered keys, and a fake MCP server. It covers RS256 and
ES256 assertion claims and signatures, assertion-only client
authentication, the secret flow (`client_secret_basic`), discovery and
direct token URL, issuer pinning (refuses a different sign-in service
and sends it nothing), the pinned endpoint surviving a hostile
challenge, the token cache (0 token requests while live), concurrent
dedupe (5 callers on 2 service instances produce 1 token request), 401
re-request, key and secret rotation, a refused personal connection,
secret redaction in responses, rows and activity, and vault cleanup on
removal.
- Red/green: with the grant-refresh machine sign-in branch and the
discovery pin guard disabled, 5 runtime tests fail (pin, cache/dedupe,
401 re-request, both rotations). Restored: 16/16 pass. The other tests
cover new routes and services that do not exist on `master`.
- `ui/src/features/connections/MachineSignInSetup.test.tsx`: 3/3.
`ConnectionSetupFlow.architecture` and `.oauth` tests: 19/19.
- `openapi-routes.test.ts`: 51/51. `runner-api.test.ts`: 2153/2153.
`mcp-http.test.ts`: 9/9. Existing `tool-access-service.test.ts` refresh
and legacy client-credentials tests: 54/54 (filtered).
- Server package `tsc --noEmit`: clean. UI `tsc`: clean except the
pre-existing generated Storybook catalog imports.
- Not run locally (machine under heavy load): full suites, repo-wide
typecheck, browser check of the live app. CI covers breadth.

## Risks

- No migration. New config lives in `config.oauth` (`grantType:
"client_credentials"` plus `clientAuth`). Older env-configured
client-credentials connections have no `clientAuth` and keep their
original path.
- The capability declaration adds one extension key only for machine
sign-in connections. Other requests are unchanged.
- Rotation has no settings-page control yet; it is available through the
API. This is a follow-up.
- Low risk to agent boot and time to first token: a live token is read
exactly like a browser sign-in token.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:10:32 +00:00
DottaandClaude Opus 5.5 1dee321844 perf(run-events): read the wake's transaction id and the replay probe in the lock read (#21528)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every native event that announces work to another part of the server
(chat progress, a final response, an evidence outcome, recovery)
registers a wake in its transaction before it writes the event
> - Such an event takes six statements: BEGIN, the run lock read, a
replay probe, a read of the transaction id for the wake, the append, and
COMMIT. A controller checkpoint save takes five: BEGIN, the owner lock,
the transaction id, the write, COMMIT
> - In the agent scaling load model these two paths are now the largest
per-agent cost after #21511 and #21520, and other events fall behind
first at 600 agents
> - This pull request returns the transaction id from the lock read that
each path already sends, and moves the replay probe into the run lock
read when it is safe
> - The benefit is four statements for such an event and four for a
checkpoint save, with the same locks, the same wake order and the same
replay rule

## Linked Issues or Issue Description

Refs #21511 and #21520 (the delta batch path). Refs #21418 (agent
scaling plan and load model).

**What existing behavior does this improve?**
The statement count of native events that register a wake
(`appendNativeSourceEvent`) and of controller checkpoint saves
(`nativeRunnerCheckpointWriter`).

**Subsystem affected**
`packages/db/src/work-signals.ts` (one new export),
`server/src/services/heartbeat-run-events.ts`,
`server/src/services/native-runtime/native-runner-checkpoint.ts`.

**Current behavior**
`signalDatabaseWork` sends `select pg_current_xact_id()` for the first
wake of each transaction. A native event with a wake reads the run lock,
then probes for its replay in a separate statement, because a statement
that waits for the lock keeps its earlier snapshot.

**Proposed behavior**
- The run lock read and the checkpoint owner lock read also return
`pg_current_xact_id()::text`. Both already lock a row, so the
transaction has an id at that point. `primeDatabaseWorkTransactionId`
gives it to the transaction's wake state, and the wake sends no
statement for it.
- For an event that may register a progress, response, reassessment or
retention wake, the run lock read also reads `next_event_seq` from its
own snapshot and checks for a replay in that snapshot. Every event
append advances `next_event_seq` in its transaction. So when the locked
row has the same `next_event_seq` as the snapshot, no append committed
in between, and the probe result is current. Otherwise, or when the
probe finds a replay, the separate probe runs as before.

**Reason and benefit**
Fewer round trips while a pool connection is held in a transaction. In
the load model these paths hold the main pool and the three-connection
checkpoint pool longest per event.

**Breaking changes**
None. No schema change.

## What Changed

- `primeDatabaseWorkTransactionId(connection, xid)` (`work-signals.ts`,
exported from `@paperclipai/db`): inside a transaction whose id is not
yet known, it records the id. Outside a transaction it does nothing. An
uncertain commit still resolves with `pg_xact_status` on the same id.
- `appendNativeSourceEvent`: the lock read returns the transaction id
when the event has any wake. For a probing wake it is one statement: a
CTE that reads `next_event_seq` from the snapshot, a CTE that locks the
run row (`for no key update`, unchanged), and `exists (...)` for the
replay. The wake is still registered after the replay check and before
the append.
- `lockNativeRunnerCheckpointOwner` returns the transaction id with its
owner lock and primes the wake state. Its return value and every caller
are unchanged.

## Verification

- Red/green on embedded PostgreSQL; each new assertion failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "registers a native
event's wakes before its write and only when the event is new": four
statements (before: six), the wake's intent and settlement are
published, and a replay publishes no wake.
- New: "finds a replay that another writer committed while the lock read
waited, and registers no wake for it". Another connection holds the run
row (the test waits until PostgreSQL reports the append waiting for that
lock), stores the same event and moves `next_event_seq`; the waiting
append returns `duplicate`, runs the separate probe, and publishes no
wake. With the `next_event_seq` comparison removed, this test fails.
- New `native-runner-checkpoint-save.test.ts`: a save is BEGIN, the
owner lock with the transaction id, the write, COMMIT (before: five
statements), and its recovery wake is published; a save after the lease
expired writes nothing and fails with
`native_runner_checkpoint_owner_lost`.
- `work-signals.test.ts`: a primed transaction sends no id statement for
its wakes; a primed transaction whose connection is killed after its
write stays pending and then resolves through `reconcileDatabaseWork`,
with no row stored.
- Added after merging `origin/master` (with #21511, #21520 and #21436)
into this branch:
- "gives the wake the transaction id of the transaction that took the
run lock": inside a caller's transaction, an event with a progress wake
sends no separate `pg_current_xact_id()` statement; the transaction's
connection is then killed, and `reconcileDatabaseWork` checks
`pg_xact_status` with exactly the id that `pg_current_xact_id()` returns
on that same transaction. Priming a wrong id makes it fail.
- "primes nothing when the lock read fails, so the transaction is not
left uncertain": a lock read that fails on `lock_timeout` leaves no
pending work; after the same failure inside a savepoint, the next signal
on that transaction still sends its own `pg_current_xact_id()`. Priming
before the lock read makes it fail.
- Both call sites pass only the value returned by the statement they
just ran on the same `tx`: `appendNativeSourceEvent` (`tx.execute(...)`
then `primeDatabaseWorkTransactionId(tx, run.xid)`) and
`lockNativeRunnerCheckpointOwner` (`tx.with(locked).select(...)` then
`primeDatabaseWorkTransactionId(tx, xid)`).
- Retest on the combined tree, one file at a time: atomic-append 28
(including the equivalence test and both race tests), the port suite
with conformance 105, checkpoint-save 2, `live-recovery-work` 51,
`work-signals` 25, runner `native-session-runtime` 297; server typecheck
passes. No code change was needed for the combination.
- Other focused suites pass, one file at a time: the port suite with its
conformance suite (100), `chat-native-progress-work` (35),
`native-observer-replay` (1), `heartbeat-run-event-sequencing` (5),
`native-event-boundary-golden` (1), `native-recovered-owner-followup`
(97), `native-restart-recovery-isolation` (1),
`heartbeat-native-restart-suspension` (115), `live-recovery-work` (51),
`work-signals` (25).
- `pnpm --filter @paperclipai/server run typecheck` and `pnpm
check:module-boundaries` pass.
- Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in
#21418). Stacked: this pull request merged locally with #21511, and the
model's local `DELTA_COALESCE=1` mode that mirrors #21520 (the model
applies the port's batching rules itself; that mode is not in any pull
request). Default pool of 10 connections; embedded PostgreSQL on the
same, shared machine; runs one after the other with a pause between
them. No run had a failed operation, a failed checkpoint save or a lost
lease. "Before" rows are `origin/master` from the same session.

| Agents | Code | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 /
12,000 | 278 / 3,895 |
| 200 | stacked | 60 s | 7.9 | 0.31 | 2,725 | 102 / 253 | 12,000 /
12,000 | 1 / 70 |
| 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 /
24,000 | 1,904 / 60,154 |
| 400 | stacked | 60 s | 8.6 | 0.60 | 5,609 | 117 / 955 | 24,000 /
24,000 | 2 / 1,274 |
| 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 /
36,000 | 10,822 / 52,171 |
| 600 | #21511 + #21520 only | 120 s | 11.4 | 0.72 | 6,730 | 1,147 /
6,335 | 61,158 / 72,000 | 11,678 / 21,212 |
| 600 | stacked | 120 s | 9.6 | 0.93 | 10,290 | 656 / 4,233 | 71,895 /
72,000 | 1,136 / 4,980 |
| 600 | stacked (repeat) | 120 s | 9.8 | 0.90 | 9,579 | 648 / 4,509 |
71,958 / 72,000 | 1,319 / 5,712 |
| 800 | stacked | 60 s | 9.7 | 0.98 | 10,971 | 920 / 3,305 | 46,244 /
48,000 | 1,974 / 3,835 |

With the three changes, one process finishes 99.9% of the offered work
at 600 agents, but its event-loop utilization is 0.90–0.93, above the
0.75 target, and p99 lateness is about 5 s. It saturates between 600 and
800 agents.
- Statements: an event with a wake 6 → 4; a checkpoint save 5 → 4 (6 → 5
where commit-time notifications are on).

## Risks

- The transaction id is read in a statement that already assigns one (a
row lock), so no transaction gets an id it did not have before.
- The probe in the lock read is used only when `next_event_seq` did not
change between the snapshot and the lock. Every writer of
`heartbeat_run_events` moves `next_event_seq` in the same transaction
(native and ordinary appends, the run coordinator store, the retry
queue). A future writer that inserts events without it would make this
probe miss a concurrent replay; the append statement then still finds
the replay and stores nothing, but the wake would already be registered
(a spurious wake, not a lost or duplicate event).
- This pull request and #21511 both edit `heartbeat-run-events.ts`, in
different functions.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:48:17 -06:00
DottaandClaude Opus 5.5 1f388ed385 Start follow-up runs normally when a warm sandbox's window ends during its idle close (#21504)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm sandbox keeps an agent's Runner alive between runs, so the
next run on the task starts fast
> - When a run ends, the server closes the idle warm Runner in-process.
A separate SQL worker stops the sandbox when its warm window ends
(#21410, merged today)
> - Both steps start at the same moment. The sandbox stop wins, so the
idle close fails. The failed owner stays registered, and the next run
throws `native_instruction_materialization_handoff_unavailable` before
any provider work
> - Recovery must just work: a follow-up run must never fail because a
predecessor's warm copy is gone
> - This pull request lets the idle close finish before the sandbox
stops, prepares instructions fresh when the warm copy cannot be handed
off, and releases an owner whose sandbox the host already gave up
> - The benefit is that automatic follow-up runs start normally after a
warm window ends

## Linked Issues or Issue Description

Refs #21472 (base of this stacked PR), #21410 (introduced the
warm-window stop).

**What happened?**
In a Daytona two-server chaos wave (`w-20261113T200824-s7106`), three
automatic follow-up runs failed before any provider work with
`native_instruction_materialization_handoff_unavailable`: run
`5f68e194`, run `d6bd5a97` and run `d1198e40`. None of them was at a
release swap. In each case, the predecessor's idle close had failed
first with `provider_state_unreadable` / "Daytona sandbox lease … is no
longer active". The cause: the warm-window stop from #21410 stopped the
sandbox at the same moment that the idle close started. For the first
run's predecessor, the close was requested at 20:30:34 and the sandbox
got SIGTERM at 20:30:37.8.

**What did you expect?**
The follow-up run prepares its instructions fresh and continues. The
warm-window stop does not interrupt the host's own idle close.

**Steps to reproduce**
1. Run a warm Daytona task until it ends, so its warm window starts.
2. Let the warm window end while the idle timer closes the Runner.
3. Start a follow-up run on the same task. It fails with
`native_instruction_materialization_handoff_unavailable`.

Two of the three tasks then ended stranded. That downstream part (the
claim-gate cancellation with `execution_reconciliation_required` and the
stranded notice) is owned and fixed by another session. It is **not** in
this PR.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
`claimWarmNativeInstructionCopy`: after it retires a stale warm owner
whose copy cannot be adopted, it returns `null`. The caller then
prepares instructions fresh. Before, it threw
`native_instruction_materialization_handoff_unavailable`.
- `server/src/services/warm-sandbox-expiry.ts`: the warm-window stop now
waits `WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS` (2 minutes) past the window.
The in-process idle close (close grace plus the 30 s proof read)
finishes first.
- `native-session-executor.ts` idle timer: when an idle close fails, it
reads the lease rows for that sandbox. If every row is no longer
`active` or `retained` (the host gave the sandbox up), it releases the
in-process owner. The next run then prepares fresh.
- `doc/execution-semantics.md`: one paragraph about the stop margin and
fresh preparation.

## Verification

Red → green (each test fails on the base `35ba76d0ef` and passes on this
branch):

- `native-session-executor.test.ts` › "hands off collection before
probing or retiring a warm owner: unavailable" — red: throws
`handoff_unavailable`; green: returns `null` (fresh preparation).
- `native-session-executor.test.ts` › "forgets a warm owner whose idle
close failed only once the host gave up its sandbox: pending_cleanup /
released / retained" — red on all three; green: the owner is released
for `pending_cleanup` and `released`, and kept (retired by the
successor) for `retained`.
- `warm-sandbox-expiry.test.ts` › "leaves the host's idle close of the
warm runner time to finish before stopping its sandbox" — red: the
sandbox stops 3 s after the window; green: it stops only after the
margin. Two existing tests ("keeps a sandbox still inside its warm
window, and wakes when the window ends", "reconciles a warm sandbox left
running across a restart") now expect the deadline with the margin.

Local runs on this branch:

- `server`: `tsc --noEmit` clean.
- `vitest run src/__tests__/warm-sandbox-expiry.test.ts
src/services/heartbeat/instruction-working-copy.test.ts`: 24/24 passed.
- `vitest run
src/services/native-runtime/native-session-executor.test.ts -t "hands
off collection before probing or retiring a warm owner|forgets a warm
owner whose idle close failed"`: 10/10 passed.

Not run: a Daytona wave with this change.

## Risks

- Warm sandboxes now stop up to 2 minutes later than the configured
window. This is a small cost increase per warm sandbox. I chose a fixed
margin over an in-process "is a close running?" check because the stop
is pure SQL. The margin works when two servers run at the same time (the
server that runs the stop may not be the server that owns the Runner),
and it needs no busy-loop or cross-server signal.
- The margin does not cover a re-armed idle timer or a close that takes
longer than 2 minutes. Fresh preparation (fix 1) and the owner release
(fix 3) cover those cases.
- Not run on Daytona. The evidence is from the wave above and the unit
tests.
- In the wave, a `restore_failed` copy-back onto the just-restarted
sandbox was also seen. I did not find its root cause. It is not changed
here.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test runs) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:47:42 -06:00
DottaandClaude Opus 5.5 a82fa96540 perf(run-events): collect deltas that arrive while a batch commits into one next batch (#21520)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane port commits streamed text deltas of a run in
batches (#21460): the first text at once, later deltas after at most 250
ms
> - Each batch is cut when its timer fires, even when the previous batch
of the same run has not committed yet
> - When the database is slow, every run then queues one small batch per
250 ms behind its slow commit. The number of statements stays the same
exactly when the database has the least room for them, and the queue
grows
> - This pull request cuts a batch only when the previous batch of the
run has committed. Deltas that arrive while a batch commits all join the
next batch
> - The benefit is fewer, larger statements under load, with no change
when the database keeps up, and no longer wait for any delta

## Linked Issues or Issue Description

Refs #21460 (batched deltas). Refs #21511 (one statement per delta
batch). Refs #21418 (agent scaling plan and load model).

**What existing behavior does this improve?**
How `PaperclipControlPlanePort` groups queued text deltas into batches
when a batch commit is slower than the batch time.

**Subsystem affected**
Server run event ingest:
`server/src/services/native-runtime/paperclip-control-plane-port.ts`.

**Current behavior**
`#flushStreamingEvents` takes the queued deltas when it runs (on the 250
ms timer, at 100 deltas, or before any other event) and chains their
commit behind the previous batch. If one commit takes two seconds, the
run queues about eight batches, each one statement, behind it.

**Proposed behavior**
`#flushStreamingEvents` chains one cut behind the previous batch and
takes the queued deltas when that cut runs. While a cut waits, later
flush requests join it. A run has at most one batch committing and one
cut waiting.

**Reason and benefit**
Under load, the number of delta statements per run falls to about one
per commit time instead of one per 250 ms. When commits are faster than
250 ms, nothing changes.

**Breaking changes**
None. Rows, receipts, order and the first-text rule are the same.

## What Changed

- `#flushStreamingEvents` queues one cut on the run's commit chain
(`#streamingFlushQueued`). The cut takes every delta queued at that
time, sets the first-text flag, commits the batch, and then releases the
pending count of that batch.
- A flush while a cut waits returns the same chain, so a caller that
waits for it (every other event, `checkpointSession`, `completeRun`,
`replayEvents`, `provesNoRunProviderWork`) still waits for every delta
queued before it.
- A batch can now hold more than 100 deltas when commits are slow. The
100-delta rule still asks for a flush; the batch is one statement with a
fixed number of bind parameters (#21460, #21511).

## Verification

- Red/green: new port test "collects the deltas that arrive while a
batch commits into one next batch, before the next other event"
(`paperclip-control-plane-port.test.ts`). A transaction holds the run
row, so the first batch waits. Six more deltas arrive 15 ms apart with a
5 ms batch time, then a non-streaming event. Before: 8 inserts (one per
timer). After: 3 inserts (the waiting batch, one batch of six, the other
event). Rows are stored 1–9 in order and every receipt resolves.
- The whole port suite with its conformance suite passes: 101 tests.
Other suites, one file at a time: `native-observer-replay` (1),
`paperclip-semantic-conformance` (1), `native-interaction-bridge` (7),
`chat-native-progress-work` (35). I ran the four streaming port tests
two more times; they pass each time.
- `pnpm --filter @paperclipai/server run typecheck` passes. `pnpm
check:module-boundaries` passes.
- Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in
#21418). The model applies the port's batching rules itself, so for
these runs I changed my local copy only (not in this pull request): with
`DELTA_COALESCE=1`, `flushDeltas` queues one cut on the agent's append
chain and takes the queued deltas when the cut runs, as the port now
does. All rows run with #21511. Default pool of 10 connections; embedded
PostgreSQL on the same, shared machine; runs one after the other with a
pause between them. No run had a failed operation, a failed checkpoint
save or a lost lease.

| Agents | Batching | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | this change | 60 s | 9.7 | 0.51 | 3,832 | 151 / 3,087 | 12,000 /
12,000 | 3 / 3,994 |
| 400 | cut on timer (before) | 60 s | 12.7 | 0.74 | 5,465 | 1,129 /
6,040 | 18,719 / 24,000 | 5,721 / 15,238 |
| 400 | this change | 60 s | 12.5 | 0.78 | 7,010 | 892 / 3,131 | 23,993
/ 24,000 | 1,896 / 4,193 |
| 600 | cut on timer (before) | 60 s | 10.0 | 0.69 | 6,158 | 2,357 /
9,483 | 18,771 / 36,000 | 2,280 / 30,985 |
| 600 | this change | 120 s | 11.4 | 0.72 | 6,730 | 1,147 / 6,335 |
61,158 / 72,000 | 11,678 / 21,212 |
| 600 | this change (repeat) | 120 s | 13.6 | 0.74 | 6,498 | 1,225 /
6,057 | 58,984 / 72,000 | 11,465 / 25,361 |
| 800 | this change | 60 s | 11.5 | 0.80 | 6,228 | 2,054 / 7,186 |
24,877 / 48,000 | 24,544 / 31,331 |

The process does more work per second with this change (more other
events finish), so statements per second rise; delta lateness falls. 600
agents are still not carried reliably: other events fall behind. #21528
removes two statements from each such event and one from each checkpoint
save; its description has the stacked results.

## Risks

- A delta never waits longer than before: it already waited for the
previous batch of its run. When the database keeps up, batches are the
same as before.
- Under load, one batch can be large. The live view then gets one larger
step instead of several small steps behind the same wait.
- A failed batch now rejects more deltas at once (every delta that
joined it). The runner resends them, as before; each resend is
idempotent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:00:08 -06:00
DottaandClaude Opus 5.5 0bba04675c perf(run-events): append in-order native events in one statement (#21511)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - After #21460, text deltas commit in batches, but each batch still
takes five statements: BEGIN, the run lock, a replay probe, one insert,
and COMMIT. A single event without a wake takes four
> - In the agent scaling load model, the process issues about 36
statements per agent per second and saturates its one thread between 200
and 400 agents
> - This pull request writes a batch of in-order native events, or one
such event, with one statement that commits on its own
> - The benefit is about half the statements per agent, less CPU per
agent on the main thread, and less time on a pool connection, with the
same rows, receipts, cursor and locks

## Linked Issues or Issue Description

Refs #21460 (batched deltas). Refs #21435 (source cursor). Refs #21418
(agent scaling plan and load model).

**What existing behavior does this improve?**
The append of native source events that continue their source's stored
cursor: batched text deltas, the first delta after another event, and
every other native event that registers no wake.

**Subsystem affected**
Server run event ingest: `server/src/services/heartbeat-run-events.ts`,
and the port comment in `paperclip-control-plane-port.ts`.

**Current behavior**
A delta batch takes BEGIN, a run lock read (`for no key update`), a
replay probe, one insert, and COMMIT: five statements. One delta that
commits at once takes four. In the load model at 200 agents the process
sends about 6,700 statements per second at event-loop utilization 0.72.

**Proposed behavior**
The same append is one statement. Its update of the run row takes the
same row lock. It requires the stored cursor to end just before the
batch, requires the `next_event_seq` that its own snapshot read, and
finds no replayed event. Then it moves the sequence and the cursor and
inserts every row. If any check fails, it writes nothing and the append
takes the existing locked transaction.

**Reason and benefit**
Fewer statements free the main thread and the shared pool for other
agents. In the load model at 200 agents, statements per second fall from
about 6,700 to about 3,300 and event-loop utilization falls from 0.72 to
0.56.

**Breaking changes**
None. No schema change. Rows, receipts and the cursor are the same.
Servers from before this change keep working on the same runs.

## What Changed

- `appendNewNativeSourceEvents` (`heartbeat-run-events.ts`) is one
statement: a CTE reads `next_event_seq` from the statement snapshot, an
`update heartbeat_runs ... returning` locks the run row and moves
`next_event_seq` and the source cursor, and an `insert ... select from
jsonb_array_elements($9)` writes every row. The text is fixed, with nine
bind parameters for any batch size.
- **Why the `next_event_seq` check.** A statement that waits for the run
row lock keeps the snapshot it took before the wait. Its replay probe
cannot see events that the lock holder committed. Every event append
(native, ordinary, the run coordinator store, the retry queue) moves
`next_event_seq` in its own transaction. So the update requires the
value that its snapshot read. PostgreSQL re-checks that condition on the
newest row version after the wait, and any append committed meanwhile
makes the statement write nothing. The unique indexes on `(run_id,
source_event_id)` and `(run_id, source_instance_id, source_seq)` are a
second guard: a unique violation from this statement also falls back.
- A source without a cursor starts at 0. The probe still finds any
stored event of that source, so a run that an older server wrote falls
back and recounts.
- `appendHeartbeatRunEventBatch` and `appendHeartbeatRunEvent` try the
one statement first for native events without a wake, outside a caller's
transaction. Every other case (a replay, a gap, an event with a wake, a
cursor that lags, a run changed meanwhile, a binding mismatch, a run
deleted meanwhile, a caller's transaction) takes the existing
transaction, unchanged.
- Inside a caller's transaction the append keeps its SAVEPOINT path. A
failed statement there would abort the caller's transaction, and the
fallback could not run.
- `assertWritable` runs before the statement is sent and after it
returns (see Risks).

## Verification

- Red/green on embedded PostgreSQL. I ran the new tests against the
previous `heartbeat-run-events.ts` and saw 9 of them fail; all 25 tests
in the file pass with the change
(`server/src/__tests__/heartbeat-run-event-atomic-append.test.ts`):
- One statement for an in-order event and for a batch (before: 4 and 5).
- A writer revoked before the send sends nothing; a statement that fails
on its third row stores no row and no sequence; the resend commits.
- A writer revoked while the statement runs: the rows stay, the receipt
is rejected, and the resend is a replay with the same rows.
- A batch waits for Stop on the run row and appends after it; a batch
and a single event that wait for a run deletion fail with
`heartbeat_run_event_binding_mismatch` and store nothing.
- An older server stores events 2 and 3 while it holds the run row,
without the cursor: the waiting batch 2–4 falls back, returns duplicate,
duplicate, committed, and the cursor ends at 4.
- Two connections append the same 200-event stream in batches of 10:
each event is stored once, in source order, sequences 1–200 without a
gap, one `committed` and one `duplicate` receipt per event.
- A 7,000-event batch (more than any bind parameter limit at 10
parameters per row) is one statement. Before: it fell back to one
statement per event and timed out at 15 s.
- A run without a cursor starts at its first event; a run with events
from an older server replays and then continues.
- Equivalence: 12 random streams of 30 steps (in-order batches, gaps,
replays of earlier ranges, fills of gaps, ordinary events, conflicting
replays). Each step goes to two runs, one through the new path and one
forced through the one-by-one locked path. Receipts, errors, rows,
`next_event_seq` and cursors are equal at every step.
- Mutation check: with the `next_event_seq` check and the
unique-violation fallback removed, the older-server race test fails.
- The port suite (`paperclip-control-plane-port.test.ts`, with the
package conformance suite) passes: 125 tests. Two tests changed: the
streamed-event test now expects one statement and the new revocation
behavior, and the batch test checks that the five deltas are one
statement sent before the boundary event.
- Other focused suites pass, one file at a time:
`heartbeat-run-event-sequencing` (5), `native-event-boundary-golden`
(1), `native-observer-replay` (1), `paperclip-semantic-conformance` (1),
`native-interaction-bridge` (7), `chat-native-progress-work` (19),
`heartbeat-native-restart-suspension` (115), `live-recovery-work` (51).
In 2 of 3 runs with this change, all 51 tests passed but the file's
`afterAll` (closing the side pools and the embedded database) hit its 30
s timeout on this busy machine; the third run passed cleanly, and one
run on `origin/master` passed. The native events in that file all
register a wake, so they do not take the new path.
- `pnpm --filter @paperclipai/server run typecheck` and `pnpm
check:module-boundaries` pass.
- Load model: `scripts/benchmark-agent-scaling-load.ts` as merged in
#21418 (per-source counts stop when the window closes). I did not change
its batching rules for this pull request. Model rates: 20 deltas/s, 1
other event/s, 0.3 API requests/s per agent; default pool of 10
connections; embedded PostgreSQL on the same, shared machine. I ran
"before" (`origin/master`) and "after" one after the other, with a pause
between runs, and I give the load average at the start of each run. The
machine was busy with other work (load 10–15 on 18 cores), so compare
rows that ran close together. No run had a failed operation, a failed
checkpoint save or a lost lease.

| Agents | Code | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 /
12,000 | 278 / 3,895 |
| 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 /
24,000 | 1,904 / 60,154 |
| 400 | after | 60 s | 12.7 | 0.74 | 5,465 | 1,129 / 6,040 | 18,719 /
24,000 | 5,721 / 15,238 |
| 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 /
36,000 | 10,822 / 52,171 |
| 600 | after | 60 s | 10.0 | 0.69 | 6,158 | 2,357 / 9,483 | 18,771 /
36,000 | 2,280 / 30,985 |
| 600 | after, `DELTA_BATCH_MS=0` | 60 s | 10.8 | 0.68 | 8,014 | 36,135
/ 52,222 | 36,000 / 36,000 | 698 / 3,422 |

At 200 agents an earlier pair gave 0.61 → 0.43 event-loop utilization
and 7,051 → 3,210 statements/s. One process with this change alone still
does not carry 600 agents: at 400 and above the 10-connection pool and
the shared database are the limit (event-loop utilization stays below
0.75 while other events fall behind). #21520 and a third pull request
for the remaining statements of other events and checkpoint saves
continue from here; their description has the stacked results.
- Statements: a delta batch 5 → 1; one delta or one native event without
a wake 4 → 1. Unchanged: an event with a wake (6) and a checkpoint save
(5).

## Risks

- **`assertWritable` window (read first).** Before, the writer check ran
inside the transaction, before the first statement and before COMMIT. A
revocation while the statements ran rolled the append back. Now a
revocation while the one statement runs does not roll it back: the rows
commit, and the caller gets the error instead of the receipt, so no
observer runs and the runner gets no ack. This check guards a controller
that hands its run to a successor at restart
(`nativeRunsDetachingForRestart`). I think this is acceptable: (1) the
earlier check was not atomic with COMMIT either, so a revocation during
the COMMIT round trip already committed; the window grows from one
COMMIT round trip to one statement round trip plus its wait for a pool
connection; (2) the append is idempotent by source event id and payload
hash, so the successor sees the committed row as a replay of the event
that the runner resends; (3) the old controller never acts on the
rejected receipt. A new test covers exactly this.
- A crash loses nothing new: the statement commits or not as a whole,
and a resend after an unknown outcome is a replay.
- The check on `next_event_seq` makes the statement write nothing
whenever any other event of the run committed after its snapshot. Under
heavy contention on one run, more appends take the fallback. The
fallback is the previous code path.
- The statement text is fixed SQL sent through the driver
(`$client.unsafe`), like the rest of the event stream statements.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:59:01 -06:00
DottaandClaude Opus 5.5 2178c3df88 fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a recovery read to a remote Runner goes unanswered, the next
attempt of the same run carries `replaceUnansweredRunner`
> - That attempt verified the Runner was alive and then refused to adopt
it anyway. On a resumed sandbox lease the reuse check then failed with
`resumed_not_suspended` and left the Runner running. Otherwise a second
Runner was launched, and its launch fence killed the busy Runner, which
lost the provider turn.
> - After a late reconnect a Runner is often slow, not dead: it is still
replaying the events it saved with no controller attached
> - This pull request makes the attempt check progress first. It adopts
a live Runner that is making progress and replaces only a Runner that
made no progress.
> - The benefit is that a slow but working Runner keeps its turn, and
the retry no longer causes the failure it was meant to fix.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13. The retry after a
first-attempt timeout failed with these codes (counts from the traced
sample of 136 failures):

- `runner_ingress_unavailable: runnerd process failed:
runner_harness_state_mismatch: resumed_not_suspended` (part of 40
`ingress` failures)
- `runner_remote_process_identity_unavailable` (25): the Runner was gone
by the time the retry checked, because the launch fence or a previous
retry had killed it
- "restored the conversation after process loss, but…" /
`provider_turn_lost_on_restore` (10): the replacement Runner restored
from the old state and the provider turn was gone

Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9
failed. Over 420 s: 14 of 15 failed. The process ID of the Runner
changed inside the same sandbox in the failed runs, which shows the
replacement.

Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt
with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit`
verifies the Runner is alive and then returns `undefined` because of
that flag, so `adoptedProcess` is `undefined`.

**Expected behavior**

A live Runner that is making progress is adopted. A Runner that made no
progress is stopped with proof and then restored from its own journal.

**Steps to reproduce**

1. Reattach a remote run whose previous attempt left a recovery read
unanswered (`replaceUnansweredRunner: true`), with the Runner alive and
its saved event sequence past the controller's last acknowledgement.
2. On master the attempt launches a second Runner, or fails with
`resumed_not_suspended` on a resumed sandbox lease.
3. The new tests in `native-session-executor.test.ts` and
`native-runner-restart-recovery.integration.test.ts` do this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): with
`replaceUnansweredRunner`, the attempt adopts the live Runner, launching
nothing and signalling nothing, when:
- the Runner's saved event sequence is past the controller's last
acknowledgement, or
- its saved state (event and command sequences, generation, lifecycle)
changes within a bounded wait of about 10 s.
- The baseline is the first successful state read; every successful read
is also compared against the controller's acknowledgement. A Runner
whose state cannot be read at all during the wait is adopted, not
replaced: missing evidence is not proof of a wedge. A Runner whose
identity changes during the wait fails closed.
- A settled exit during the wait keeps the attempt allowance that
admitted the replacement, so it also restores on the extra fourth
attempt.
- Only a Runner that made no progress is replaced. The relaunch's
existing launch fence stops it and its process lineage with proof, and
the new Runner restores its own journal. A resumed sandbox lease no
longer refuses that relaunch with `resumed_not_suspended`.
- A Runner that exits during the wait follows the existing exited-Runner
rules: a settled suspension restores, and a mid-turn exit stays
fail-closed.
- The wait runs only on this recovery path, never on a normal run start.
- `doc/execution-semantics.md`: describes adopt versus replace.
- The claim flag `replaceUnansweredRunner` is set in claim logic that
another session owns. This PR does not change it; it changes only what
the executor does with it.

## Verification

Red on master, green on this branch:

- `native-session-executor.test.ts`
- `adopts a live Runner that emitted events past the controller's last
acknowledgement`
- `adopts a live Runner whose saved state advances within the bounded
wait`
- `hands a live Runner that made no progress to the relaunch, whose
launch fence stops it, without signalling it here`
- `restores from the journal of a Runner that suspended and exited
during the wait`
- `stays fail-closed for a Runner that exited mid-turn during the wait`
- `native-runner-restart-recovery.integration.test.ts`: `reattaches a
live %s Runner whose recovery read went unanswered on a resumed sandbox
lease` (progressing, wedged)
- Review fixes, `native-runner-restart-recovery.integration.test.ts`
(the first four were red before the fix):
- `adopts a live Runner whose first state read failed and whose later
reads advance`
- `adopts a live Runner whose first readable state is already past the
controller's acknowledgement`
- `does not replace a live Runner whose state could not be read at all
during the wait`
- `restores from the journal of a Runner that suspended and exited
during the wait on the admitted fourth attempt`
- `fails closed when the Runner's identity changes during the wait`
(pins existing behavior)
- `native-remote-retry-admission.test.ts`: unchanged, still passes.

Local runs on this branch: `tsc --noEmit` is clean for `server` on this
branch (master c6b497d831, which contains #21442).
`native-runner-restart-recovery.integration.test.ts`, the
adopt-or-replace block: 12 of 12 pass. The full
`native-session-executor.test.ts` and
`native-remote-retry-admission.test.ts` runs did not finish on this
shared machine (load 20 to 96); CI ran them on this head and they
passed.

Conflict resolved: master added a relaunch branch for a Runner that
exited for a fresh bootstrap (#21374) at the same place. Both branches
are kept; the cheaper check for an unanswered Runner that made no
progress comes first.

Daytona results that exist. They ran on a combined branch with this PR,
#21450, #21442 (merged) and #21362 (merged), so they cannot be credited
to this PR alone:

- G5 baseline (master d1d71a7755; 480 s delay injected on the first
identity write per run): one Claude run reproduced the staging chain:
authentication 419 ms, identity +423.5 s, attempt 1 `native session
provider recovery timed out after 900000ms`, then the retry failed in 6
s with `runner_harness_state_mismatch: resumed_not_suspended`.
- G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished
hands-off. The chain did not recur. All runs that crossed swap 2 were
adopted on the same Runner process ID.
- Caveat: the adopt-or-replace retry path that this PR changes was never
exercised in a cell. In G5c no first attempt failed, because all
identity writes started together 4 to 5 s after the claim. G5c also ran
before the phase timing fix in #21450.

## Risks

- Not proven on Daytona: the adopt-or-replace path itself (see the
caveat above).
- Without #21442 (merged) an adopted Runner can still time out on its
first command while it replays a backlog. This PR stops the replacement;
#21442 (merged) makes the adopted Runner answer.
- A Runner whose state cannot be read during the whole wait is adopted.
If it is in fact wedged, the adopted attempt times out again and the
attempt limit bounds it.
- A wedged Runner that still emits a few events counts as progressing
and is adopted. The existing attempt limit bounds this.
- The progress check adds a wait of up to about 10 s on this recovery
path only. A normal run start has no added wait.
- The claim construction (`native-restart-recovery.ts`) is owned by
another session and is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:56:11 -06:00