Commit Graph
8051 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 28761f9c25 perf(workspaces): read a downloaded snapshot archive once, in the encrypt pass (#21683)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote agents work in sandboxes, and the server saves each sandbox
workspace as an encrypted snapshot in object storage
> - One Node main thread on the server controls every run, so per-agent
byte work on that thread limits how many agents one server can hold
> - The save path read every downloaded archive twice: a main-thread
SHA-256 re-hash after the download, then a full read in the off-thread
encrypt pass that already computes the same digest
> - This pull request removes the main-thread re-hash and checks the
encrypt pass's digest and size against the sandbox receipt instead
> - The benefit is one full archive read per save instead of two, and no
archive hashing on the main thread, with the same integrity guarantee

## Linked Issues or Issue Description

Refs the 1,000-concurrent-agents plan in
`doc/plans/2026-11-13-thousand-concurrent-agents.md` (Phase 1,
"Workspace snapshot" row).

**What existing behavior does this improve?**
Saving a remote workspace snapshot after a run.

**Subsystem affected**
Server: remote workspace snapshot save (`remote-workspace-transport.ts`,
`remote-workspace-snapshot-storage.ts`, `remote-workspaces.ts`).

**Current behavior**
`captureSandboxWorkspaceArchive` streams the whole downloaded archive
through `createHash` on the main thread to compare it with the sandbox
receipt. Then `uploadWorkspaceSnapshot` reads the whole archive again on
a worker thread to encrypt it, and computes the same plaintext digest.

**Proposed behavior**
After the download, the server checks only the file size against the
receipt. The encrypt pass compares its plaintext digest and size with
the receipt. A mismatch fails with the same
`remote_workspace_capture_transfer_mismatch` error before the upload
attempt is recorded and before any object is stored.

**Reason and benefit**
One full read and one SHA-256 of each archive leave the server's main
thread (about 0.36 ms of main-thread time per MB on a loaded dev
machine; 37 ms for a 100 MB archive, at event-loop utilization 1.0 for
that time).

## What Changed

- `captureSandboxWorkspaceArchive` no longer re-hashes the downloaded
archive. It checks with `lstat` that the download is a file with the
receipt's byte size, and fails with
`remote_workspace_capture_transfer_mismatch` when it is not.
- `uploadWorkspaceSnapshot` accepts an optional `expectedArchive`
(`sha256`, `byteSize`). After the encrypt pass, it compares the sealed
plaintext digest and size with it. A mismatch throws
`remote_workspace_capture_transfer_mismatch` before the attempt enters
`metadata.uploads` and before `putFile`. The scratch ciphertext is
removed as before.
- `saveRemoteWorkspace` passes the sandbox receipt as `expectedArchive`
when it captured an archive.
- The save-level `verifyMs` timing field is removed, because that step
no longer exists. The storage-level `verifyMs` stays.
- `doc/run-log-events.md` says that the encrypt step is the only full
read and where a mismatch fails.

Callers that relied on the early hash keep their behavior:
- Unchanged-snapshot reuse (`findUnchangedWorkspaceSnapshot`) compares
the sandbox receipt, not the local hash. When it reuses the parent, the
local copy is used only to list directories, and
`readRemoteWorkspaceArchiveDirectories` verifies the manifest digest.
- Reuse of an earlier upload attempt or an already uploaded snapshot
does not read the archive. Those objects were verified when they were
stored.
- The read-only disk-full fallback builds its archive on the server, so
its receipt always matches. Its behavior does not change.
- The error message and the failure path do not change, so the save
failure code, retry and fallback stay the same.

## Verification

- Red/green: `remote-workspace-transport.test.ts` "leaves the downloaded
archive's single full read to the encrypting upload" counts main-thread
read streams of the downloaded archive during capture. On master it is 1
(fails). With this change it is 0 (passes). The encrypt pass is the one
remaining read.
- New: digest mismatch and size mismatch in `uploadWorkspaceSnapshot`
reject with the transfer-mismatch error, do not call `putFile`, and
leave the row `pending` with no recorded attempt. A retry with the right
receipt saves.
- New: an end-to-end `saveRemoteWorkspace` with a transfer that flips
one byte (same size) fails with the transfer-mismatch error, stores
nothing, keeps the writer and the previous snapshot, and a retry saves
and restores the right content.
- New: a download whose size differs from the receipt fails in capture.
- `cd server && npx vitest run
src/services/remote-workspace-snapshot-storage.test.ts
src/services/remote-workspace-transport.test.ts
src/services/remote-workspaces.integration.test.ts
src/services/remote-workspace-transfer-timeout.test.ts
src/services/remote-workspace-readonly-capture-budget.test.ts
src/services/sandbox-disk-full-fault.test.ts` passes: 6 files, 360
tests. `src/services/remote-workspace-snapshot-reuse.test.ts` passes: 32
tests.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. A same-size corrupt download is now detected in the encrypt
pass instead of right after the download. It is still detected before
any attempt is recorded or any object is stored.
- When the sandbox reports an unchanged workspace, a corrupt local copy
no longer fails the save. The save reuses the verified parent snapshot,
which matches the sandbox's content. The local copy is used only for its
manifest, and the manifest digest is verified.
- The save-level `saveTiming.verifyMs` field is gone. No code reads it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:41:29 -06:00
DottaandClaude Opus 5.5 cf2a4221e3 perf(server): skip the per-request stale instance-admin purge when no row exists (#21680)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Cloud-hosted instances authenticate each request through trusted
Cloud headers in the actor middleware
> - That path deletes stale `instance_admin` rows for the user on every
request, static files and live-events upgrades included
> - On an idle hosted instance this delete ran about 1.2 times a second,
which is a write on an instance-wide table for each request
> - Almost always there is no row to delete, so the statement is wasted
work and a database round trip
> - This pull request finds out if a stale row exists inside the
membership read that the request already does, and deletes only when a
row exists
> - The benefit is one read and no write for each unchanged
authenticated request, with the same purge on the very next request when
a stale row appears

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Cloud trusted-header authentication in the actor middleware.

**Subsystem affected**
Server authentication (`server/src/middleware/auth.ts`).

**Current behavior**
`resolveCloudTenantActor` in `server/src/middleware/auth.ts` sends
`delete from instance_user_roles where user_id = $1 and role =
'instance_admin'` for every trusted-header request. The actor middleware
runs before the static UI, so asset requests run it too. Each
live-events WebSocket upgrade runs it again through `resolveCloudActor`.
The retry wrapper runs it again after a transient connection error.

**Proposed behavior**
An unchanged request sends no write. A stale `instance_admin` row is
still removed on the next request after it appears.

**Reason and benefit**
Idle hosted instances send about one delete for each request. This costs
a round trip and a write statement on the main database for no change.

**Breaking changes**
None. The resolved actor and the authorization decisions do not change.

## What Changed

- The Cloud trusted-header path reads the user's active memberships
together with an `exists` check for an `instance_admin` row, in one
statement.
- The purge runs only when that check finds a row, when the user has no
active membership row yet (first request), or when the read fails.
- The session path and the Cloud path share one membership filter
function, so both still resolve the same access set.
- New test `server/src/__tests__/cloud-tenant-actor-statements.test.ts`
meters the statements of one authenticated request on the embedded test
database.

## Verification

- `cd server && npx vitest run
src/__tests__/cloud-tenant-actor-statements.test.ts
src/middleware/cloud-tenant-actor.test.ts
src/__tests__/cloud-tenant-actor-round-trips.test.ts`: 41 passed.
- Red/green: with the `origin/master` version of `auth.ts`, the new test
fails. An unchanged member request sends 2 statements, and one of them
is the delete. With this change it sends 1 read and 0 writes.
- The test inserts a stale `instance_admin` row between two requests.
The next request deletes it, and the request after that sends no write
again.
- The test changes the stack role from member to admin and back. The
membership role and the experimental-settings access change on the very
next request each time.

## Risks

- Low risk. The actor that the middleware returns is the same.
Authorization decisions do not change.
- The purge now runs after the membership read, not before the tenant
sync. The sync does not touch `instance_user_roles`, so the order does
not matter. The request still waits for the purge before it continues.
- A race between the read and a new row insert existed before too
(delete, then insert). The next request removes such a row.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:34:30 -06:00
DottaandClaude Opus 5.5 dcba9628c7 perf(recovery): keep a finished task's settled attention settled and halve graph hint fan-out (#21676)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Issue recovery records "recovery actions" on tasks. A sweep resolves
the active action of a task that is already done or cancelled.
> - On an idle hosted instance, a debug-log run (from #21640) named a
ping-pong. A native worker re-surfaces a board attention for 13 finished
tasks. Recovery resolves it as `source_terminal`. The unique "one active
action" index then lets the worker insert it again, about every 26
seconds per task. That built 45,008 rows since 2026-10-21. Each insert
and each resolve also signals run dispatch and the task's recovery
graph, which wakes both sides again.
> - Separately, every task-graph recovery hint is published twice per
task queue: once as the graph hint and once as a `{companyId, backstop:
true}` twin. That is half of all idle signals, for a backstop that is
time-gated anyway.
> - This pull request breaks the ping-pong from the recovery side and
removes the backstop twin.
> - The benefit is that a finished task's settled attention stays
settled and wakes nothing. Each graph hint costs half the signals.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

With zero runs active, recovery and native workers kept re-creating and
re-resolving the same board attention on 13 done or cancelled tasks. The
rows are `active_run_watchdog`, owner `board`, cause
`native_restart_recovery_blocked`. The loop added about one row per task
every ~26 s and about 2.6 work-signal commits per second.

**Expected behavior**

Once recovery settles a finished task's attention, the same attention
identity is not surfaced again, and settling it wakes nothing that a
finished task cannot use.

**Steps to reproduce**

1. Create a cancelled task.
2. Surface the native "controller process space unverified" attention
for it (same cause and fingerprint each time), then run
`reconcileActiveRecoveryActions`.
3. Repeat step 2. On master each cycle inserts a new active row and
resolves it, and each resolve signals the task queues and run dispatch.

**Paperclip version or commit**

master at 36101ebcb9.

## What Changed

- `server/src/services/issue-recovery-actions.ts`:
- **No-match skip.** `resolveActiveForIssue` first reads whether any
action matches. When none does, it returns `null` with no transaction
and no signal. Before, `writeExecutionControl` always published
run-dispatch, graph-recovery and execution hints first.
- **Settled-attention guard.** `upsertSourceScoped` returns the existing
resolved action instead of inserting when ALL of these hold:
    - the task is done or cancelled;
    - it has no active action;
- an action with the same cause and fingerprint was already resolved
with `source_terminal`.
The guard writes no row and sends no signal, and the caller gets that
resolved action. It covers only the exact identity that recovery itself
settled because the task finished. A new fingerprint (renewed controller
authority), a different cause, or a reopened task still creates an
action.
- **Quiet terminal resolve.** A new `sourceTerminal` resolve mode. It
updates the row only while the task is still done or cancelled. It still
notifies native recovery notices, unsafe-workspace export recovery, and
a preserved native finalization waiter. It does NOT signal run dispatch,
the task's recovery graph, or the execution disposition/continuation
queues: a finished task has no run to dispatch and no assignment to
recover.
- `server/src/services/recovery/service.ts`: the `source_terminal`
branch of `reconcileActiveRecoveryActions` uses `sourceTerminal: true`.
- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery` publishes one graph hint per task queue and
no longer adds a `{companyId, backstop: true}` hint per queue. A new
`issueRecoveryBackstopHint(companyId)` builds that hint string.
- `server/src/services/recovery/work.ts`: when a task queue loads a
graph scope, it arms its own backstop in process
(`worker.invalidate(issueRecoveryBackstopHint(companyId))`). The #21552
guarantee stays: any graph activity arms the company backstop, which
runs at most once per backstop interval after the last full pass. It no
longer costs one signal per queue per hint.
- `server/src/__tests__/issue-recovery-actions.test.ts`: the fake-DB
unit tests accept the guard's extra read.
- `server/src/__tests__/issue-recovery-signal-fanout.test.ts` (new, real
PostgreSQL), six tests:
  - one graph hint per task queue and no backstop hint;
  - a resolve with no matching action returns null and signals nothing;
- an escalated action on a finished task is still resolved and still
signals;
- the ping-pong cycle: surface then reconcile, five times, leaves one
row and no stranded or active-recovery signals;
- settling a finished task's attention sends no task-queue, run-dispatch
or active-recovery signals;
- a renewed authority and a reopened task still get a new attention,
which stays active on a reopened task.

## Verification

- New test file:
  - master: 4 of 6 fail (red).
    - 8 task-queue hints per graph hint instead of 4.
    - A no-match resolve publishes 18 settled signals.
- Cycle 2 inserts a new active row; master builds a new row every cycle.
- Settling a finished task's attention signals the task queues and run
dispatch.
  - This branch: 6 of 6 pass (green).
- Regression on real PostgreSQL:
- heartbeat-native-restart-suspension, native-recovery-notice-work,
issue-recovery-deadlines (including the #21552 backstop tests),
assigned-work-invariant, execution-control-work, issue-recovery-actions,
issue-recovery-work-lifecycle, issue-recovery-assignments,
execution-recovery-resolution, automatic-run-brake-recovery-passes: all
pass (381 tests).
- Four unit tests in issue-recovery-actions use a hand-written fake DB.
They needed `innerJoin` on the fake query and an expected count of two
selects, because the guard adds one read.
  - recovery-deadline-acceptance: passes.
- `tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- The guard returns a resolved action to a caller that asked to surface
one. I checked the callers: the one that loops
(`surfaceNativeControllerNamespaceAttention`) ignores the return value.
The others only reach the guard for a finished task whose identical
attention recovery already settled, where re-surfacing it would be
resolved again within one sweep.
- The terminal resolve no longer wakes run dispatch or the task's graph.
Completing or cancelling a task already sends those signals itself.
- This does not stop the native worker from trying to surface attention
for finished tasks. That fix belongs in the native owner-release code,
which another change owns.
- The ~45k historical resolved rows on the hosted instance stay as they
are. A cleanup would need a migration or an operator script that deletes
resolved `native_restart_recovery_blocked` rows whose source is done or
cancelled, keeping the newest per (task, fingerprint) so the guard still
matches. That is not part of this change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21552, #21621, #21640.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:06:38 -06:00
DottaandClaude Opus 5.5 97a167093d fix(queue): send a message into a starting run without a cancel or recovery errors (#21661)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can queue a message while an agent works, then send it into
the running turn
> - A run that is still starting takes queued messages into its first
turn, and Interrupt hands off to them
> - Both rules only accepted a queue that a user started
> - A user's message can merge into an earlier automatic wake that waits
behind the task, and that wake keeps its system requester
> - Then the starting run did not take the message, Interrupt cancelled
the boot, and the task showed "Couldn't finish this attempt" with Retry
and "Waiting for execution recovery" under the composer until the
cancelled boot settled
> - This pull request makes both rules judge the messages and the
recorded Interrupt intent, not the wake's requester, and shows an
Interrupt hand-off as sent
> - The benefit is that sending a queued message feels immediate and
shows no error

## Linked Issues or Issue Description

**What happened?**
A run started from a resolved blocker. The user queued a message 20
seconds later, while the runner was still starting. The message merged
into an older automatic deferred wake. The queue offered Interrupt
instead of sending it. Interrupt cancelled the starting run. For about 7
seconds the task showed "Couldn't finish this attempt. Your message is
saved." with Retry, and the composer showed "Waiting for execution
recovery. Your message is saved." Then the message was delivered.

**Expected behavior**
The starting run takes the message into its first turn. When an
Interrupt does stop a run, the message shows as sent, with no error
banner, Retry, or recovery text.

**Steps to reproduce**
1. Have an automatic deferred wake for a task (for example from an
earlier blocked execution).
2. Start a run for the task with a Paperclip runner agent.
3. While the runner is still starting, queue a message and click the
send control.

## What Changed

- `deferredWakeJoinsFirstTurn`: a system or unattributed wake qualifies.
Each message must still be the responsible user's own
(`firstTurnEligibleComment`). A wake from another user or an agent keeps
its own delivery.
- Task interruption presentation: find the Interrupt queue by its
recorded `queuedCommentInterrupt` intent, not by `requestedByActorType =
user`. The hand-off now shows as working for a merged queue too.
- Queued-comment read: while a deferred queue carries an Interrupt
intent, report an `interrupt_handoff` wait instead of the recovery
message.
- UI (`TaskChatQueuedMessages`, `IssueDetail`): an `interrupt_handoff`
response counts as sent. The messages stay out of the queue and no
status or error line shows.

## Verification

Red/green tests. Each failed before the fix and passes after it:
- `server/src/__tests__/issue-queued-comments-routes.test.ts`: "gives a
booting run's first turn the user's message merged into an earlier
automatic wake"; the Interrupt fallback case now expects
`interrupt_handoff` on the response and on a poll. 144/144 pass.
- `server/src/__tests__/task-interruption-query.test.ts`: the hand-off
case runs for a user-started and a system-started queue. 32/32 pass.
- `ui/src/components/task-chat/TaskChatQueuedMessages.test.tsx`: "shows
an Interrupt as sent while the stopped turn hands off". 43/43 pass.
- `first-turn-queued-input.test.ts` 8/8, `issue-queued-comment-queue`
7/7. Server and UI typecheck pass.

Root cause from the staging wake diagnostics: the run started at
20:26:54, the message was coalesced at 20:27:17 into a deferred
`issue_execution_deferred` automation wake from the previous day, the
Interrupt at 20:27:20 recorded `execution_recovery`, and the
false-positive hold resolved at 20:27:26.

Not run: local runner E2E steering cells. The OpenCode chat-features
cells require the trusted CI host. The Codex and Claude
`revise-while-running` chat cells failed before steering on a fixture
workspace path (`workspaces/` vs `isolated-workspaces/`), which this
change does not touch. Staging verification follows the deploy.

## Risks

- Low. A wider set of deferred wakes can join a starting run's first
turn. Message eligibility, the claim receipt, and exactly-once delivery
are unchanged.
- An Interrupt intent that never delivers shows "sent" in the composer.
The task notice still shows a failure with Retry after its existing
grace period.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:06:15 -06:00
DottaandClaude Fable 5.2 ac114d9479 refactor: cut DEVELOPING.md to a setup and workflow guide (#21670)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `AGENTS.md` tells every contributor and agent to read
`doc/DEVELOPING.md` for setup, commands, testing, and worktrees
> - That file grew to 5,495 lines. Most of it was notes on how one
subsystem behaves, added one pull request at a time
> - Readers could not find the commands. Agents also read behavior notes
as rules
> - This pull request cuts `doc/DEVELOPING.md` to a 660-line setup and
workflow guide. It moves the subsystem notes to focused docs
> - The benefit is a guide that a person or an agent can read in full,
and a clear place for each kind of note

## Linked Issues or Issue Description

**Issue type**

Outdated or confusing documentation.

**Where is the issue?**

`doc/DEVELOPING.md`

**What's wrong?**

The file was 5,495 lines. It opened with 112 lines about case handoffs
before it reached Prerequisites. Unrelated sections sat below the
"OpenClaw Docker UI One-Command Script" heading. Statements about
current behavior read like policy.

**Suggested fix**

Keep setup, commands, and workflow in `doc/DEVELOPING.md`. Move each
subsystem note to a doc for that subsystem. Add an index.

## What Changed

- `doc/DEVELOPING.md` is now 660 lines: prerequisites, start dev, health
checks, local run, test drive, Docker, instance layout, database,
storage, secrets, worktrees, test commands, Storybook, CLI, and an index
of the other docs.
- 15 new docs hold the moved text without wording changes:
`ci-workflows`, `test-infrastructure`, `worktrees`,
`hot-restart-deploys`, `heartbeat-internals`, `runner-internals`,
`workspace-repositories`, `workspace-storage`,
`skills-and-teams-catalog`, `ui-internals`, `case-handoffs`,
`chat-connector-development`, `instance-configuration`,
`integration-api-notes`, `database-backups`.
- "Publish a branch Storybook" moved to `doc/STORYBOOK-DEPLOYMENT.md`.
The privacy rollout notes moved to `doc/ISSUE-PRIVACY.md`.
- Deleted about 280 lines: the story-by-story Storybook notes, "Agent
Artifact Uploads" (a copy of `doc/AGENT-ARTIFACTS.md`), two pointer-only
sections, "Workspace feature checks (2026-10-09)", and "Chat and task
navigation caching". The test-drive build-lock paragraph is now four
lines.
- Repointed every inbound link and every script or workflow comment that
named a moved section.
- `AGENTS.md` now says where a behavior note goes, so the guide does not
grow again.

## Verification

- A script split the file by line range. It asserts that each of the
5,495 source lines is kept, moved, or dropped exactly one time.
- A link check on every changed Markdown file found 0 missing files and
0 missing anchors.
- `node scripts/check-maps.mjs` passes.
- `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail.
- `node --check` passes on the three scripts with changed comments.
- `pnpm verify` was not run. This worktree has no installed
dependencies, and the change has no runtime code.

## Risks

- Low risk. The change is documentation and comments only.
- A link to a moved section from outside this repository will land on
the top of `doc/DEVELOPING.md`.
- The new docs are not in the audited prose list in
`.github/scripts/ci-inputs.mjs`. An edit to one of them runs full CI. An
edit to `doc/DEVELOPING.md` does not. This PR does not change that list.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool
use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 18:02:27 -06:00
DottaandClaude Opus 5.5 648b238be1 perf(native-runtime): stop re-finalizing native results whose finalization error is recorded (#21674)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime releases owners of failed runs. When the owner's
controller is in a process space that Paperclip cannot verify, it shows
controller namespace attention on the task instead.
> - On a hosted instance with zero active runs, this writer made about
21% of all work signals. It inserted a new active attention row for the
same task about every 26 seconds.
> - The tasks were already done or cancelled. Recovery resolved each row
as `source_terminal`. The active slot was then free, so the next
owner-release pass inserted a new row and published a graph recovery
signal again. The loop did not stop.
> - A live task had the same defect: after the fresh-continuation
fallback resolved its attention, the next pass made the same incident
again.
> - This pull request stops the writer from surfacing attention for
finished tasks. It also stops the writer from making an incident again
when that run authority was resolved after its lease expired. Those
owners are no longer release candidates and do not set the release
deadline.
> - The benefit is that an idle instance does not write or publish
anything for these runs. Attention still shows once for a live task.

## Linked Issues or Issue Description

**What happened?**
The failed-run owner release (`settleFailedRunOwner`, expired-owner
path) called `surfaceNativeControllerNamespaceAttention` for every
expired foreign owner that had no active attention. It did this even
when the source task was `done` or `cancelled`. Recovery resolved that
attention as `source_terminal`. The next pass found no active attention
and inserted a new board row. Each insert published
`notifyIssueGraphRecovery` and the other execution-control signals. The
stranded-task check then ran again for the task graph. A hosted instance
collected about 45,000 rows on 13 finished tasks in this way.

**Expected behavior**
A pass that changes nothing writes nothing and publishes nothing. A
finished task never gets controller namespace attention. A live task
gets one incident for each exact run authority. A resolved incident
stays resolved until the authority changes.

**Steps to reproduce**
1. Make a native run that failed, with an `observed` coordinator whose
foreign lease has expired and whose controller host is not recorded.
2. Mark the task done or cancelled, or let the fresh-continuation
fallback resolve the attention of a live task.
3. Run the execution-finalization worker with the issue recovery workers
for 30 minutes.
4. On master, each task gets about 40 to 60 new attention rows. The
writer publishes thousands of signals.

## What Changed

- `surfaceNativeControllerNamespaceAttention` (shared helper): before it
creates attention, it checks for a resolved incident for this run that
was resolved at or after the owner's current lease expiry. If one
exists, it writes nothing.
- New `settledNativeControllerNamespaceAttention` SQL predicate. It is
shared by the helper, the release candidate page, and the release
deadline query, so all three use the same rule.
- `settleFailedRunOwner` (expired-owner path) does not surface attention
when the task is `done` or `cancelled`. The restart-recovery claim path
has the same guard.
- `releaseAbandonedNativeOwners` and `nextAbandonedNativeOwnerReleaseAt`
skip foreign owners whose task is finished or whose incident is settled.
A pass that has nothing to do selects nothing, and no immediate deadline
is set again. Local owners keep the normal proof-based release.
- The owner release proofs did not change. A foreign owner is still
never released without proof.
- Tests:
`server/src/__tests__/native-failed-run-owner-release-idle.test.ts`
(real PostgreSQL, real coordinator and workers, fake clock, statement
and signal counters) and three new cases in
`native-failed-run-owner-release.test.ts`.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/native-failed-run-owner-release-idle.test.ts
src/services/native-runtime/native-failed-run-owner-release.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/services/native-runtime/native-restart-recovery-isolation.test.ts`:
80 tests pass.
- Red on master, green with this change. The test seeds one done task
and one cancelled task (each with an earlier resolved `source_terminal`
incident) and one live task, then runs 30 idle minutes:
- master: 126 attention inserts, 249 attention writes, 7,024 signals,
8,025 statements.
- this branch: 1 attention insert (the live task, surfaced once), 2
attention writes (that insert and the fallback's resolution), 36
signals, 350 statements. All of this is startup work. Nothing repeats.
- After the fresh continuation settles a live task's attention, six
unrelated finalization wakes over 30 minutes:
- master: 3 attention inserts, 90 signals, 551 statements, and the
release deadline stays due.
- this branch: 0 attention writes, 0 run writes, 6 signals (the test's
own wakes), 153 statements, and the release deadline is null.
- New unit cases: a done or cancelled task never surfaces, and surfaces
once when it is reopened. A settled incident stays settled over repeated
passes. A renewed authority (a new lease expiry) surfaces one new
incident.
- The three new unit cases fail on master and pass on this branch.
- `tsc --noEmit` for `server` passes.

## Risks

- Low risk. The change only removes writes. It does not weaken a release
proof, an authorization, a budget, a pause, or an approval check.
- An incident that a person resolved or dismissed after the lease
expired is not made again for the same authority. A renewed authority (a
later lease expiry) still makes a new incident.
- A superseded incident (status `cancelled`) still shows again when the
superseding action clears. This keeps the current behavior.
- The historical rows that the loop made are not removed. This change
only stops new rows.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21621, #21616, #21617

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1115.0-canary.0
2026-11-14 17:50:45 -06:00
DottaandClaude Opus 5.5 64c7122dae perf(native-runtime): stop re-surfacing settled controller namespace attention (#21675)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime releases owners of failed runs. When the owner's
controller is in a process space that Paperclip cannot verify, it shows
controller namespace attention on the task instead.
> - On a hosted instance with zero active runs, this writer made about
21% of all work signals. It inserted a new active attention row for the
same task about every 26 seconds.
> - The tasks were already done or cancelled. Recovery resolved each row
as `source_terminal`. The active slot was then free, so the next
owner-release pass inserted a new row and published a graph recovery
signal again. The loop did not stop.
> - A live task had the same defect: after the fresh-continuation
fallback resolved its attention, the next pass made the same incident
again.
> - This pull request stops the writer from surfacing attention for
finished tasks. It also stops the writer from making an incident again
when that run authority was resolved after its lease expired. Those
owners are no longer release candidates and do not set the release
deadline.
> - The benefit is that an idle instance does not write or publish
anything for these runs. Attention still shows once for a live task.

## Linked Issues or Issue Description

**What happened?**
The failed-run owner release (`settleFailedRunOwner`, expired-owner
path) called `surfaceNativeControllerNamespaceAttention` for every
expired foreign owner that had no active attention. It did this even
when the source task was `done` or `cancelled`. Recovery resolved that
attention as `source_terminal`. The next pass found no active attention
and inserted a new board row. Each insert published
`notifyIssueGraphRecovery` and the other execution-control signals. The
stranded-task check then ran again for the task graph. A hosted instance
collected about 45,000 rows on 13 finished tasks in this way.

**Expected behavior**
A pass that changes nothing writes nothing and publishes nothing. A
finished task never gets controller namespace attention. A live task
gets one incident for each exact run authority. A resolved incident
stays resolved until the authority changes.

**Steps to reproduce**
1. Make a native run that failed, with an `observed` coordinator whose
foreign lease has expired and whose controller host is not recorded.
2. Mark the task done or cancelled, or let the fresh-continuation
fallback resolve the attention of a live task.
3. Run the execution-finalization worker with the issue recovery workers
for 30 minutes.
4. On master, each task gets about 40 to 60 new attention rows. The
writer publishes thousands of signals.

## What Changed

- `surfaceNativeControllerNamespaceAttention` (shared helper): before it
creates attention, it checks for a resolved incident for this run that
was resolved at or after the owner's current lease expiry. If one
exists, it writes nothing.
- New `settledNativeControllerNamespaceAttention` SQL predicate. It is
shared by the helper, the release candidate page, and the release
deadline query, so all three use the same rule.
- `settleFailedRunOwner` (expired-owner path) does not surface attention
when the task is `done` or `cancelled`. The restart-recovery claim path
has the same guard.
- `releaseAbandonedNativeOwners` and `nextAbandonedNativeOwnerReleaseAt`
skip foreign owners whose task is finished or whose incident is settled.
A pass that has nothing to do selects nothing, and no immediate deadline
is set again. Local owners keep the normal proof-based release.
- The owner release proofs did not change. A foreign owner is still
never released without proof.
- Tests:
`server/src/__tests__/native-failed-run-owner-release-idle.test.ts`
(real PostgreSQL, real coordinator and workers, fake clock, statement
and signal counters) and three new cases in
`native-failed-run-owner-release.test.ts`.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/native-failed-run-owner-release-idle.test.ts
src/services/native-runtime/native-failed-run-owner-release.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/services/native-runtime/native-restart-recovery-isolation.test.ts`:
80 tests pass.
- Red on master, green with this change. The test seeds one done task
and one cancelled task (each with an earlier resolved `source_terminal`
incident) and one live task, then runs 30 idle minutes:
- master: 126 attention inserts, 249 attention writes, 7,024 signals,
8,025 statements.
- this branch: 1 attention insert (the live task, surfaced once), 2
attention writes (that insert and the fallback's resolution), 36
signals, 350 statements. All of this is startup work. Nothing repeats.
- After the fresh continuation settles a live task's attention, six
unrelated finalization wakes over 30 minutes:
- master: 3 attention inserts, 90 signals, 551 statements, and the
release deadline stays due.
- this branch: 0 attention writes, 0 run writes, 6 signals (the test's
own wakes), 153 statements, and the release deadline is null.
- New unit cases: a done or cancelled task never surfaces, and surfaces
once when it is reopened. A settled incident stays settled over repeated
passes. A renewed authority (a new lease expiry) surfaces one new
incident.
- The three new unit cases fail on master and pass on this branch.
- `tsc --noEmit` for `server` passes.

## Risks

- Low risk. The change only removes writes. It does not weaken a release
proof, an authorization, a budget, a pause, or an approval check.
- An incident that a person resolved or dismissed after the lease
expired is not made again for the same authority. A renewed authority (a
later lease expiry) still makes a new incident.
- A superseded incident (status `cancelled`) still shows again when the
superseding action clears. This keeps the current behavior.
- The historical rows that the loop made are not removed. This change
only stops new rows.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21621, #21616, #21617

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:45:02 -06:00
DottaandClaude Opus 5.5 fc9e4cbbb3 perf(ui): show a returning task's saved chat before any server response (#21671)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI opens a task as a chat. A returning browser restores a
saved read cache from local storage before the first render.
> - #21650 made a returning hard load share the one chat bootstrap. But
the saved cache held no chat, so the first chat frame still waited for
that response.
> - The owner approved saving recent comments in the browser on
2026-11-14, so that cached tasks show their chat at once.
> - This pull request saves the first chat frame of the 20 most recently
opened tasks, with size, age, revocation and isolation guardrails. The
restored chat shows at once and revalidates through the bootstrap.
> - It also adopts the hard-load bootstrap at app start and sends the
other live-run readers through it, so readers that start before the task
page mounts do not send their own requests.
> - The benefit is that a hard load of a task the user opened before
shows its conversation before any server response.

## Linked Issues or Issue Description

**Describe the enhancement**
A hard load of a task that the user opened before still waits for one
server round trip before its chat appears. The saved read cache did not
include comments.

**Current behavior**
With every API response delayed by 800 ms, a returning load shows the
chat at 892 ms (median). The frame waits for the chat bootstrap
response.

**Proposed behavior**
The saved read cache also keeps the first chat frame of recently opened
tasks. A returning load shows that frame before any response, then
revalidates through the bootstrap.

**Why**
The owner approved saving recent comments in the browser for this on
2026-11-14. Cold and cached loads must both be as fast as possible.

Related: #21650 (returning load shares the chat bootstrap), #20594 and
#20591 (returning inbox from the saved cache), #20592 and #21557 (cold
task load from one bootstrap).

## What Changed

- `ui/src/lib/persisted-query-cache.ts`: saves the reads the first chat
frame waits for (`taskChatFirstFrameRef`): recent comments, their input
facts, recent activity, the recent run window, interactions, attachment
metadata, live and active runs, work products and the plan document.
- Only for the 20 saved recent tasks, in the record of the task's own
user and company.
- At most 50 comments per task (the newest; `hasMore` is set), 192 KiB
per task, 1 MiB in total, newest tasks first.
- If storage refuses a record, the record is saved again without chat,
so chat never displaces the rest of the cache.
- All other field exclusions (tokens, credentials, environment values,
embedded documents) still apply. Only `comments` is allowed, and only
for these keys.
  - Expiry is the same as every other entry: 24 hours after the read.
- A 401, 403 or 404 on the task or on any of its chat reads removes the
task and its saved chat from memory and storage at once.
- `ui/src/lib/issue-chat-bootstrap.ts` and `ui/src/main.tsx`:
`adoptHardLoadChatBootstrap` adopts the bootstrap that the HTML started,
at app start, under the task's restored id and identifier. Readers that
start before the task page mounts share it.
- `IssueDetail.tsx`, `LiveRunWidget.tsx`, `IssueRunLedger.tsx`,
`useTaskPluginPanels.ts`: their `live-runs` reads share the loading
bootstrap. Before, whichever observer mounted first could send its own
request.
- `doc/DEVELOPING.md`: the read-cache rules now state what is saved, the
bounds, why, and the owner's approval on 2026-11-14.
- Tests: `ui/src/lib/saved-task-chat.test.ts` (guardrails), a new case
in `restored-task-chat-bootstrap.test.ts`, and an always-on perf test:
"a returning task load shows its saved chat before any API response,
then revalidates".

## Verification

Local production build, local_trusted instance, CDP delay on every API
request, fresh browser context per run, JavaScript in the HTTP cache.
Time to the revealed history with the latest message and an editable
composer. Saved read cache present against cleared (other storage kept),
5 alternating pairs each.

| API delay | Build | Cleared | Present | Requests before frame
(present) |
| ---: | --- | ---: | ---: | ---: |
| 800 ms | master `36101ebcb9` | 1,911 ms (1,895–1,920) | 892 ms
(887–898), released by the bootstrap | 33 |
| 800 ms | this PR | 1,922 ms (1,912–1,936) | 571 ms (556–588), before
any response | 36 |
| 300 ms | master | 916 ms (907–920) | 570 ms (564–576) | 37 |
| 300 ms | this PR | 928 ms (910–998) | 586 ms (568–622) | 37 |

- At 300 ms the bootstrap answers before the page mounts on both builds,
so they are the same. At 800 ms the saved chat removes the wait for the
bootstrap: −321 ms against master.
- The new perf test holds every response for 2.5 s. It fails on master
(frame at 2,595 ms, first response at 2,524 ms) and passes with this PR.
- Guardrail tests, red/green: I disabled each guardrail in turn in
`persisted-query-cache.ts` and ran `saved-task-chat.test.ts`. Each
mutant made its test fail, and all 19 tests pass with the code. Mutants:
no chat saved, no 50-comment cap, no 20-task cap, no per-task budget, no
total budget, no retry without chat when storage refuses, no 24-hour
expiry, no revocation on a chat read, chat stored in the wrong company
record, and no field exclusions.
- Sign-out and other-user isolation use the existing per-user records
and their clearing (no new code). The tests cover them for saved chat.
- "Never overwrites newer data" is the query cache's hydrate rule; the
test covers it. "The bootstrap replaces saved data" uses the
restored-entry rule from #21650; the test covers it.
- App-start adoption test: it fails when the adoption call is removed,
and passes with it.
- Focused Vitest: 25 files that use the changed modules, including
`IssueDetail.test.tsx` and `TaskChatThread.test.tsx`: 1,156 passed.
`pnpm --filter @paperclipai/ui typecheck` passes.
- Browser journeys against this build: `layout-stability.spec.ts`,
`plan-startup.spec.ts`, `cold-bootstrap.perf.spec.ts`: 20 passed (2
opt-in benchmarks skipped).

Leftover early reads on staging (one run after #21650 started
`issues/:ref?view=core` and `issues/:ref/live-runs` before the frame): I
could not reproduce them locally. That included sidebar Recents and a
bootstrap delayed to 1.2 s with CDP initiator stacks. Both reads would
go to the server only if no bootstrap was being tracked for the task, or
if an observer used a direct read. This PR removes both openings: the
bootstrap is adopted at app start, and every `live-runs` observer shares
it. I will check the request timeline on staging after deploy.

## Risks

- Privacy: recent comments of up to 20 recent tasks now stay in local
storage for up to 24 hours, or until sign-out. The owner approved this
on 2026-11-14. The bounds, the revocation and the field exclusions are
tested. Saved data is per user and company, and sign-out clears it in
every tab.
- Storage: saved chat is limited to 1 MiB in total. A record that
storage refuses is saved again without chat.
- A saved chat can be up to 24 hours old when it shows. It is marked for
revalidation, and the bootstrap answer replaces it within one round
trip. A refused revalidation removes it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser automation through
Playwright and Chrome).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:38:28 -06:00
DottaandClaude Opus 5.5 36101ebcb9 fix(evals): grade five cohort-14 chat cases by saved state and clicked links (#21667)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat-feature eval catalog runs real chat journeys. It grades
saved state, rendered links and run ledgers.
> - Chat-eval cohort 14 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `d4498ae7ac`)
failed five cases where the agent behaviour was correct or acceptable.
> - Four failures came from graders that read free-form reply text or
required a setup label to be absent. One failure came from a harness
race with a client-side route rewrite.
> - This pull request fixes those graders and that harness step. It does
not change the product.
> - The benefit is that these cases measure the feature again: the
retained correct replies pass, and the relevant broken behaviour still
fails.

## Linked Issues or Issue Description

**What happened?**
In chat-eval cohort 14 (run 43637708886), five cases failed although the
agent behaved correctly:
- `projects.share-project`: a regex over the free-form reply rejected a
correct board-only handoff that linked the project access settings.
- `teams.install-team`: the grader required the literal skill slug, name
or key. The reply named the blocked skill "document-maintenance"; the
slug is `doc-maintenance`.
- `steering.discard-queued-message`: the final plan removed the contact
section but cited the setup label in a "Latest direction" note. The
grader required the label to be absent everywhere.
- `workspaces.inspect-git`: the grader required the reply to name
`existing-notes.txt`. The person asked about "website files", and the
agent correctly left the unrelated file out.
- `runs-adapters.reset-session`: the harness opened the agent actions
menu on `/agents/<id>/runtime`. The app then rewrote the route to
`/agents/<urlKey>/runtime` and redrew the page, so the "Reset sessions"
click timed out.

**Expected behavior**
Correct agent behaviour passes. Grading uses saved state, setup literals
and rendered or clicked links, not reply wording. Broken behaviour (an
agent-made grant, a kept contact section, a changed unrelated file, a
missing first reply) still fails.

**Steps to reproduce**
1. Use the retained cohort-14 evidence for the five cases.
2. Run the old graders against the retained replies and revisions. Each
case fails.
3. For reset-session, open an agent profile by id while the app rewrites
the route after a delay, then open the actions menu before the rewrite
completes.

**Paperclip version or commit**
master `d4498ae7ac` (cohort 14); fixed on top of current master.

## What Changed

- `projects.share-project` (`internal-privacy.ts`): removed the reply
regex. The grade now requires (a) a rendered link to the project's
configuration page (where Manage access lives) that the harness clicks
and that opens the Manage access control, and (b) saved state with no
agent-made access grant: Riley is absent from the membership before the
owner acts, and no agent `project.access_member_added` or visibility
`project.updated` activity exists. The owner step still completes when
the link is missing, so later phases keep their evidence.
- `teams.install-team` (`external-team-conditional.ts`): removed
`namesTeamSkill`. The case contract is "reports conflicts or partial
failures truthfully"; truthfulness is meaning, so a person reviews it.
The partial outcome is already graded from saved state: the catalog
skill is still missing beside the local skill that holds its slug, and
the installer recorded a warning. A skill link was not chosen because
the blocked catalog skill has no installed page to link, and the
contract only asks for links to the installed agents and project. The
check is now "a first reply was delivered".
- `steering.discard-queued-message` (`runtime-discard.ts`): the later
contact request now names a contact person and an attempt-specific phone
(a setup change in the harness act step, not the YAML). The grade
collects the contact section's distinctive whole lines from the revision
that saved it: lines in the section whose heading carries the label,
contact name or phone, plus lines that carry one of those markers,
excluding lines that also appear elsewhere in that revision. None of
those whole lines may remain in the final revision, and the contact name
and phone may not appear anywhere in it (substring match on those
distinctive literals). A note that quotes the label inside a longer
line, and shared lines such as `---`, do not count.
- `workspaces.inspect-git` (`runtime-git.ts`): stopped requiring
`existing-notes.txt` in the reply. The question asks about website
files, and the unrelated file's bytes are still checked unchanged. No
YAML change, so no TestOps manifest change.
- `runs-adapters.reset-session` (`runtime-support.ts`):
`openRuntimeAgentProfile` now waits until the URL settles on the agent's
canonical route key (`urlKey`, or the same `deriveAgentUrlKey` the app
uses) before callers use profile controls. This also covers the other
callers of this helper.

## Verification

- Red/green tests:
- `internal-privacy-share.test.ts`: the retained cohort-14 reply and a
reworded reply pass. An agent grant, an agent visibility change, a
pre-owner membership, a root-only link and an unopened link fail. The
old regex returns false on the retained reply.
- `external-team-conditional.test.ts`: the retained cohort-14 reply
passes. The same reply fails when saved state shows no blocked skill or
no installer warning. A missing first reply fails.
- `runtime-discard.test.ts`: the retained cohort-14 plan passes with the
label still cited in a note. Shared lines remaining passes. Removing
only the contact name and phone fails. Keeping the heading line, a
contact detail line, or reformatted details with the contact name fails.
I also ran the new check on the full retained revisions: retained
passes, kept section fails.
- `runtime-git.test.ts`: the retained cohort-14 reply passes. A changed
unrelated file still fails.
- `runtime-support-profile-route.test.ts` (real Chromium): the profile
rewrites the route after 0, 400 and 1500 ms. With the fix, "Reset
sessions" is clicked on the settled page. Without the fix, the click
times out (the cohort-14 failure).
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog`: 291 files passed.
- `pnpm test:e2e:runner:unit`: all passed (one unrelated hook timeout
under load passed on rerun).
- `pnpm test:e2e:runner:typecheck`: passed.
- Narrow paid run of the five full case ids on this branch: run
43643474282 (results are added in a comment).

## Risks

- Low risk. Only eval graders, fixtures and one harness wait change.
- `teams.install-team` no longer checks how the reply names the blocked
skill. A person reviews that meaning; saved state still proves the
partial outcome.
- `projects.share-project` now needs a link to the project configuration
page. A reply that links only the project overview fails
`owner-access-page-linked`.
- `steering.discard-queued-message` does not detect a contact section
that the final revision rewrites line by line with new wording. A person
reviews that. The later request text changed (it now lists a contact
name and phone), so earlier retained runs used a different request.

## Model Used

- Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use
and code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.18
2026-11-14 16:35:41 -06:00
DottaandClaude Opus 5.5 172c267338 fix(runner): count returned validation rejections and give a copyable call in repeated-failure corrections (#21666)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents change Paperclip records through the `call_api` tool, and the
native runtime runs each tool call through a repeated-identical-failure
guard (#21360, #21398)
> - Some model hosts drop arguments. A model then sends the same failing
call again and again, and each attempt costs a model turn
> - In chat-eval cohort #14 (OpenCode, `deepseek-v4.1-flash`), two cases
sent a body-less `PATCH /api/issues/{id}` 7 and 10 times with one input
hash
> - The guard counted these refusals and stopped running the call after
the second failure. But its correction showed only a `"body": {...}`
placeholder, so the model had no concrete call to copy. Also, the guard
did not count a route's own 400/422 when it came back as an `ok: false`
result
> - This pull request puts a copyable call in each correction and counts
returned 400/422 rejections. It never counts transient failures
> - The benefit is that a model gets the exact call to send on the
second failure, and the third identical validation failure does not run,
whether the rejection is thrown or returned

## Linked Issues or Issue Description

Refs #21360, #21398

**What happened?**

In chat-eval cohort #14, `questions-and-approvals.review-plan` (run
470002dc) and `tasks.private-task` (run bb76a832) sent `call_api PATCH
/api/issues/{id}` with no `body` 7 and 10 times, all with the same input
hash. The pre-dispatch check (`assertBodyPresent`) refused each call and
returned HTTP 400. The guard corrected the second failure and refused
the rest. But the correction said only `Resend with a top-level body
argument: {"operationId": ..., "pathParams": {...}, "body": {...}}`. The
model kept sending the identical call, and each refusal cost another
model turn: 2 min 18 s in review-plan and 24 s in private-task. Also,
when the route itself returns a 400/422 validation response, `call_api`
gives back `{ ok: false, status: 422, ... }` as a result and does not
throw. The guard did not count that, so an identical invalid request
reached the route every time.

**Expected behavior**

On the second identical deterministic failure, the agent gets a
correction with a call it can copy that passes validation. On the third,
Paperclip refuses to run the call and tells the agent to change the
arguments or to stop and report. Transient failures (5xx, timeouts, rate
limits, conflicts) always run again.

**Steps to reproduce**

1. Run a native-runtime agent on a host that drops the `body` argument.
2. Have it update a task with `call_api PATCH /api/issues/{id}`.
3. Look at the tool results. They contain only a `{...}` placeholder. A
route's 422 returned as `ok: false` is never counted.

**Paperclip version or commit**

master d4498ae7ac (cohort #14), reproduced against ecdeed5e7e.

## What Changed

- `runner-api-client.ts`: Add `runnerApiCallExample`. It builds a
copyable call from the operation's catalog schema: the path parameters
the model sent, plus a body with the required fields. If no field is
required, the body has one free-text field, for example
`{"operationId":"PATCH /api/issues/{id}","pathParams":{"id":"<sent
id>"},"body":{"title":"<title>"}}`. The "requires body" correction and
the "body must be a JSON object, not a JSON-encoded string" correction
now include this example in place of the `{...}` placeholder.
- `repeated-tool-failure-guard.ts`: Count a result with `ok: false` and
HTTP 400/422 as a deterministic failure, in the same way as a thrown
400/422. The first returned rejection passes through unchanged. The
second gets a `guidance` note that says the call will not run again. The
third and later identical calls are refused without running, and the
refusal quotes the response.
- Never count transient results: 5xx, 408, 429, 409, 404, and a missing
status (timeouts or unknown outcome).
- A route can reject a request because of the state it found. So the
guard forgets a returned rejection after a different call succeeds. A
refusal before sending depends only on the arguments, and the guard
keeps it for the run, as before.
- Tests: the guard tests cover returned 422 (correct, then refuse), a
changed call that still runs, returned 500/503/429/408/409/404/no-status
and thrown 429/409/timeout that keep running, and a returned rejection
forgotten after a state change. The runner API tests check that the
example in each recorded body-less correction is sent when copied as is,
that the issue-update example passes `updateIssueSchema`, and that every
catalog write with a required body (more than 200) has an example its
request schema accepts.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/services/native-runtime/runner-api.test.ts`: 2190 passed.
- Red/green: with the master guard, the 2 returned-rejection tests fail.
With the master client, 6 example tests fail. With this change, all
pass.
- `node scripts/typecheck-layers.mjs` (server): no errors.
- Per-call cost (median of 5 × 20,000 calls, local M-series): guard on a
successful call 2.13 µs before, 2.19 µs after. Transient-failure path
2.12 µs. Building an example takes 0.67 µs and happens only on a refused
call. Nothing is added before the turn starts. The guard is created with
the run's tool authority, as before.
- Paid narrow run: see the comment below.

## Risks

- Low. The change affects only calls that fail validation twice with
identical arguments.
- A route that returns 400/422 because of state can succeed later with
the same arguments. To cover that, the guard forgets a returned
rejection after any different call succeeds. A model that retries with
no call in between is refused after two attempts and must change the
arguments or report.
- The example uses `<field>` placeholders for free text and for ids that
the model must look up. If a model copies a placeholder verbatim, the
route stores that text. The correction says to send the fields the model
means to change.
- This does not make a model stop resending a refused call. It only
gives better guidance and keeps the refused call from running.

## Model Used

- Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use,
code execution, and extended reasoning).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.17
2026-11-14 15:55:09 -06:00
Nicky LeachandClaude Opus 5.5 c603f591dc fix(runner): bound ACPX instructions only where the sandbox delivers them (#21646)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner prepares a private sandbox for each ACPX agent before it
opens the provider session
> - `validateProviderPolicy` in that sandbox stage refuses system
instructions over 32 KiB for every ACPX agent
> - Only Pi gets the text from the sandbox stage. Claude, Codex, Grok
and Antigravity do not use it there. The host accepts 256 KiB
> - An agent with a large instruction bundle fails every run at
`session.open`. The error says only `classification=admission_sandbox`,
so the operator cannot see the cause
> - This pull request applies the 32 KiB sandbox bound only to Pi, and
gives every instructions rejection its own closed admission code
> - The benefit is that Claude and Codex agents with large instructions
can start, and an over-bound rejection names its real cause

## Linked Issues or Issue Description

Fixes: #20814

Refs #14430 (this pull request added the 32 KiB check, with no stated
reason for the other providers)

Related: #21644 (also fixes #20814, but only for Claude, and does not
change the classification)

**What happened?**
Every run of a Claude ACPX agent failed with
`provider_session_open_rejected`. The message was `ACPX sidecar command
session.open was rejected (retryable=false,
classification=admission_sandbox)`. A reproduction in a sandbox from the
same image showed that all the filesystem steps of
`prepareAcpxRuntimeSandbox` succeed. The real cause: the agent's
instructions entry file was 54,734 bytes. That is more than the 32 KiB
check in `validateProviderPolicy`. The host bound (`boundedInstructions`
in `runtime-host.ts`) is 256 KiB. Agents with smaller instructions (12.7
KB) work.

**Expected behavior**
The sandbox stage must refuse only the text that it delivers. #14430
added the check when only Pi consumed `systemInstructions` in the
sandbox (as the `PAPERCLIP_PI_SYSTEM_INSTRUCTIONS` env var). The check
runs before any per-provider branch, so it also refuses Claude, Codex,
Grok and Antigravity. Those providers get the text in the session
request. When a bound refuses the text, the error must name the
instructions, not the sandbox.

**Steps to reproduce**
1. Create a Claude ACPX agent whose composed instructions are more than
32 KiB (for example, an `AGENTS.md` of 54 KB).
2. Start a run of the agent on the Paperclip Runner.
3. See that `session.open` fails with
`classification=admission_sandbox`.

**Paperclip version/commit**
`master` at 163d9f323c.

**Deployment mode**
Paperclip Runner with ACPX providers (any deployment).

Where each provider consumes `systemInstructions` today:

| Provider | Delivery | Bound |
| --- | --- | --- |
| Pi | `PAPERCLIP_PI_SYSTEM_INSTRUCTIONS` env var (sandbox) | 32 KiB in
`validateProviderPolicy` (kept) |
| Cursor | `PAPERCLIP_CURSOR_INSTRUCTIONS` env var (sandbox) | 32 KiB in
`cursorInstructionBinding` |
| Kimi | `PAPERCLIP_KIMI_INSTRUCTIONS` env var (sandbox) | 32 KiB in
`kimiInstructionBinding` |
| Copilot | `copilot-instructions.md` (host, after the sandbox stage) |
32 KiB in `composeCopilotSystemInstructions` and
`refreshCopilotSystemInstructions` |
| Claude, Codex, Grok, Antigravity | session request
(`systemPrompt.append` or the provider's own session admission) | host
bound, 256 KiB |

## What Changed

- `runtime-sandbox.ts`: `validateProviderPolicy` gets the agent. It
applies the 32 KiB bound only to Pi. The type and NUL checks stay for
all agents. Cursor, Kimi and Copilot keep their own 32 KiB checks.
- `admission-diagnostics.ts`: new closed admission code
`ACPX_ADMISSION_INSTRUCTIONS`, and `acpxInstructionsAdmissionError`,
which tags the error with that code. The tag wins over the stage code
that the host records later.
- The instructions bound and NUL rejections in `runtime-sandbox.ts`,
`runtime-host.ts` (`boundedInstructions`), `cursor-instructions.ts`,
`kimi-instructions.ts` and `copilot-profile.ts` use the new tagged
error. The messages contain no instruction text and no provider output.
- `acpx_sidecar_transport.rs`: `ACPX_ADMISSION_INSTRUCTIONS` maps to
`classification=admission_instructions`. The classification set stays
closed.
- `antigravity-connection-probe.ts`: the closed classification pattern
accepts `admission_instructions`.
- `copilot-installation.test.ts`: the v70 identity test compared two
historical source hashes with the live files. Since #18325, Runner
source changes do not advance profile identities. The test now pins the
hashes that the v70 record holds.

## Verification

- Red-green: the new tests failed on `master` (11 failed) and pass with
the fix.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/acpx/runtime-sandbox.test.ts
src/drivers/acpx/runtime-host.test.ts
src/drivers/acpx/kimi-instructions.test.ts
src/drivers/acpx/cursor-instructions.test.ts
src/drivers/acpx/copilot-profile.test.ts
src/drivers/acpx/copilot-runtime-host.test.ts
src/drivers/acpx/pi-acp-runtime.test.ts src/cli`: 29 files, 489 tests
pass.
- New tests:
- The sandbox admits 54,734-byte instructions for Claude, Codex, Grok
and Antigravity, and does not put them in the launch environment.
- The host opens a Claude runtime with 54,734-byte instructions and
passes the full text to the provider.
- Pi and Cursor sandboxes refuse more than 32 KiB with
`ACPX_ADMISSION_INSTRUCTIONS`. NUL also gives that code.
- Through the host and the sidecar error mapping
(`acpxSidecarErrorCode`), oversized Pi (32 KiB + 1), Copilot (32 KiB +
1) and Claude (256 KiB + 1) instructions give
`ACPX_ADMISSION_INSTRUCTIONS` before the provider launches.
- The Rust classification test maps `ACPX_ADMISSION_INSTRUCTIONS` to
`admission_instructions`.
- The Antigravity probe logs `admission_instructions` as a closed
classification.
- `cargo test -p paperclip-runner-core --lib
classifies_only_allowlisted_internal_sidecar_failures` passes. `cargo
fmt --check` passes.
- `tsc -p packages/paperclip-runner/tsconfig.json --noEmit` passes.
- `cd server && npx vitest run
src/__tests__/antigravity-connection-probe.test.ts`: 11 tests pass.
- I did not do a live provider run. I did not run the full local suite.
CI supplies that coverage.

## Risks

- Low risk. The bound gets less strict only for providers that do not
get the text from the sandbox stage. Those providers stay under the host
bound (256 KiB). Pi, Cursor, Kimi and Copilot keep 32 KiB.
- An over-bound rejection now reports `admission_instructions` and not
`admission_sandbox` (or `admission_configuration` for Copilot). A
dashboard or alert that matches the old classification for this case
must also match the new one. Server code that checks a specific
classification does not check these values, except the Antigravity
probe, which this pull request updates.
- The NUL rejection message in `validateProviderPolicy` changes from
"exceed their bounded size" to "are invalid".

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:40:20 -08:00
DottaandClaude Opus 5.5 ecdeed5e7e perf(ui): open a task restored from the saved read cache through the chat bootstrap (#21650)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI opens a task page as a chat. A hard load of a task URL
starts one chat bootstrap request before the app's JavaScript runs.
> - A returning browser also restores a saved read cache from local
storage. This cache holds the core fields of recently opened tasks, but
no chat history.
> - The task page started the bootstrap only when the core task was not
cached. A restored core task skipped it. The early bootstrap request
ran, but nothing used its answer.
> - So every first-frame read went to its own endpoint, and the comment
input facts waited for the recent comments. A returning load was slower
than a cold load: on staging, 3.6–3.8 s against 1.9–2.4 s.
> - This pull request treats a task that only the saved cache holds as
cold for the chat. The page adopts the bootstrap, the restored task's
own revalidation shares it, and the answer replaces the restored data.
> - The benefit is that a returning load is now faster than a cold load,
not slower.

## Linked Issues or Issue Description

**Describe the bug**
A hard load of a task page is slower when the browser has a saved read
cache than when it has none. Measured on staging on 2026-11-14 (hard
load, JavaScript in the HTTP cache, 3 alternating pairs on one task):
saved cache cleared 1.9–2.4 s, saved cache present 3.6–3.8 s, until the
chat composer exists.

**Steps to reproduce**
1. Open a task page so that the saved read cache
(`paperclip:query-cache:v1:*`) holds it.
2. Close the tab and open the same task URL again.
3. Compare the time to the first chat frame with a load where the saved
read cache is cleared.

**Expected behavior**
A load with a saved read cache is at least as fast as a load without
one. The chat's first-frame reads share the one bootstrap request.

**Actual behavior**
With a saved read cache, the bootstrap request runs but its answer is
not used. About 16 more per-endpoint reads run before the frame, and the
frame waits for a second, serial wave of reads.

Related: #20355 (chat bootstrap endpoints), #20592 (cold task open from
one bootstrap), #20591 and #20594 (returning Mine inbox from the saved
read cache), #21557 (cold-load perf spec).

## What Changed

- `ui/src/lib/persisted-query-cache.ts`: records which queries a page
restored from the saved read cache. `isRestoredSavedRead(client, key)`
is true until a read or a local write in this page replaces that data.
- `ui/src/pages/IssueDetail.tsx`: starts (or adopts) the chat bootstrap
when the core task is missing or only restored. It passes the restored
task's id and identifier as aliases, so reads keyed by the id share the
request while it loads.
- `ui/src/lib/issue-chat-bootstrap.ts`: `readFromIssueChatBootstrap`
lets a key that holds only restored data share the bootstrap, like an
empty key. `startIssueChatBootstrap` accepts aliases.
- `ui/src/lib/issueChatBootstrapCache.ts` and
`ui/src/lib/issueDetailCache.ts`: the bootstrap answer replaces restored
data (core task under every ref). The restored core task's revalidation
shares the bootstrap. A core task that was read in this page keeps its
own small read.
- `ui/src/hooks/useIssuePlanDocument.ts`: the first plan read shares the
bootstrap's plan projection instead of reading `documents/plan` before
the frame.
- `tests/perf/issue-detail/cold-bootstrap.perf.spec.ts`: an always-on
request-shape test for a returning load with a saved read cache, and an
opt-in benchmark of alternating pairs (saved cache present and cleared).
The harness now clears storage after the warm-up page closes, because
closing the app saves its read cache again.
- `ui/src/lib/restored-task-chat-bootstrap.test.ts`: unit tests for the
restore-then-bootstrap path.
- `tests/perf/issue-detail/README.md`: documents the returning-load test
and benchmark.

## Verification

Local production build (`PAPERCLIP_UI_DEV_MIDDLEWARE=false`),
local_trusted instance, CDP delay of 300 ms on every API request, fresh
browser context per run, JavaScript in the HTTP cache. Time from
navigation start to the revealed history with the latest message and an
editable composer.

| Build | Saved read cache | Runs | Median | Spread | API requests
before frame | Frame released by |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| master `95a6ab31e6` | cleared | 5 | 913 ms | 908–957 | 33 | company
list (shell stage 2) |
| master `95a6ab31e6` | present | 5 | 1,204 ms | 1,178–1,215 | 49 |
`activity?view=comment-inputs` (serial second wave) |
| this PR | cleared | 7 | 942 ms | 925–997 | 35 | company list (shell
stage 2) |
| this PR | present | 7 | 573 ms | 566–658 | 37 | app boot and route
mount (all task data is the bootstrap) |

- Paired delta with this PR (present − cleared): median −376 ms over 7
pairs, all negative (−267 to −400). On master it was +292 ms over 5
pairs, all positive.
- With an 800 ms API delay (this PR, 5 pairs): present 890 ms, released
by the bootstrap response; cleared 1,910 ms.
- Cold load with no app storage (existing benchmark, 7 runs): master 910
ms, this PR 939 ms. The host load average was about 20 during the PR run
(other work on the machine). Requests before the frame are 35 in both. I
make no claim of a cold change.
- Request-shape tests (`cold-bootstrap.perf.spec.ts`): the new
returning-load test fails on master (the timeline shows `?view=core`,
`recent-comments`, `activity`, `runs`, `interactions`, `attachments`,
`live-runs` before the frame) and passes with this PR. The existing cold
test passes.
- Unit tests: `restored-task-chat-bootstrap.test.ts` has 10 cases. With
the behavior changes reverted (the restore marker kept so the file
imports), 3 fail: the bootstrap refreshes the restored task and fills
the chat, id-keyed reads share the loading bootstrap, and a partial
answer. All 10 pass with this PR. The other 7 cases are guards that hold
on both: a stale refresh reads the server, a failed bootstrap falls
back, a 403 removes the restored task from memory and storage, another
company's answer is not seeded, an answer is not used after another user
signs in, and another user's saved task is not restored.
- Focused Vitest: 27 files that use the changed modules, including
`IssueDetail.test.tsx` and `TaskChatThread.test.tsx`: 1,208 passed.
- Browser layout journeys against the PR build:
`layout-stability.spec.ts` and `plan-startup.spec.ts`, 17 of 17 passed
(reveal order, reading anchor, older history, Back, comment links,
drafts during background refresh, failed and stalled reads with Retry,
plan projection with stalled plan reads).
- `pnpm verify` exit 2 (the README is a global CI input), so required CI
supplies the broad coverage.

Commands:

```sh
cd ui && npx vitest run src/lib/restored-task-chat-bootstrap.test.ts
PAPERCLIP_ISSUE_PERF_BASE_URL=http://127.0.0.1:<port> pnpm exec playwright test --config tests/perf/issue-detail/playwright.config.ts cold-bootstrap
PAPERCLIP_ISSUE_PERF_BASE_URL=http://127.0.0.1:<port> PAPERCLIP_ISSUE_PERF_COLD_RUNS=7 pnpm exec playwright test --config tests/perf/issue-detail/playwright.config.ts cold-bootstrap -g returning
```

## Risks

- Low risk. The change only affects data that the saved read cache
restored and that no read in this page has confirmed. Data read in this
page keeps the old rule: a refresh asks the server.
- The saved read cache content is unchanged. It still excludes comments
and bootstrap payloads, so the first chat frame still needs one response
(the bootstrap). It cannot appear before any task response without
saving chat history, which the documented allowlist excludes.
- If the bootstrap fails, stalls or omits the task, each read falls back
to its own request, as on a cold load. A 403 or 404 on the restored
task's fallback read removes it from memory and storage, as before.
- Account changes: a bootstrap answer is bound to the session's user, as
before. The saved cache restores data only for the same user.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser automation through
Playwright).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 15:16:43 -06:00
Nicky LeachandClaude Fable 5.2 0ab73d4e30 feat(native-runtime): ask the Runner for provider-started turns on native ACPX Claude runs (#21456)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner can keep Claude while a background shell is running and
give the cycle of Claude to the controller as a turn that the provider
started. Core can wait for that turn. The transport can ask for it
(#21607), and a controller restart during it is safe (#21612).
> - No run does this yet, because Core does not ask the Runner for it.
> - This pull request makes the native runtime of Core ask, for ACPX
Claude. It is the only part that changes what a production run does.
> - The benefit is the goal of the whole work: a run with a background
command that is still running continues when the command finishes, and
does not fail with `native_finalization_missing`.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request turned the behavior on and had grown too large to
review. It is now three stacked pull requests:

1. #21607 (merged): the transport can ask the Runner for
provider-started turns. No caller asks.
2. #21612: the recovery after a controller restart during a wait or
during a provider-started turn.
3. **This pull request:** Core sets the option for native ACPX Claude
runs.

This is part 3. Its base branch is `fix/provider-turn-restart-recovery`.
Refs #19912 (design).

**What existing behavior does this improve?**

What a native ACPX Claude run does after a turn that ends with no result
while a background command is still running.

**Subsystem affected**

Core, the native runtime:
`server/src/services/native-runtime/native-session-executor.ts` and
`native-provider-turn-wait.ts`. ACPX Claude only.

**Current behavior**

- The run ends at the end of the turn. The Runner restarts Claude, the
background command stops, and the run fails with
`native_finalization_missing`.

**Proposed behavior**

- The native runtime sets `providerStartedTurns: true` in the transport
options for ACPX Claude, at the two places that build those options. One
function, `nativeProviderStartedTurns`, decides this and also decides if
the consumer gets the wait options, so the two cannot be different.
- A turn that ends with no result while a background command is still
running now waits, for a maximum of 10 minutes, for the turn that Claude
starts by itself. That turn is consumed as the next turn of the same
run, and its result settles the run.
- Other providers and other ACPX agents do the same as today.
- Limit of the Runner (see #21421): after a follow-up turn, the Runner
waits again only while a background command is still running. A command
that finishes while Claude reports on an earlier command gets no
follow-up.

**Reason and benefit**

- The opt-in is in the last part, so each earlier part can merge with no
change for a production run.

## What Changed

- `server/src/services/native-runtime/native-provider-turn-wait.ts` (5
lines added, 1 removed): `nativeProviderStartedTurns`.
- `server/src/services/native-runtime/native-session-executor.ts` (4
added, 2 removed): the option at the two places that build the transport
options, and the same function for the wait options.
- `server/src/services/native-runtime/native-session-executor.test.ts`
(4 added, 2 removed): the two native paths set the option for ACPX
Claude and not for other providers.

## State saved by the previous release

- No persisted state changes in this part.
- **A Runner that does not know the field** ignores it, and the run does
the same as today.
- **A deploy during a run:** a turn that started before the deploy has
no opt-in and ends as today. The next turn has the opt-in.
- **Rollback of Core:** the previous Core does not set the option, so
the Runner does not wait.

## Verification

The items below were run on the tree that was verified before the split
into three parts (commit `1b15ded1c4` has that tree, byte for byte). The
head of this branch (`23e2c13ad3`) is that commit on the new head of
#21612; its three server files are the same.

- `pnpm typecheck` and `node
scripts/verify-runner-vendor-dependencies.mjs` in `server`; `npx tsc
--noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`,
`pnpm run typecheck:typescript` and `pnpm run build:typescript` in
`packages/paperclip-runner`: all passed.
- `server/src/services/native-runtime/native-session-executor.test.ts`
and `native-runner-restart-recovery.integration.test.ts`: 1,732 passed,
21 skipped.
- The end-to-end behavior is tested in parts 1 and 2 with the real
`paperclip-runnerd`, where the test sets the option in the way that this
part does.
- Live run with a real Claude account, not in CI, at the adapter level
of the Runner (see #21316): the cycle of Claude was given as a turn with
its own usage, and each Paperclip tool call was admitted.

Not run: a live run through Core, `paperclip-runnerd` and the real
sidecar with a real Claude account (that path needs an API key, not a
local sign-in); a remote sandbox; the full suite. CI supplies the broad
coverage.

## Risks

- **This part turns the behavior on for each native ACPX Claude run.** A
run with no background command is not changed: the Runner waits only
when a shell is still running at the end of a turn that has no result.
- **This part must not merge before #21612.** Without #21612, a
controller restart during a wait or during a provider-started turn fails
the run.
- **A run can now stay open up to 10 minutes longer** when the
background command does not finish or Claude does not start a turn. The
agent computer stays active for that time.
- **A waiting run can show as "suspicious" after 5 minutes of silence**
in the run watchdog. The watchdog only labels a native run; it does not
stop it.
- **Follow-ups that the Runner cannot wait for are lost,** as the limit
above says.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 13:10:20 -08:00
DottaandClaude Opus 5.5 6ba46fd080 fix(github): read only the webhook secrets each delivery step uses (#21651)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents receive GitHub work through a GitHub App chat channel. GitHub
sends each event as a signed webhook delivery.
> - Each delivery passes three steps: durable staging, the durable
replay worker, and the replayed request's preflight under the credential
lease.
> - Each step resolved every stored GitHub App secret. A GitHub App
connection stores seven secrets, so each delivery decrypted and audited
about 21 secrets. The steps use only the signing secret, the bound
installation id and the App id.
> - On our staging instance, these reads are now the largest source of
secret reads and `secret_access_events` rows.
> - This pull request makes each webhook step read only its own keys,
and only when it needs them.
> - The benefit is fewer vault reads and audit inserts per delivery, and
the App private key and OAuth client secret stay sealed on the webhook
path.

## Linked Issues or Issue Description

Refs #21641 (fixed the other source: review-check polls now reuse cached
installation tokens).

**What happened?**
On our staging instance (serving 163d9f323c), from 2026-11-14 20:23:52Z
to 20:32Z, 32 `github_webhook_ingress` chat actions arrived. In the same
window, `secret_access_events` recorded 97 resolutions of
`credentials.webhookSecret`. It recorded the same count for
`privateKey`, `appId`, `installationId`, `clientId`, `clientSecret` and
`slug`. That is about 3 full credential resolutions per delivery, and
about 21 audit rows per delivery (roughly 70–80 rows per minute). In the
same window, 84 review-check polls caused no secret reads, because
#21641 fixed that source.

**What did you expect?**
A delivery reads only the secrets it uses. An unsigned or forged
delivery reads only the signing secret.

**Steps to reproduce**
1. Configure a GitHub App chat endpoint.
2. Send one signed `issue_comment` delivery for the bound installation.
3. Count the new `secret_access_events` rows for the connection. Before
this change, each stored key has 3 rows. A bad signature still reads
every key once.

The three call sites per delivery:
1. `server/src/services/chat-channels/github/webhook-ingress.ts`
`stageGitHubWebhookIngress` (`resolveCredentialRefs` with all refs).
2. `server/src/services/chat-channels/github/webhook-ingress.ts`
`processGitHubWebhookIngress` (`resolveCredentialRefs` with all refs, to
re-sign the retained body). This was the unattributed third resolution.
3. `server/src/services/chat-channels/github/webhook-authentication.ts`
preflight and ping (`resolveCredentials` with no key filter).

## What Changed

- Added `githubCredentialRefsFor(refs, keys)` in the GitHub webhook
ingress module. It selects the secret refs for named keys.
- Staging reads `webhookSecret` first. After the signature holds, it
reads `installationId` (only when the event names an installation and is
not an `installation` event) and `appId` (it scopes the recovery-receipt
denial query).
- The replay worker reads `webhookSecret` to re-sign the body. It reads
`appId`, `privateKey` and `installationId` only for a recovered delivery
that needs a source recheck.
- The preflight and ping read `webhookSecret` through the existing
`resolveCredentials` key filter. The preflight reads `installationId`
only after the signature holds and only for an event that names an
installation.
- The failed-delivery recovery scan reads only `appId`, `privateKey` and
`installationId`.
- Unchanged: the timing-safe compare, the credential-mutation lease, the
re-read of current endpoint state under the lease, the status and fence
checks, and the superseded-credential retry. No cache of decrypted
values.
- Per valid delivery, secret reads go from 7 keys × 3 (21 on staging) to
6: `webhookSecret` × 3, `installationId` × 2, `appId` × 1. A bad
signature goes from all keys to 1 read.
- `webhook-ingress.ts` was already above 1,000 lines. It grows by about
30 lines because the key selection belongs with the steps that use it.
`chat-channels.integration.test.ts` holds the GitHub webhook fixtures,
so the new test goes beside the existing signature tests.

## Verification

- Red on master, green with this change: `pnpm exec vitest run
server/src/__tests__/chat-channels.integration.test.ts -t "resolves only
the secrets a"`.
- `current`: before, `{webhookSecret: 3, installationId: 3, appId: 3,
privateKey: 3}` (the fixture stores 4 keys). After, `{webhookSecret: 3,
installationId: 2, appId: 1}`.
- `invalid_signature`: before, every key once. After, `{webhookSecret:
1}`.
- Regression: GitHub webhook, signature, rotation, ping, recovery,
installation, paused and revoked tests in
`chat-channels.integration.test.ts` (filter `-t
"[Ww]ebhook|signature|rotat|[Pp]ing|recovery|installation|GitHub
callback|revoked|paused"`): 304 passed, 1 failed. The failure is a Slack
test (`materializes direct external-chat finals once…`, matched by the
`ping` filter in "bookkeeping"). It passes when run alone, so it is a
load flake and not related to this change.
- `pnpm exec vitest run
server/src/services/chat-channels/github/webhook-ingress.test.ts
server/src/services/chat-github-provider-stress.test.ts
server/src/services/chat-github-webhook-config.test.ts`: 105 passed.
- `tsc --noEmit -p server/tsconfig.json`: passed.
- `pnpm verify` selects 2,726 of 3,554 Vitest files for this change (the
chat integration file pulls in a wide dependency set). I did not run
that broad set locally. Required CI supplies that coverage for the final
revision.

## Risks

- Low risk. Each step reads a subset of the same refs from the same
fenced record, so the values it uses do not change.
- A missing `webhookSecret` ref still fails the signature check as
before. A missing `installationId` ref still makes an
installation-bearing event "ignored" as before.
- The preflight now resolves `webhookSecret` and `installationId` in two
reads under the same credential lease. The lease blocks credential
mutation between them.
- Installation lifecycle events (`installation-lifecycle.ts`) still
resolve the full set, because they re-verify and re-persist all App
credentials. They are rare.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run in Claude Code
with tool use (shell, file edits, test execution) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 15:06:29 -06:00
Nicky LeachandClaude Fable 5.2 ddfbbfff20 fix(runner): recover provider-started turns after a controller restart (#21612)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - The Runner can keep Claude while a background shell is running and
give the cycle of Claude to the controller as a turn that the provider
started. Core can wait for that turn. #21607 lets a caller ask the
transport for it.
> - A controller can restart at any time, for example at a deploy. A
wait can last 10 minutes and a provider-started turn can last longer, so
a restart during one of them is a normal event.
> - Tests with the real `paperclip-runnerd` showed that such a restart
failed the session. The checkpoint that Core had was the checkpoint of
the turn before, and the recovery of the driver refused or mislabelled
the turns that it did not know.
> - This pull request makes that restart safe. It changes no production
run, because no caller asks for provider-started turns before part 3.
> - The benefit is that part 3 can turn the behavior on without a failed
run at each deploy.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

#21456 turned the behavior on and had grown too large to review. It is
now three stacked pull requests:

1. #21607 (merged): the transport can ask the Runner for
provider-started turns. No caller asks.
2. **This pull request:** the recovery after a controller restart during
a wait or during a provider-started turn.
3. #21456: Core sets the option for native ACPX Claude runs. Only that
part changes what a production run does.

This is part 2. Its base branch is `master`. Refs #19912 (design),
#21456.

**What existing behavior does this improve?**

None in production. It makes a controller restart safe for a run that
asked for provider-started turns.

**Subsystem affected**

Paperclip Runner package: the consumer
(`src/native-session-runtime.ts`), the session backend
(`src/backends/harness-driver-backend.ts`), the driver
(`src/drivers/codex/`) and the transport (`src/live/`). Only for a run
that asked for provider-started turns.

**Current behavior**

- The consumer saves no checkpoint when a wait starts or when a
provider-started turn starts. During a wait and during a
provider-started turn, the checkpoint that Core has is the checkpoint
from the start of the turn before. It names that turn as active.
- A restart from that checkpoint fails: the driver has the old turn as
active, gets the events of a provider-started turn, and stops the
session with `turn_binding_mismatch`.
- The recovery of the driver adopts one turn that it does not know as a
disposition recovery turn, and refuses two or more. With two or more, it
closes the transport, and that stops the Runner and Claude.
- The Runner does not send again an event that the controller
acknowledged. If the start of a provider-started turn was acknowledged
before the restart, the new driver gets the other events of that turn
with no start.
- A turn that ends waiting for a provider-started turn sets the terminal
of the session, as a turn that ends the run does. After a restart, if
the provider-started turn already has its result when recovery
completes, the consumer finds a result and a terminal, settles the run,
and does not read the events of that turn. The start, the items, the
usage event and the terminal of that turn do not reach Core.
- The recovered driver reads notifications from its construction, and
`reconcile()` maps the terminal of the saved turn only after its reads.
If the saved checkpoint is still the checkpoint of the controller turn
and the start of the provider-started turn is already queued, the driver
judges that start too early and stops the session with
`turn_binding_mismatch`.
- A checkpoint that waits can also have a consumed disposition marker (a
recovery turn after an earlier restart ended waiting). The driver
restores that session as closed and reads no notifications, and the
consumer settles the run on the earlier terminal with no result.

**Proposed behavior**

- **Checkpoints.** The consumer saves a checkpoint when a wait starts
and when a provider-started turn starts, with the function that it
already uses for a governed wait. A checkpoint that fails stops the
turn, as for a governed wait.
- **A checkpoint as of the event that Core stored.** The driver gives
each event its sequence when it reads it from the provider, before the
consumer stores it. A snapshot that is taken later can be ahead of Core.
For these two checkpoints, the driver captures its snapshot when it has
handled the notification that made the event (the `turn.completed` that
waits, or the `turn.started` of the provider), with the sequence of that
event. The consumer asks for the snapshot as of the event that it has
just appended. The saved state and cursor are then those of an event
that Core has.
- **Session terminal.** A turn that ends waiting for a provider-started
turn does not set the terminal of the session (`HarnessNativeSession`).
The terminal of the session is the end of the run, and that turn does
not end the run. A checkpoint that is saved during a wait or during a
provider-started turn has no terminal, so a recovered session settles
only from the events that it reads, in order, to the terminal of the
turn that has the result.
- **Driver, a checkpoint that waits.** Recovery keeps the wait. It does
not adopt and does not refuse the turns after the last terminal that it
knows. The Runner sends again, in order, each event that was not
acknowledged, and the driver admits each provider-started turn from that
stream, as it does with no restart.
- **Driver, a checkpoint with an active turn.** `reconcile()` accepts
one turn in progress that comes after the saved active turn, when the
saved turn ended with the wait marker. It refuses each other unexpected
active turn as before, and it refuses this one too when the reconciled
turn settled a result.
- **Driver, a start that is queued before recovery reconciled the saved
turn.** A `turn/started` of the provider that arrives while the session
does not wait for one first waits until `reconcile()` has settled, and
is then judged as before. `reconcile()` judges the live state after its
reads; if the stream already settled the saved turn into a wait, it
accepts exactly one turn in progress after the last terminal turn and
does not adopt it. The driver admits that turn at its start.
- **A wait with a consumed disposition marker.** The driver keeps
reading notifications for a restored session that waits, as it does for
a wait with no restart. It still refuses a second controller-started
turn. The consumer does not take the consumed-marker shortcut for a
recovered snapshot that waits, so the wait is resumed.
- **Transport, the start comes first.** For each driver, the transport
gives the `turn/started` of a provider-started turn before each other
notification and before a tool call of that turn. If it did not give
that start to this driver, it builds it from the `turn.started` event in
its stored events. It does not do this for the turn that the recovered
driver already has as active (`resumeActiveTurnId`). It binds the events
and the tool call of a provider-started turn to that turn, not to the
turn that the Runner reports as active at the time of the recovery read.
- **Transport, which start owns an event.** The stored events can have
gaps: the window removes old events but keeps the latest of each anchor
type, and the controller checkpoint drops streamed deltas. The owner of
an event is the last stored `turn.started` before it, if it is a
provider start, and if the first turn terminal that is stored after that
start is not before the event and is the terminal of the same provider
turn (`providerTurnId`). If the first stored terminal after the start is
the terminal of a different turn, the end of the provider turn is not
known, and the transport builds no start.

**Reason and benefit**

- The checkpoint is the cause. With a true checkpoint, the two states of
a restart are states that recovery can handle: a wait, or an active
turn.
- The Runner already sends again each event that was not acknowledged.
Recovery uses that stream and does not build the turns again from a
read.
- The window removes a terminal only while a later terminal of the same
type is stored. Thus the first stored terminal after a start shows if a
terminal of that turn was removed. A rule that only ignores the gaps is
not safe: the Runner can make a `turn.failed` for a turn that did not
emit its start (`provider_turn_lost_on_restore`).

## What Changed

Production code: 113 lines added, 19 removed.

- `src/native-session-runtime.ts` (15 added, 4 removed): the two
checkpoints in `consumeTurn`, with the sequence of the appended event,
and the resumed wait with a consumed disposition marker.
- `src/drivers/codex/codex-session-state.ts` (24 added, 1 removed),
`codex-session-terminal.ts` (2 added), `codex-session-notifications.ts`
(2 added), `codex-harness-session.ts` (24 added, 7 removed),
`codex-app-server-driver-impl.ts` (4 added, 1 removed): the snapshot at
the event, and the recovery of the driver.
- `src/contracts/harness-driver.ts` and
`src/contracts/native-session-backend.ts` (1 added each): an optional
`boundarySnapshot` on a driver session, and an optional sequence in the
snapshot options. Other drivers do not implement it and are not changed.
- `src/backends/harness-driver-backend.ts` (7 added, 5 removed): the
session terminal, and the snapshot as of a sequence.
- `src/live/runnerd-codex-transport.ts` (15 added, 1 removed) and
`src/live/provider-turn-tool-order.ts` (18 added): the start-first rule
and the owner of an event.
- Tests: `src/live/runnerd-acpx-provider-turn.test.ts` (572 added, 5
removed),
`src/drivers/codex/codex-app-server-driver.provider-turn.test.ts` (348
added), `src/live/runnerd-codex-transport.test.ts` (137 added),
`src/control-plane/durable-prp-control-plane.test.ts` (84 added),
`src/live/provider-turn-tool-order.test.ts` (new, 61),
`src/native-session-runtime.provider-turn.test.ts` (92 added, 6
removed), `src/backends/harness-driver-backend.test.ts` (56 added).
- `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary
only, 123 added, 14 removed): a delay before a provider-started turn, a
goal state at its terminal, a list of provider-started turns for one
session, a gap between them, a turn that stays open until the new
controller reads the snapshot, and a turn that starts after that read.
The modes that existed are not changed, and the fake reads its input on
one thread as before.

## State saved by the previous release

- No new persisted field. The driver session already has
`awaitingProviderTurn` (#21453).
- **More checkpoints:** one when a wait starts and one when a
provider-started turn starts. An ordinary turn saves the same number of
checkpoints as before (4 in the unit test, measured before and after).
- **No terminal in a checkpoint of a wait.** A checkpoint that is saved
during a wait or during a provider-started turn has `terminal: null`.
When a wait ends with no provider-started turn (the limit, a comment of
a person, or the Runner ends the run), the last checkpoint of the run
also has `terminal: null`; before, it had the terminal of the turn that
waited. One reader uses that field with no result: the text of a cleanup
failure (`nativeCheckpointRecordsCompletedTurn`), which then does not
say that the turn completed. In the tests with the real daemon (the
limit, the provider process lost, the sidecar exits), the run still ends
on the path of today (`native_finalization_missing`).
- **A checkpoint of the previous release** has no waiting state.
Recovery takes the path of today.
- **Rollback of the Runner package during a wait:** the previous code
reads the checkpoint, ignores `awaitingProviderTurn`, and takes its
recovery path of today.

## Verification

With nothing from part 3. The Rust checks and the build checks in the
first item were run before the last seven commits; these commits change
no Rust. The other items were run at the head of this branch
(`36d5e5e587`), or at an earlier commit of this branch for a suite whose
files the later commits do not change.

- Before the last seven commits: `pnpm run typecheck:typescript`, `pnpm
run build:typescript` and `node --test
test/acpx-sidecar-contract.test.mjs` (18 passed) in
`packages/paperclip-runner`; `node
scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`; `cargo fmt
--check`, `cargo build`, the 17 `acpx_*` suites (241 passed) and the
`acpx` unit tests (102 passed): all passed.
- `npx tsc --noEmit -p tsconfig.json` and `npx tsc --noEmit -p
tsconfig.test.json` in `packages/paperclip-runner`; `pnpm typecheck` in
`server`: all passed.
- `src/live/runnerd-acpx-provider-turn.test.ts`, 4 runs with the
production code of this head, 26 of 26 passed in each. The Core of these
tests follows the production append (`appendNativeSourceEvent`): it
stores an event that is out of order, keeps its contiguous cursor below
a gap, and reports a duplicate or a conflict for an event that it has.
It does not reject a gap in the sequence; production does not. The
restart cases, all with the real `paperclip-runnerd`:
- **From the checkpoint that the real consumer saved.** The first
`executeNativeSession` is fenced and detached as the server does for a
restart, and a second one recovers from the last checkpoint that the
first one saved. (a) Restart during the wait; the provider-started turn
starts after the new controller is attached. (b) Restart while a
provider-started turn is in progress. In each: the test asserts the
shape of that checkpoint, the run settles with the result of the
provider-started turn, each event is appended one time and in order, the
turn has one `turn.started` with `origin: "provider"`, and the session
closes with no error.
- **Without the two checkpoints, each of these cases fails:** the saved
checkpoint is the checkpoint of the turn before, and the second
controller fails with `turn_binding_mismatch` or with a replay conflict.
- **A slow `appendEvent`, with events that arrive behind the stored
event** (3 cases, each run 3 times with the fix and 3 times without it,
with the same result in each run). R7: the append of the
`turn.completed` that waits is slow, and the provider-started turn
starts behind it. With the snapshot at the event, the checkpoint is the
waiting state at the sequence that Core has, and Core gets the
`turn.started` of the turn one time. Without it, the checkpoint names
the provider-started turn as active, the transport builds no start, and
Core never gets that `turn.started`. R8: the first controller answers
the `paperclip_finish` call while the append of the provider
`turn.started` is slow. With the snapshot at the event, the checkpoint
has no result and the run ends with `native_finalization_missing` (see
Risks). Without it, the checkpoint has a result that Core does not have,
and recovery stops with "session cannot start another turn". R6: the
append of the provider `turn.started` is slow and a delta is queued
behind it. With the snapshot at the event, the new controller continues
the sequence that Core has. Without it, the only difference is one
sequence number that Core never gets.
- **The start was acknowledged before the restart** (3 cases: one turn
open across the restart; two turns; a turn that waits for the answer to
its `paperclip_finish` call). The new driver gets the start of each turn
one time before its other events, the finish call is answered exactly
one time, and the session closes with no error.
- **The turn ran with no controller attached** (2 cases, with and with
no result) and **two turns in one outage** (2 cases): the turns arrive
in order after the recovery.
- **A provider-started turn that is active in the checkpoint** gets no
second start, and the next provider-started turn arrives after it.
- **The turn already has its result when recovery completes** (3 cases:
the turn made its `paperclip_finish` call with no controller attached;
the turn waited for the answer to that call at the restart; two turns
across the restart). The test asserts the exact list of events that Core
gets: each provider-started turn has one start, its usage item, its
message item and its terminal, and then `run.result.accepted` and
`run.terminal`. `completeRun` is called one time with the turn that has
the result. Before the change to the session terminal, the second
controller appended only `run.result.accepted` and `run.terminal` in
each of these cases.
- **The wait checkpoint is not saved, then a restart** (R10). The last
checkpoint is the checkpoint of the controller turn, Core has the
terminal of that turn, and the start of the provider-started turn is
queued. The run settles with the result of the provider-started turn,
Core has one `turn.started` for it, and the finish call is answered one
time. Without the wait in the driver, recovery stops with
`turn_binding_mismatch`.
- **An ordinary turn**, with no option. One that waits for the answer to
its finish call at a restart: Core gets its result, its terminal and the
run terminal one time. One whose checkpoint after `startTurn` is ahead
of Core at a restart (R9): the run settles, and Core does not have only
the one event that the first controller did not store. These cases pass
with and without this pull request.
- **No checkpoint with a terminal and no result** is saved by the two
controllers in the cases above.
- `src/live/provider-turn-tool-order.test.ts` and
`src/live/runnerd-codex-transport.test.ts`: 345 passed. The helper tests
cover a stored start with removed events after it, a dropped delta, the
terminal of the turn before and after the event, a terminal of a
different turn after a removed terminal, and a controller start. With no
option, with a different agent or provider, and for a turn that the
controller started, the notifications of the transport are the same as
before this pull request.
- `src/control-plane/durable-prp-control-plane.test.ts`: 230 passed.
Three new tests use the real commit path: a full window of 4,096 events
removes the events after a provider start and the start is found; the
window removes the `turn.failed` of a provider turn after a later
`turn.failed` with no start, and no start is found; a restored
checkpoint that dropped the deltas of the turn, and the start is found.
- `src/drivers/codex` and `src/backends/harness-driver-backend.test.ts`:
491 passed. The driver tests cover a provider start that is queued
before `recoverSession()` is called, the wait state with one turn in
progress at `reconcile()` (and the two cases that it must still refuse),
a restored wait with a consumed disposition marker (the provider turn is
admitted and `startTurn` is refused), the snapshot at the two events
(also after the driver moved on, and for a restart continuation that
ends waiting), one later turn in progress, one completed and a second in
progress, two completed with a result, and the cases that `reconcile()`
must still refuse. The backend tests cover the session terminal and the
snapshot as of a sequence.
- `src/native-session-runtime.provider-turn.test.ts` and
`src/native-session-runtime.test.ts`: 333 passed. Two new cases resume a
restored wait with a consumed disposition marker: the provider-started
turn arrives and settles the run, or the wait ends at its limit and the
run takes the path of today with no new turn.
- `server`: `native-session-executor.test.ts`,
`native-provider-turn-wait.test.ts` and
`native-runner-restart-recovery.integration.test.ts`: 1,743 passed, 21
skipped.
- Mutation checks, each made named tests fail: no checkpoint at the
start of a wait; no checkpoint at the start of a provider-started turn;
the consumer does not pass the sequence (R6, R7 and R8 fail: a hole in
the sequence, no `turn.started` in Core, a different failure); the
backend ignores the sequence; the driver has no snapshot at the event;
no start-first rule; no exclusion of the active turn of the recovered
driver; the turn binding only at the first delivery; the contiguous
sequence required again; no check of the first terminal after the start;
the adoption of later turns put back; no exception in `reconcile()`; no
check after the settlement in `reconcile()`; the terminal of the session
set again by a turn that ends waiting; no wait of the provider start for
`reconcile()`; no exception in `reconcile()` for a session that waits;
the pump deferred for a restored wait; the consumed-marker shortcut
taken for a snapshot that waits.

Not run: the full suite. CI supplies that coverage.

## Risks

- **No production effect.** Each change is behind the waiting state, a
provider `turn.started`, or the option of the transport. No caller sets
the option before part 3.
- **An event that the first controller received but Core did not store
is not sent again after a restart.** This is the same on `master` and is
not changed here. Only the terminal of a turn is built again. If the
lost event is the result of a provider-started turn (the first
controller answered the `paperclip_finish` call and stopped before Core
stored the proposal), the run ends on the path of today for a turn with
no result. R8 shows this. R9 shows the same loss for one event of an
ordinary turn.
- **A second `turn.started` for one provider-started turn can reach
Core.** This occurs if the restart comes after Core stored the start and
before the checkpoint of that start is saved (the time of one checkpoint
write). The last checkpoint then waits, and the transport builds the
start again for the new driver. Seen with the real daemon when the
checkpoint write is held; not in a committed test. The usage accounting
of the server uses the turn id as its key, so there is no second count,
but the second start resets the "restored" mark of that turn. The
transcript does not read `turn.started`.
- **The older checkpoints can also save a cursor that is ahead of Core**
(measured for the checkpoint after `startTurn` of an ordinary turn).
Production accepts the gap in the sequence that follows, so this is not
a failure there. It is on `master` and is not changed here.
- **When the terminal of a provider-started turn was removed from the
stored events,** the transport builds no start for an event of that
turn, and the session stops on that event, as before. This needs a later
terminal of the same type with no start of its own (a turn that was lost
while it started) and a full window.
- **The last checkpoint of a wait that ends with no provider-started
turn has no terminal,** as the section above says. The change is only in
a run that waits, and no run waits before part 3.
- **A wait with a consumed disposition marker has no test with the real
daemon.** The state needs two restarts in one run, and the test helper
does one. Unit tests of the driver and of the consumer cover it.
- **A provider start that waits for `reconcile()`** waits as long as the
reads of `reconcile()` take. Each read ends at the command deadline of
the Runner (30 seconds by default) or when the transport closes.
- **`item.delta` events across a restart are not asserted.** A streamed
delta that the first controller received is not sent again. The
completed items are asserted.
- **The driver keeps a maximum of 8 snapshots at events** and drops each
one when the consumer asks for it or for a later one. If the consumer is
more than 8 such events behind, the checkpoint uses the current
snapshot, as before this change.
- **A second provider-started turn that also ends during the same
outage** is not tested with the real daemon: the fake gives each
terminal the same goal revision, and the Runner then holds the second
terminal until a controller is attached. The end of the reconnect time
of the Runner is not tested here.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, decided the designs, reviewed the diffs, and ran a
part of the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as Claude Code subagents with
tool use (shell, file edits, local test runs), wrote the change and the
tests, reproduced the review findings with the real daemon, and verified
this slice alone.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
canary/v2026.1114.0-canary.16
2026-11-14 12:51:52 -08:00
DottaandClaude Opus 5.5 d4498ae7ac fix(runner): settle every recorded tool result and free sessions blocked by a closed turn's pending call (#21490)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call Paperclip tools through the Runner. The Runner records
each call and returns the controller's result to the provider (here
OpenCode)
> - In chat-eval cohort 13, a `list_workflows` result was recorded but
never reached OpenCode. OpenCode waited about 290 s, and the call stayed
pending
> - The next run's `run.attach` then failed with
`durable_pending_tool_calls`, so the person's follow-up question never
got an answer
> - This pull request fixes the result shape that caused the rejection,
makes the Runner settle every recorded result, and lets the next run
settle a call that a closed turn left pending
> - The benefit is that a tool result always reaches the agent, and a
stuck call can no longer block every later turn

## Linked Issues or Issue Description

**What happened?**
Chat-eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`). In
`cases.case-history` and `cases.create-case`, the CEO called
`list_documents` (or `list_projects`) and `list_workflows` (call id 5)
together. The server answered `GET /companies/{id}/workflows` with 200
in 584 ms. The OpenCode MCP bridge diagnostic for call 5 says `outcome:
"unanswered"`, `written: false`, `handlerMs: 289038` (290723 in
create-case). The sibling call was answered in about 1 s. The run ended
with `runner_settlement_incomplete`: `semanticTools.pending` held call 5
and its `semantic_tool.result` delivery command had status `failed`. The
next question started a run whose `run.attach` was refused: `run.attach
requires a settled Codex provider session: durable_pending_tool_calls`.
The run failed in 17 s, and the question was not answered.

Cause: `executeNativeWorkflowTool` returned the Runner API result
unchanged: `{ ok: true, status: 200, operationId: "GET
/api/companies/{companyId}/workflows", ... }`. Every other API-backed
tool renames that field to `apiOperationId`. The Rust tool bridge
(`semantic_response_value`) reads an object with a boolean `ok` and an
`operationId` as a semantic envelope that must name this tool and call.
The REST route did not match, so `apply_result` failed with `semantic
result envelope does not match its provider call`. Nothing was sent to
the provider, and the call stayed pending forever. The red test
reproduces this exact error. The parallel call is not the trigger: the
sibling call settled normally, and the OpenCode MCP bridge answered
other parallel pairs in the same runs. `list_workflows` first appears in
cohort 13, so cohorts 11 and 12 did not hit it. All four workflow tools
had this shape.

**Expected behavior**
Every recorded tool result reaches the provider. When a recorded result
cannot be delivered, the provider gets an error at once and the call
settles. A call that a closed turn left pending does not block the next
run.

**Steps to reproduce**
1. Run an OpenCode agent with Cases enabled on the durable Runner.
2. Ask a question that makes the agent call `list_workflows`.
3. See the call wait about 290 s. Then ask a follow-up question and see
`run.attach` fail with `durable_pending_tool_calls`.

**Paperclip version or commit**
`master` at `df1b12ad8e`.

## What Changed

- `server/src/services/native-runtime/native-workflow-tools.ts`:
workflow tool results name the route `apiOperationId`, like case tools
and `call_api`.
- `provider_bridge.rs`: add `settle_undeliverable_result`. When the
bridge rejects a recorded result for a call it still holds, the call
settles with a bounded, non-retryable runner error. Conflicting or
unknown results still fail as before.
- `provider_backend.rs` (`deliver_semantic_result`): use that
settlement, deliver the error to the provider, and record a
`semantic_tool_result_undeliverable` diagnostic with the call,
operation, reason and the rejected result's digest.
- `provider_bridge.rs` and `provider_backend.rs` (`attach_run`): add
`expire_closed_turn_calls`. When nothing is live (no active or ambiguous
turn, no provider request waiting), `run.attach` settles calls that a
closed turn left pending as an expired failure with an unknown outcome,
and records `semantic_tool_result_expired`. The provider stopped waiting
when its turn ended, and the new run's controller never delivers the old
run's results. A call that a live turn still waits on still blocks
attachment. This follows the existing `tool_call_interrupted_by_restart`
rule: never execute the operation again, and record an unknown outcome
when no result can ever arrive.
- Docs: `doc/run-log-events.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe both
diagnostics.

## Verification

- Red/green, Rust (`cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --test codex_provider --
a_rejected_recorded_result run_attach_expires`):
- Before:
`a_rejected_recorded_result_still_answers_the_provider_and_frees_the_session`
fails with `semantic result envelope does not match its provider call`
(the production error).
`run_attach_expires_a_call_left_pending_by_a_closed_turn` fails with
`run.attach requires a settled Codex provider session:
durable_pending_tool_calls` (the production refusal).
- After: both pass. The fake provider logs `tool-response:failure`, so
it got an answer at once. The next `run.attach` succeeds.
- Parallel calls (`cargo test -p paperclip-runner-core --lib --
provider_bridge`):
`a_rejected_result_in_a_parallel_pair_still_settles_its_own_call` sends
the cohort 13 pair (`list_documents` + `list_workflows`). The rejected
call settles, its sibling keeps its own result, and the next turn
starts. `only_a_closed_turn_expires_its_pending_calls` checks that an
open turn expires nothing.
- Upgrade and mid-run cutover:
`run_attach_expires_a_call_left_pending_by_a_closed_turn` saves state
with a pending call after the turn ends, stops that Runner, and attaches
with a new Runner process from the saved state.
`durable_backend_settles_pending_tools_when_recovery_finds_the_turn_ended`
(unchanged) still passes: a late result in the same run still settles as
`settled_after_turn`.
- Red/green, server (`cd server && npx vitest run
src/services/native-runtime/native-case-tools.integration.test.ts -t
workflows`): before, `expected { ok: true, status: 201, … } to not have
property "operationId"`; after, 2 passed.
- `cargo test -p paperclip-runner-core`: 643 lib tests pass. One lib
test (`fenced_receipt_output_keeps_prefix_when_stdout_never_closes`)
failed once with `EPERM` while signaling a process group under parallel
load. It passes alone 3 of 3 and does not touch this change. The
integration test binaries pass.
- Paid narrow run for `cases.case-history` and `cases.create-case`: see
the comment below.

## Risks

- Previous-release story: no persisted shape changes. State saved by the
previous Runner with a call pending after its turn closed carries
forward. The new Runner settles that call at the next `run.attach`, and
the session continues (tested from saved state with a new process). A
previous-release Runner that receives the corrected `apiOperationId`
result delivers it normally, because the result is no longer read as an
envelope.
- Behavior change: a call left by a closed turn is now settled at the
next run's attachment, not kept forever. Its outcome is recorded as
unknown, and the operation is never run again. A provider turn that is
still live keeps the old refusal (`durable_pending_tool_calls`,
`provider_side_effect_pending`), and the controller still never replaces
the session.
- A recorded result that the bridge rejects now becomes an error for the
agent, not an endless wait. The agent is told the action may have
completed and must check the current state before repeating it.
- Latency: nothing is added before turn start. The attach-time check
walks the in-memory pending calls (usually none) and adds no I/O beyond
the existing state save. The rejected-delivery path now answers in the
same command instead of after the provider's ~290 s client timeout, and
the follow-up run no longer fails after 17 s.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21398 (first release where `list_workflows` was called in this
eval), #19733 (the attach refusal), #20886 (earlier OpenCode delivery
fix).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:21 -06:00
DottaandClaude Opus 5.5 9b581d689b fix(chat): report an outcome once and create no run when an earlier reply gave it (#21489)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In agent chat, a finished delegated task or a declined tool action
wakes the chat agent so it tells the person
> - #21401 stopped a second completion report when an earlier reply
already gave it
> - But the suppressed completion still created a zero-second heartbeat
run, and declined tool actions still got a second report
> - Run accounting must count only real turns, and the person must get
each outcome once
> - This pull request settles an already-reported outcome before it
becomes a run, and applies the same rule to declined tool actions
> - The benefit is one report per outcome, no phantom run, and no extra
model call

## Linked Issues or Issue Description

Refs #21401

**What happened?**
Chat eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`)
showed three problems after #21401.
1. `questions-and-approvals.duplicate-answer`: the completion wake was
suppressed, but it still created a 0-second run (`runtimeMode: legacy`,
no usage). The bounded-native-runs check rejected that run.
2. `skills.studio-test`: the CEO's single reply reported the Studio
result after it read the test task done (0 passed, 1 failed, links
COM-2). The completion was then suppressed, but the eval waited for a
separate completion run and timed out after 240 s.
3. `questions-and-approvals.decline-tool`: the person declined the tool
review one second after the turn for their message started. That turn
read the declined action and said so. The decline continuation, which
waited behind that turn, then posted the same declined outcome again.

**Expected behavior**
Each finished task or declined tool action is reported once in the
conversation. An outcome that an earlier reply already reported creates
no heartbeat run.

**Steps to reproduce**
1. In chat, delegate a short task, or start a governed tool read that
needs approval.
2. While it waits, send a message that wakes the chat agent.
3. Let the task finish, or decline the tool review, while that turn
runs. The turn reads the outcome and reports it.
4. See the completion wake become a 0-second run, or see a second
decline report from the decline continuation.

**Paperclip version or commit**
master `df1b12ad8e`

## What Changed

- `server/src/services/chat-completion-delivery.ts`:
`settleReportedChatCompletions` marks each undispatched completion that
an earlier reply reported as delivered, with that reply as its response.
It returns true only when nothing else is owed. The outbox `deliver`
also checks this before it requests a wake, so no wake and no run exist
for an already-reported completion.
- `server/src/modules/wake-queue`: a new transaction port,
`skipAlreadyReportedChatWake`, runs in the release drain before a
deferred wake is promoted. When the earlier reply already reported every
outcome of the wake, the wake ends `skipped` ("Already reported by an
earlier reply in this conversation") and no run is created. Wakes that
carry queued comments or a failed-chat retry are never skipped.
- `server/src/services/chat-interaction-outcome-report.ts` (new): the
same structural rule for declined tool actions. A conversation agent run
that lists its interactions records `chat_interaction.declined_read` for
each declined tool action. The decline continuation is settled when the
same agent replied in the conversation after the decline, from a run
with that receipt, posted after the read. The rule covers only declined
tool-action confirmations. Answers, plan reviews and accepted actions
keep their own turn.
- `server/src/routes/issues.ts`: `GET /issues/:id/interactions` calls
the recorder for agent runs.
- The run-start check from #21401 stays as a fallback for a completion
turn that was admitted before the earlier reply landed.
- `tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.ts`:
the Studio oracle (v5) accepts either the completion turn's reply or an
earlier CEO reply as the delivery. The earlier reply counts only from
saved state: no completion run exists, the replying run has a
`chat_handoff.completed_read` receipt for the test task, the reply was
posted after that read, it is the only such reply, and it links the
saved result. No wording check is added.
- `doc/run-log-events.md`: documents both read receipts and the skipped
wake outcome.

## Verification

- `cd server && npx vitest run
src/__tests__/chat-completion-delivery.test.ts
src/modules/wake-queue/adapters/chat-declined-outcome-reported.test.ts
src/modules/wake-queue/application/use-cases.test.ts`: all pass.
- Red: with the drain check and the outbox check disabled, 5 tests fail:
"starts no completion wake or run ...", "settles a deferred completion
wake without a run ... (true)", "settles the decline continuation with
no run ...", and both "settles an already-reported chat wake without a
run" cases.
- Negative controls stay green: a linking reply without a done read, a
decline the earlier turn never read, a non-tool-action card, and an
accepted card all still run their continuation.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.test.ts`:
34 pass. Red on the old oracle: "accepts the earlier reply the server
took as the completion report" fails. New negative cases reject no done
read, a read of another task, a reply before the read, a reply from
another run, two replies, an extra completion run, and a missing task
link.
- A narrow paid eval run of `duplicate-answer`, `studio-test` and
`decline-tool` on this branch follows in a comment.

## Risks

- A completion turn admitted before the earlier reply landed (a narrow
race) still ends as a short succeeded run with no model call, as in
#21401.
- If an earlier reply read the outcome but did not mention it, the
person does not get a second message. The read receipt, the
reply-after-read order and, for completions, the task link make this
unlikely.
- The decline rule covers only declined tool-action confirmations. Other
interaction outcomes are unchanged.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:16 -06:00
DottaandClaude Opus 5.5 64f3abbdef test(evals): write the hard-stop setup note in plain words (#21487)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks that the chat agent explains a budget
hard stop to a person
> - The `budgets-costs.explain-hard-stop` setup puts a note on the
conversation. The note said the receipt was "an imported synthetic
accounting fixture, not live model-provider billing"
> - In chat-eval cohort 13, the agent repeated that eval jargon to the
person
> - This pull request rewrites the setup note in plain words, the same
way #21408 and #21429 did for spend-breakdown
> - The benefit is that the eval tests the explanation a real person
would get, without setup jargon to copy

## Linked Issues or Issue Description

Refs #21408
Refs #21429

**What happened?**
Chat-eval cohort 13 ran OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017` on master `df1b12ad8e`.
The `budgets-costs.explain-hard-stop` reply gave a correct diagnosis,
but it said "The charge is an imported accounting fixture, not live
model-provider billing". That wording came from the eval's own setup
note.

**Expected behavior**
The setup note uses plain words. It keeps the references the scenario
needs, and it does not give the agent test jargon to repeat.

**Steps to reproduce**
1. Run
`chat-features.runner-opencode.local.budgets-costs.explain-hard-stop`.
2. Read the conversation note and the reply.

## What Changed

- `budgetIncidentSourceDescription` in
`tests/runner-e2e/chat-catalog-domains/identity-budget-incidents.ts`
builds the note. The note names the task, the recorded hard-stop
incident and the separately paused agent. It no longer says that the
receipt is an imported synthetic fixture.
- The grader is unchanged. It already proves the receipt's origin from
saved state (`receiptOrigin`), not from the reply. The note's task
identifier, incident reference and paused-agent name are unchanged. The
grader matches no literal from the note.
- TestOps manifest: no change is necessary. The case's oracle version,
source file list, query, context and expected outcome are unchanged.

## Verification

- `npx vitest run -c tests/runner-e2e/vitest.config.ts
chat-catalog-domains/identity-budget-incidents.test.ts` (16 tests pass).
- Red/green: the new test checks that the note keeps the task, incident
and paused-agent references and contains none of the setup jargon. It
fails on master (no plain-words note) and passes with this change.
- A narrow paid run of `budgets-costs.explain-hard-stop` (with the two
sibling cohort-13 PRs) ran on a combined branch. The results are in a PR
comment.

## Risks

- Low risk. Only eval setup text changes. The scenario difficulty is the
same: the note states the same facts about the stop and the manual
pause, without the test-origin remark.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:11 -06:00
DottaandClaude Opus 5.5 8a97037bd2 fix(runs): say who started a retry run in the issue run read (#21486)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat agent answers questions about failed runs by reading the
task's run list (`GET /api/issues/{id}/runs`)
> - For a retry run, that read returned only raw invocation fields
(`on_demand`, `manual`) and an owner id. It did not say who started the
retry
> - In chat-eval cohort 13, the agent told the person "Paperclip retried
automatically" about a run that the board started with Retry
> - This pull request adds a plain-words `retriedBy` field to each retry
run, and tells the chat agent to use it
> - The benefit is that people get a true account of who retried their
work, with a name and no raw ids

## Linked Issues or Issue Description

**What happened?**
Chat-eval cohort 13 ran OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017` on master `df1b12ad8e`.
In `runs-adapters.inspect-failed-run`, the reply said "Paperclip retried
automatically on a supported model". The successful run was a person's
manual Retry: `invocationSource: on_demand`, `triggerDetail: manual`,
wake reason `retry_failed_run`, and the responsible user was the board.
The agent's own reasoning hedged ("probably automatic"), because the run
read did not say who started the retry.

**Expected behavior**
The run reads that the agent uses say who started each retry: a person,
by display name, or Paperclip's automatic recovery, in plain words and
with no raw ids.

**Steps to reproduce**
1. Let a run fail on a task.
2. Press Retry on the failed run as a board user.
3. Read `GET /api/issues/{id}/runs`. Before this change, nothing in the
retry row says that a person pressed Retry.

## What Changed

- `runsForIssue` (`server/src/services/activity.ts`) reads the wake
request of each retry run. It adds `retriedBy` to each row:
- `{ kind: "person", personName, summary }` when a user wake with reason
`retry_failed_run` or `task_retry` started the run. The name comes from
the user record, or "Board" for the local board, with the same fallback
as activity actor names.
  - `{ kind: "automatic", summary }` for an `automation` retry run.
- `null` for a run that is not a retry, or a retry whose starter is not
recorded. The read does not guess. It does not use `responsibleUserId`,
because automatic retries keep the original responsible user.
- The OpenAPI description of `GET /api/issues/{id}/runs` and the chat
agent directive tell the agent to use `retriedBy` and not to call a
retry automatic unless `retriedBy` says so.
- The UI `RunForIssue` type has the new optional field.

## Verification

- `cd server && npx vitest run src/__tests__/activity-service.test.ts
src/__tests__/agent-conversations.test.ts
src/__tests__/activity-routes.test.ts
src/__tests__/openapi-routes.test.ts` passes.
- Red/green: the new test "says who started each retry run" fails
without the service change (`expected undefined to be null`) and passes
with it. It covers a board Retry, a named member's Retry, an automatic
retry that keeps the board as responsible user, a retry with no recorded
wake, and no raw ids in the output.
- `npx tsc --noEmit -p server` passes.
- A narrow paid run of `runs-adapters.inspect-failed-run` (with the two
sibling cohort-13 PRs) ran on a combined branch. The results are in a PR
comment.

## Risks

- Low risk. The change adds one read-only field and one bounded query by
primary key on the wake requests of retry runs in the page.
- No persisted state changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:04 -06:00
DottaandClaude Opus 5.5 95a6ab31e6 feat(db): attribute work signals by caller behind a debug flag (#21640)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Background queues wake from work signals. A transaction calls
`signalDatabaseWork(topic, hint)`, and the delivery coordinator wakes
the queue for that topic with that hint.
> - On a hosted instance with zero runs, the stranded-assignment
recovery queue re-checks tasks about 7 times a second. About 2.6 hinting
commits per second drive it. HTTP traffic, chat review polls and local
fixtures with all heartbeat workers do not explain the writer.
> - Statement statistics show what runs, but not who signalled it.
Nothing records the caller of a work signal or why a queue woke.
> - This pull request adds an opt-in attribution log for work signals
and recovery-queue wakes, off by default.
> - The benefit is that one short run on an affected instance names the
writer of any idle signal loop, without code changes.

## Linked Issues or Issue Description

No issue exists. This describes the gap with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

Observability of `signalDatabaseWork`
(`packages/db/src/work-signals.ts`) and of the issue recovery queues
(`server/src/services/recovery/work.ts`).

**Current behavior**

A queue that wakes several times a second at idle shows up only as
repeated statements. Nothing tells you which caller published the hint,
whether the wake came from another process, a reconnect, a direct
settled publish or a stored deadline, or which load and check it caused.

**Proposed behavior**

An environment variable, `PAPERCLIP_DEBUG_WORK_SIGNALS`, read once when
the module loads. It holds a comma-separated topic allowlist, and each
topic can carry `:N` for its stack sampling rate. When it is unset or
empty, the signal path only checks one boolean and does nothing else.

**Reason and benefit**

Several idle-load loops this week took hours of statement analysis
before the writer was found. Attribution by caller turns that into one
log query.

## What Changed

- `packages/db/src/work-signal-debug.ts` (new):
- `parseWorkSignalDebugSpec` reads `topic[:N],…`. An optional numeric
suffix is parsed from the right, so topics that contain colons (for
example `live:<companyId>`) work with or without `:N`. N defaults to 20;
malformed entries are ignored.
- `createWorkSignalDebug` returns `enabled`, `allows(topic)`,
`signal(topic, hint, entry)`, `event(topic, name, fields, stack, entry)`
and `flush()`. A disabled instance constructs no `Error`, captures no
stack, builds no line and starts no timer.
- Stack capture (`captureWorkSignalStack(entry)`) raises
`Error.stackTraceLimit` to 40 and restores it in a `finally`. It calls
`Error.captureStackTrace(holder, entry)`, so V8 cuts every frame above
the public entry (`signalDatabaseWork`, `publishDatabaseWorkSettled`, or
the recovery `wake()`).
- Frames are dropped by file (`node_modules`, `node:internal`, the
signal wrapper files) and by function name (`signalDatabaseWork`,
`notifyDeliveryWork`, …), so trimming works on the bundled `index.mjs`.
Positions are kept raw (`index.mjs:L:C`) for offline source-map lookup.
At most 10 frames are kept.
- Each (topic, event) pair has its own budget of 10 lines per second,
checked BEFORE any stack is captured. A `suppressed` line names the
topic, the dropped event and the count. Frequent `check` lines therefore
never drop `signal` or `direct_wake` lines.
- Stack sampling (1 in N) counts per (topic, event), so `check` and
`load` do not shift the phase of `signal` sampling. `direct_wake` always
captures its caller's stack.
- Lines go to an in-memory buffer that an unref'd 1-second timer flushes
to stderr. The signal path does no I/O and no await.
- `workSignalDebug` is the process instance, built from
`process.env.PAPERCLIP_DEBUG_WORK_SIGNALS` once at module load.
- `packages/db/src/work-signals.ts`: when enabled, it records:
  - `signal` in `signalDatabaseWork`;
  - `remote_notification` and `notification_reconnect`;
- `settled_publish` (with a sampled stack) for
`publishDatabaseWorkSettled`;
  - `uncertain_settled` when an uncertain transaction resolves.
- `server/src/services/recovery/work.ts`: under the same flag:
  - each task-queue `load` logs its scope key;
- each `check` logs the issue and company id, `dueAt`, and `wake:
"deadline"` or `"load"`. The marker rides on the queued entry itself,
and only while the flag is on, so nothing is tracked otherwise and
nothing outlives the entry;
- `wake()` logs ONE `direct_wake` line listing the queues it actually
invalidates (assigned-work only when registered), under the first
allowlisted one, with its caller's stack.
- **No request route is logged.** The server has no request async
context to read one synchronously, and this change does not add one.
- `docs/deploy/environment-variables.md`: documents the flag, internal
ids only, raw positions, and that enabling or changing it requires a
restart.
- Tests:
  - `packages/db/src/work-signal-debug.test.ts`, 12 tests:
- strict no-op when unset or empty (no `Error`, no `captureStackTrace`,
no stack capture, no sink);
    - the process instance is disabled without the env;
    - allowlist filtering, and spec parsing including colon topics;
    - per-(topic, event) sampling, and always-on direct-wake stacks;
- source-tree trimming, and a synthetic single-file bundle stack with
raw positions kept and wrappers dropped by name;
    - real capture below the entry with the limit raised and restored;
- checks at about 8/s alongside signals at about 3/s for 5 seconds, with
every signal line kept;
    - suppressed records never capture a stack;
    - buffered, timer-driven output.
- `packages/db/src/work-signals.test.ts`: on real PostgreSQL, the signal
and settled-publish paths never call the debug log while the flag is
off.
- `server/src/__tests__/issue-recovery-work-debug-attribution.test.ts`,
on the recovery fixture with real PostgreSQL:
    - no recovery log calls while the flag is off;
- with the flag on, a run graph hint logs its signal, the graph-scoped
load and the check;
- `wake()` logs exactly one `direct_wake` that lists the queues and
carries the caller's stack.

## Verification

- `packages/db`: `vitest run src/work-signal-debug.test.ts
src/work-signals.test.ts`: 39 passed. `tsc --noEmit` passes.
- `server`: `vitest run
src/__tests__/issue-recovery-work-debug-attribution.test.ts
src/__tests__/issue-recovery-deadlines.test.ts`: 17 passed. `tsc
--noEmit` passes.
- Staging values:
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:10`, then restart.
- Sample output from the synthetic single-file bundle test:

```
{"msg":"paperclip work signal debug","at":1000000,"event":"signal","topic":"stranded-assignments","hint":"{\"companyId\":\"c\",\"graphRunId\":\"r\"}","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","at async notifyRunDispatch (file:///app/server/dist-runtime/index.mjs:120501:3)","at async PostgresJsDatabase.connection.transaction [as transaction] (file:///app/server/dist-runtime/index.mjs:4021:22)","at async Object.settleRunProjection (file:///app/server/dist-runtime/index.mjs:300777:5)","at async file:///app/server/dist-runtime/index.mjs:310002:17"]}
{"msg":"paperclip work signal debug","at":1000000,"event":"check","topic":"stranded-assignments","issueId":"i","companyId":"c","wake":"load","dueAt":1}
{"msg":"paperclip work signal debug","at":1000000,"event":"direct_wake","topic":"stranded-assignments","queues":"stranded-assignments,orphan-blockers","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","…"]}
{"msg":"paperclip work signal debug","at":1001000,"event":"suppressed","topic":"stranded-assignments","suppressedEvent":"check","count":17}
```

## Risks

- Low risk. It is off by default. While off, each signal costs one
property read and one boolean check.
- While on, the log can show internal ids from hints (company, issue,
run). It holds no secrets and no user identifiers, and the topic
allowlist limits what is logged.
- Stack capture costs CPU. Sampling (default 1 in 20) and the 10 lines
per second cap bound it.
- The file owner of `packages/db/src/work-signals.ts` reviews this
before merge.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21621.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:54:00 -06:00
DottaandClaude Opus 5.5 163d9f323c perf(github): reuse repository installation tokens until shortly before expiry (#21641)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel lets an agent review pull requests and
answer comments through a GitHub App
> - Each GitHub call gets a repository-scoped installation token from
`githubBotRepositoryToken`
> - That function minted a new token on every call. To mint, it resolved
every App secret through the secret service, and each resolution wrote
one audit row
> - The review-check head poll calls it about every 5 minutes for each
waiting check. On staging this made the secret audit insert one of the
busiest statements in the database
> - This pull request keeps a minted token in memory until shortly
before it expires, and checks the connection's current state on every
call
> - The benefit is far fewer database writes and reads, fewer GitHub
token mints, and an audit log that records real secret use

## Linked Issues or Issue Description

**What happened?**
On paperclip.staging (commit a18479bbba, measured at 19:05Z on
2026-11-14), `secret_access_events` got about 285 rows in 60 s. These
were 44 resolutions each of the GitHub App connection's
`credentials.appId`, `privateKey`, `installationId`, `webhookSecret`,
`clientId` and `clientSecret` (+21 `slug`), with `consumer_type`
`tool_connection` and actor `system`. That is about 4.7 audit inserts/s.
The matching `secret_versions` and binding reads ran at about 3.6/s each
in Neon. These were among the busiest statements on the database.

**Expected behavior**
A GitHub installation token is valid for one hour. Paperclip should
reuse it and resolve the App secrets only when it mints a new token.

**Steps to reproduce**
1. Connect a GitHub App chat endpoint and get pull request reviews into
the `received` review-check state (staging had 84 `github_review_check`
actions).
2. Let the review-check worker poll the pull request head (23 polls/min
measured on staging).
3. Count `secret_access_events` rows per minute. Each poll adds one row
per App secret and mints a new token at GitHub.

**Paperclip version or commit**
master at a18479bbba (the commit serving paperclip.staging at the time
of measurement).

**Where in the code**
`githubBotRepositoryToken` in
`server/src/services/chat-github-client.ts`. Callers:
`chat-github-checks.ts` (review-check head poll),
`chat-github-response-comments.ts` and `chat-github-reviews.ts`.

## What Changed

- `githubBotRepositoryToken` keeps a minted token in a per-process,
in-memory cache (at most 512 entries, least recently used first out). It
never persists or logs tokens.
- The cache uses the token until 5 minutes before GitHub's `expires_at`
(capped at one hour). If GitHub does not give a usable expiry, the token
is not kept.
- The cache key holds the company, endpoint, repository id, the exact
permission set, and an authority revision. The revision holds the
connection id, the endpoint's runtime generation (pause, revoke and
availability changes advance it), and for each mint credential its
secret ref, version selector, secret status, current version,
`updatedAt`, and binding. The installation id is a secret, so its secret
ref and version identify it in the key.
- Each call reads the current authority from the database in two narrow
statements: one row from the endpoint and connection join (status,
enabled, refs, runtime generation), and the three mint secrets with
their bindings (status and version columns only, no secret values). The
endpoint and connection status checks run on every call, as before.
- If a status check fails, all cached tokens for that endpoint are
dropped. A token cannot survive a revoke, pause, disable or credential
change.
- Secrets are resolved only when a token is minted, and only the three
that minting uses (`appId`, `privateKey`, `installationId`). So the
audit log gets three rows per mint instead of one row per App secret per
call.
- New `githubBotRepositoryClient` sends repository requests. If GitHub
returns 401 for a cached token, it drops that token, mints a new one,
and sends the request once more. A 401 for a newly minted token is not
retried. The review-check, response-comment and review paths use this
client.
- The token policy stays in the GitHub chat client module
(`rules/module-boundaries.mdc`). The callers only change how they get a
request function.

## Verification

- New test file
`server/src/__tests__/github-repository-token-cache.test.ts` (embedded
Postgres, real secret service, mocked GitHub fetch):
- `mints once and resolves the App's secrets only when it mints`: red on
master (second call minted `installation-token-2`), green now. It also
checks that only the three mint secrets are audited.
- `refreshes a token shortly before GitHub's expiry`: reused at +54 min,
minted again at +56 min.
  - `never keeps a token GitHub issues for less than the safety margin`
- `mints again after a credential rotation and uses the new credential`:
rotates the installation id secret and checks that the new mint uses it.
  - `mints again after the connection's credential refs change`
- `never serves a cached token across a pause, revoke or disabled
connection`: includes a pause and resume with no call in between.
  - `mints a separate token for each repository`
- `replaces a cached token GitHub rejects and retries the request once`:
also checks that a 401 for a new token is not retried.
- On master, 5 of the 8 tests fail. All 8 pass with this change.
- Focused suites pass locally: every server test file that matches
`github`, plus `chat-sdk-runtime.adapters`, `chat-provider-inventory`,
`idle-completed-history` and `connection-management.integration` (41 of
42 files, 955 of 956 tests). The one failure,
`github-connection-enrollment-work.test.ts`, also fails on unchanged
master (`expected +0 to be 1`), so this change did not cause it.
- `pnpm --filter @paperclipai/server typecheck` passes.
- CI supplies the full typecheck, build and test coverage for the final
revision.

## Risks

- Low risk. A cached token is used only while the endpoint, connection
and mint secrets are unchanged, and the status checks still read current
database state on every call.
- A GitHub-side revoke of a token (for example, an App uninstall) makes
GitHub return 401. The client then mints again once. If the installation
is gone, the mint fails as before.
- The cache is per process. Each server process mints its own token, at
most once per hour per repository and permission set.
- Two concurrent first calls for the same key can both mint. This is the
same as before and is rare.
- No schema, migration or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.15
2026-11-14 13:42:06 -06:00
DottaandClaude Opus 5.5 a18479bbba perf(workspaces): start a full terminal workspace pass only at startup and on the hourly backstop (#21630)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep walks every candidate workspace in a
full pass. On staging that is about 4,000 finished remote workspaces,
and each one costs a persistence probe and a work product read.
> - #21617 limited full passes to startup and an hourly backstop that
only activity arms. But three other paths in the worker still started a
full pass at once:
>   - a wake whose hints do not parse to workspace or task targets;
>   - more than 50 pending targets;
>   - any run that finds no targets.
> - Across one staging deploy the probe count rose by about 2.8 full
passes in 26 minutes. At 1,000 concurrent agents, more than 50
run-finish hints between passes would be normal, so the overflow path
alone would bring back passes proportional to history.
> - This pull request works through targets in batches of 50 and turns
hintless wakes into the backstop. It gives the one caller that needs a
broad look, a workspace provider plugin becoming ready, its own hint
that inspects only plugin workspaces.
> - The benefit is that a full pass happens only at startup and at most
once per backstop hour, whatever the activity.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On staging, the hourly steady state holds (one full pass per hour, no
probes between). But across a deploy the probe count rose by 11,351 in
under 26 minutes, about 2.8 full passes over the roughly 4,000 finished
remote workspaces.

**Steps to reproduce**

The new test file
`server/src/__tests__/terminal-workspace-rotation-limits.test.ts` runs
the real coordinator and worker against real PostgreSQL on a fake clock,
with finished remote workspaces that cannot be archived (the staging
state). These are the counts on master. The numbers are workspaces
checked by full passes; a pass over N workspaces checks N:

| Scenario | Full-pass checks on master | Expected |
| --- | --- | --- |
| A burst of 120 run-finish `issue:` hints, 300 workspaces | 600 (2
extra passes) | 0 |
| One hintless wake and one unparseable hint, 120 workspaces, 30 minutes
| 1,490 (about 12 passes) | 0, then 1 at the backstop |
| A workspace provider plugin becomes ready (hintless notify from
`plugin-registry`), 123 workspaces | 123 (1 pass) | 0; only the 3 plugin
workspaces |
| Restart: old instance drains while hints arrive, new instance starts
and gets the plugin hint and 120 run-finish hints, 202 workspaces, 30
minutes | 2,424 (12 passes) | 202 (the startup pass) |

**Cause of the passes across a deploy:**
- The new instance runs its startup pass.
- Each workspace provider plugin that turns `ready` sends a hintless
notification (`plugin-registry.ts`), which started a full pass.
- Run recovery at startup finishes many runs, and their `issue:` hints
overflow 50, which started another full pass.
- Any later wake without targets also started a full pass, including the
coordinator's 5-minute retry timer. In the fixture this produced two
back-to-back passes every 5 minutes.
- The draining old instance starts nothing, because the worker's
`canRun` is false during drain. The test confirms this.

**Expected behavior**

A full pass runs only at startup and when the activity-armed backstop is
due, at most once per hour. Targets are worked through in targeted
batches.

## What Changed

- `server/src/services/terminal-workspace-work.ts`:
- **Pending targets.** Workspace and task targets go into pending sets
with no cap. Each run inspects at most 50 of them, and the rest stay
queued for the next run, which is scheduled at once. A failed targeted
run puts its targets back.
  - **Hintless or unparseable wakes** only arm the hourly backstop.
  - **No targets, nothing due:** the run does nothing.
- **Due deadlines:** the due workspaces are queued as targets, including
isolated re-home and nested clone deadlines keyed by a workspace. A due
non-workspace deadline (the isolated source orphan sweep, task directory
work) runs a targeted sweep for its prelude and arms the backstop.
- **A failed plugin workspace listing** restores the plugin refresh
flag, so the coordinator's retry lists again.
- **Full passes** happen only at startup, when the backstop is due, and
on continuation pages of a pass already under way.
- **New hint `plugin-workspaces`** (`TERMINAL_WORKSPACE_PLUGIN_HINT`):
it queues the ids of plugin-provider candidate workspaces from
`listPluginTerminalWorkspaceIds`.
- The worker now always tells the service `targeted: true`, so a hint
during a pass no longer requests a rescan lap.
- `server/src/services/execution-workspaces.ts`:
- New `listPluginTerminalWorkspaceIds` reads the plugin-provider
candidates.
- A targeted sweep reads all matches of its at most 50 targets, with no
page limit and no rescan request.
- A failed targeted sweep no longer requests a rescan lap, because the
worker re-queues its targets.
- `server/src/services/plugin-registry.ts`: the two workspace-provider
notifications send `plugin-workspaces` instead of no hint.
- `server/src/__tests__/terminal-workspace-work.test.ts`: a new test,
`retries the plugin workspace listing after it fails instead of waiting
for the backstop`, fails without the retry fix. Three existing tests now
describe the new behaviour:
  - unscoped commits go to the backstop;
  - more than 50 hinted workspaces are inspected in batches of 50 and 1;
- a due deadline on a service without targeting still runs a full pass.
- `server/src/__tests__/terminal-workspace-rotation-limits.test.ts`:
four new tests, the scenarios above.

The startup pass stays. The 1,000-workspace test in
`terminal-workspace-deadlines.test.ts` shows that durable deadlines keep
an idle instance quiet after startup. But only the startup pass
reconciles workspaces whose hints were lost while no instance was
running, so removing it would need a stored "last full pass" marker
first.

## Verification

- The four new tests fail on master with the counts in the table and
pass after the change:
- 0 full-pass checks for the burst, worked as targeted batches of at
most 50 (120 checks in total);
- 0 full-pass checks for 30 minutes after hintless and unparseable
wakes, then 1 pass at the backstop;
  - only the 3 plugin workspaces are inspected after the plugin hint;
- exactly 1 full pass across the restart: none on the draining instance,
only the startup pass on the new one.
- Prompt archiving and the missed-hint backstop keep their existing
tests in `terminal-workspace-deadlines.test.ts`:
  - the release of a holding run inspects its workspace at once;
- a run made terminal outside `writeRunStatus` is archived within 1 hour
1 minute;
  - 1,000 finished remote workspaces stay unread at idle.
- `npx vitest run` in `server/` on `terminal-workspace-rotation-limits`,
`terminal-workspace-work`, `terminal-workspace-deadlines`,
`terminal-workspace-idle`, `terminal-workspace-page-batch`,
`execution-workspaces-service`, `plugin-workspaces.integration`,
`workspace-operations-reconciliation`, `execution-workspace-reopen`,
`isolated-task-directory-retention`, `git-workspace-expiration` and
`workspace-delivery-cleanup.integration`.
- `pnpm check:module-boundaries` passes.

## Risks

- **A notification that relied on an immediate broad pass now waits up
to an hour.** The only caller I found that sent such a notification on
purpose is the workspace provider readiness in `plugin-registry.ts`, and
it now has its own targeted hint. Other hintless notifications arm the
backstop.
- **The isolated source orphan sweep** runs only at the start of a full
pass. Its due deadline now arms the backstop instead of starting a pass
at once. This only matters when the isolated source re-home feature is
enabled.
- **A targeted sweep has no page limit.** Its at most 50 targets bound
the matched workspaces to those tasks, their ancestors and their
execution workspaces.
- **A burst of N hints costs N/50 targeted runs, back to back.** The
cost scales with activity, not with history.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.14
2026-11-14 12:19:43 -06:00
DottaandClaude Opus 5.5 890dcbdd66 fix(native): wait for a live warm session owner during a deploy overlap (#21627)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap: the outgoing server still
serves while the incoming server starts new runs
> - A warm agent session stays open in the server process that ran the
last turn, after that run finished
> - The incoming server took over a warm session while the outgoing
server was alive and still held it, because the finished run's
finalization lease ends at commit
> - The takeover then failed, the task went blocked with a Retry, and in
one case a retry loop ran 27 times on the same sandbox
> - This pull request makes the incoming server wait while the server
that holds the warm session is alive, makes the outgoing server close
idle warm sessions as soon as it leaves new runs to the replacement, and
stops the retry loop when the promised fresh sandbox does not come
> - The benefit is that a follow-up during a deploy finishes without a
person, and no Runner ever gets two controllers

## Linked Issues or Issue Description

**What happened?**
A zero-downtime release-swap test (Gate B, wave B1, master aaec67b891)
found two failures with one cause.
Task 1: run A finished on the outgoing server at 15:13:56. Its warm
session stayed idle in that server. The incoming server claimed the
follow-up at 15:13:57. `inspectRemoteWarmSessionRecovery` asked
`evaluateNativeControllerTakeover`, which allowed the takeover with
`remote_controller_lease_released_or_expired`. That lease covers the
finished run, not the warm session. The outgoing server was alive until
15:18:18. `runner.prp.authenticate` failed after 62.3 s, the run failed
with `remote_warm_session_unproven`, the failure was saved as ambiguous
and owned by the board, and the task stayed blocked with a Retry.
Task 2: the same shape on a later swap started 27 automatic runs in 13
minutes. Fourteen of them said "The next attempt starts on a fresh
sandbox", but each one resumed the same sandbox and got the same refusal
(`journal_unsettled`).

**Expected behavior**
A follow-up claimed during an overlap waits for the warm session owner
to let go, then continues with no Retry banner and no second controller.
If the outgoing server dies with the session open, the takeover
continues after its ownership ends. A message that promises a fresh
sandbox is true, or the retries stop after a bounded number with a true
state.

**Steps to reproduce**
1. Run the restart-chaos release-swap preset with a warm (opencode or
codex) staging mix.
2. Let a run finish on the outgoing server after the traffic switch,
with a follow-up comment already queued.
3. See the follow-up fail with `remote_warm_session_unproven` and the
task go blocked.

**Paperclip version or commit**
master aaec67b891 and 45531b719e.

**Deployment mode**
Hosted deploy overlap (release swap) with Daytona sandboxes.

## What Changed

- `remote-warm-session-recovery.ts`: before live adoption, the run
proves that the server process that ran the finished turn is gone. It
uses the shared owner proof (`proveOwnerGone`): a PID check in the same
process space, or the `controller_instances` lease in another process
space. While that server is alive, the run defers with the existing
`native_controller_recovery_waiting_for_owner` wait and checks again
after 5 s. The wait ends when the server exits or is fenced, or when the
finished turn's warm window (idle timeout plus two minutes) ends.
- `native-session-executor.ts` and `run-start-handover.ts`: a server
that leaves new run starts to a replacement closes its idle warm
sessions after 1 s, through the existing idle-timeout close. A session
that becomes idle after that closes after 1 s too. This is the same
close that shutdown already does, done earlier.
- `predecessor-retirement-fallback.ts`: when a retry meets an unretired
owner on the same sandbox that its source already asked to replace, it
does not ask again. The run fails without another automatic retry, and
the task shows a Retry.
- `predecessor-retirement.ts`: the error no longer says "The next
attempt starts on a fresh sandbox", because the replacement is only a
request.
- `doc/execution-semantics.md`: documents the owner wait, the hand-off
at the swap, and the retry stop.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts`
- New: "waits while the outgoing server that ran the finished turn still
holds its warm session, then takes it over once that server exits" (red
on master: master returned a takeover while the owner was alive).
- New: "waits for a live outgoing server in another process space until
its controller lease ends" (red on master).
- New: "stops waiting for a live outgoing server once the finished
turn's warm window has closed" (bound check).
- `cd server && npx vitest run
src/__tests__/heartbeat-native-predispatch-retry.test.ts`
- New: "stops retrying with a truthful failure when the next attempt met
the same unretired sandbox instead of a fresh one" (red on master:
master scheduled another retry on the same sandbox).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"replacement server starts"`
- New: "closes a warm session promptly when a replacement server starts
this server's runs (before/after the turn ends)" (both red on master).
- Focused suites: native-session-executor, remote-warm-session-recovery,
native-controller-recovery, native-recovered-owner-followup,
native-runner-restart-recovery, heartbeat-native-predispatch-retry,
environment-runtime-reusable-resume-replacement,
task-interruption-presentation, sandbox-runner-port-holders.
- `npx tsc --noEmit -p server/tsconfig.json` passes. `pnpm verify` exits
2 (a doc is a global CI input), so required CI supplies broad coverage.
- Not run here: the restart-chaos release-swap wave. The maintainer runs
it after merge.

## Risks

- A follow-up can now wait up to the outgoing server's remaining overlap
when that server cannot close its idle session. Before, it failed and
needed a Retry. The wait is bounded by the owner's exit or the warm
window.
- The outgoing server closes idle warm sessions earlier during an
overlap. The next turn then resumes from a suspended Runner instead of a
live one, as it already does after a shutdown.
- A retry chain that meets the same unretired sandbox twice now stops
one attempt earlier than the shared cap. The task shows a Retry instead
of looping.
- Rows without a recorded process space keep the existing one-release
local proof, so they behave as before outside the same host.
- Previous-release state: this change adds no persisted field.
Finalization rows saved by the previous release already record the
controller boot, PID, process space and start time, so they carry
forward. A follow-up already deferred by the previous release resumes
through the same wait code.
- Runtime cost: the owner proof adds one `ps` start-time read (same
process space) or one `controller_instances` row read (other process
space) per inspection. It runs once per attempt, at most every 5 s while
a live owner holds the session.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.13
2026-11-14 11:59:17 -06:00
DottaandClaude Opus 5.5 45531b719e perf(recovery): stop re-asserting a failed legacy run whose decision is recorded (#21621)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Stranded-work recovery checks tasks that are assigned to an agent
but have no live run. It runs as a scoped-deadline queue that a change
notification or a stored deadline wakes.
> - For a failed legacy run, the check calls
`terminalizeLegacyExecution` with the run's current status. When the
run's recovery decision is already recorded (for example "continue in a
fresh run from saved work"), that call changes nothing. But it still
updated the run row and notified about 18 delivery queues, including the
run's own recovery graph.
> - The graph notification re-checked the same task, which re-asserted
the same run again. The loop ran about once a second per waiting task,
forever, with zero runs active. On a hosted instance this shows up as
constant idle transactions.
> - This pull request makes a re-assertion that changes nothing return
before it writes or notifies.
> - The benefit is that an idle instance does no stranded-recovery work
for tasks that wait on a recorded decision. A real change (another run
fails, a comment, a status change) still notifies and recovers at once.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

With zero runs active, the server runs many transactions per second. A
CPU profile groups most samples under `reconcileStrandedAssignedIssues`
(`server/src/services/recovery/service.ts`). A local fixture on real
PostgreSQL reproduces it: a task in `in_progress`, assigned to an agent,
whose latest legacy run failed and whose recovery decision is recorded.

Each check reaches the branch that terminalizes a failed legacy run that
needs reconciliation (`service.ts`, the
`legacyExecutionNeedsReconciliationWithEvidence` branch, and the same
call in `enqueueStrandedIssueRecovery`). `terminalizeLegacyExecution`
then:
- updates the run with the same status (only `updated_at` changes);
- calls `notifyExecutionRunWork`, `notifyInstructionCollection` and
`notifyUnsafeWorkspaceExportRecovery`, which hint the run graph of the
task and about 18 queues;
- finds the recorded decision and returns.

The graph hint re-checks the same task after the queue's 1-second pass
interval, so the loop never ends.

**Expected behavior**

A check that finds nothing to change stores nothing and wakes nothing.
Only a real change re-checks the task.

**Steps to reproduce**

1. Create a task in `in_progress` assigned to an agent, with a failed
legacy run (`adapter_failed`).
2. Start the issue-state recovery work. The boot pass records a
fresh-continuation decision for the run.
3. Leave the instance idle. On master: 591,045 statements in 30 minutes
for 10 such tasks (fake clock, real PostgreSQL), and one no-op `update
"heartbeat_runs"` per task per second.

**Paperclip version or commit**

master at 7db92ea319.

## What Changed

- `server/src/services/legacy-execution-recovery.ts`:
- `terminalizeLegacyExecution` returns the unchanged run, without a
write or a notification, when all of these are true: the call re-asserts
the current status with no patch; the run has no workspace restore
failure or required workspace recovery; the task's execution and
checkout locks do not point at the run; and either the run needs no
reconciliation (`reconcileIfNeeded`), no assignee or reviewer owns a
decision, or the decision is already recorded.
- Why the short-circuit is safe: stranded-work and quota checks
re-assert the run's unchanged terminal status each time they revisit its
task. Once the decision is recorded, that re-assertion changes nothing.
If it wrote or notified anyway, the notifications would re-check the
same task, which would re-assert the run again, about once a second for
as long as the task waits.
- `findRecordedLegacyDecision` returns the resolved decision already
recorded for the run (a reconciliation for this run, an explicit user
continuation with a real successor, or a no-replay hold).
`decisionOwner` says whether the task's assignee or current reviewer
owns a decision about the run.
- The recorded-decision query moves into `findRecordedLegacyDecision`,
and the owner condition into `decisionOwner`. The full path uses both
helpers, so its behavior does not change.
- `server/src/__tests__/issue-recovery-recorded-decision-idle.test.ts`:
two new tests on real PostgreSQL. They use the real coordinator, the
real issue-state recovery work and the production 5-minute backstop on a
fake clock.

## Verification

- `stays quiet for half an hour once each failed run has its decision`:
10 waiting tasks, zero runs. The test first checks that each failed run
has a recorded decision, then counts statements over 30 idle minutes.
  - master: 591,045 statements (red).
- this branch: 0 statements (green). No run is re-written or started,
and no task moves.
- `still recovers promptly when another run of a waiting task fails`:
after 10 idle minutes, a new legacy run of one task fails and notifies
like a finished run.
- this branch: within 5 seconds, recovery creates the run's
`legacy-execution:<run>` hold. Within the next 6 minutes, the
fresh-continuation fallback resolves that hold with
`executionReconciliation.runId` set to the new run. Over the next 30
minutes there is no `update "heartbeat_runs"` and fewer than 5,000
statements (the assigned-work invariant keeps its own spaced re-checks).
- master: 18,000 no-op `update "heartbeat_runs"` statements in the same
window (red).
- Neighbouring suites pass locally: `legacy-workspace-restore-recovery`,
`legacy-workspace-restore-lock-contention`,
`legacy-continuation-authority`, `heartbeat-process-recovery`,
`issue-recovery-assignments`, `recovery-deadline-ordinary-bootstrap`,
`issue-recovery-deadlines` (600 of 601 in one parallel run; `resolves a
run's task graph through index lookups` failed once under machine load
on a planner row count, and passes alone; it does not use the changed
code).
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The short-circuit applies only when the full path would also
write nothing except `updated_at`. Every other case (status change,
patch, workspace restore evidence, an owned lock, no recorded decision)
takes the unchanged full path.
- A re-assertion no longer re-sends notifications. Queues that relied on
that repeated hint as a side effect lose it. The first terminalization
of a run still sends all notifications, and run, comment and status
writes still send their own.
- No schema or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21552, #21616, #21617.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:30:03 -06:00
Nicky LeachandClaude Opus 5.5 68d5c9c120 feat(runner): let a caller ask the transport for provider-started turns (#21607)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner can already keep Claude while a background shell is
running and give the cycle of Claude to the controller as a turn that
the provider started (#21423, #21316). Core can already wait for that
turn (#21453, #21454, #21455). No run does this yet, because no
controller asks the Runner for it.
> - The request goes through the transport of the Runner package. This
pull request gives the transport an option for it and the behavior that
the option needs.
> - No caller sets the option in this pull request. It changes no
production run.
> - The benefit is a small base for the two pull requests after it: the
recovery after a controller restart, and the switch that turns the
behavior on.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

#21456 turned the behavior on and had grown too large to review. It is
now three stacked pull requests:

1. **This pull request:** the transport can ask the Runner for
provider-started turns. No caller asks.
2. #21612: the recovery after a controller restart during a wait or
during a provider-started turn.
3. #21456: Core sets the option for native ACPX Claude runs. Only that
part changes what a production run does.

Refs #19912 (design), #21456.

**What existing behavior does this improve?**

None in production. It adds the support that parts 2 and 3 use.

**Subsystem affected**

Paperclip Runner package, the controller side:
`src/live/runnerd-codex-transport.ts`. ACPX Claude only, and only for a
caller that sets the new option.

**Current behavior**

- The transport has no way to ask the Runner for provider-started turns.

**Proposed behavior**

- The transport options have a new field, `providerStartedTurns`. The
transport uses it only for ACPX with the agent `claude`. In each other
case the field has no effect.
- With the option, the transport sends `providerStartedTurns: true` in
`turn.start`. It does not send it for a compaction turn.
- With the option, a tool call in a provider-started turn waits until
the transport gave each earlier event to the driver. Without this, a
`paperclip_finish` of that turn could reach the driver before the
`turn.started` of that turn, and the driver would refuse it.
- With the option, the transport gives a `run.terminal` of the Runner to
the driver as a marked internal notification (`paperclip/runTerminal`).
The driver uses it only during a wait, so a wait ends at once when the
Runner says that the run is over.
- With no option, the transport does the same as before this pull
request. Live sessions, evals, the CLI runner and the probes do not set
the option.

**Reason and benefit**

- The opt-in is a choice of the caller, not a property of the agent.
Only a consumer that can wait for a provider-started turn must ask for
one. `CapabilityLiveSession` completes a turn at the first
`turn/completed`, so it must not get the behavior because its agent is
Claude.

## What Changed

- `src/live/runnerd-codex-transport.ts` (31 lines added, 1 removed): the
option, the field in `turn.start`, the order of the tool call, and the
`paperclip/runTerminal` notification.
- `src/live/provider-turn-tool-order.ts` (new, 16 lines): finds the
source sequence of a tool call that belongs to a provider-started turn.
- `src/live/runnerd-acpx-provider-turn.test.ts` (new, 239 lines): 7
cases with the real `paperclip-runnerd`, the fake ACPX sidecar, and the
real consumer of Core.
- `src/live/live-session.test.ts` (12 lines added): a Claude live
session gives the transport no such option.
- `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary
only): the mode `provider-result-resolved`. In this mode the
provider-started turn calls `paperclip_finish` and waits for the answer
of Core through the Runner (`tool.resolve`), as the real sidecar does.
The mode `provider-result` is not changed.

## State saved by the previous release

- No persisted state changes in this part.
- **A Runner that does not know the field** ignores it.
- **Rollback of the Runner package:** the previous transport does not
send the field, so the Runner does not wait.

## Verification

At the head of this branch (`2f2cb14b9c`), with nothing from parts 2 and
3:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript` and `pnpm run
build:typescript` in `packages/paperclip-runner`; `pnpm typecheck` and
`node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node
--test scripts/__tests__/runner-release-crate-cache.test.mjs`; `node
--test test/acpx-sidecar-contract.test.mjs` (18 passed): all passed.
- `src/live/runnerd-acpx-provider-turn.test.ts`, 3 runs, 7 of 7 passed
in each. The cases:
- background work, then the turn of Claude calls `paperclip_finish`;
Core answers; the Runner gives the answer to the sidecar with exactly
one `tool.resolve`; the turn completes; the run finalizes with one
`run.result.accepted`;
- a Claude run with no option from the caller takes the path of today:
no `providerStartedTurns` in `turn.start`, no `paperclip/runTerminal`,
no wait, and no `session.provider_turn` to the sidecar;
- the limit ends the wait, the run takes the path of today, and the
provider processes had exited before the cleanup of the test;
- the provider process is lost during the wait, and the run settles in
seconds;
  - the sidecar exits during the wait, and the run settles in seconds;
  - a turn with no background work does not wait;
- an ACPX agent that is not Claude is never opted in, also when the
caller sets the option.
- Each case asserts the result of the close of the session: clean in 6
cases, and a named failure in the case where the sidecar is killed. Each
clean case also asserts that there is no
`native_session_settlement_incomplete` diagnostic and no "conflicting
duplicate tool result" in the durable state.
- `src/live/live-session.test.ts`,
`src/live/runnerd-codex-transport.test.ts`,
`src/live/runnerd-acpx-provider-turn-wait.test.ts`,
`src/native-session-runtime.provider-turn.test.ts`: 434 passed.
- Rust, because the fake sidecar changed: `cargo fmt --check`, `cargo
build`, the `acpx_*` suites (244 passed) and the `acpx` unit tests (102
passed).
- Mutation checks, run on the full change before the split: with the
option ignored, 6 of the 7 assertions of the case with no option fail;
with the old mode of the fake, the case with the result fails at the
close of the session.

Not run: the full suite. CI supplies that coverage.

## Risks

- **No production effect.** No caller sets the option. Core sets it in
part 3.
- **The transport is shared by each provider.** The three new behaviors
are behind one condition: the option, ACPX, and the agent `claude`. With
the condition false, the code path is the path of today.
- **The tool-call ordering uses a 10 ms poll** while earlier events are
not delivered, only in a provider-started turn.
- **A controller restart during a wait or during a provider-started turn
is not handled in this part.** Part 2 handles it. Part 3 must not merge
before part 2.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.12
2026-11-14 08:16:24 -08:00
DottaandClaude Opus 5.5 7db92ea319 perf(workspaces): read a sweep page's task trees and active runs in two statements (#21618)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep inspects candidate workspaces 50 per
page. For each workspace it read the task tree with its own recursive
query. For many of them it also ran one or two queries to check for an
active run.
> - The cost of a page therefore grew with the page: about 50 to 110
statements. On the hosted instance, building the SQL text of these
statements costs about as much main-thread time as parsing their
results.
> - The archive statement re-checks the terminal tree and the absence of
a live run in the same statement, and the remote and plugin cleanups
re-check active runs under the lifecycle lock. So these early reads are
only pre-filters.
> - This pull request reads the task trees and the active-run workspaces
of a whole page in two fixed-text statements.
> - The benefit is that a page costs a constant number of these reads,
whatever its size.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The per-page reads of the terminal workspace sweep
(`sweepTerminalWorkspaces` in
`server/src/services/execution-workspaces.ts`).

**Current behavior**

Each candidate calls `listWorkspaceIssueTree`, one recursive query. A
candidate that reaches the active-run checks calls
`workspaceHasActiveRun`: one query for the linked tasks' run ids, then
one for live runs.

**Proposed behavior**

- `readPageIssueTrees` reads the tree rows of every root on the page in
one statement. Its SQL text is fixed: the roots are one `jsonb`
parameter. The trees are then built in memory and keep the same row
shape.
- `readPageActiveRunWorkspaces` returns, in one fixed-text statement,
the page workspaces that have a `queued` or `running` run on a linked
task.
- The two pre-filter checks in the loop use these results. A workspace
whose source task changed earlier in the same pass (rehome) still reads
its own tree.
- The authoritative checks are unchanged: the archive `UPDATE` with its
terminal-tree and live-run predicates, the active-run check inside the
remote and plugin cleanup under the lifecycle lock, and the
per-workspace check before a stale reopen flag is cleared.

**Reason and benefit**

One rotation pass should issue a bounded number of statements per page.
For a page of 50 workspaces (40 with open trees, 10 held by a run), the
tree reads fall from 50 to 1 and the active-run reads from 20 to 1.

## What Changed

- `server/src/services/execution-workspaces.ts`: add
`readPageIssueTrees` and `readPageActiveRunWorkspaces`, and use them for
the per-page pre-filter.
- `server/src/__tests__/terminal-workspace-page-batch.test.ts`: two new
tests on real PostgreSQL.

## Verification

- `reads the task trees and active runs of a page in a fixed number of
statements`: 2 companies, 50 workspaces, each with a child task, 10 held
by a `running` run.
  - Before the change it fails: 50 tree reads.
- After the change: one tree read, one active-run read, fewer than 10
statements for the page. The results are the same (`checked: 50`,
`skippedNonTerminalTree: 40`, `skippedActiveRun: 10`).
- `does not archive a workspace whose run starts after the page was
read`:
- The page is read first. Then, during that workspace's close-readiness
check, a `running` run is inserted and set as its task's checkout run.
- The sweep counts the workspace `eligible: 1`, `archived: 0`,
`skippedRace: 1`, and it stays `active`.
  - After the run succeeds, the next sweep archives it (`archived: 1`).
- `npx vitest run src/__tests__/terminal-workspace-page-batch.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/git-workspace-expiration.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/nested-clone-migration.test.ts
src/__tests__/execution-workspace-compaction.test.ts
src/__tests__/git-restore-point-expiration.test.ts
src/__tests__/plugin-workspaces.integration.test.ts` (in `server/`).
- `pnpm check:module-boundaries` passes.

## Risks

- **The pre-filter reads happen once, at the start of the page.** A page
can take long, because cleanups do remote and Git I/O. A run that starts
mid-page on a workspace the pre-filter saw as idle is still caught by a
check made at the destructive step itself, in the same statement or
under a lock. Line numbers are on this head:
- **Archive** of an active workspace: the archive `UPDATE` in
`sweepTerminalWorkspaces` carries `NOT EXISTS (… live_run.status IN
('queued', 'running'))`
(`server/src/services/execution-workspaces.ts:4291`). It changes 0 rows
and the workspace counts as `skippedRace` (`:4340`).
- **Remote cleanup** (destroying the sandbox):
`runTerminalWorkspaceCleanup` re-runs `workspaceHasActiveRun` under the
lifecycle lock before `cleanupRemoteWorkspaceAllocation` (`:2702`).
- **Plugin workspace removal**: the same check under the lifecycle lock
before `removePluginWorkspace` (`:2706`).
- **Git cache expiration** and **isolated task directory expiry**:
`isGitWorkspaceExpirationEligible` re-reads the tree and refuses any run
that is not finished, inside the locked transaction
(`server/src/services/git-workspace-expiration.ts:311` and `:416`,
`server/src/services/isolated-task-directory-retention.ts:244` and
`:290`).
- A run that ends mid-page leaves its workspace for the run-release hint
or the next inspection.
- **Tree rows are now deduplicated** (`UNION` instead of `UNION ALL`).
Parent cycles therefore cannot repeat rows.
- Low risk otherwise. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.11
2026-11-14 08:51:28 -06:00
DottaandClaude Opus 5.5 081dacba1d perf(workspaces): inspect only due workspaces and wait for a holding run's release (#21617)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep stores one deadline per workspace:
cooldown end, compaction, cleanup retry, Git idle.
> - When any deadline was due, the worker ran a full rotation over every
open candidate workspace. That costs a task tree read and more for each
one.
> - A workspace held by a `running` run kept a past deadline, so the
5-minute deadline rotation came back for it forever. A run that never
finishes would keep the instance awake indefinitely.
> - This pull request makes a due deadline inspect only the due
workspaces. A held workspace waits for its run instead of for a timer.
> - The benefit is that an idle instance does work only for workspaces
whose deadline really arrived. The cost no longer grows with the number
of open workspaces.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

How the terminal workspace worker
(`server/src/services/terminal-workspace-work.ts`) acts on due deadlines
and on workspaces held by an active run.

**Current behavior**

`hasPending` reads `nextTerminalWorkspaceSweepAt`. When the earliest
deadline is past, the next run starts a full keyset rotation, 50
workspaces per page with 1 second between pages, over every candidate. A
workspace whose issue still has a `queued` or `running` run is skipped,
but the archive deadline set earlier in the same pass stays. It is
already past, so the rotation repeats every 5 minutes.

**Proposed behavior**

- When the run fires for a due deadline, the worker takes the due
workspace ids from the service (`takeDueTerminalWorkspaceTargets`) and
runs a targeted inspection of only those workspaces. The service first
clears the deadlines of the targeted workspaces; the inspection sets
fresh ones.
- If more than 50 workspaces are due, or a due deadline belongs to the
isolated source, nested clone or task directory work, the worker still
runs a full rotation. Startup still runs a full rotation.
- A workspace skipped because a run holds it drops its deadline.
- `writeRunStatus` (`server/src/services/heartbeat/run-lifecycle.ts`)
publishes `issue:<runIssueId>` to the terminal workspace queue when a
run reaches a terminal status. That targeted inspection reaches the
workspace the run held.
- **Backstop.** About 200 code paths write `heartbeat_runs` directly
(reapers, recovery terminalization, bulk cancels), and they do not pass
through `writeRunStatus`. So the worker keeps a full rotation as a
backstop, armed only by activity, like the recovery backstop from
#21552:
  - Any terminal-workspace hint arms it.
- Once armed, a full rotation runs at most once per hour
(`TERMINAL_WORKSPACE_BACKSTOP_MS`) after the previous full rotation.
  - An idle instance with no hint schedules nothing.
- A missed release hint therefore delays archiving by at most one hour
after activity, without a restart.
- The hour is a compromise. On master today, the 5-minute rotation
re-reads every candidate. On the hosted instance that is about 1,000
finished remote workspaces, roughly 3,000 statements per lap. The hourly
backstop re-reads them at most once per hour while there is activity,
about 1 statement per second averaged.

**Reason and benefit**

Background work must be driven by deadlines and change notifications,
and its cost must not grow with the backlog. Every guarantee keeps a
named trigger:
- cooldown end, compaction, retry backoff and Git idle each have a
stored deadline, and a due deadline inspects exactly that workspace;
- the release of a holding run publishes a hint;
- task status changes and workspace writes still publish hints;
- startup still reconciles everything.

## What Changed

- `server/src/services/execution-workspaces.ts`:
  - New `takeDueTerminalWorkspaceTargets()`.
- A targeted inspection clears the deadlines of its targets before it
inspects them.
  - A workspace skipped for an active run drops its deadline.
- `server/src/services/terminal-workspace-work.ts`: a due run uses the
due targets instead of a full rotation, unless they overflow or a
non-workspace deadline is due. A due run that finds nothing due does
nothing. Any hint arms the hourly backstop rotation.
- `server/src/services/heartbeat/run-lifecycle.ts`: a terminal run
status hints the run's task to the terminal workspace queue.
- `server/src/__tests__/terminal-workspace-deadlines.test.ts`: four new
tests on real PostgreSQL. They use the real coordinator and worker on a
fake clock.
- `server/src/__tests__/terminal-workspace-idle.test.ts` (from #21616):
its idle window now bounds the number of sweeps instead of requiring
zero targeted sweeps, because a due deadline is now a targeted
inspection.

## Verification

- `inspects only the workspace whose cooldown ends, however many trees
stay open`: 60 workspaces with open task trees and 1 done workspace
whose 1-day cooldown ends 1 hour later.
- Before the change: after startup, every due deadline starts a full
rotation (27 full sweeps in 2 hours).
- After the change: no full rotation; the done workspace is inspected
alone (`checked: 1`) once its cooldown ends; fewer than 200 statements
in 2 hours.
- `waits for the release of the run that holds a workspace instead of
rechecking it`: a done workspace held by a `running` run.
- Before the change: a full rotation every 5 minutes (22 sweeps in 2
hours).
- After the change: 0 statements in 2 idle hours. When the run is set to
`succeeded` through the lifecycle, the workspace is inspected (`checked:
1`, `skippedActiveRun: 0`).
- `leaves a thousand finished remote workspaces unread at idle`. This is
the hosted state: 1,000 `active` remote workspaces with a location, a
committed snapshot and a task done 5 days ago, and zero runs.
- After startup and 2 idle hours: no full rotation. No remote workspace
is read again; the only inspections are of the one due local workspace.
Statements stay within 10 per inspection.
  - On master, every due deadline re-reads all 1,000.
- `archives a workspace whose run ended outside the run lifecycle within
the activity-armed backstop`:
- A workspace held by a `running` run. The run is set to `failed` by a
direct `UPDATE`, so no release hint is published, and one unrelated hint
arrives.
- The workspace is archived within one hour and one minute, without a
restart.
  - With the backstop scheduling disabled, it stays `active`.
- `npx vitest run src/__tests__/terminal-workspace-deadlines.test.ts
src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/git-workspace-expiration.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/nested-clone-migration.test.ts
src/__tests__/execution-workspace-compaction.test.ts
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 333
tests pass.
- `pnpm check:module-boundaries` passes.

Local fixture with #21616 applied as the base, 30 idle minutes on a fake
clock, real PostgreSQL. Each fixture holds open trees, a stranded
`running` holder, cooldown, done local workspaces with missing files,
closed remote workspaces with and without a committed snapshot:

| Workspaces | Before: sweeps | Before: statements | After: sweeps |
After: statements |
| --- | --- | --- | --- | --- |
| 140 | 20 (full rotations) | 1,371 | 21 (targeted only) | 866 |
| 420 | 84 (full rotations) | 6,597 | 33 (targeted only) | 3,272 |

After the change, no sweep reads the task tree of a workspace whose tree
is still open (`skippedNonTerminalTree` falls from 1,440 to 0 at 420
workspaces). What remains comes only from workspaces with a due retry or
inspection backoff, so it grows with the number of stuck workspaces, not
open ones. Per inspected workspace a pass issues the same 3–6 statements
as before; batching them per page is a separate pull request.

## Risks

- **A run that leaves `running` without `writeRunStatus`** does not
hint. The activity-armed backstop catches it within one hour of
activity, and startup reconciles. This pull request does not decide that
a stranded run is gone; it waits for a status change.
- **A workspace whose deadline is reset to "now"** (a reopen-pending
flag within its grace period, task directory budget overflow) is
inspected alone on each due run. The inspection backoff in
`nextTerminalWorkspaceSweepAt` still spaces these runs.
- **Workspaces without a stored deadline** (for example an undelivered
workspace whose deadline was deleted) are inspected again only when a
hint or the startup rotation reaches them. Before this change they were
also reached by the rotation that a different workspace's deadline
started.

## Open decision

**Should a terminal workspace whose local files are gone be archived, or
should it wait?** This pull request does not change the behaviour; it
leaves the decision to the owner.

Today such a workspace fails its close-readiness inspection and is
reported as `skippedUndelivered`. A non-remote workspace keeps its past
deadline, so the 5-minute inspection backoff re-inspects it again and
again. That costs about 3–4 statements per stuck workspace every 5
minutes (from the local fixture: 255 inspections, 866 statements), and
it never ends while the files stay missing. The cause is typically a
replaced container.

The two options:
- **Archive it.** Nothing remains on disk to keep.
- **Store a "files missing" state.** It is then inspected again only
when a write announces a change.

Either option removes this cost. Archiving also removes the row from
later rotations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.10
2026-11-14 08:23:49 -06:00
DottaandClaude Opus 5.5 20ccf03676 fix(workspaces): settle closed remote workspaces after one confirmed destroy (#21616)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server cleans up closed execution workspaces in a background
sweep (`terminal-workspaces`). For a remote workspace, cleanup destroys
the sandbox lease after it confirms that the saved snapshot is
committed.
> - A closed remote workspace stays a sweep candidate while
`metadata.remoteWorkspace.location` is an object. A confirmed destroy
did not clear that location.
> - The cleanup write goes through the lifecycle gateway, and the
gateway published a `workspace:<id>` hint for the same workspace. That
hint started a new inspection, which cleaned the workspace again and
published again.
> - On a hosted instance with no runs, this loop kept three statements
at about 15–24 per second each and a dedicated lock connection per
inspection. Every pass also asked the sandbox provider to destroy a
sandbox that was already destroyed.
> - This pull request clears the location after a confirmed destroy,
skips the provider call when the stored lease already has a destroy
receipt, and stops the sweep from hinting itself for its own writes.
> - The benefit is that an idle instance with closed remote workspaces
settles: each workspace is cleaned once, and the provider is called at
most once per lease.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

With no runs, the terminal workspace sweep never became idle. On the
hosted instance these statements ran all the time:
`workspaceHasActiveRun` (22.8/s), the task tree read (22.6/s) and the
snapshot persistence read in `inspectRemoteWorkspacePersistence`
(14.8/s). `SET application_name` for the
`paperclip-workspace-finalization-lock` connections ran at 9.6/s.

**Steps to reproduce**

1. Create closed (`archived`) remote workspaces that have a committed
snapshot, a clean authority and a `location`.
2. Start the terminal workspace worker with no other activity.
3. Watch the sweeps. Each cleanup reports `archived`, the gateway
publishes `workspace:<id>`, and the next targeted inspection cleans the
same workspace again.

Locally, a fixture with 140 workspaces (40 of them in this state) ran
568 sweeps, 136,888 statements and 23,880 lease destroy calls in 30 idle
minutes. The destroys always hit the same 40 workspaces.

**Expected behavior**

A closed remote workspace is cleaned once. After the destroy is
confirmed it leaves the candidate set, and an idle instance issues no
statements between real deadlines.

## What Changed

- `server/src/services/remote-workspace-lifecycle.ts`
(`cleanupRemoteWorkspaceAllocation`):
- Read the stored lease first. When it already carries a destroy receipt
for the same environment, lease and provider lease, do not call
`destroyLease` again.
- After a confirmed destroy, set `metadata.remoteWorkspace.location` to
`null`. One fixed-text `jsonb_set` update does this, and only when the
stored location is still the location that was destroyed.
- `server/src/services/execution-workspaces.ts`: the gateway takes
`sweepOwned`. The sweep's own cleanup, cleanup-failure and reopen-flag
writes set it, and then the gateway does not publish a terminal
workspace hint for that workspace. All other gateway callers still
publish.
- `server/src/__tests__/terminal-workspace-idle.test.ts`: two new tests
on real PostgreSQL.

## Verification

- `settles after one cleanup of each closed remote workspace and
inspects nothing on its own writes`:
- Fixture: 40 workspaces (open trees, a stranded `running` holder,
cooldown, done, closed remote with a committed snapshot, closed remote
without a snapshot).
- Runs the real coordinator and worker on a fake clock for 2 minutes,
then 30 idle minutes.
- Asserts: each closed remote workspace is destroyed exactly once; no
targeted inspection in the idle window; fewer than 5,000 statements in
30 minutes; every cleaned workspace has `location: null`.
- Before the change it fails: 850 destroy calls instead of 10 in the
warm-up alone. After the change it passes.
- `clears the location of an already destroyed lease without calling the
provider again`: the stored lease has a destroy receipt. Before the
change, the sweep calls `destroyLease`. After the change the provider
mock is never called, the location is cleared, and the next inspection
finds no candidate.
- `npx vitest run src/__tests__/terminal-workspace-idle.test.ts
src/services/remote-workspace-lifecycle.test.ts
src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts` (in
`server/`).

Local fixture, 30 idle minutes, 140 workspaces (40 closed remote with a
committed snapshot):

| | Before | After |
| --- | --- | --- |
| Sweeps | 568 (all targeted, continuous) | 20 (one deadline rotation
every 5 minutes) |
| Statements | 136,888 | 1,371 |
| Lease destroy calls | 23,880 | 40 (once per workspace) |
| `workspaceHasActiveRun` reads | 45,440 | 260 |
| Snapshot persistence reads | 22,720 | 40 |

The remaining rotation every 5 minutes comes from deadlines of
workspaces that cannot make progress (a stranded active-run holder,
cleanup backoff). A separate pull request handles it.

## Risks

- **Who reads `remoteWorkspace.location`.** For a closed, clean
workspace with a committed snapshot and a destroyed lease, none of these
readers needs the location:
- the live run binding checks (`remote-workspace-lifecycle`,
`fresh-continuation-fallback`, `workspace-recovery-fallback`,
`workspace-recovery-source`) need a current writer or a dirty run
reference;
- the abandonment and hold paths (`remote-workspace-epoch-abandonment`,
`remote-workspace-abandoned-files-hold`,
`remote-workspace-unchanged-proof`) need a dirty or held allocation;
- `reusable-sandbox-resume-replacement` treats `null` as no conflicting
sandbox;
- `remote-workspace-files` reads the saved snapshot when the location is
`null`. With a destroyed location it would fail as unavailable.
- The lease-release path
(`clearRemoteWorkspaceLocationForReleasedLease`) already writes the same
`null` for the same state. This change makes the sweep stop depending on
that side effect.
- **Provider calls.** Before this change every loop pass re-claimed the
lease and called the driver's `destroyRunLease` again, through
`destroyClaimedSandboxLease` in
`server/src/services/environment-runtime.ts`. After it, a lease with a
stored destroy receipt causes no provider call.
- **Existing backlog after deploy.** The startup rotation and the next
deadline rotation visit each closed remote workspace that still has a
location once, 50 per page with 1 second between pages. A lease that
already has a destroy receipt costs no provider call, and its location
is cleared. Only leases without a receipt get one destroy call. The
workspaces then leave the candidate set.
- **No self-hint.** The sweep already handles the workspace it writes,
and an inspection's deadline bookkeeping is unchanged. Other writers
still publish hints. The git restore point and task directory paths
depend on task or explicit hints, not on these writes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.9
2026-11-14 07:23:35 -06:00
DottaandClaude Opus 5.5 c1451814c1 perf(chat): scope GitHub review check run hints to the issue and throttle company-wide rebuilds (#21610)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel keeps each unresolved "Paperclip Review"
check in step with its task and run. A background queue
(`chat-github-review-checks`) rebuilds its checks when a hint arrives
> - Each run milestone (`notifyChatRunMilestones`) sends this queue a
company-wide hint, also when the milestone knows the issue. Each
company-wide hint rebuilds every unfinished check of the company and
reads its full state again. Passes are limited to one a second
> - So in a company with a steady stream of run milestones, each
unresolved check reads its whole state about once a second, although
nothing about it changed
> - On a hosted test database the five state statements of a review
check ran 10.3 times a second each. #21608 removes the endless 5-second
head poll. This is the second cause
> - This pull request sends the issue with the hint, rebuilds only the
checks of that issue, and rebuilds for a company-wide hint at most once
every 30 seconds
> - The benefit is that run activity no longer makes every unresolved
check of the company read its state each second. A run change of the
check's own task still publishes at once

## Linked Issues or Issue Description

Refs #21608 (backs off the head poll of an unresolved check; the other
cause of the same hosted load). Refs #21580 (the same 30-second rule for
the GitHub response sweeps). Refs #18384 (the same rule for external
objects).

**What happened?**

On a hosted test database with zero active runs, the five statements of
`githubReviewCheckState` (`chat_endpoints`, `chat_github_reviews`,
`chat_message_links`, the `heartbeat_runs` `wakeCommentId` lookup and
`chat_deliveries`) ran 10.3 times a second each. One source is the
rebuild of the review check queue for company-wide hints.

**Current behavior**

`notifyChatRunMilestones` sends `{"companyId": ...}` to
`chat-github-review-checks` for every milestone, also when its scope has
an issue. The queue scope has no issue field. So each milestone rebuilds
every received and retryable check of the company, and calls
`githubReviewCheckState` for each received check.

**Proposed behavior**

A milestone with an issue sends `{"companyId", "issueId"}`. The queue
matches a check to an issue through the issue of its latest review for
the pull request head. This is the same review that
`githubReviewCheckState` reads. A company-wide hint (company ID only)
rebuilds at most once every 30 seconds. All narrower hints, the startup
scope (`*`) and saved deadlines are not changed.

**Steps to reproduce**

1. Admit a GitHub review check that stays unresolved (a review with no
run that the check can find).
2. Send a run milestone for the company once a second (for example,
status changes of other runs).
3. Count statements. Each hint rebuilds the check and reads its full
state.

## What Changed

- `review-check-work.ts`: the scope has an optional `issueId`. `page()`
reads the issue of the latest review for the check head
(`reviewIssueId`, through the existing `chat_github_reviews_pull_idx`
index) and filters on it for an issue scope. `matches()` uses the same
value.
- `review-check-work.ts`: set `reconstructionIntervalMs` to
`githubReviewCheckReconstructionIntervalMs`: 30 seconds for a
company-only scope, 0 for all other scopes.
- `chat-run-milestone-notifications.ts`: send the issue with the review
check hint when the milestone has one.
- `chat-channels.integration.test.ts`: two regression tests on the real
queue, the real `dispatch` and embedded PostgreSQL with a controlled
clock. The test helper now counts only the state read of
`chat_github_reviews`, not the new subquery in `page()`.

## Verification

Red/green regression tests:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "company-wide hints
at most once per interval|at once from its issue milestone"`
- Red (without the fix): `expected [ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 ] to
deeply equal [ 1, +0, +0, +0, +0, +0, +0, +0, …(2) ]`. Each of 10
company-wide hints read the check state. The milestone hint was
`{"companyId"}` only.
- Green: 2 of 2 pass. The first company-wide hint reads the state once.
The next 9 read nothing. After 30 seconds the waiting hint reads it
once. A hint for the check's issue reads it at once each time. A hint
for another issue or another company reads nothing. A real
`notifyChatRunMilestones` with the check's issue publishes `in_progress`
on the next pass and resets the head poll interval.

Statement counts for one unresolved check with one company-wide hint
each second (120 simulated seconds, real queue and `dispatch`, embedded
PostgreSQL):

| | Statements a second | Check state reads a second |
| --- | --- | --- |
| Before | 6.0 | 1.0 |
| After | 0.2 | 0.033 |

In the real-coordinator reproduction of #21608, one hint each second
raised one stuck check from 0.70 to 1.67 state reads a second and from
8.4 to 15.2 statements a second.

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: passed,
including all review check queue tests.
- `cd server && npx vitest run
src/__tests__/native-status-reassessment-work.test.ts
src/__tests__/chat-run-milestone-work.test.ts
src/services/native-runtime/external-chat-wait.integration.test.ts
src/services/chat-github-review-repair.test.ts
src/__tests__/idle-completed-history.test.ts
src/__tests__/chat-admission-condition.test.ts
src/__tests__/app-private-hostname-gate.test.ts`: 518 passed.
- `cd server && node ../scripts/run-tsc.mjs` and `node
scripts/typecheck-layers.mjs` (with test layers): passed.
- `pnpm verify --explain` retains 2,718 test files conservatively, so I
did not run it locally. Required CI supplies that coverage.

## Risks

- A change that only a company-wide hint reports can take up to 30
seconds to reach a check. Company-wide hints now come from milestones
without an issue (a run with no task, a comment merged into an active
run) and from company authority changes. A run with no task cannot be
the run of a review check. For a merged comment, the run's next status
change sends an issue hint. The saved head poll deadline still applies.
Each dispatch reads the full state again before it publishes.
- An issue hint finds a check through the issue of its latest review for
the head. If a run of another issue were the check's run, its change
would wait for the next company-wide rebuild (30 seconds at most) or the
next head poll. Run status and run identity milestones use the run's
task issue, which is the review's issue for a GitHub review task.
- The deadline reconciliation snapshot reports the queue as not
reconciled while a company-wide hint waits (up to 30 seconds). This is
the same as the GitHub response sweeps and the external-object refresh
sweep.
- `page()` reads one more indexed subquery for each row. It uses
`chat_github_reviews_pull_idx`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local embedded PostgreSQL
reproductions).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.8
2026-11-14 05:46:48 -06:00
DottaandClaude Opus 5.5 a7453eba60 perf(chat): back off GitHub review check head polls while the check cannot change (#21608)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel publishes a "Paperclip Review" check for
each admitted pull request head. A background queue
(`chat-github-review-checks`) keeps each unresolved check in step with
its task and run
> - While a check is queued or running, the queue polls GitHub for the
pull request head every 5 seconds. This finds a push whose webhook was
lost
> - The poll never stops while the check is unresolved. A check whose
run never comes, or whose run the check cannot find, stays queued
forever. It re-reads its whole state and calls GitHub every 5 seconds,
also when nothing changes
> - On a hosted test database with zero active runs, the five state
statements of this check ran 10.3 times a second each
> - This pull request doubles the head poll interval for each poll that
finds the same state, up to 5 minutes. A state change resets it to 5
seconds
> - The benefit is that an idle unresolved check costs about one poll
each 5 minutes, not one each 5 seconds. Local changes still publish at
once

## Linked Issues or Issue Description

Refs #17662 (added the 5-second head reconciliation for unresolved
checks). Refs #21580 (the same kind of fix for the GitHub response
sweeps).

**What happened?**

On a hosted test database (Neon, 488-second window, zero running or
queued runs, 2 runs created in 30 minutes), these five statements ran at
exactly 10.3 a second each: the `chat_endpoints` source read, the
`chat_github_reviews` read, the `chat_message_links` comment read, the
`heartbeat_runs` `wakeCommentId` lookup and the `chat_deliveries` state
read. All five come from `githubReviewCheckState`
(`server/src/services/chat-channels/github/review-check-state.ts`).

**Expected behavior**

A check whose state cannot change makes no recurring database reads or
GitHub calls until something that can change it happens: a local hint, a
lease expiry, or a saved deadline. The missed-webhook head check stays,
at a bounded cost.

**Steps to reproduce**

1. Admit a GitHub review check. Bind a task review to it, but give it no
run that the check can find (for example, the run was never created, or
its wake comment is not the delivery's comment).
2. Start the review check queue and leave the instance idle.
3. Count statements. The check is due again 5 seconds after each poll,
for ever. Each poll evaluates the check state about 3 to 4 times (before
dispatch, in dispatch, after dispatch, and in the rebuild that the
publication lease release starts). Each poll also asks GitHub for an
installation token and the pull request.

**Cause**

`dispatch` (`server/src/services/chat-github-checks.ts`) saved `retryAt
= now + 5s` after each poll. `entry()` in `review-check-work.ts` uses
that deadline when the state is unchanged. So an unresolved check was
due every 5 seconds without end.

## What Changed

- `chat-github-checks.ts`: save `unchangedPolls` in the check result. A
poll that finds the same state as the saved state adds one (up to 6). A
changed state, a first publication and a recovered failure set it to 0.
The next head poll is due after `min(5 minutes, 5 seconds ×
2^unchangedPolls)`.
- Export `GITHUB_REVIEW_CHECK_HEAD_POLL_MS` (5 seconds) and
`GITHUB_REVIEW_CHECK_MAX_HEAD_POLL_MS` (5 minutes).
- `chat-channels.integration.test.ts`: two regression tests. They use
the real review check queue, the real `dispatch`, embedded PostgreSQL, a
controlled clock and one pass each second. The fixture has the hosted
state: a review with no run, a delivery with a linked comment, and no
run for that comment.

## Verification

Reproduction with the real delivery coordinator, the real queue and
embedded PostgreSQL. One queued check with no findable run, 30 seconds
of real time, before this change:

| Company-wide hints | Statements a second | Check state reads a second
| GitHub pull reads a second |
| --- | --- | --- | --- |
| None | 8.4 | 0.70 | 0.17 |
| One each second (`notifyChatRunMilestones`) | 15.2 | 1.67 | 0.17 |

So one stuck check costs about 0.7 state reads a second from the head
poll alone. The hosted rate (10.3 a second) is consistent with about 15
such checks, or fewer checks with frequent company-wide hints.
Company-wide hints are a separate cause and a separate pull request.

Red/green regression test, 600 simulated seconds with one pass each
second:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "unresolved GitHub
review check cannot change|publishes a local change at once"`
- Red (without the fix): `expected 120 to be less than or equal to 7`.
Over 600 seconds: 5,160 statements, 360 check state reads, 120 GitHub
pull reads.
- Green: 258 statements, 18 check state reads, 6 GitHub pull reads (8.6
→ 0.43 statements a second, a 95% drop). The saved interval stops at 5
minutes.
- Correctness (green): after the back-off, a restarted queue makes no
early GitHub call. A run bound to the review publishes `in_progress` at
the next pass and resets the interval. A pushed head is still found
within 5 minutes, the check is cancelled and the review is superseded.

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed,
including all 12 review check queue tests.
- `cd server && npx vitest run
src/services/chat-github-review-repair.test.ts
src/__tests__/idle-completed-history.test.ts
src/__tests__/chat-admission-condition.test.ts
src/__tests__/app-private-hostname-gate.test.ts`: 109 passed.
- `cd server && node ../scripts/run-tsc.mjs` and `node
scripts/typecheck-layers.mjs` (with test layers): passed.
- `pnpm verify --explain` retains 2,718 test files conservatively, so I
did not run it locally. Required CI supplies that coverage.

## Risks

- A push whose webhook is lost is found later: after up to 5 minutes,
not 5 seconds, when the check has not changed for a while. The new
head's own webhook still creates its new check at once. Only the old
head's check stays queued or running on GitHub a little longer before it
is cancelled.
- Local changes are not delayed. `entry()` still makes a check due at
once when its state differs from the saved state, and run, review,
enqueue and lease hints still rebuild it.
- Saved actions from an older release have no `unchangedPolls`. They
start at 0. An invalid saved value also starts at 0. No migration.
- Restart recovery is unchanged: the deadline is saved on the action,
and startup rebuilds all unfinished checks.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local embedded PostgreSQL
reproductions).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 05:22:35 -06:00
DottaandPaperclip 7c0c3c4f31 fix(mcp): preserve Tasks sleep authority and re-enable recovery (#21603)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected apps can retain unfinished provider work through MCP
Tasks.
> - Managed defaults can enable Tasks without a stored experimental
flag.
> - The sleep check used the stored flag and could miss an active
provider task.
> - Enabling Tasks also failed to notify an owner that had become idle.
> - This pull request uses the effective setting and wakes the existing
owner after the setting commits.
> - Pending work keeps its recovery path. Completed history can become
quiet.

## Linked Issues or Issue Description

Refs #21263, which added durable provider Tasks. The current roadmap
covers the governed MCP Tool Gateway. This change repairs two existing
paths.

**What happened?**

The gateway could enable MCP Tasks through managed defaults while the
idle check reported no pending work. After an operator disabled and
enabled Tasks, an existing task could stay idle until a separate task
write woke its owner.

**Expected behavior**

The gateway and sleep check must use the same effective setting.
Enabling Tasks must notify the existing owner. It must retain the task's
original deadline and backoff.

**Steps to reproduce**

1. Retain a task with `next_poll_at`. Use managed Tasks defaults without
a stored true flag. Read the normal sleep projection.
2. Disable Tasks, retain the same task, then enable Tasks through the
settings service without writing the task again.

## What Changed

- Read the effective Tasks flag inside the same read-only sleep
transaction. Bind that value to the existing pending-task query.
- Register the existing Tasks queue notification before the settings
transaction writes. The normal commit protocol owns the wake.
- Add two real-PostgreSQL regressions to the existing nine-case suite.
Extend eight mock transactions with the read projection they now need.
Keep their assertions unchanged.
- Publish the Tailscale enrollment fixture state by rename so its CLI
watcher cannot parse a truncated JSON file. Existing enrollment
assertions and time limits remain unchanged.

## Verification

The two regressions fail against the preceding current-master products:
the pending-work projection reports `none`, and re-enable does not rearm
the owner within the original two-second oracle. After the fix, all nine
PostgreSQL cases pass at the recorded source, including completed
history, disabled Tasks, credential debt and fail-closed query errors.

At current `72bc32f`, the normal DB build and migrations, all nine
PostgreSQL cases, 155 affected mock cases, and the complete normal
Server typecheck/planner pass. The mock run includes three selected
safety families and six complete files; it explicitly excludes 196
existing durable-PG cases. All original case deadlines and assertions
remain. The startup file's earlier 133-case pass is limited to its
unchanged mock contract; a new hot-restart implementation is not
relabeled as tested. Root independently verified all current outputs and
process closure. The MCP product and regression files remain
byte-identical at `321c4b2ae2f3`; its additional change is the Tailscale
test fixture repair below. Exact current-head CI passes: 55 successful
checks, four explicit skips and one neutral result; Snyk succeeds. Fresh
Greptile is 5/5 for this exact head. The complete latest edited summary
and all paginated threads were inspected, with no open findings.

An earlier additional whole-eight-file run reached its 180-second
aggregate watchdog while the existing mixed PostgreSQL suite was still
running. It produced no individual failing assertion. Its startup file
passed 133 cases. The original failure remains recorded. The follow-up
selects the three affected mock families and runs the other affected
mock files whole; it does not claim that the interrupted PostgreSQL file
passed. No test assertion or case deadline changed. Full required CI
remains a merge gate.

The first remote run at `72bc32f` passed 51 checks but failed one
Tailscale enrollment case and an interrupted browser shard; two
aggregate gates then failed. The browser log records a runner shutdown
signal; its underlying cause is unknown, and one specific-job recovery
was requested. The Tailscale case failed its status CLI on both original
CI attempts. A local control then held publication until the original
file watcher observed an empty state: the CLI threw `Unexpected end of
JSON input`, and the existing enrollment assertion failed. Publishing
the state atomically passed the same control. All nine original
enrollment cases now pass in 4.393 seconds with the fixture repair. The
recorded remote interleaving remains unknown; the original logs and the
initial local non-reproduction are retained. No retry count, assertion,
timeout or product behavior was weakened. The later exact-head run
passes all required checks. A subsequent server shard was interrupted by
a runner shutdown and later recovered at the same source; the
interruption and unknown underlying cause remain recorded.

Use Node24/pnpm9, one worker, an owned external PostgreSQL test database
and the existing fixture/admin URL options. The two new regressions are
in `server/src/__tests__/idle-sleep-diagnostics-postgres.test.ts`.

These checks do not establish deployed suspension or a live provider's
availability.

## Risks

- The read adds the normal effective-settings projection to the existing
bounded snapshot. Query errors must still block sleep.
- A settings update now notifies the existing task owner. Transaction
rollback must not publish that notification.
- This adds no schema, provider protocol, timer or replacement task
framework. Existing active task polling remains required until its work
settles.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI `gpt-6.1-sol` implemented the regressions and advised the fix.
GPT-6-based Codex also reviewed and integrated it. The host does not
expose every supervising model ID or context-window size. The agents
used source editing, Git, code execution and regression tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (the focused scope above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (the
existing pending-task contract remains accurate)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-14 05:00:50 -06:00
DottaandClaude Opus 5.5 aaec67b891 fix(native): close idle warm sessions in parallel across agents at shutdown (#21597)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - At shutdown, the server closes its idle warm Runner sessions before
it prepares the hot-restart snapshot
> - It closed them one at a time, and each full close (checkpoint, then
two stopped-copy collections) takes 3-5 s
> - In a zero-downtime swap test, five closes used the whole 20 s
pre-drain budget. Hot-restart preparation timed out, a checkpoint was
cut mid-way, and shutdown logged "hot-restart shutdown preparation
failed"
> - The same test logged 34 "Delivery reconciliation failed" errors, all
from directory cleanup retries that found the agent directory lock busy
> - This pull request closes idle warm sessions in parallel across
agents, one at a time within an agent, and treats a busy directory lock
as a quiet retry
> - The benefit is that shutdown preparation fits its budget with
several warm sessions, and expected lock contention no longer looks like
a failure

## Linked Issues or Issue Description

**What happened?**
`closeIdleWarmNativeSessions` awaited each warm session's full close in
turn. In swap 2 of a release-swap test, five idle sessions closed at
22:28:42, :46, :51, :56 and :59 (one "transport_close_requested"
diagnostic each). The `hot_restart_prepare` step then timed out at 19.9
s. Separately, `recoverDirectoryRelease` threw
`AgentDirectoryOperationBusyError` when another close held the agent
directory lock. The delivery coordinator logged each one at ERROR level.

I first checked if these sessions should have taken the cheap
released-session detach. They could not: `entry.released` is set only
after a successful `session.release()` through a release/resume-capable
plugin Runner task. The Runner state files from all sandboxes in the
test recorded only `run.prepare`, `session.open`, `turn.start`,
`runner.drain` and `runner.suspend` commands, and no `runner.release`.
The Daytona daemon logs show only `/process/execute` and
`/process/session` calls, and no Runner-task API. So no session in that
test was ever released, and a full close was the only correct shutdown
for them.

**Expected behavior**
Idle warm sessions of different agents close at the same time. Two
sessions of the same agent never close together, because they collide on
the agent directory advisory lock. A cleanup retry that finds that lock
busy stays pending until its claimed retry time and logs no error.

**Steps to reproduce**
1. Keep five or more idle warm sessions on remote sandboxes.
2. Send SIGTERM.
3. The closes run one after another and `hot_restart_prepare` times out.

**Paperclip version or commit**
`master` at bbed32450c.

## What Changed

- `closeIdleWarmNativeSessions` groups sessions by company and agent. Up
to four agents close at once, and each agent's sessions close one at a
time. The per-session close logic is unchanged.
- Idle-sleep retirement (`prepareIdleWarmNativeSessionsForSleep`) keeps
a concurrency of one, so it still checks drain authority before each
retirement.
- `recoverDirectoryRelease` catches `AgentDirectoryOperationBusyError`,
keeps the deferred retry time, and returns the pending entry instead of
throwing.

## Verification

- Red, then green (each failed against `master` source and passes with
the fix):
- `closes idle warm sessions of different agents in parallel and one at
a time within an agent` in
`server/src/services/native-runtime/native-session-executor.test.ts`.
Before the fix, the second agent's close was never started while the
first agent's close was pending.
- `keeps a cleanup pending without an error while another server holds
the agent directory lock` in
`server/src/__tests__/agent-directory-working-copies.test.ts`. It holds
the real Postgres advisory lock from a second connection. Before the
fix, the coordinator received an `AggregateError` with
`AgentDirectoryOperationBusyError`.
- `native-session-executor.test.ts` (1502 passed) and
`agent-directory-working-copies.test.ts` pass in full.
- `pnpm typecheck` in `server` passes.

## Risks

- Up to four warm closes now run at once, so shutdown makes more Runner
and database calls at the same time. The cap keeps this small.
- The order of entries in `failures` can change when several closes
fail.
- A busy release now waits for its deferred retry silently. A lock that
stays busy for a long time shows as a stuck pending cleanup, not as
repeated errors.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.7
2026-11-14 04:15:19 -06:00
DottaandClaude Opus 5.5 12ac7cfd9f perf(recovery): read only the run keys that active execution recovery checks use (#21602)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Live recovery checks each running run when recovery work wakes it.
The check is `recoverActiveExecutions` in
`server/src/services/heartbeat/recovery.ts`.
> - Each check read the full `heartbeat_runs` row and the agent's
`adapter_config`. That includes `context_snapshot` and
`runner_profile_json`, which are large JSON columns.
> - Most checks only defer the run to its lease deadline. They use a few
JSON keys and never use the rest of the row.
> - Recovery wakes come from runner checkpoints, run events and company
work signals. Each running run is checked about 4.5 times per minute
under that load, so this read grows with the number of running runs.
> - This pull request reads only the JSON keys that the check gates use.
A run that recovery settles as lost reads its full context, runner
profile and adapter config before the settle.
> - The benefit is about 97% fewer bytes per check, with the same
recovery decisions.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The read that live recovery does for each running run
(`recoverActiveExecutions`).

**Subsystem affected**
Server: heartbeat run recovery
(`server/src/services/heartbeat/recovery.ts`).

**Current behavior**
Each check selects the whole `heartbeat_runs` row plus
`agents.adapter_config`. With a realistic row (`context_snapshot` 115 KB
of text, 23 KB stored; `runner_profile_json` 48 KB of text, 10 KB
stored), one check receives 175 KB from Postgres. A server with 20
running runs did 90 checks per minute, which is 15.7 MB per minute from
this one statement. That was 99% of all bytes in the measured window.

**Proposed behavior**
The check selects all scalar run columns, a small JSON object with only
the keys its gates read, and no output excerpts or adapter config. The
process-lost settle path reads the full `context_snapshot`,
`runner_profile_json` and `adapter_config` for that one run before it
decides on a retry.

**Reason and benefit**
The staging server spends 21-24% of main-thread CPU on postgres.js
result parsing at about 26 KB per statement. Whole-row reads in
recurring loops cause that overfetch. This change removes about 171 KB
from each check.

**Breaking changes**
None. No schema, API or contract change.

**Additional context**
I examined the `+1 s` re-arm (when revocation finds a child process that
the inventory snapshot did not have). I kept it. It only fires when the
revocation sees a changed row. The revocation also writes a new
60-second lease for a random boot ID, so the next pass defers on that
live lease. It does not loop on unchanged state. In the 60-second
measurement it did not fire.

## What Changed

- `recoverActiveExecutions` selects `activeRecoveryRunColumns`. This is
all `heartbeat_runs` columns, except that `context_snapshot` holds only
`wakeReason`, `issueId` and `nativeQuestionCancellation`,
`runner_profile_json` holds only `nativeStartupContext`,
`nativeWorkspaceSync` and `remoteWorkspace`, and the stdout/stderr
excerpts are null. It does not select `agents.adapter_config`.
- New `readActiveRecoverySettlement` reads the full `context_snapshot`,
`runner_profile_json` and `adapter_config` of one run. The process-lost
settle uses that data for the conversation-adapter check, the retry
budget, the process-loss diagnostic, the stop metadata and the
first-heartbeat flag.
- New tests in `server/src/services/heartbeat/recovery.test.ts`:
- A regression test counts the bytes Postgres sends during one deferred
check through a TCP proxy. It must be below 32 KB.
- A settle test (two cases) gives a lost run its retry budget,
first-heartbeat flag and conversation adapter only in the JSON keys
outside the narrow read. The test checks that the settle still uses
them.
- A test checks that a run with `nativeQuestionCancellation` in its
context is still skipped.

## Verification

- Red/green: the byte test fails on `origin/master` (`expected 454426 to
be less than 32768`) and passes with this change. The other three new
tests pass both before and after.
- Focused local runs pass: `recovery.test.ts`,
`live-recovery-work.test.ts`, `native-startup-cleanup-recovery.test.ts`,
`ownership-reconciliation-report.test.ts` (4 files, 255 tests).
- `pnpm verify` selected 2,715 of 3,541 files (conservative retention).
On this machine 1,689 files passed and 32 failed. The failures were
`afterAll` hook timeouts (30 s) and unrelated tool-access, email and
runner artifact suites under local load. The recovery suites above pass
when run alone. Required CI gives the full coverage.
- Before/after measurement (local embedded Postgres, a wire proxy that
assigns each server response to its statement, the real live-recovery
scheduler, 60 s window, 20 running runs: 10 native runs held by a live
controller and 10 legacy runs with a live peer lease; a run recovery
signal every 5 s for each run, a company work signal every 10 s, legacy
lease renewal every 20 s):

| | Before | After |
| --- | --- | --- |
| Checks per minute | 90 | 90 |
| Bytes per check (inventory statement) | 174,954 | 4,470 |
| Inventory bytes per minute | 15,745,819 | 402,298 |
| All bytes per minute in the window | 15,848,137 | 504,640 |

## Risks

- Low risk. The narrow read keeps the keys that the gates read. The
settle path reads the full row data before it uses it.
- The settle path now reads `context_snapshot`, `runner_profile_json`
and `adapter_config` after the lease revocation, not at the start of the
check. The values can be some milliseconds newer. The revocation already
uses the committed row state, so this agrees with the rest of the
settle.
- The settle path has one more query. It only runs when a run is settled
as lost.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as an agent in
Claude Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:58:47 -06:00
DottaandClaude Opus 5.5 30fc64e212 fix(instructions): collect a waiting working copy as soon as its stop proof is written (#21596)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run gets a private working copy of the agent's instructions or
agent files. A queue collects the copy after the provider is proven to
have stopped.
> - #21586 stopped this queue from re-reading copies forever: a copy
that waits for proof of the stop is now checked again after half of its
wait, capped at 6 hours.
> - With only that change, a copy whose stop is recorded T after its run
finished is collected up to T/2 later (at most 6 h). Before #21586 this
took at most 30 seconds. A sandbox that takes minutes to stop therefore
delays the save of an agent's instruction changes, and copies of
stranded runs wait up to 6 h after their stop is finally recorded.
> - This pull request makes the write that records the proof bring the
run's waiting copy forward, so the queue checks it on its next pass.
> - The benefit: collection is prompt again when the proof arrives,
without polling copies that are still waiting.

## Linked Issues or Issue Description

Follow-up to #21586. No public issue exists. Bug description:

**What happened?**
After #21586, a working copy waiting for proof of the provider stop
keeps its saved backoff deadline when the proof arrives. Proof is a
`native.local_process_stopped` event or a remote lease termination
receipt. A copy whose run finished 30 days ago is collected up to 6
hours after its stop is recorded.

**Expected behavior**
When a stop proof for a run is written, that run's waiting copy is
checked on the next queue pass. A write for a run without a waiting copy
costs one indexed no-op UPDATE and sends no work signal.

**Steps to reproduce**
1. Create a working copy for a native run that finished 30 days ago and
has no stop event, then let the collection queue check it once. Its next
check is about 6 h later.
2. Record the local stop (`recordNativeLocalProcessStopInTransaction`),
or release the run's remote lease with a termination receipt.
3. On master the copy stays in `pending_collection` until the old
deadline.

**Paperclip version or commit**
`origin/master` at 633dec7e43.

## What Changed

- New `notifyInstructionCopyStopEvidence(tx, { companyId, runId })` in
`instruction-collection-notifications.ts`.
- It reads the run's copy by its primary key (`run_id`, scoped by
`company_id`), and only if that copy waits in the collection queue or
the agent-files cleanup queue.
  - If there is no such copy, it stops: one SELECT, no write, no signal.
- If there is one, it first registers the run-scoped hint for both the
collection and the cleanup queue, and then runs one UPDATE by the
primary key. The UPDATE sets `next_attempt_at` to `now()` (or keeps an
earlier value) and sets a new `updated_at`. All of this happens in the
producer's transaction.
- The collection claim now compares `updated_at` with the value the
worker read (millisecond precision). If a stop proof is written between
the worker's evidence read and its backoff write, the claim fails and
the worker returns the copy as due, so the next pass collects it.
Without this fence, the worker could save a 6 h deadline over the proof.
- The writes that record stop proof call it in their own transaction:
- `native.local_process_stopped` receipts: `appendLocalProcessStop`
(used by `recordNativeLocalProcessStop` and its transaction form),
`recordUnidentifiedLocalRunnerStop`, and the operator confirmation in
`confirmUnidentifiedLocalRunnerStopped`.
- Remote lease termination receipts: `notifyEnvironmentCleanup` when it
has a run scope. Every lease release (`releaseLease`) and every
`writeEnvironmentCleanup` write calls it, which covers the receipt
writes in `environment-runtime`, `environment-task-runtime`,
`native-workspace-export-resume`, `sandbox-stop-and-retain` and
pending-cleanup recovery. The one remaining receipt write (sandbox
absence in `sandbox-stop-and-retain`) calls it directly.
- Orchestration's own stop path already calls `collectStopped` directly
after it joins the provider, so it collects without the queue and needs
no change.

## Verification

- `server/src/__tests__/agent-instruction-copy-retry-backoff.test.ts`, 3
new tests (embedded PostgreSQL, real queue registration):
- A local copy whose run finished 30 days ago (next check about 6 h
away) is collected within 10 seconds after
`recordNativeLocalProcessStopInTransaction` records the stop.
- A remote copy with the same backoff is collected within 10 seconds
after `environmentService.releaseLease` writes a termination receipt.
- For a run without a waiting copy, the transaction contains one SELECT
and nothing else, and no hint is sent. For a run with a waiting copy,
the only hint is `{"companyId":…,"runId":…}` (run-scoped, not
company-wide).
- A stop recorded between the worker's evidence read and its claim
(injected in the claim's transaction hook) is still collected on the
next pass.
- Red first: with the producer changes reverted, the two collection
tests fail (the copies stay in `pending_collection` until the old
deadline). Without the `updated_at` fence, the race test fails.
- Existing suites, one at a time: see the PR comment for the list and
results.
- `pnpm check:module-boundaries` passes.

## Risks

- Every lease cleanup write with a run scope now runs one extra
primary-key SELECT on `agent_instruction_working_copies`, and an UPDATE
only when a waiting copy exists.
- For agent-files cleanup (scratch files only, the saved data is not at
risk), the same race is not fenced. A missed advance there waits for the
cleanup backoff.
- A lease cleanup write that does not carry a termination receipt can
also bring a waiting copy forward. The queue then checks the stop
evidence once more (3–5 statements) and moves the copy back to its
backoff if the stop is still not proven.
- A release written with only a lease scope (no run) does not bring a
copy forward. That copy is still collected at its saved deadline.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:57:36 -06:00
DottaandClaude Opus 5.5 0ab63cf4d9 fix(restart-chaos): ignore a done the server rejected as incomplete (#21598)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness runs staging-mix waves and grades each
task for duplicated side effects and extra turns
> - The grader counts a run as a completer when its last accepted result
reports `done`
> - The server can reject that `done` as incomplete. The status arbiter
then queues a same-agent continuation wake with key
`native-completion-incomplete`
> - The grader ignores the rejection. It reports a false duplicate
completion and anchors completion on the rejected run, so the
continuation run that finished the work shows as an extra turn
> - This pull request marks a `done` as rejected when a
`native-completion-incomplete` wake for the issue follows it, and
ignores that `done` unless the same run posted the DONE comment
> - The benefit is that wave results stop reporting false duplicates and
false extra turns, so real regressions stay visible

## Linked Issues or Issue Description

**What happened?**
In wave `w-20261114T081656-s7105`, task t005, run d80d3ab3 reported
`done` with no DONE comment. The server rejected it as incomplete and
queued a `native-completion-incomplete` continuation wake. Run a777038c
then posted the only DONE comment and completed the issue. The grader
reported "duplicate completion: runs d80d3ab3, a777038c" and listed
a777038c as an extra turn.

**Expected behavior**
The grader reports one completer (a777038c), anchors completion on
a777038c, and lists no extra turn.

**Steps to reproduce**
1. Build wake evidence where run A has an accepted `done` result and no
DONE comment.
2. Add a continuation wake with payload `continuationIdempotencyKey:
"native-completion-incomplete"` created after A's result.
3. Add run B from that wake. B posts the DONE comment and reports
`done`.
4. Grade the task with `gradeStagingMix`, `completedAtAnchor` and
`runsAfterDone`.

**Additional context**
This is a test-harness change only. It changes no product code.

## What Changed

- `tests/restart-chaos/follow-up-delivery.ts`: add `rejectedDoneRuns`.
It attributes each `native-completion-incomplete` wake to the latest run
whose accepted `done` came at or before the wake.
- `tests/restart-chaos/wake-evidence.ts` (`completionEvidence`) and
`tests/restart-chaos/staging-mix-retry.ts` (`evidenceCompletion`): set a
new optional `doneRejected` flag on each completion run.
- `tests/restart-chaos/staging-mix.ts`: `completedAtAnchor` and
`completionRunIds` ignore a rejected `done`, unless that run posted the
DONE comment.
- `tests/restart-chaos/staging-mix-retry.test.ts`: add the t005
red/green test, a test for a rejected run that posted DONE itself, and a
G5c t004 guard.
- `tests/restart-chaos/README.md`: document the rule.

## Verification

- Red on master, green with the fix: `a done the server rejected as
incomplete > anchors on the run that finished the work, with one
completer and no extra turn (w-20261114T081656-s7105 t005)`. On master
the anchor is the rejected run's result time (1794646168026) instead of
the finisher's (1794646218506).
- Guard, green on master and with the fix: `a done the server rejected
as incomplete > keeps a run woken by recovery after a real completion as
an extra turn and a duplicate completion (G5c t004)`.
- `npx vitest run --config tests/restart-chaos/vitest.config.ts
--maxWorkers=3`: 71 files, 816 tests pass.
- `tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean.
- The real t005 evidence from the wave now grades as one completer
(a777038c) with no extra turns.
- `pnpm verify` exits 2 because `follow-up-delivery.ts` is a global CI
input. Required CI supplies the broad coverage.

## Risks

- Low risk. The change affects only the restart-chaos grader.
- The grader trusts the wake key. If the server queued
`native-completion-incomplete` for another reason, the grader would skip
the latest `done` before it. The server queues this key only from the
incomplete branch of `arbitrateNativeStatus`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:56:51 -06:00
github-actions[bot]andlockfile-bot 0b2b7aa737 chore(lockfile): refresh pnpm lockfiles (#21604)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.
- Resync the SDK public source proof's lock pins from the refreshed
lock.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when every added
package version is at least 3 days old on npm. Otherwise the PR waits
for a maintainer, and the daily refresh tries again. The bot merges only
the exact commit this workflow generated from manifests already reviewed
on master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
canary/v2026.1114.0-canary.6
2026-11-14 09:49:06 +00:00
DottaandClaude Opus 5.5 d25667f370 perf(db): keep postgres.js connection queues proportional to live connections (#21601)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - The driver keeps its pool connections in small FIFO queues (`open`,
`busy`, `full` and others) in `src/queue.js`
> - The `open` queue keeps an empty slot for every connection it hands
out until it is completely empty, and `remove` scans every slot
> - A busy pool always has some idle connections, so `open` never
empties. Each query then scans one slot for every query the process has
already run
> - The main thread of a long-running server therefore spends more time
on each query as the process ages
> - This pull request compacts the queue once empty slots outnumber
queued items, and makes `remove` search only the queued items
> - The benefit is that query dispatch cost stays flat for the life of
the process instead of growing linearly

## Linked Issues or Issue Description

**What happened?**
A hosted Paperclip server spends more of its main thread in the
postgres.js dispatcher the longer it runs. A CPU profile 14 minutes
after start showed the driver's `handler` (`src/index.js:350`) at 9.6%
self time, plus 3.8% more in the same inlined queue work. The server had
about 117 open sockets, a pool maximum of 500, and about 1,000
statements per second.

The cause is in `src/queue.js` (and `cjs/src/queue.js`). `shift` resets
its backing array only when the queue is completely empty. Otherwise it
leaves an `undefined` slot. `handler` takes a connection with
`open.shift()`. Then `move` calls `open.remove(c)` for a connection that
`shift` already took out, so `indexOf` scans the whole backing array and
finds nothing. When the query finishes, `move(c, open)` pushes the
connection back. The backing array grows by about one slot per query.

**Steps to reproduce**
1. Open a pool with more connections than the number of concurrent
queries (for example 24 connections and 8 concurrent queries).
2. Run 200,000 `select 1` queries.
3. Measure main-thread CPU per query for each block of 20,000 queries.
It grows from 30.6 µs to 48.7 µs.
4. Or run the same queue operations without a database: with 117
connections and 8 in flight, the cost per query is 16 µs at 200,000
queries and 151 µs at 1,000,000 queries.

**Expected behavior**
The cost to dispatch a query depends on the number of connections in the
queues, not on how many queries the process has run.

**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18, hosted and local servers.

## What Changed

- `patches/postgres@3.4.9.patch`: adds one hunk each to `src/queue.js`
(ESM, which the server loads) and `cjs/src/queue.js` (CommonJS). All
existing hunks are unchanged. The added hunk is:

  ```diff
       remove: (x) => {
  -      const index = xs.indexOf(x)
  -      return index === -1
  +      const at = xs.indexOf(x, index)
  +      return at === -1
           ? null
  -        : (xs.splice(index, 1), x)
  +        : (xs.splice(at, 1), x)
       },
  ...
         if (index === xs.length) {
           index = 0
           xs = []
  +      } else if (index > 1024 && index * 2 > xs.length) {
  +        xs = xs.slice(index)
  +        index = 0
         } else {
           xs[index - 1] = undefined
         }
  ```

The compaction copies only the queued items, and only after more than
1,024 shifts and after empty slots outnumber queued items. Its cost is
therefore amortized O(1) per `shift`. FIFO order, `length`, `push`,
`shift` (including `shift` on an empty queue) and `remove` (including
`null` for an absent item) keep the same results.
- `packages/db/src/postgres-queue.test.ts`: for both the ESM and
CommonJS builds, checks FIFO order, `length` and `remove` results. It
compares the queue with a plain array over 300,000 random operations
that cross the compaction threshold. It also runs 1,000,000 queries in
the pattern where `open` never empties, and checks that the slots
scanned by `remove` and the backing array size stay bounded by the live
connections.
- `packages/db/src/__fixtures__/postgres-queue-pool.mjs` and
`packages/db/src/postgres-queue-pool.test.ts`: run a real pool (24
connections, 8 concurrent queries, 20,000 queries) against embedded
PostgreSQL for both builds. They check that a dispatch scans no more
slots than there are connections.
- `packages/db/scripts/benchmark-postgres-queue.ts`: reports per-query
cost for the queue simulation (1,000,000 queries) and for the real pool
(200,000 queries, 10 blocks).
- `doc/DATABASE.md`: describes the fix, the tests and the benchmark
command.
- `pnpm-lock.yaml` is not changed. The lockfile refresh workflow updates
the patch hash, as it did after #21432.

## Verification

- Red before the fix: `pnpm --filter @paperclipai/db exec vitest run
src/postgres-queue.test.ts src/postgres-queue-pool.test.ts` failed 4
tests. The unit test scanned 13,340 slots (limit 125). The real pool
scanned 5,017 slots in its first 5,000 queries (limit 24). The semantic
tests passed before and after the fix.
- Green after the fix: the same command, plus
`src/postgres-receive-buffer.test.ts` and
`src/postgres-connection-recovery.test.ts`, passes 32 of 32 tests.
- `pnpm --filter @paperclipai/db exec tsc --noEmit` passes.
- `pnpm verify` exits 2 because the patch is a global CI input. Required
CI supplies the broad coverage.
- Benchmark: `pnpm --filter @paperclipai/db exec tsx
scripts/benchmark-postgres-queue.ts` on Apple Silicon, Node.js 26.4,
embedded PostgreSQL 18:

  | Measurement | Before | After |
  | --- | --- | --- |
| Simulation, µs per query at 200k / 400k / 600k / 800k / 1M queries |
16.0 / 50.4 / 81.3 / 128.4 / 150.7 | 0.070 / 0.070 / 0.071 / 0.073 /
0.072 |
| Real pool, main-thread CPU µs per query, first vs last 20k of 200k |
30.6 vs 48.7 (+59%) | 22.8 vs 14.6 (first block includes warm-up) |
| Real pool, CPU µs per query, blocks 2 to 10 | 29.6 rising to 48.7 |
13.3 to 15.9, flat |

## Risks

- Low risk. The change is limited to the driver's internal queue. The
results of every queue operation are the same, and the semantic test
compares them with a plain array over 300,000 random operations in both
builds.
- `remove` now searches from the first queued item. Slots before it are
always empty, so it cannot miss a queued item. Before this change,
`remove(undefined)` could remove an empty slot. The driver never calls
it that way.
- Compaction allocates a new array with only the queued items. It runs
at most once per 1,024 shifts, so memory use falls.
- The lockfile refresh workflow updates the patch hash in
`pnpm-lock.yaml` after merge, as it did after #21432.
- A possible follow-up is to offer the same fix to the upstream
postgres.js project. This pull request does not do that.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:46:41 -06:00
DottaandClaude Opus 5.5 009c8884a4 perf(attention): read blocker retention state once per page (#21595)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `attention-retention` queue archives old attention items. One
kind is `blocker_attention`: a blocked task whose terminal blocker needs
a person.
> - When the queue loads a page of blocked tasks, it reads `keep` and
`archived_at` from `decision_retention` for each of them, one full-row
`select` per task.
> - A company hint walks every page. On a hosted instance with 2,112
blocker attention items and one active run, these per-task reads ran at
about 16 per second.
> - This pull request reads the retention state for the whole page in
one narrow statement. It also scopes three remaining run-path attention
hints to the task graph.
> - The benefit is one retention read per page instead of one per
blocked task, with the same decisions.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The `blocker_attention` eligibility read in
`server/src/services/attention-retention.ts` (`eligible()`), and three
run-path producers of company-wide attention hints.

**Subsystem affected**

Server background retention (`attention-retention.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-review-participant.ts`,
`native-runtime/native-workspace-reconstruction.ts`).

**Current behavior**

- For each blocked task whose blocker attention is terminal (and for the
owner variant), `eligible()` ran `select <all columns> from
decision_retention where company_id = $1 and source_kind = $2 and
source_id = $3`. It did this during `load`, once per task, on every
company hint.
- The hosted statement is identified by that column prefix: `id,
company_id, source_kind, source_id, source_activity_at, …`.
- I checked every other full-row reader of `decision_retention`. None
fires per run event:
- `syncItems`: attention summary and list requests, which a board calls
once per load; retention `check` and the sweep, which run for due
entries only; Keep and Archive actions.
  - `getState`: user actions only.
  - `archiveReviewedManifest`: proposal acceptance only.
  - `decisions.list`: only when archive proposals exist.
- `native-provider-capacity-retry`, `native-review-participant` and
`native-workspace-reconstruction` sent `{ companyId }` attention hints,
which make the queue walk every page.

**Proposed behavior**

- `pageEvidence` reads `source_id`, `keep` and `archived_at` for every
blocker root and every terminal blocker on the page in one statement
(ids passed as one JSON parameter). `eligible()` uses that map when it
has page evidence.
- A missing row means "not kept, not archived", exactly as the per-task
read returning no row did.
- The per-entry `check` path still reads one row, now only `keep` and
`archived_at`.
- The three producers send `notifyIssueAttentionRetentionWork` with
their task (and run). That is the task, its parents and the tasks it
blocks, with a company fallback past 100.
  - A capacity retry changes the failed-run items of its task.
  - A review participant claim changes the review of its task.
- A workspace reconstruction resolves the recovery action of its task,
which can also change the blocker attention of the tasks it blocks.
- The configuration, lifecycle, import and fallback producers keep their
company hints.

**Reason and benefit**

Background work must not grow with the backlog. A page now costs one
retention read instead of up to 100.

## What Changed

- `server/src/services/attention-retention.ts`: batched and narrowed
blocker retention read; `PageEvidence.blockerRetention`.
-
`server/src/services/native-runtime/native-provider-capacity-retry.ts`,
`native-review-participant.ts`, `native-workspace-reconstruction.ts`:
scoped attention hints.
- Tests in `server/src/__tests__/attention-retention-work.test.ts` and
`server/src/services/native-runtime/native-workspace-reconstruction.test.ts`.

## Verification

- `reads blocker retention state once per page, not once per blocked
task` (I saw it fail first). 160 blocked tasks, each behind its own
unassigned blocker; the blockers' retention rows are kept (40), archived
(40), plain (40) or missing (40). The full-size response capture and the
statement log measure the load of both pages:
- before: one `decision_retention` read per task (160), 26,240 result
bytes;
- after: 2 reads (one per page), 5,480 bytes, none of them selecting
`source_activity_at`.
  - The same tasks are listed: only plain and missing.
  - `check` still drops the kept and archived ones.
- `scopes the attention retention hint of a reconstruction to the task
graph` (I saw it fail first): before, the reconstruction sent `{
companyId }`; after, it sends a hint that names the task.
- `batches 100 blocker graphs with exact terminal timestamps and fresh
terminal visibility` pinned 200 retention reads per page (one per root
and variant). It now expects 1; the rest of that test is unchanged and
passes.
- Not separately tested: the capacity-retry and review-participant hint
changes. Each is one line that passes the task the function already
holds, the same pattern as the reconstruction test.
- `attention-retention-work` (58), `attention-retention-candidates`,
`native-workspace-reconstruction`, `native-review-participant` and
`heartbeat-run-dispatch-work` pass, each file run alone (in `server/`,
against dedicated local `_test` databases).
- `pnpm check:module-boundaries` passes.

## Risks

- The page read happens once, before the per-entry decisions of that
page. Before, each decision read its row just in time. A Keep or Archive
that lands between the page read and an entry's decision is seen by the
next load or by `check`, which re-reads under a row lock, as before.
- A run-path hint that changes more than its task graph would now miss
the wider effect. The 90-day archive delay makes a missed reload cheap.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:05:59 -06:00
DottaandClaude Opus 5.5 cf09024292 fix(heartbeat): count an early native hand-off only after its suspend commits (#21572)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each live Runner run to the next server, so
the agent keeps working without a retry
> - At shutdown start, the outgoing server hands off native runs early,
in parallel with the rest of shutdown
> - The early hand-off put a run in the "handed off" set before its
suspend finished, and the drain counted every run in that set as
suspended without waiting
> - The early hand-off shares the 20 s pre-drain budget, and one Runner
detach can take 20-30 s, so the process could exit with the suspend
unfinished
> - Then the finalization lease stays live, the next server gets no
hand-over signal, and the run waits for lease expiry or the reaper
> - This pull request makes the drain wait for each in-flight early
hand-off, and fall back to its own hand-off when the early one failed or
did not finish
> - The benefit is that a run counts as handed over only after its
hand-over is committed, so the next server can always reach it at once

## Linked Issues or Issue Description

**What happened?**
A passing zero-downtime swap test showed a latent race in the shutdown
hand-off. `handOffNativeRunsForShutdown` added a run to
`nativeRunsHandedOffEarly` before `suspendNativeRunForShutdown`
finished. `drainRunningRunsForShutdown` then reported every run in that
set as `restartSuspended` without awaiting it. Because the early
hand-off shares the 20 s pre-drain budget and a Runner detach can wait
20-30 s, the process can exit before the lease release commits. Also, a
timed-out `prepareHotRestartShutdown` could still write the hot-restart
snapshot and detach sessions after the fallback drain started. In the
same test, swap 1's early hand-off took 2.08 s, and no log showed which
phase was slow.

**Expected behavior**
A run counts as handed over only after its suspend commits. If the early
suspend fails or does not finish, the drain hands the run off itself. A
timed-out hot-restart preparation does no further work after the drain
starts. The hand-off logs its phase timings.

**Steps to reproduce**
1. Start a native Runner run on a remote sandbox.
2. Send SIGTERM while the Runner detach is slow (or fails).
3. The drain reports the run as restart-suspended while its suspend is
still pending, and the process can exit before the lease release.

**Paperclip version or commit**
`master` at bbed32450c.

**Deployment mode**
Hosted, zero-downtime release swap.

## What Changed

- `nativeRunsHandedOffEarly` now holds each run's in-flight suspend
promise, not only its id.
- The shutdown drain awaits that promise (bounded at 30 s). If it fails,
returns false or does not finish, the drain runs its own suspend, as it
does for a run without an early hand-off.
- `runBoundedShutdownStep` and the pre-drain budget pass an
`AbortSignal` to each step and abort it on timeout.
- `prepareHotRestartShutdown` takes that signal and stops before the
snapshot write, the run events, and the session detach when the step
timed out.
- `handOffNativeRunsForShutdown` logs one structured line, "native run
shutdown hand-off phase timings": intent read, controller identity,
select, max per-run detach, max per-run transaction, and total.

## Verification

- Red, then green (each failed against `master` source and passes with
the fix), all through the public shutdown functions in
`server/src/services/heartbeat/recovery.test.ts`:
- `does not count an early hand-off as suspended while its detach is
still pending at the shutdown drain`
- `hands off a run at the shutdown drain when its pending early hand-off
fails`
- `writes no hot-restart snapshot and detaches nothing after a timed-out
preparation hands over to the drain`
- New test in `server/src/shutdown.test.ts`: `aborts the signal of a
step that outlives its bound, and only that step`.
- `pnpm vitest run src/shutdown.test.ts
src/services/heartbeat/recovery.test.ts
src/__tests__/heartbeat-native-early-hand-off.test.ts` passes (145
tests).
- `tsc --noEmit` for `server` passes. `pnpm verify` passes for the
selected tests.

## Risks

- When an early hand-off does not finish within 30 s, the drain starts a
second suspend. The second detach joins the first, because the session
memoizes its restart detach. The run can then get two "suspended" run
events. That is preferred to reporting a hand-over that did not happen.
- The drain can now wait up to 30 s for a slow early hand-off before the
process exits. Before, it exited without the hand-over.
- The abort signal only stops work between steps of the preparation. A
snapshot write that already started still completes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:02:25 -06:00
DottaandClaude Opus 5.5 633dec7e43 fix(instructions): back off working copies that cannot make progress (#21586)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run gets a private working copy of the agent's instructions or
agent files. Background queues collect the copy after the provider stops
and clean up its scratch directory.
> - When the queue cannot prove that the provider stopped, or cannot
finish a cleanup, it re-arms the same copy every 30 seconds with no
limit. The attempt count does not increase on the "stop not proven"
branch.
> - Every re-arm is a transaction that sends work signals to its own
queue, to the stranded-agent-status queue and to the skill-retention
queue. A self-signal also reloads the copy's scope. On a hosted instance
with only a few live runs, these copies were one of the largest sources
of SQL calls and kept the main thread busy.
> - This pull request makes a copy that cannot progress wait on a
growing, capped backoff. It also removes the work signals from a write
that only moves a deadline later, and reads the copy once per check.
> - The benefit: a copy that waits for proof of the provider stop costs
about 1/350 of the old steady-state load, and a copy that becomes
collectable is still collected automatically.

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened?**
A hosted instance had only a handful of live agent runs, but its server
main thread was about 95% busy. An analysis of its most-called SQL found
the instruction working-copy queues re-reading the same unresolved
copies forever, at about 25–30 statements/s. A copy whose run is
terminal but whose provider stop is not proven was re-armed every 30 s.
A completed agent-files copy whose remote scratch cleanup cannot finish
(its lease is released but not destroyed, or the lease row is gone) was
also re-armed every 30 s. Each re-arm was a transaction with work
signals that woke three queues.

**Expected behavior**
A copy that cannot progress is not re-read until a growing, capped
backoff elapses. A write that only moves a deadline later does not wake
other queues. A copy that becomes collectable is still collected
automatically, and no data is lost.

**Steps to reproduce**
1. Create working copies of terminal native runs that have no
`native.local_process_stopped` event, and completed agent-files copies
with `cleanupPending: true` whose remote lease does not exist.
2. Register the instruction-collection and directory-cleanup queues.
3. Count statements while time advances. With 300 + 300 such copies,
master sends 252 statements/s, 38.7 transactions/s and 38.7 work
signals/s.

**Paperclip version or commit**
`origin/master` at 3c3e6c2d75.

## What Changed

- `instructionCopyRetryAt(waitingSince)` sets the next check to half of
the time the copy has waited. The minimum is `INSTRUCTION_COPY_RETRY_MS`
(30 s) and the maximum is `INSTRUCTION_COPY_MAX_RETRY_MS` (6 h). An
unproven stop measures the wait from the run's `finished_at`. A cleanup
measures it from `process_stopped_at`. The deadline stays in the
existing `next_attempt_at` column, so a restart keeps it. There is no
schema change.
- The collection check now reads the copy once, checks the stop
evidence, and then claims. For an unproven stop, the claim writes the
backoff deadline (and the move from `prepared` to `pending_collection`)
in the same UPDATE. Before, the check was two reads plus two signalling
transactions.
- `tryPatch` skips all work signals when a write only moves deadlines
later and keeps the copy in the same queues (collection, cleanup, commit
and the stranded-agent predicate). A write that changes queue membership
or moves a deadline earlier still signals as before. The write stays in
its own transaction, so the existing claim and race tests keep their
hook.
- A cleanup that cannot finish gets the same capped backoff after
`release` re-arms it. The saved copy and its cleanup receipt are kept.
The claim does not change `attempts`, because a concurrent foreground
collection uses `attempts` in its compare-and-set.
- A remote agent-files copy that stopped but was not destroyed, and an
unavailable agent-files copy that waits for a destroy receipt, also get
the capped backoff instead of a 30 s re-arm.
- No copy is parked or dropped. A copy with an unproven stop stays in
the queue with the 6 h cap. When proof of the stop arrives later (a
`native.local_process_stopped` event, a lease termination receipt, or a
recovery that records that the host is gone), the next backoff check
collects it automatically. No manual call is necessary.

## Verification

- New suite
`server/src/__tests__/agent-instruction-copy-retry-backoff.test.ts`
(embedded PostgreSQL, real queue registration, statement meter):
- The backoff constants and their values at 0 s, 30 min, 2 h, 12 h and
30 days.
- A copy that has waited 2 days: two checks in 12 h, 5 statements and 1
transaction per check, and zero work signals.
- A copy owned by a live older server (run finished 5 minutes ago): it
is checked every few minutes. When the older server records the stop,
the copy is collected within 10 minutes, without a manual call.
- A copy that has waited 30 days: it is collected within one 6 h cap
after the stop is recorded.
- A backlog of 300 copies that have waited 30 days: one pass drains it
with zero work signals, and the hour after it is silent.
- A write that moves the deadline earlier, and a write that changes
queue membership, both still send the signal.
- A cleanup that cannot finish, with release returning and with release
throwing: at most 32 deadline writes in the first 6 h, at most 4 in the
next 6 h, zero work signals, and the copy is still saved with its
cleanup pending. Before the fix, the throwing case made 74 writes in 6
h.
- Red first: on master this suite fails 5 of 7 tests (the 30 s re-arm,
150 statements in the drain window, extra signals, a cleanup checked
every 30 s). The two "collects after stop" tests pass on master and
still pass: the positive behavior is kept.
- Steady state with 300 unproven stops plus 300 stuck cleanups, each
waiting for 1 day (10-minute window after warm-up):

| | statements/s | transactions/s | work signals/s | stop checks/s |
cleanup checks/s |
  | --- | --- | --- | --- | --- | --- |
  | master | 251.6 | 38.7 | 38.7 | 9.7 | 9.6 |
  | this PR | 0.45 | 0.10 | 0 | 0 | 0.03 |

- Per check: an unproven stop goes from 12 statements, 2 transactions
and 2 work signals to 5 statements, 1 transaction and 0 work signals. A
stuck cleanup goes from about 14 statements, 2 transactions and 2 work
signals to about 15 statements, 3 transactions and 0 work signals.
Cleanup checks are exponentially less frequent.
- Backlog drain: queue batches stay at 20 per pass, with one pass per
second at most. An overdue copy moves to its long backoff on its first
check. The first drain of N stuck copies therefore takes about N/20
seconds. For example, 17,000 copies drain in about 15 minutes with no
work signals.
- Existing suites, one at a time: `agent-instruction-working-copies`
(50/50), `agent-directory-working-copies` (120/120),
`heartbeat-instruction-working-copy` (4/4),
`heartbeat-agent-directory-busy` (8/8), `agent-file-checkpoint-locking`
(9 passed, 1 skipped), `agent-file-durability` (17/17),
`idle-sleep-safety` (226/226), `heartbeat-process-recovery` (459/459),
`agent-instructions-routes` (26/26), `services/heartbeat/agent-status`
(42/42).
- `pnpm check:module-boundaries` passes.

## Risks

- Product-visible choice: the 6 h cap. A copy that has waited for a long
time is collected at most 6 h after its stop proof arrives (half of the
wait while the wait is shorter than 12 h). During that time, the
uncollected candidate shows in the agent's instruction candidates
(`pending_collection`, `INSTRUCTION_STOP_UNCONFIRMED`) for legacy
copies, and in the run's agent-files diagnostics for agent-files copies.
Its bytes are kept on disk. An explicit `collectStopped` from
orchestration still collects immediately.
- There is no seven-day park. A park at `attempts = 3` would leave the
queue predicate, and a later stop proof would not reach the copy without
changes in files that another fix owns. The copy therefore stays in the
queue with the 6 h cap. The steady cost is about 5 statements per copy
every 6 h.
- During the 40-minute deploy overlap, a copy owned by a live older
server waited only a short time, so its backoff is still 30 s to a few
minutes. The test covers this case.
- A write that only moves a deadline later no longer wakes the
stranded-agent-status and skill-retention queues. The stranded-agent
predicate is compared before and after the write, and any change still
signals.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.5
2026-11-14 02:47:06 -06:00
DottaandClaude Opus 5.5 c3f9a0c040 fix(retention): evaluate native retention blockers for many sessions in one statement (#21588)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep runner checkpoints and harness backups so that a
session can resume. Retention expires them after the session can no
longer resume.
> - A retention blocker decides whether a session can still resume: a
live or recent run, an open or recently closed task, an unsettled
finalization, or a live lease.
> - The startup checkpoint pass evaluated that blocker in its own
transaction, once for each retained checkpoint, with three reads. A
session that stays blocked (for example, a long-lived open task) was
evaluated again for each of its runs on every server start. On a hosted
instance with frequent deploys and a 40-minute overlap of two servers,
this pass ran almost continuously at about 7 checkpoints/s, about 21
statements/s.
> - Backup reconciliation also used four reads and a transaction for
each run hint that reached a backup.
> - This pull request evaluates the blocker for many sessions in one
fixed-text statement. Only candidates that are not blocked enter the
existing locked transaction.
> - The benefit: a restart costs 2 statements for each page of 200
blocked checkpoints instead of 5 statements and 1 transaction for each
checkpoint, and a backup hint costs 2 statements with no transaction.

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened?**
A hosted instance had only a handful of live agent runs, but its server
main thread was about 95% busy. An analysis of its most-called SQL found
three retention-blocker reads (`company_id, agent_id from
heartbeat_runs`; the session's runs; the session's issues) at exactly
7.0/s each, each in its own transaction. The startup checkpoint pass
sent them once for each retained checkpoint older than 7 days whose
session was still blocked, and it repeated the full pass after every
start.

**Expected behavior**
A checkpoint or backup that is blocked from expiring costs no
transaction and no statements of its own, because one statement
evaluates its session together with the rest of its page. It expires as
soon as its blocker clears: checkpoints on the next startup pass (the
current design), and backups at their exact deadline.

**Steps to reproduce**
1. Create 100 native sessions with 3 finished runs each (finished 10
days ago), settled finalizations, a checkpoint header for each run, and
an open task.
2. Register the startup checkpoint expiration and count statements. Then
register it again, as a restart.
3. On master each start costs 1,502 statements and 300 transactions, and
nothing expires.

**Paperclip version or commit**
`origin/master` at 3c3e6c2d75.

## What Changed

- `SESSION_RETENTION_BLOCKERS_SQL` (new module
`native-runtime/native-session-retention-blockers.ts`) is one fixed-text
statement, sent through `$client.unsafe`. It takes a JSON list of
sessions and returns, for each one, the same blocker reasons as
`sessionRetentionBlocker`, in the same order: missing run, live run,
incomplete native recovery, recent run (checkpoint mode), unknown, open,
executing or recently closed task, pending finalization, live lease. It
also returns the session's resume pointer, run, task and lease
dependencies, and the latest close time.
- `expireCheckpoints` and `expireDurableBackups` evaluate each page of
candidates with this statement. They skip blocked candidates and
candidates whose session still has a resume pointer. A candidate that is
not blocked still enters the existing transaction. That transaction
checks the blocker again under row locks before it retires anything.
- `reconcileBackup` uses the statement for its dependencies and its
deadline. It replaces three dependency reads plus a blocker transaction.
The lease references use one index-backed arm for each reference kind
(run, backup stamp, task, identity lease).
- The unused `lock: false` path of `sessionRetentionBlocker` and the
`backupDependencies` helper are removed.
- The existing index test now checks the plan of the new statement.
Every arm that reads `environment_leases` uses an index in both the
custom and the generic plan.

## Verification

- New suite
`server/src/__tests__/native-retention-blocker-batch.test.ts` (embedded
PostgreSQL, real coordinator registration, statement meter):
- 300 blocked checkpoints: a start costs 4 statements and 0
transactions, and a restart costs the same. On master it is 1,502
statements and 300 transactions for each start. The checkpoints are
kept.
- When a session's task is closed long enough ago, its checkpoints
expire on the next startup pass. The other sessions are not touched.
- A session with a resume pointer keeps its checkpoints for 2 statements
in total.
- Two servers that run the startup pass at the same time expire every
checkpoint exactly once with no failures, because the locked transaction
checks again and the retirement header is compared before deletion.
- 50 run hints that reach 50 blocked backups: at most 2 statements for
each hint (the scope lookup and one blocker statement) and no
transaction. On master it is 9 statements and 1 transaction for each
hint. The backups are kept.
- Existing suites, one at a time: `native-harness-backup-expiration`
(109/109, including the backup expiry flows for task close, reopen and
lease release), `native-harness-backup-expiration-config` (7/7),
`native-harness-backup-storage.integration` (73/73),
`native-harness-backup-publication` (4/4),
`native-retired-backup-source.integration` (38/38).
- `pnpm check:module-boundaries` passes.

## Risks

- The startup pass still runs on every server start. It also runs on
both servers during the deploy overlap. Its new cost is 2 statements for
each page of 200 candidates (the candidate page and one blocker
statement). A page of candidates that are not blocked still uses one
locked transaction for each candidate, as before.
- Two servers can run the pass at the same time. They cannot retire the
same session twice or retire a blocked one: the unlocked statement only
filters, and the existing transaction locks the finalizations, checks
the blocker again, and compares the retirement header before it deletes.
The concurrency test covers this.
- If the SQL blocker and the TypeScript blocker ever disagree in the
"not blocked" direction, the locked transaction still refuses. If they
disagree in the "blocked" direction, retention is only delayed. When
more than one task blocks a session, the reported reason can differ from
the TypeScript loop, which reads the tasks in no fixed order.
- Checkpoints keep their current design: they are considered again on
the next startup, not at a deadline. Backups keep their exact deadline
after a task closes.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:47:01 -06:00
DottaandClaude Opus 5.5 f452b449fd fix(recovery): stop re-checking settled native runs that cannot be released (#21590)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server releases the process identity (`process_pid`,
`process_group_id`, `process_started_at`) of a finished native run when
it can prove that the runner process stopped (#17374)
> - A background queue (`settledNativeProcesses`) finds these runs and
checks them. A check that cannot finish ("skipped" or "pending") writes
nothing, so the run stays in the queue and every recovery hint checks it
again
> - A finished run that a replaced container recorded can never pass the
check, because only the host that recorded the process can prove that it
stopped. Each deploy adds more of these runs
> - On one hosted instance this loop ran about 10 checks per second and
about 55 SQL statements per second, and the server main thread was about
95% busy with only a few live runs
> - This pull request moves each "cannot make progress" reason into the
queue's SQL predicate, adds a growing backoff for "pending" runs, and
makes one check cost one locking statement
> - The benefit is that runs which cannot progress cost nothing until
something that can change the answer happens, and a run that becomes
releasable is still released at once

## Linked Issues or Issue Description

No public issue exists. Bug report:

**What happened?**
The settled native process recovery queue checks the same finished
native runs again on every recovery hint and every 30 seconds, forever.
With 300 such runs and one company-wide hint per second it ran 9 checks
per second and 84 SQL statements per second (9.4 statements per check,
2.2 ms of server CPU per check) on embedded PostgreSQL. A hosted
instance showed the same rates. The runs that never finish are: runs
recorded by a host that is gone (each redeploy starts a new container
with a new hostname), runs whose coordinator phase is not settled, runs
with no checkpoint and no provider evidence, and runs whose local
process is still alive (they defer 30 seconds with no limit). Startup
discovery (`discoverSettledNativeRunProcesses`) also opens one
transaction for every such run on every boot.

**Expected behavior**
A run that cannot make progress is not read again until something that
can change the answer happens. A run that becomes releasable (its phase
settles, provider evidence arrives, its own host checks it, its sandbox
receipt arrives, its process exits) is released promptly.

**Steps to reproduce**
1. Create a failed native run with a recorded local process identity
from another host (or an unsettled coordinator phase, or no checkpoint
and no provider evidence, or a live process).
2. Register the residual recovery work and send company-wide
`notifyLiveRunRecovery` hints.
3. Count the run's `for update` checks: on `master` every hint checks
the run again.

`server/src/__tests__/settled-native-process-rechecks.test.ts` automates
these steps.

**Paperclip version or commit**
`master` at 3c3e6c2d75.

**Deployment mode**
Any. The cost grows with every deploy of a hosted instance.

## What Changed

- `server/src/services/native-local-process-stop.ts`
- New `settledNativeProcessCandidates`: one fixed-text statement that
finds releasable runs. It reads only runs recorded in this server's PID
space (`process_host_hash`), sandbox runs (`remote`, with a non-local
lease), and rows saved before the hash (their latest identity event must
name this host). It excludes runs with an unsettled coordinator phase
and runs with no checkpoint and no provider evidence. Each PID-space
branch reads its own range of the new partial index.
- `releaseSettledNativeRunProcess` now reads the run, phase, evidence,
lease facts and identity match in one `for update` statement (it was 6
statements). It reads the lease rows again only for a sandbox run. The
new `inspectSettledNativeRunProcess` also returns whether a remote stop
acknowledgement can apply.
- Startup discovery uses the same candidate statement, so it opens no
transaction for an excluded run.
- `server/src/services/heartbeat/live-recovery-work.ts` (settled kind
only)
- The load runs the candidate statement and a separate indexed statement
for cancelled `cursor_cloud_api` runs whose Stop is still `requested`.
- The check uses the loaded entry. It no longer runs the load again for
each entry.
- A "pending" run backs off 30 s, 60 s, 120 s, up to 15 min. A
run-scoped hint (identity event, lease cleanup, stop receipt) resets the
backoff.
- A "skipped" run that the predicate cannot exclude (for example a birth
time mismatch on this host) waits for a run-scoped hint or a restart.
- `server/src/services/heartbeat/recovery.ts` (the settled handler
only): returns the outcome. It calls `acknowledgeRemoteStop` only when
the run has a non-local lease or is a cancelled `cursor_cloud_api` run.
For any other run that function reads two rows and writes nothing.
- `packages/db`: migration 0487 builds
`heartbeat_runs_settled_process_identity_idx` and
`heartbeat_runs_pending_api_stop_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. The migration is `online_compatible` and is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. `doc/DATABASE.md` describes both
indexes.

End state for each reason that a check cannot finish:

| Reason | Before | After | What re-admits the run |
| --- | --- | --- | --- |
| Recorded by another PID space | checked on every hint | not read |
only a server in that PID space reads it (a live older server in a
deploy overlap still releases its own runs) |
| Coordinator phase not settled | checked on every hint | not read | the
phase write; the finalizer sends a company hint in the same transaction
|
| No checkpoint and no provider evidence | checked on every hint | not
read | the next hint after a provider or turn event |
| Live local process, unreleased lease, missing sandbox receipt | every
hint and every 30 s | backoff 30 s to 15 min | a run-scoped hint resets
the backoff |
| Other same-host skip | checked on every hint | parked | a run-scoped
hint (an identity event sends one) or a restart |

## Verification

- `pnpm exec vitest run
src/__tests__/settled-native-process-rechecks.test.ts` (new, 12 tests,
real PostgreSQL, real queue and coordinator, fake clock):
- each exclusion: replaced container, unsettled phase, no evidence: 0
checks over 4 hints (red on `master`: 3 to 5 checks);
- a live older server and a stopped older server (different hash): the
new server never reads the run and writes no receipt; this server's own
run is released at start;
- re-admission: phase settles, late provider evidence, legacy rows with
no hash (this host released, another host not read), sandbox run
released from any server with its receipt;
- live process: at most 4 checks over 8 hints (red: 17), released on the
run-scoped hint after the process exits;
- one check is BEGIN, one `for update` statement and COMMIT (3
statements); the next hints do not check it; a run-scoped hint checks it
once more (red: 5 statements and a check on every hint);
- a restart costs one check for each pending run; startup discovery with
50 excluded runs sends 1 statement (red: 302).
  - On `master` 11 of the 12 tests fail.
- Measurement on embedded PostgreSQL, 300 stuck runs, one company hint
per second for 300 s (fake clock), same harness before and after:
- before: 9.0 checks/s, 84.3 statements/s, 9.4 statements per check, 5.9
s CPU (2.2 ms per check);
- after: 0 checks/s, 3.2 statements/s for all seven residual queues
together, 0.16 s CPU.
- Load plan with 12,000 excluded finished runs (10% without a hash) in a
31,000-row `heartbeat_runs`: the candidate statement reads only index
entries for this server's hash, `remote` and `unknown`, and a bitmap
scan of the 1,200 legacy rows: 2.7 ms, 4,416 buffers. Without the index
the same exclusion read all 31,000 rows (31,127 buffers).
- Existing suites, one file at a time: all pass.
`live-recovery-work.test.ts` 51/51 (an earlier run under machine load
hit the `afterAll` hook timeout after all tests passed; the rerun passed
clean, and `native-safe-replacement.test.ts` hits the same hook timeout
on the base commit), `native-settled-process-release.test.ts` 31/31,
`native-local-process-stop.test.ts` 5/5, `heartbeat/recovery.test.ts`
107/107, `native-runner-restart-recovery.integration.test.ts` 231 passed
/ 14 skipped, `native-restart-recovery.test.ts` 49/49,
`native-controller-recovery.integration.test.ts` 30/30,
`retained-native-session-cleanup-work.test.ts` 29/29,
`native-provider-failure.integration.test.ts` 16/16,
`native-safe-replacement.test.ts` 70/70 (with the `afterAll` hook
timeout that also occurs on the base commit),
`native-remote-retry-admission.test.ts` 93/93.
- `pnpm check:module-boundaries`, `pnpm check:migrations` (packages/db),
server typecheck: pass.
- Not run: the full suite and browser suites.

## Risks

- Unresolved by design: a finished run whose server was replaced keeps
its process fields indefinitely and is never released, because only the
host that recorded a process can prove that it stopped, and the shared
prover refuses a controller-lease proof for a run with a process group.
This pull request only stops paying for it. These runs keep no stop
receipt (`native.local_process_stopped`), so what waits for that receipt
still waits: accounting close of the run's usage source, retained native
session cleanup, and instruction or directory working copies that wait
for proof that the run stopped. A follow-up must decide the ownership
rule for a replaced container.
- Rows saved before `process_host_hash` exist are read through the
legacy branch, which checks the latest identity event for each of them
on each load. On an instance with many such rows the load costs one
index lookup per row. The count can be measured with `select count(*)
from heartbeat_runs where process_host_hash is null and process_pid is
not null and status in
('failed','cancelled','timed_out','interrupted')`.
- A run that this server recorded but whose `process_host_hash` changed
(a new PID namespace on the same host, for example a container restart)
is no longer read here. Before, it was read and skipped, so no release
is lost.
- The backoff and parked state live in memory. A restart checks each
pending run once.
- The migration builds two partial indexes concurrently on
`heartbeat_runs`. It holds no long lock.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:29:23 -06:00
DottaandClaude Opus 5.5 e8c79235ca perf(attention): scope run lifecycle retention hints and read only used columns (#21579)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `attention-retention` queue archives old attention items (failed
runs, reviews, questions, recovery actions, blocked tasks). A company
hint makes it reload every candidate of the company, 100 per page, with
eligibility reads for each page.
> - Several run lifecycle producers sent that company hint: every agent
status change after a run, the execution-control writes and legacy run
terminalization when they name only a run, scheduled retries, and the
Runner's task monitor tool.
> - Each page also read whole rows it did not use: full `heartbeat_runs`
rows (recent ones carry tens of KB of `context_snapshot` and
`runner_profile_json`), full `issues` rows (descriptions), and the whole
agent roster with its configuration.
> - This pull request names the task graph or the source kind in those
hints, and reads only the columns each eligibility decision uses.
> - The benefit is fewer company reloads per run event and far fewer
result bytes per reconstruction, with the same decisions.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`. Stacked on #21552 (it builds
on that pull request's change to `execution-control-notifications.ts`);
review that one first.

**What existing behavior does this improve?**

The attention retention queue
(`server/src/services/attention-retention.ts`) and the producers that
wake it on run lifecycle paths.

**Subsystem affected**

Server background retention (`attention-retention.ts`,
`retention-work-notifications.ts`, `execution-control-notifications.ts`,
`heartbeat/agent-status.ts`, `heartbeat/retries.ts`,
`native-runtime/paperclip-runner-tool-authority.ts`).

**Current behavior**

- `heartbeat/agent-status.ts` sends `{ companyId }` on every agent
status change after a run (two places). `heartbeat/retries.ts` sends it
for each scheduled retry. `writeExecutionControl` and
`notifyExecutionRunWork` send it when the scope names only a run. The
Runner's `set_task_monitor` tool sends it for one task.
- On a local fixture with 1,500 open tasks and run churn, the queue
loaded 47 company candidate rows per second from these hints.
- `pageEvidence` read `select *` from `heartbeat_runs`,
`issue_recovery_actions` and (through `getTableColumns`) `issues`;
`reviewEligibility` read `select *` from `issues`;
`interactionEligibility` read the full agent roster; the per-entry
`check` repeated full-row reads.

**Proposed behavior**

- Agent status changes send `{ companyId, sourceKind:
"agent_error_alert" }`. The status moves between `running`, `idle` and
`error`. All three are invokable for blocker attention
(`BLOCKER_ATTENTION_INVOKABLE_AGENT_STATUSES`) and for questions
(`agent-eligibility.ts`). Failed-run attention only excludes terminated
agents, and this path never sets that. So only agent error alerts can
change.
- Run-scoped execution-control writes, legacy run terminalization and
scheduled retries use `notifyIssueAttentionRetentionWork` with the run
id (the run's task, its parents and the tasks it blocks; company
fallback past 100). This is the same helper as #21521. The task monitor
tool scopes to its task.
- `notifyIssueAttentionRetentionWork` matched the run's saved task id
with `issue.id::text = …`. It now validates and casts the id, so the
primary key is used, as #21552 does for recovery.
- Narrow reads, same decisions:
- runs: `id`, `agent_id`, `created_at` and the saved task id computed in
SQL with the same rules as `readRunIssueId` (`issueId` unless it is
absent or JSON null, then `taskId`; only non-empty strings).
  - recovery actions: `id`, `source_issue_id`.
- issues for evidence and `visibleIssue`: the blocker-attention input
columns plus `unblock_descriptor` and `blocked_transition_at`.
- review issues: `id`, `company_id`, `status`, `execution_state`,
`assignee_user_id`.
- agent roster for question addressees: `id`, `company_id`, `name`,
`reports_to`, `status` (the `AgentOrgRow` that
`evaluateAgentInvokability` takes).

**Reason and benefit**

A run event no longer reloads every attention candidate of the company.
A reconstruction no longer transfers whole run, task and agent rows.

## What Changed

- `server/src/services/attention-retention.ts`: column lists for runs,
recovery actions, issues and the roster; the run probe type replaces
whole rows.
- `server/src/services/heartbeat/agent-status.ts`,
`heartbeat/retries.ts`, `execution-control-notifications.ts`,
`native-runtime/paperclip-runner-tool-authority.ts`: scoped attention
hints.
- `server/src/services/retention-work-notifications.ts`: indexed lookup
of a run's saved task.
- `server/src/__tests__/attention-retention-work.test.ts`: two tests.

## Verification

- `reads only the fields its decisions use from large runs, reviews and
agents` (I saw it fail first): 50 failed runs, 50 tasks in review with
large descriptions, 10 questions to a paused agent, about 40 KB of
payload in each run's snapshot, profile and stdout and in the agents'
configuration. It loads the three kinds with the full-size response
capture (`full-database-response-capture.ts`). Before: `heartbeat_runs`
6,317,700 bytes, `agents` 84,336 bytes, `issues` 4,235,810 bytes. After:
under the test bounds of 20,000, 5,000 and 60,000 bytes. The decisions
are the same before and after: 50 failed runs, 40 reviews and 10
questions.
- `scopes run lifecycle attention hints to the run's task graph and
agent status to agent alerts` (I saw it fail first):
`writeExecutionControl` and `notifyExecutionRunWork` with a run id send
the hint for the task and its parent. `finalizeAgentStatus` sends the
agent-alert hint. Before, all three sent `{ companyId }`.
- `attention-retention-work` (57 tests),
`attention-retention-candidates`, `attention-service`,
`execution-control-work` and `heartbeat-agent-status-recovery` pass,
each file run alone (in `server/`, against a dedicated local `_test`
database).
- `pnpm check:module-boundaries` passes.

Not done in this pull request, and why:

- I did not reproduce `syncItems` re-syncing unchanged rows from the
retention `check`. On my fixture (300 stalled blocked tasks, 4,500 old
recovery-action and blocker retention rows, run churn), `syncItems` ran
0 times in a 180-second window and `decision_retention` reads were 0 per
second. `check` calls `syncItems` only for entries that are already due
(older than the 90-day archive delay), and a due entry is archived and
leaves the candidates. So I did not batch `syncItems` or skip checks of
unchanged rows.
- `readIssues` still selects the `issues` columns that blocker attention
needs. I did not narrow `issueService.listBlockerAttentionBatch`.

## Risks

- A producer that changes more than the run's task graph would now miss
the wider effect. Company-level changes (agent configuration, company
status, budgets) keep their company hints. The 90-day archive delay
makes a missed reload cheap: the item is reloaded on the next hint that
covers it.
- The saved task id now comes from SQL. The SQL mirrors `readRunIssueId`
for strings, absent keys and JSON null. A non-string `issueId` value now
yields no task, as before.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:17:21 -06:00
DottaandClaude Opus 5.5 a1c37b5cc4 perf(chat): rebuild GitHub response sweeps for company-wide hints at most every 30 seconds (#21580)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel has two background sweeps for request
response comments: acknowledgements, and terminal responses. Each sweep
rebuilds its candidate set when a hint arrives, and it reads the latest
run of each candidate request (`readGitHubResponseRun`) on each rebuild
> - A run milestone for a run with no task sends a company-wide hint to
both sweeps. Examples: each status change of a run without a task
(`setRunStatus`), and a comment merged into an active run in the wake
queue. In a busy company these hints arrive more than once a second
> - So both sweeps rebuild the whole company about once a second (their
minimum pass interval). Each rebuild reads every parked request again,
although nothing about those requests changed. On a hosted test
database, `readGitHubResponseRun` ran 6.7 times a second
> - This pull request rebuilds a GitHub response sweep for a
company-wide hint at most once every 30 seconds. Issue, request,
conversation, endpoint and lease hints still rebuild immediately.
`external-objects.ts` uses the same rule for its refresh sweep
> - The benefit is that a steady stream of company-wide hints no longer
makes each sweep read every parked request every second

## Linked Issues or Issue Description

Refs #21577 (makes each `readGitHubResponseRun` call cheap; this pull
request reduces how often the sweeps call it). Refs #18384 (the same
reconstruction interval for company-wide external-object hints).

**What existing behavior does this improve?**
The candidate rebuild of `registerGitHubAcknowledgementWork`
(`acknowledgement-work.ts`) and `registerGitHubTerminalResponseWork`
(`terminal-response-work.ts`) for company-wide hints.

**Subsystem affected**
GitHub chat channel: request acknowledgement and terminal-response
recovery sweeps.

**Current behavior**
`notifyChatRunMilestones` calls `notifyChatResponseAvailability` with
its scope. When the scope has no issue (only a company, or a company and
a run), the hint for both GitHub response queues is company-wide
(`{"companyId": ...}`), because the response scope has no run field.
Each company-wide hint makes each sweep load every candidate of the
company and call `readGitHubResponseRun` for each one. Passes are
limited to one a second, so with a steady hint stream the cost is about
(parked candidates) reads a second.

**Proposed behavior**
Both sweeps set `reconstructionIntervalMs`. A company-wide scope
(company ID only) waits 30 seconds after its last rebuild before it
rebuilds again. All narrower scopes, the startup scope (`*`), and saved
deadlines are not changed. Entries that are already loaded stay
available for dispatch while a company-wide hint waits, and each check
still reads the request again from the database.

**Reason and benefit**
On the hosted test database, `readGitHubResponseRun` ran 6.7 times a
second for no change in state. In a local reproduction, the call rate
drops from about one call for each parked request each second to one
call for each parked request each 30 seconds.

**Breaking changes**
None.

## What Changed

- Add `response-sweep-scope.ts` with the shared scope type of the two
GitHub response sweeps and `githubResponseReconstructionIntervalMs`: 30
seconds for a company-wide scope, 0 for all other scopes.
- Use it as `reconstructionIntervalMs` in the acknowledgement sweep and
the terminal-response sweep. Both files now import the shared scope type
instead of declaring it.
- Add `response-sweep-scope.test.ts`. It registers both real sweeps on a
real database with 6 parked requests (3 for each sweep) and a controlled
clock. It sends company-wide hints and issue hints, and counts
`readGitHubResponseRun` statements on each pass.

## Verification

Cause, from a local reproduction with the real delivery coordinator and
both real sweeps on embedded PostgreSQL 18.1, with 8 parked requests (4
for each sweep), each with a succeeded native run that waits for its
presentation:

| Hints | Hints a second | `readGitHubResponseRun` calls a second |
| --- | --- | --- |
| None | 0 | 0 |
| Company-wide run milestone (`{ companyId, runId }`) | 9.7 | 8.75 |
| Company-wide run milestone | 1.0 | 7.98 |
| Run milestone for an unrelated issue | 9.7 | 0 |

So the call rate is about one call for each parked request each second,
from company-wide hints only. The hosted rate (6.7 a second) is
consistent with about 7 parked GitHub requests in one company.

Red/green regression test:

- `cd server && npx vitest run
src/services/chat-channels/github/response-sweep-scope.test.ts`
- Red (without the two `reconstructionIntervalMs` lines): `expected [ 6,
6, 6, 6, 6 ] to deeply equal [ 6, +0, +0, +0, +0 ]`. Each of 5
company-wide hints rebuilt all 6 parked requests.
- Green: 2 of 2 pass. The first company-wide hint rebuilds once (6
reads). The next 4 hints read nothing. After 30 seconds, the waiting
hint rebuilds (6 reads). Issue hints still read exactly the one request
of that issue at once, and a hint for another company reads nothing. No
acknowledgement is sent for a parked request.

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed
(all GitHub chat-channel integration tests, including the
acknowledgement and terminal-response sweeps).
- `cd server && npx tsc --noEmit`: passed.
- After the rebase onto #21577: both new test files pass (4 of 4), and
the server typecheck passes.
- `pnpm verify` selected most of the server suite, because the sweep
modules are imported widely. I stopped that broad local run. Required CI
supplies that coverage.

## Risks

- A change that only a company-wide hint reports can take up to 30
seconds to rebuild a GitHub response sweep. I checked the callers.
Changes to one request use issue, delivery, conversation or lease hints:
run status changes with a task, native presentation commits
(`status-decision-committer.ts`, `native-run-finalizer.ts`,
`native-chat-review-presentation.ts`), publications, fast responses, and
the publication lease. Those stay immediate. Company-wide hints come
from runs with no task, from a comment merged into an active run (the
wake queue sends only the run ID), and from company authority changes. A
run with no task cannot change the latest run of a GitHub request's
task. For a merged comment, the active run's next status change sends an
issue hint, and staging the request's response sends a delivery hint;
both rebuild at once. Only a mid-run ownership update can wait up to 30
seconds. For an authority change, each check reads the request and its
authority again before it acts.
- The deadline reconciliation snapshot reports these two sweeps as not
reconciled while a company-wide hint waits (up to 30 seconds). This is
the same as the external-object refresh sweep.
- The startup rebuild (`*`) and failure retries do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local PostgreSQL
reproductions).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:05:45 -06:00
DottaandClaude Opus 5.5 340157d1d6 perf(recovery): scope run lifecycle recovery hints to the run's task graph (#21552)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues in the background. Two of
them, `stranded-assignments` and `assigned-work-invariant`, find
assigned tasks that have no live run and restart them.
> - Every run dispatch and every run state change sent a recovery hint
for the whole company. On each hint these queues reload every eligible
open task of the company and check each one again, at most every 30
seconds per queue.
> - With live runs the hint comes all the time, so the queues never
stop. Their cost is (run activity) x (open-task backlog). On a hosted
instance with about 1,400 open tasks and 5 live runs, the server main
thread stayed at 87-92% of a core.
> - A run event can only change the tasks of its own task graph. This
pull request makes run and task producers name that graph, and keeps one
slow full pass as a backstop.
> - The benefit is that a run event costs a few checks, whatever the
size of the backlog, and a missed hint still delays recovery by at most
five minutes.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The issue recovery queues in `server/src/services/recovery/work.ts`
(`stranded-assignments`, `assigned-work-invariant`,
`resolved-dependencies`, `stale-issue-locks`) and the producers that
wake them on run dispatch, run start, run state change, run finalization
and accounting settlement.

**Subsystem affected**

Server background recovery (`server/src/services/recovery/`,
`server/src/services/scoped-deadline-work.ts`, the notification helpers
in `server/src/services/*-notifications.ts`).

**Current behavior**

`notifyRunDispatch`, `notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` sent `notifyIssueRecovery(tx, {
companyId })`. Most run lifecycle paths call one of them, often two or
three in the same transaction (for example `notifyIssueReviewState`).
Some other run paths sent the company hint directly: the wake status
update, the execution-control deadline writes, legacy run
terminalization, scheduled retries, accounting settlement, task checkout
and task document writes. Each company hint makes each task queue reload
all eligible open tasks of the company, 100 per page, and check each
task again. A company hint is reconstructed at most every 30 seconds per
queue, so one run event per 30 seconds is enough to keep a full pass
running all the time. `issue-recovery-deadlines.test.ts` pinned this: a
company hint re-checks every eligible task.

**Proposed behavior**

Design. For each task queue, a run event of run R on task T can change:

| Queue | What the check reads | What one run event can change |
| --- | --- | --- |
| `stranded-assignments` | T's own runs, wakes, handoffs, questions,
recovery actions, goal, assignee eligibility | T (its runs). The parent
of T and the tasks T blocks when the event also writes T's status. |
| `assigned-work-invariant` | the same plus open blockers and open
children | T, the parent of T (open children), the tasks T blocks (open
blockers) |
| `resolved-dependencies` | blocked tasks whose blockers are all done |
the tasks T blocks |
| `stale-issue-locks` | tasks whose checkout or execution run is R | T |

The assignee's agent status (`running`, `idle`, `error`) does not change
whether the agent is invokable
(`packages/shared/src/agent-eligibility.ts`), and a freed concurrency
slot starts the agent's next queued run through the
`queued-run-dispatch` queue, which keeps its company hint. So the
agent's other tasks are not in the set.

- `notifyIssueGraphRecovery(tx, { companyId, runId?, issueId? })` sends
`{ companyId, graphRunId?, graphIssueId? }` to the four task queues. It
runs no query in the producer transaction.
- When a task queue loads that hint, it resolves the graph with one
bounded, indexed query (`resolveIssueGraph`): the task of the run
(`issue_id`, `native_issue_id`, or the saved `issueId`/`taskId`,
validated and cast so the primary key is used), its parent, and the
tasks it blocks. The four queues share the result for 2 seconds. With no
task or more than 100 tasks, the queue turns the hint into the plain
company hint, so task-less runs coalesce under its 30-second gate
instead of each starting its own full pass. The lookup runs after
commit, so it sees the written state. Graph hints do not wait for the
30-second reconstruction gate (they load at most 100 tasks); hints for
the same run coalesce while dirty, and the lookup is shared across the
four queues, so it runs about once per pass instead of once per producer
call. An earlier round ran the lookup in every producer transaction: 34
lookups/s at 1,500 open tasks and 70/s at 5,000.
- The three company-level queues (`orphan-blockers`,
`active-recovery-actions`, `task-watchdogs`) keep the company hint. They
hold one entry per company or per watchdog, not per open task.
- `notifyRunDispatch`, `notifyScheduledRetryDispatch`,
`notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` use it when the scope names a run
or a task. A scope with only a company keeps the company hint.
- Run paths that knew the run or the task but sent only the company now
pass it. This includes three `writeExecutionControl` deadline writes,
legacy run terminalization (`notifyExecutionRunWork`) and native safe
replacement. Those helpers also forward the scope to
`notifyLiveRunRecovery` and `notifyLiveResourceRecovery`, so their hints
to the live-run queues now carry the run id.
- Recovery action changes (`issue-recovery-actions.ts`: create,
supersede, update, exhaust, resolve) name their source task. A recovery
action is a fact of its source task for these queues. The company-level
`active-recovery-actions` queue still gets the company hint. In the
fixture, failing runs create and resolve recovery actions every few
seconds, and that alone kept a full pass running.
- `writeExecutionControl` and `notifyExecutionRunWork` send the
attention retention hint for a task through
`notifyIssueAttentionRetentionWork` (the task, its parents and the tasks
it blocks), a superset of the single-task hint they sent before. A
recovery action of a blocker can change the blocker attention of its
dependents, so the narrower recovery action scope needs this.
- `promoteDeferredWorkIntents` scopes `lock_released` with a task to
that task, and `accounting_settled` to the run when the company has no
active hard-stop budget policy. Only a reached hard stop can refuse work
because of accounting (`evaluateAdmissionInTransaction`), so without
such a policy the settlement of one run cannot admit another task. The
other hold releases keep the company hint.

Backstop. Each scoped hint also sends `{ companyId, backstop: true }` to
the same four queues. Nothing else arms it. The queue runs it as a full
company pass, but not before 5 minutes after the last full pass of that
queue for that company or of the startup pass. It is a scheduler
deadline (`nextWakeAt` reports it), but it holds no idle intent and does
not mark the queue unreconciled. So:

- Startup still reconstructs every company.
- A failed scoped load retries with the existing back-off (30 seconds to
5 minutes). The backstop is pending in any case.
- A graph with no task or more than 100 tasks loads the whole company at
once.
- A task that a scoped hint missed is checked at most
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes) after the event. While runs
continue, each queue runs at most one full pass per company per 5
minutes. With no activity, nothing is scheduled.

Why arming on activity only cannot leave a stranded task undetected when
the instance then goes quiet. A task becomes stranded only through a
write or through a run that stops without a write:

1. The write names the task's graph (run and task producers). The scoped
pass checks the task at once.
2. The write sends a company hint (pause, budget, agent and company
configuration, tree holds, the producers listed below). The company pass
checks it within the 30-second gate, as before.
3. The write is a run event whose graph does not contain the task. That
same event armed the backstop. The backstop deadline stays scheduled
while the instance is quiet and checks the task at most 5 minutes later.
4. A run stops without any write (process lost, controller gone). The
existing deadline paths find it: `stale-issue-locks` waits for the
stored controller lease, and `active-run-silence` and
`orphan-active-leases` act on their own deadlines. Each of them
terminalizes the run, which is a run write and falls under 1 or 3. The
new test `restarts a task whose run vanished without a final event once
its lease lapses` covers this with the production backstop.
5. The process restarts or a quiet instance is woken by a restart. The
startup pass reconstructs every company.
6. A check that found a task waiting (grace, back-off, monitor) returns
its own deadline. That entry keeps its deadline whatever hints arrive.

Five minutes is the existing upper bound of recovery back-off in this
area: the maximum retry delay of a failed scoped-deadline check
(`scoped-deadline-work.ts`), and `DEFERRED_WAKE_PENDING_BOUND_MS`, the
time `assigned-work-sweep.ts` already trusts a deferred wake before it
calls a task stranded.

Producers that keep the company hint, and why. They can change
eligibility of tasks outside one task graph, or I could not show that
their effect stays in the graph. None of them fires once per run event
in the fixture.

- Eligibility for many tasks: company status (`companies.ts`), budgets
(`budgets.ts`), agent lifecycle and configuration (`agent-lifecycle.ts`,
`agent-configuration.ts`), instance settings (`instance-settings.ts`,
`*`), tree holds (`issue-tree-control.ts`), company deletion and import
(`company-deletion.ts`, `company-portability.ts`), and the hold releases
`pause_lifted`, `tree_hold_released`, `scheduling_resumed` and
`reconciler_backstop`. `accounting_settled` also keeps it when the
company has an active hard-stop budget policy.
- Task writes that can move a task between graphs or change eligibility:
task creation and a task update that changes the parent in `issues.ts`
(other task updates and checkout releases now name the task), document
delete and restore paths in `documents.ts`, task watchdogs
(`task-watchdogs.ts`), system and workspace recovery
(`system-recovery.ts`, `task-retry.ts`,
`workspace-recovery-capture-policy.ts`,
`workspace-recovery-stop-proof.ts`).
- Conversation and chat paths: `agent-conversations.ts`, the daily-cap
release in `heartbeat/queue.ts`, the Telegram attachment failure,
`chat-attachment-reuse.ts`, `native-runner-file-handoff.ts`,
`notifyChatRunMilestones` without a task or run (company authority
change), and `notifyNativeStatusReassessment` from `access.ts`.
- `writeExecutionControl` and `notifyExecutionRunWork` callers that pass
only a company.

**Reason and benefit**

Background work must not grow with the backlog of open tasks. The checks
a run event causes now depend on its task graph, not on the company.

## What Changed

- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery`, `resolveIssueGraph`, the scope matcher and
decoder, `ISSUE_RECOVERY_MAX_ISSUE_IDS` (100) and
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes).
- `server/src/services/recovery/work.ts`: the task queues resolve a
graph hint when they load it (shared for 2 seconds), remember the last
full pass per company, and gate the backstop hint on it.
`registerIssueRecoveryWork` takes an optional `backstopMs`.
- `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects`
accepts `issueIds`.
- `server/src/services/scoped-deadline-work.ts`: an optional
`backstop(scope)` that returns the earliest time of a backstop hint. A
pending backstop hint keeps its deadline but does not count as pending
work or as unreconciled.
- Producers: `issue-recovery-actions.ts`,
`execution-control-notifications.ts`, `run-dispatch-notifications.ts`,
`native-status-reassessment-notifications.ts`,
`external-conversation-settlement-notifications.ts`,
`work-intent-outbox.ts`, `run-cost-accounting.ts`,
`heartbeat/run-lifecycle.ts`, `heartbeat/runtime-execution.ts`,
`heartbeat/retries.ts`, `heartbeat.ts`, `legacy-execution-recovery.ts`,
`issues.ts` (checkout), `documents.ts` (task document writes),
`modules/wake-queue/adapters/postgres.ts` (recovery runs for a task),
`heartbeat/queue.ts` (wake enqueue), `recovery/service.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-safe-replacement.ts`,
`modules/run-dispatch/adapters/workspace-busy-wait.ts` (scheduled
retries).
- `bounds subsidiary scans for100 assignment candidates`: a resolved
recovery action now adds one scoped load per task queue (bound 8 to 11),
and the coalescing check sends one company hint before the burst of 100,
because the first full pass after the resolve is now a backstop pass
with its own key.
- `checks only blocked tasks whose blockers are done`: uses the shared
recording harness and runs out the backstop before its quiet check,
because the blocker's status change is now a graph hint (its 300
dependents overflow to a company load).
- Tests. `issue-recovery.test-fixture.ts` and
`live-recovery-work.test.ts` register the queues with `backstopMs: 0`,
so their quiet checks keep the earlier timing. The new tests use the
production value.

## Verification

New tests. I saw each red test fail before the change.

- `issue-recovery-deadlines.test.ts` `re-checks only the tasks a run
lifecycle event can affect, whatever the backlog` (red before): a run of
the agent on a task with a parent and a dependent, 1,000 backlog tasks
of another agent, and a second open task of the same agent. Seven real
producers (`notifyNativeStatusReassessment`, `notifyExecutionRunWork`,
`notifyRunDispatch`, `promoteDeferredWorkIntents` lock release and
accounting settlement, recovery action create and resolve) check exactly
the 3 tasks of the graph. The recovery action steps were red before the
third commit. With 4,000 more backlog tasks the event issues the same
number of statements. Before the change it checked 1,004 tasks.
- `resolves a run's task graph through index lookups, not the backlog`
(red before the second commit, from a Greptile finding): with 2,000
backlog tasks the graph lookup read 2,005 issue rows
(`pg_stat_xact_user_tables`) because it compared `id::text`. It now
validates and casts the saved id, reads fewer than 50 rows, and returns
exactly the run's graph.
- `finds a stranded task outside the affected set within the backstop
bound` (red before): a task inserted without a hint is not checked by
the run event, and is checked at most 5 minutes after the last full
pass. After 20 quiet minutes, a run event checks a new unrelated task
within 5 seconds, because the last full pass is older than 5 minutes.
Then the queues are quiet.
- `keeps the company pass for hold releases that can admit any task`
(guard, passes before and after): accounting settlement with a hard-stop
policy and a lifted pause still check every task.
- `falls back to the company pass when a run affects more tasks than a
hint names` (guard): a run task that blocks 100 more tasks checks the
whole company.
- `limits an ordinary comment to its issue among one thousand open
assignments`: the old final assertion (a company hint checks all 1,000
tasks) stays, and the test now also runs out the backstop and ends
quiet.
- `issue-recovery-assignments.test.ts`, real heartbeat, real producers,
production backstop (guards; they also pass with the old company hint):
a task whose run failed, a task whose run the system cancelled, a run
that vanished without a final event and is found when its controller
lease lapses, and an agent's queued task that starts when its previous
run ends. Each one recovers within 15 seconds, well before the backstop.
- `scoped-deadline-work.test.ts` `keeps a backstop hint's deadline
without holding the worker busy` (red before).

Suites, each file alone, in `server/` (issue recovery, live recovery,
recovery action and attention suites against dedicated local `_test`
databases): `issue-recovery-deadlines` (13/13),
`issue-recovery-assignments` (7/7), `issue-recovery-work-lifecycle`
(14/14), `scoped-deadline-work` (24/24), `live-recovery-work` (51/51),
`assigned-work-invariant` (32/32), `assigned-work-sweep-round-trips`
(2/2), `saved-work-recovery-overdue` (4/4),
`issue-recovery-dependencies` (5/5), `issue-recovery-conversations`
(3/3), `issue-recovery-actions` (112/112), `attention-retention-work`
(55/55). `recovery-deadline-acceptance` skips without its external
database. `pnpm check:module-boundaries` passes.

Local benchmark. The real server ran against PostgreSQL with
`log_statement=all` and instrumented counters (not committed). Fixture:
1 company, 60 agents, open tasks at 150 / 1,500 / 5,000 (52% in review,
39% blocked behind open blockers, 3% in progress, 2% to do), with
process agents that fail at once so recovery re-dispatches runs all the
time. The tables are analyzed after loading. Each window is 180 s,
starting 150 s after the server is ready, so each "after" window
contains one backstop pass. Base is `master` at 54c3616ad9; after is
this branch at dcd170e29f. The last commit (graph hints without the
30-second gate, task updates and checkout releases scoped) is covered by
the tests below but not re-measured. The machine was shared (load
average 9-16, one base window at 45). Run churn differs between runs, so
the table gives counts per run event (runs created in the window).

| Open tasks | Runs created (/s) base → after | Stranded checks per run
base → after | Assigned-work checks per run base → after |
`agent_task_sessions` reads (/s, 4 shapes, stranded check) | Statements
per run event base → after |
| --- | --- | --- | --- | --- | --- |
| 150 | 0.08 → 0.14 | 45.1 → 6.2 | 2.00 → 0.96 | 3.75 → 1.27 | 2,260 →
1,257 |
| 1,500 | 1.09 → 1.00 | 22.0 → 3.9 | 1.24 → 1.53 | 26.5 → 7.2 | 1,313 →
1,060 |
| 5,000 | 1.12 → 1.99 | 20.4 → 1.6 | 4.10 → 0.91 | 29.3 → 8.1 | 1,400 →
829 |

New statements: the task graph lookup (`with seed(id) ...`) at 0.8 / 9.3
/ 18.5 per second, and scoped loads of the task queues (`select "id",
"company_id" from "issues" ...` grew from 0.7 / 6.6 / 7.8 to 2.9 / 20.6
/ 34.4 per second, each a lookup of at most 100 ids). Stale-lock checks
grew from 0 / 50 / 52 to 22 / 190 / 445 per window: each run event now
checks the locks of its own task at once instead of waiting for the next
full pass. In an earlier round the graph lookup ran in every producer
transaction: 34/s at 1,500 and 70/s at 5,000.

With no activity the server issues no statements before and after this
change (quiet checks in `issue-recovery-deadlines`,
`issue-recovery-assignments`, `live-recovery-work`).

## Risks

- A task outside the graph of the run whose state the event changes is
now found by the backstop, at most 5 minutes later, instead of within 30
seconds. One known case: from #21427, `assigned-work-invariant` counts a
held recovery wake as pending while its agent has queued recovery runs.
When the last one starts, the continuation queue delivers the held wake;
the invariant check of that task waits for the backstop.
- A producer that names a task but changes something wider would now
miss the wider effect. Producers that change eligibility for many tasks
(pause, budget, agent and company configuration, tree holds) still send
the company hint.
- The graph is resolved after commit, from the written state. A parent
change is not a run event and keeps its company hint.
- The live-recovery and issue-recovery suites opt out of the backstop
deadline with `backstopMs: 0`.
- The execution-control deadline writes and legacy run terminalization
now pass the run id. `writeExecutionControl` also forwards that scope to
the live-run queues (`active-run-silence`, `native-process-sessions` and
others), so those hints now name the run instead of the company. Those
queues already handle run-scoped hints from native finalization.
- Test-only option: `registerIssueRecoveryWork(..., { backstopMs })`.
The shared recovery fixtures pass `0` so their quiet checks keep the
earlier timing; the new tests use the production value.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
nightly/v2026.1114.0-nightly.0 canary/v2026.1114.0-canary.4
2026-11-14 01:29:22 -06:00
DottaandClaude Opus 5.5 c1d3f90b37 fix(recovery): keep the resolved-dependency probes per task whatever the statistics (#21574)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `resolved-dependencies` recovery queue loads the blocked tasks
whose blockers are all done (#21517). It runs this load on recovery
hints.
> - The load has two correlated subqueries on `issue_relations`.
PostgreSQL turns them into a semi join and an anti join, and lets the
statistics choose how.
> - After a bulk change in a company that the statistics have not seen
(a large import, a restore, a big cleanup), the planner estimates one
row for the company. It then picks nested loops whose inner side scans
every relation of the company and filters by task id. On a local copy
with 5,000 open tasks the query ran for more than 20 seconds (once 19
minutes) until the table was analyzed; after `ANALYZE` it takes 3.7 ms.
> - This pull request keeps both subqueries as per-task subplans, so
each one is an index probe by (company, task id) whatever the statistics
say.
> - The benefit is that the load stays proportional to the blocked tasks
it reads, also right after bulk changes. This is prevention: the hosted
instance does not show this query as slow today.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a local database that had been analyzed with a smaller dataset and
then loaded with 5,000 open tasks in a new company, the
`resolved-dependencies` load query did not finish within 20 seconds; in
a benchmark run it stayed active for 19 minutes after its server
stopped.

**Steps to reproduce**

1. Load a company with about 1,500 tasks and run `ANALYZE issues,
issue_relations`.
2. With autovacuum off for these tables, load another company with 5,000
tasks, 1,900 of them blocked with `blocks` relations.
3. Run the load query of `resolved-dependencies` for the new company.

Plan with stale statistics (abridged):

```
Limit
  ->  Nested Loop Semi Join  (rows=1)
        Join Filter: (issues.id = r.related_issue_id)
        ->  Nested Loop Anti Join  (rows=1)
              Join Filter: (r_1.related_issue_id = issues.id)
              ->  Index Scan using issues_company_id_uq on issues  Index Cond: (company_id = ...)
              ->  Nested Loop
                    ->  Index Scan using issue_relations_company_related_issue_idx on issue_relations r_1
                          Index Cond: (company_id = ...)
                    ->  Index Scan using issues_company_id_uq on issues blocker  Index Cond: (company_id = ...)
        ->  Index Scan using issue_relations_company_related_issue_idx on issue_relations r  Index Cond: (company_id = ...)
```

The index condition holds only `company_id`. The task id is a join
filter, so each blocked task rescans every relation (and every task, for
the blocker) of the company.

**Expected behavior**

Each blocked task probes its own relations through
`issue_relations_company_related_issue_idx` (company_id,
related_issue_id) and its blockers through `issues_company_id_uq`
(company_id, id).

## What Changed

- `server/src/services/recovery/work.ts`: the candidate predicate is a
named export, `resolvedDependencyCandidate`, and both subqueries end
with `offset 0`. PostgreSQL does not pull a subquery with `OFFSET` up
into a join, so it stays a correlated subplan. Its index condition then
includes the outer task id. The predicate and its results are otherwise
unchanged.
- `server/src/__tests__/resolved-dependency-candidates.test.ts`: two
tests.

Indexes that the subplans use (all existing):
`issue_relations_company_related_issue_idx` (company_id,
related_issue_id) for both relation probes, and `issues_company_id_uq`
(company_id, id) for the blocker status.
`issue_relations_company_issue_idx` (company_id, issue_id) is not needed
here.

## Verification

- `selects the same tasks as the earlier predicate`: 40 blocked tasks
with a done blocker, an open blocker, a done and a cancelled blocker, or
no blocker. The new predicate returns the same 10 tasks as the earlier
one.
- `probes the relations of each task, whatever the planner statistics
say`: analyzes a small company, then loads 3,000 blocked tasks in a new
company. The plan (`EXPLAIN (FORMAT JSON)`) has two subplans and no semi
or anti join, and the load returns 100 rows within a 5-second statement
timeout. I saw it fail first with the earlier predicate: the plan had a
semi join and an anti join.
- Local plan on the stale database: EXPLAIN ANALYZE shows `SubPlan 1`
and `SubPlan 2` with `Index Cond: ((company_id = issues.company_id) AND
(related_issue_id = issues.id))`, 1,933 probes each, 4.9 ms.
- `issue-recovery-deadlines.test.ts` and
`issue-recovery-dependencies.test.ts` pass (14 tests). `pnpm
check:module-boundaries` passes.

## Risks

- With fresh statistics the planner could use a merge or hash join for a
large page; the subplan form gives up that choice. Each blocked task now
costs two index probes, which is the same order as the fresh-statistics
plan (3.7 ms before, 4.9 ms after on 1,933 blocked tasks, measured once
each on a loaded machine).
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:28:55 -06:00
DottaandClaude Opus 5.5 6b6c94a4ea perf(chat): read the GitHub response run by issue run indexes (#21577)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel keeps one response comment for each GitHub
request. Two background sweeps (acknowledgements and terminal responses)
and the acknowledgement step read the latest run of the assigned agent
for the request's issue with `readGitHubResponseRun`
> - That function sends two statements. Both match runs with
`coalesce(native_issue_id::text, context_snapshot->>'issueId') = <issue
id>`. No index has that expression, so each statement reads every run of
the agent and decompresses the large `context_snapshot` of each legacy
run
> - On a hosted test database, these two statements were the two most
expensive statements by database time. In a 701 s window, each ran 6.7
times a second, for 203 s and 137 s of database time (about 30 ms and 20
ms for each call). Together they kept about half of one database session
busy
> - This pull request reads the latest run with two index lookups
(native binding, and legacy context without a native binding), the same
way #20671 changed the terminal-response sweep. It decides request
ownership on that one run in the same statement
> - The benefit is that each call reads a few index entries in one
statement instead of the agent's whole run history in two statements.
The results do not change, and no migration is necessary

## Linked Issues or Issue Description

Refs #20671 (the same rewrite for the terminal-response discovery page;
its Risks section names this function as the remaining case).

**What existing behavior does this improve?**
`readGitHubResponseRun` in
`server/src/services/chat-channels/github/response-run-state.ts`. The
GitHub acknowledgement sweep calls it once for each candidate request.
The terminal-response sweep and the acknowledgement step call it through
`githubResponseExecution`.

**Subsystem affected**
GitHub chat channel: request acknowledgement and terminal-response
recovery.

**Current behavior**
The function sends two statements:

1. The latest run of the agent for the issue that woke for the request:
`heartbeat_runs` joined to the inbound `chat_message_links` row of the
delivery on `wakeCommentId` or `wakeCommentIds`, ordered by `created_at
desc`, limit 1.
2. The latest run of the agent for the issue, with the
`responseRunColumns` projection, ordered by `created_at desc, id desc`,
limit 1.

Both statements filter on `coalesce(native_issue_id::text,
context_snapshot->>'issueId') = $issueId`. The planner can only use
`heartbeat_runs_company_agent_started_idx` or
`heartbeat_runs_company_created_at_desc_idx`. It reads all runs of the
agent and evaluates the `coalesce` on each row, which decompresses the
context of each legacy run. On the hosted test database,
`heartbeat_runs` has about 31,000 rows and 1.96 GB, because
`context_snapshot` is large.

**Proposed behavior**
One statement:

- The latest run ID comes from two disjoint branches, each `order by
created_at desc, id desc limit 1`: runs with `native_issue_id =
$issueId` (through
`heartbeat_runs_company_native_issue_contract_id_uq`), and runs with
`native_issue_id is null and context_snapshot->>'issueId' = $issueId`
(through `heartbeat_runs_company_ctx_issue_created_idx`). Together the
branches are exactly the rows where the old `coalesce` equals the issue
ID.
- The statement reads that one run by primary key with the same
`responseRunColumns` projection.
- An `exists` subquery on the inbound `chat_message_links` row of the
delivery decides whether that run woke for the request. It uses the same
`wakeCommentId` / `wakeCommentIds` match as before, but on one run only.
- When the issue ID is null, the function returns no run without a
query. The old statements could not match a null issue ID either.

**Reason and benefit**
These were the two most expensive statements on the hosted test
database. On a local fixture with the hosted shape, the median call goes
from 111.6 ms to 1.03 ms (about 108 times faster), and the buffers for
one call go from about 73,000 to 98.

**Breaking changes**
None. The function signature, the returned run, and the ownership result
do not change.

## What Changed

- `readGitHubResponseRun` reads the latest run of the agent for the
issue through the two index-backed branches, and decides request
ownership on that run in the same statement. Two statements become one.
- Add `response-run-state.test.ts`. It runs the old two-statement
function (kept in the test as the oracle) and the new function on the
same inputs, and it requires equal results. It also requires a plan
bound: one call must read fewer than 150 `heartbeat_runs` rows while the
agent has 1,500 unrelated runs.
- Add `scripts/benchmark-github-response-run-state.ts`. It loads the
baseline module from a given commit, compares the complete output of
both versions for every input, measures both, and prints the plans.

## Verification

Why the old statements were slow, and the thesis: each statement filters
the agent's whole run history with an expression that no index has, and
detoasts the context of every legacy run. If each precedence branch uses
its own issue index and ownership is checked on the one latest run, the
cost no longer depends on the agent's run count, and the results stay
identical.

Red/green regression test:

- `cd server && npx vitest run
src/services/chat-channels/github/response-run-state.test.ts`
- Red (test file on the old `response-run-state.ts`): the plan bound
fails with `expected 3597 to be less than 150`. The equivalence test
passes, because the oracle is the old code.
- Green (this change): 2 of 2 pass.

The equivalence fixture has two companies, two agents each, 60 issues
each, and one or two GitHub requests for each issue. Runs cover these
cases: native binding, legacy context without a native binding, a native
binding to another issue while the legacy context names this issue,
another agent, another company, `wakeCommentId`, `wakeCommentIds` as an
array, `wakeCommentIds` as a JSON string, a link to the outbound message
of the delivery, a link to another company's comment, a link to the
sibling request of the same issue, a request with no inbound comment,
ties on `created_at`, and a null issue ID. It also calls with another
company's endpoint. The test requires more than 20 results in each
outcome (owned, latest run not owned, no run) and both binding kinds.

Local benchmark with the hosted shape:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-github-response-run-state.ts --baseline-ref=3c3e6c2d75
```

Fixture (synthetic): one company, 4 agents with about 7,750 runs each
(31,000 runs), 2,000 issues, 300 GitHub requests. Each run has about 60
KiB of incompressible context, and `heartbeat_runs` is 1,932 MB. Runs
are native or legacy at random, and about half link to the request
comment. Embedded PostgreSQL 18.1 on an Apple laptop. The hosted
database is Neon, so absolute times are local measurements only. The
benchmark compares the complete output for all 600 inputs (262 owned,
329 latest run not owned, 9 no run): equal. It then times 20 inputs with
runs (two agents), with 2 warmups and 7 alternating samples.

| | Statements per call | Median ms per call (range) | Buffers for one
call |
| --- | --- | --- | --- |
| Before | 2 | 111.6 (100.9–240.5) | 18,358 hit + 26,498 read, then
14,997 hit + 13,489 read |
| After | 1 | 1.03 (0.81–1.39) | 80 hit + 18 read |

Plans for one call. Before: statement 1 is a `Bitmap Heap Scan` on
`heartbeat_runs_company_agent_started_idx` that removes 7,674 rows by
filter (75.7 ms). Statement 2 is an `Index Scan` on
`heartbeat_runs_company_created_at_desc_idx` that removes 19,801 rows by
filter (60.2 ms). After: one `Index Scan` on `heartbeat_runs_pkey`. Its
`InitPlan` is a `Merge Append` of two `Limit` index scans on
`heartbeat_runs_company_native_issue_contract_id_uq` and
`heartbeat_runs_company_ctx_issue_created_idx`, and its `SubPlan` is one
`Index Scan` on `chat_message_links_delivery_idx` (0.128 ms).

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed
(all GitHub chat-channel integration tests, including the
acknowledgement and terminal-response sweeps).
- `cd server && npx tsc --noEmit`: passed.
- `pnpm verify` exited 2 because a file under `scripts/` changed, which
requires the broad CI coverage. Required CI covers the rest.

Why the function runs 6.7 times a second. The acknowledgement sweep and
the terminal-response sweep rebuild their candidate set when a hint
arrives, and they call this function once for each candidate on each
rebuild. A run milestone for a run with no issue (for example, every
status change of a run without a task in `setRunStatus`, or a comment
merged into an active run in the wake queue) sends a company-wide hint
to both sweeps. So in a busy company, both sweeps rebuild about once a
second (their minimum pass interval), and each rebuild reads every
parked candidate again, although nothing about those candidates changed.
In a local reproduction with 8 parked candidates (4 for each sweep) and
the real coordinator and workers, 10 company-wide milestone hints a
second gave 8.75 calls a second; 1 hint a second gave 7.98 calls a
second; 10 hints a second for an unrelated issue gave 0 calls; and no
hints gave 0 calls. So the hosted rate is consistent with about 7 parked
GitHub requests in one company and a steady stream of company-wide run
milestones. This pull request makes each call cheap. A separate pull
request will address the repeated rebuilds.

## Risks

- Low risk. The change is one function. The run match is the same set of
runs: `native_issue_id::text = <uuid text>` is equal to `native_issue_id
= <uuid>` for the canonical UUID text that callers pass (the issue IDs
come from UUID columns), and a null `native_issue_id` makes the old
`coalesce` use the legacy context.
- Ownership: the old code compared the latest run that woke for the
request (ordered by `created_at` only) with the latest run (ordered by
`created_at, id`). That is true exactly when the latest run woke for the
request, except when two runs that both woke for the request tie on
`created_at` with the latest run. In that case the old result depended
on the physical row order. The new code always returns the deterministic
answer (the latest run woke for the request).
- The number of calls does not change in this pull request.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 00:59:25 -06:00
Nicky LeachandClaude Opus 5.5 e977c3b922 feat(native-runtime): end the wait for the provider's own turn when a person comments (#21455)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner part of this work is complete in #21316 and the pull
requests below it: when a controller sends `providerStartedTurns` on
`turn.start`, the Runner keeps Claude while a background shell is
running, and gives the cycle of Claude to the controller as a turn that
the provider started.
> - Part 2 gave the consumer a wait with a limit and two inputs: a hook
that ends the wait early, and a start time.
> - This pull request gives both from the server for a native ACPX
Claude run. A new comment from a person ends the wait. The start time
comes from an event that is already saved, so a restart of Core does not
make the wait longer.
> - The benefit is that a person who comments does not wait up to 10
minutes for the agent to see it.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one of four parts on the Core side. They are
stacked on #21316:

1. The driver admits a turn that the provider started.
2. The consumer waits for that turn, for a limited time.
3. The server ends the wait when a person comments, and keeps the clock
across a restart.
4. The transport sends the opt-in. Only this part changes what a
production run does.

This is part 3. Its base branch is `feat/core-provider-turn-consumer`.
Refs #19912 (design).

**What existing behavior does this improve?**

Nothing visible yet. What Core gives to the consumer of a native ACPX
Claude run.

**Subsystem affected**

Server: `server/src/services/native-runtime`.

**Current behavior**

- A comment from a person during a run is queued as a deferred wake. The
next run delivers it. This does not change.
- Core has no wait for a provider-started turn.

**Proposed behavior**

- For a native ACPX Claude run, Core gives `executeNativeSession` the
option `providerTurnWait`.
- One delivery worker does the person-comment check for all runs that
wait. The server registers it at startup with the shared delivery work
coordinator, as it registers the cost accounting worker. The worker
listens to the existing notification for deferred wakes. On each notice
it runs one query for each run that waits: a comment from a person on
this issue, made after the start of the run, that a deferred wake of
this agent names. A comment that the query finds ends the wait of that
run. The run then takes the path of today, and the next wake delivers
the comment as today.
- If a database read fails, that run stays owed a check, and the
coordinator runs the worker again after 5 seconds, as for each other
delivery worker. A read that fails for one run does not stop the check
of the other runs. When no run is owed a check, the worker has no
deadline and does no query.
- The start time of the wait is the time of the first saved
`turn.completed` event of the run that has `awaitingProviderTurn`. With
no such event it is empty.
- Each query has the company id in its condition.

**Reason and benefit**

- The comment window starts at the start of the run, not at the start of
the wait. A comment made during the prompt turn must not wait for the
limit.

## What Changed

- `server/src/services/native-runtime/native-provider-turn-wait.ts`
(new, 109 lines) and 4 lines in `native-session-executor.ts`.
- `server/src/index.ts` (4 lines): the registration of the worker, its
stop at server close, and its place in the list of delivery workers that
must be ready. `delivery-work-notifications.ts` (1 line): the queue key
of the worker. The key needs no database change, because each topic uses
the one shared channel.
- `server/src/vendor/paperclip-runner/index.ts`: one type export.
- Tests: `native-provider-turn-wait.test.ts` (new, 252 lines) and one
case in `native-usage-accounting.test.ts` (25 lines).

## State saved by the previous release

- No new persisted state and no schema change. The start time is read
from `heartbeat_run_events`.
- **A run that the previous release started:** it has no waiting
terminal, so the start time is empty.
- **Core restarts during a wait:** the new process reads the start time,
so the limit is the original one. The consumer then continues the wait
(#21454), and the worker checks the comments again from the start of the
run, so a comment that was made before the restart still ends the wait.
- **Rollback of Core during a wait:** the previous Core stops at the
waiting terminal and finalizes the run as `native_finalization_missing`,
as today.

## Verification

At the head of this branch:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json` and `pnpm run build:typescript` in
`packages/paperclip-runner`; `node
scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in
`server`; `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed.
- `native-provider-turn-wait.test.ts`, 10 tests on embedded Postgres
with the real coordinator: a comment from a person in a deferred wake
ends the wait; a comment from an agent, a comment before the run, and a
comment of a different company do not; the start time comes from the
first waiting terminal, and a run of the previous release has none.
- The 6 new tests: a read that fails one time is tried again after 5
seconds and finds a comment that is already saved, with no new notice; a
notice during a read that fails gives a new check; a read that fails for
one run does not hold back a second run, and each run ends only for its
own comment; a run that starts to wait during a check is checked; a wait
that is aborted is removed, nothing checks it again and no deadline
stays; with no run owed a check there is no deadline and no query in one
hour of test time.
- The test helper now registers the wake intent before it writes the
deferred wake, in the same transaction. This is the order of each
production writer (`modules/wake-queue/adapters/postgres.ts` and
`services/heartbeat/queue.ts`). Production code did not change for this.
- Mutation checks: a read that fails clears the owed check (3 tests
fail); an aborted wait is not removed (the abort test fails).
- Also run, unchanged and green: `delivery-work-coordinator.test.ts`,
`budget-enforcement-worker.test.ts`,
`server-startup-feedback-export.test.ts`, `startup-timeline.test.ts`.
The 7 files together: 1,699 passed, 7 skipped.
- `native-usage-accounting.test.ts`: a prompt turn and the
provider-started turn after it are each counted one time, with the
events as the executor receives them. The cost of the provider-started
turn is the session total in the cumulative channel, and it is not added
two times. The accounting code is not changed.

Not run: a run through the executor with a database; the full suite. CI
supplies the broad coverage.

## Risks

- **One more query at each start of a native ACPX Claude run,** on
`heartbeat_run_events` by company and run.
- **The worker is registered in each server process, and it is idle with
no wait.** A notice with no run that waits does no query. No wait starts
before part 4.
- **After an abort during a retry, the 5 second retry of the coordinator
runs one more time.** It finds nothing owed and does no query. The
coordinator has no call to cancel the retry of one worker.
- **A process that does not register the worker does no comment check.**
The server registers it with no condition at startup, and native runs
execute only in that process. Only a unit test can be in this state.
- **A comment ends the wait but does not interrupt a turn that is
running.** That is the rule of part 2.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:55:11 -08:00
DottaandPaperclip 2102492806 fix(discord): refresh due command registration after task drains (#21573)
## Thinking Path

> - Paperclip runs agents and connects their work to chat services.
> - Discord command registration uses a shared worker and saved
deadlines.
> - A task drain can hold that worker while registration is already due.
> - The coordinator can keep a 30-second fallback after the drain ends.
> - This change refreshes the command owner on drain release or observed
expiry.
> - Normal admission, earliest-time and provider retry checks still
apply.

## Linked Issues or Issue Description

Refs: #17557, #21562. Related: #15532 checks credential reads when
registration is not due. This change handles resumption of due work in
the separate command worker.

**What happened?**

A due command registration can wait for the coordinator's fallback after
a task drain ends. The worker retains the request but has no
drain-release subscription. This delays command work and its return to
an idle state.

**Expected behavior**

Resume due command work when admission returns. Preserve future earliest
times and provider failure backoffs. Leave an empty or stopped worker
idle.

**Steps to reproduce**

1. Register the normal command worker while an idle task drain is
active.
2. Retain a due registration and capture the coordinator's fallback.
3. Release the drain, or observe its expiry, before that fallback.
4. Check that the due endpoint is checked before the fallback. Both
regression cases fail before this change.

**Paperclip version or commit**

Main base `3c15acd3fc1b200ef0e36ac904a03cd5fcba00c0`.

**Deployment mode**

Built from source. The focused tests use the real coordinator and
command owner with controlled registration operations. They do not
contact Discord.

## What Changed

- Subscribe the command owner to normal drain changes and observed
expiry.
- Wake it only when current admission allows work and its existing next
time is due.
- Remove both subscriptions through the normal worker stop callback.
- Add eight controls for release, expiry, another closed gate, future
earliest time, provider backoff, empty state and both stop paths.

## Verification

Local checks used the integrated branch that exposed this contract gap.
This main port contains the same two tested file afterimages. Its
admission, scheduler and fairness modules match those local inputs.
Main's coordinator and DB modules have other changes. Fresh CI on
`aec6ecc65717aa6372bad526e044ec4aa3888bec` now qualifies this main
context.

- Both release and expiry cases fail at the intended missing
endpoint-check assertion before the product change. No setup or timeout
failure is used as causal proof.
- All eight focused cases pass after the change. No case is excluded.
- The complete normal Server typecheck and affected-file program checks
pass on the integrated branch.
- Normal committed Source SDK verification passes on this main branch.
- Exact-head main CI passes, including typecheck, build, the general and
serialized suites, and all eight new controls (142 ms) in server shard
2/12. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/43565579653).
A full local repository build and suite were not run in this checkout.
- Greptile reviewed the exact head at 5/5 with no actionable findings.
The complete edited summary and all paginated review threads were
inspected; no unresolved threads remain.

Run the focused controls with:

```sh
pnpm --filter @paperclipai/server exec vitest run src/services/chat-channels/discord/command-work.test.ts --maxWorkers=1 --no-file-parallelism --retry=0
```

The existing admission-drain contract remains accurate. This fix adds no
configuration or command, so no documentation change is required. Full
transport sleep acceptance and deployed suspension are separate claims.

## Risks

A release event can request an earlier pass for already due work. The
existing `canRun`, earliest-time, batching and per-item backoff checks
remain active. Both worker stop and coordinator stop must remove the
subscriptions; the controls cover each path. No scheduler policy, claim
or provider deadline changes.

## Model Used

OpenAI `gpt-6.1-sol` assisted with implementation and independent
review. The Codex managing agent performed technical review and
verification. The host does not expose its exact runtime model ID or
context-window size. Work used source inspection, code edits and local
test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (not
required: the existing contract remains accurate)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-14 00:46:49 -06:00
DottaandClaude Opus 5.5 3c3e6c2d75 fix(retention): stop rescanning unchanged pinned workspace snapshots (#21571)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote workspaces keep saved versions in the
`execution_workspace_snapshots` ledger, and a retention sweep retires
versions that no reference holds
> - Every execution notification wakes this sweep, and the sweep
examined every pinned version again on each pass
> - Some pins never change. A migration cohort supersession leaves
failed operations with `lastFailure` and no `retiredSave`, and these
hold their snapshots until a separate review releases them
> - On a busy server the sweep ran four statements per pinned row, many
times each second, on a main thread that was already saturated
> - This pull request records the reference that pins a version and
leaves that version out of the candidate query until that reference
releases
> - The benefit is that an execution notification costs one candidate
query, not one transaction per pinned row

## Linked Issues or Issue Description

No public issue exists. Bug report:

**What happened?**
A hosted instance ran four statements at 13.4 calls per second each: the
workspace read, the importing-parent read and the operation read in
`pinned()`, and the locked ledger row read. The sweep in
`server/src/services/remote-workspace-snapshot-retention.ts` caused
them. A pinned superseded row keeps `status = 'superseded'` with
`delete_after = null`, so the candidate query selects it on every pass.
Each pass examines up to 50 rows. A truncated page runs again after one
second. Each run-work notification (`notifyExecutionRunWork`) sets
`rescanRequested`, which starts the scan again. The sequence stopped at
the operation read because a failed operation with `lastFailure` and no
`retiredSave` pinned these rows.
`supersedeExpiredRemoteWorkspaceMigrationCohort` writes such operations,
and they hold their snapshots permanently by design (see
`doc/DATABASE.md`, "Workspace cohort admission proofs").

**Expected behavior**
A pinned version is examined again only when the reference that pins it
can have changed. An idle or busy server does not repeat reads for
unchanged pinned history. A released version is retired on its normal
deadline.

**Steps to reproduce**
1. Create 300 superseded snapshots. Pin each with a failed
`remote_workspace_migration` operation that has `lastFailure` and no
`retiredSave`.
2. Register the retention worker on a delivery coordinator and let the
first pass finish.
3. Send `notifyExecutionRunWork` about 13 times each second for 60
seconds.
4. Count statements with `pg_stat_statements`.

**Paperclip version or commit**
`master` at 6dd0349d52

**Additional context**
The local reproduction showed the same four statements at equal call
counts, as on the hosted instance (see Verification).

## What Changed

- `pinned()` returns the first reference that holds the version (current
pointer, importing parent, operation, unfinished child, run, or recovery
action) and its ID, not a boolean.
- The sweep stores that reference in `metadata.retentionPin` when it
pins a version. It writes again only when the reference changes or a
deadline must be cleared.
- The candidate query leaves out a version with no deadline while its
recorded reference still holds. Each pin condition is one SQL builder.
`pinned()` and the candidate query use the same builder, so the
exclusion can never hold a version that `pinned()` would release.
- When the recorded reference releases, the version comes back, and
`pinned()` checks every reference again. It records the next reference
that holds, or schedules or retires the version.
- The record is removed when the sweep schedules or claims a version.
- The worker coalesces wakes for up to one second
(`REMOTE_WORKSPACE_SNAPSHOT_COALESCE_MS`). Retention deadlines are days,
so this delay has no effect on retirement.
- `doc/DATABASE.md` describes the record.
- No schema migration. The record is in the existing `metadata` JSONB
column.

## Verification

Red/green regression tests in
`server/src/services/remote-workspace-snapshot-retention.test.ts`:
- `leaves %s-pinned history out of later passes, across restarts, until
that pin is released`. This runs once for each of the six pin kinds. It
pins the version and checks that a new service instance (a restart)
examines zero rows. Then it releases the pin and checks that the version
is deleted.
- `re-examines a pinned snapshot when its recorded pin is released but
another still holds it`.
- `revisits a failed save's recorded pin once its abandoned upload is
settled`.
- `does not re-examine unchanged pinned history on execution
notifications`. This uses the coordinator and `notifyExecutionRunWork`.

On the parent commit, the first eight new tests fail (6 pin kinds, the
second-pin case, and the notification case). With the fix, all 31 tests
pass.

Related suites pass locally (212 tests):
`remote-workspaces.integration.test.ts`,
`remote-workspace-snapshot-reuse.test.ts`,
`remote-workspace-cohort-migration.test.ts`,
`snapshot-attempt-upload-abort.test.ts`, and
`remote-workspace-snapshot-retention.test.ts`. `tsc --noEmit -p
server/tsconfig.json` is clean.

Load measurement. Local PostgreSQL 18 with `pg_stat_statements`. 300
pinned rows. About 13 run-work notifications each second for 60 seconds,
after the first pass finished:

| | Before | After |
| --- | --- | --- |
| Pinned rows examined again | 30,250 – 39,000 / min | 0 / min |
| Sweep statements | ~151,000 – 196,000 / min | 88 / min |
| All statements, including the test's own notification transactions |
153,327 – 197,205 / min | 1,647 / min |
| Sweep database execution time | ~1,285 ms / min | ~52 ms / min |
| Candidate query | 0.07 ms, ~10 / min | 0.88 ms, 58 / min |

Before: `begin`, the locked row read, the workspace read, the
importing-parent read and the operation read each ran 30,250 times in 60
seconds. This is the same signature as on the hosted instance. After:
the sweep ran 58 candidate queries and 30 deadline reads. With
per-notification wakes and no coalescing, the sweep ran 776 candidate
queries and 776 deadline reads (1,552 / min).

I did not verify the change on a hosted instance.

## Risks

- A recorded pin is checked against only its own reference. The check
uses the same SQL condition as `pinned()`, so a version cannot stay
excluded after that reference releases. The parametrized test covers
each pin kind.
- A version can be pinned by more than one reference. When the recorded
reference releases, the sweep checks all of them again and records the
next one.
- A producer that rewrites the row's metadata and drops the record
causes one extra examination. A producer that sets a deadline makes the
record inactive, because the exclusion applies only when `delete_after`
is null.
- Versions that have no record yet, which includes all existing pinned
rows, are examined once after deploy. The examination writes the record.
- The candidate query does one primary-key lookup for each pinned row.
It takes about 0.9 ms with 300 pinned rows, and the time grows linearly
with the number of pinned rows.
- Pinned history from failed migration cohorts stays pinned. This change
does not alter that hold.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, local PostgreSQL)

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 00:05:32 -06:00