Commit Graph
4274 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 28761f9c25 perf(workspaces): read a downloaded snapshot archive once, in the encrypt pass (#21683)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote agents work in sandboxes, and the server saves each sandbox
workspace as an encrypted snapshot in object storage
> - One Node main thread on the server controls every run, so per-agent
byte work on that thread limits how many agents one server can hold
> - The save path read every downloaded archive twice: a main-thread
SHA-256 re-hash after the download, then a full read in the off-thread
encrypt pass that already computes the same digest
> - This pull request removes the main-thread re-hash and checks the
encrypt pass's digest and size against the sandbox receipt instead
> - The benefit is one full archive read per save instead of two, and no
archive hashing on the main thread, with the same integrity guarantee

## Linked Issues or Issue Description

Refs the 1,000-concurrent-agents plan in
`doc/plans/2026-11-13-thousand-concurrent-agents.md` (Phase 1,
"Workspace snapshot" row).

**What existing behavior does this improve?**
Saving a remote workspace snapshot after a run.

**Subsystem affected**
Server: remote workspace snapshot save (`remote-workspace-transport.ts`,
`remote-workspace-snapshot-storage.ts`, `remote-workspaces.ts`).

**Current behavior**
`captureSandboxWorkspaceArchive` streams the whole downloaded archive
through `createHash` on the main thread to compare it with the sandbox
receipt. Then `uploadWorkspaceSnapshot` reads the whole archive again on
a worker thread to encrypt it, and computes the same plaintext digest.

**Proposed behavior**
After the download, the server checks only the file size against the
receipt. The encrypt pass compares its plaintext digest and size with
the receipt. A mismatch fails with the same
`remote_workspace_capture_transfer_mismatch` error before the upload
attempt is recorded and before any object is stored.

**Reason and benefit**
One full read and one SHA-256 of each archive leave the server's main
thread (about 0.36 ms of main-thread time per MB on a loaded dev
machine; 37 ms for a 100 MB archive, at event-loop utilization 1.0 for
that time).

## What Changed

- `captureSandboxWorkspaceArchive` no longer re-hashes the downloaded
archive. It checks with `lstat` that the download is a file with the
receipt's byte size, and fails with
`remote_workspace_capture_transfer_mismatch` when it is not.
- `uploadWorkspaceSnapshot` accepts an optional `expectedArchive`
(`sha256`, `byteSize`). After the encrypt pass, it compares the sealed
plaintext digest and size with it. A mismatch throws
`remote_workspace_capture_transfer_mismatch` before the attempt enters
`metadata.uploads` and before `putFile`. The scratch ciphertext is
removed as before.
- `saveRemoteWorkspace` passes the sandbox receipt as `expectedArchive`
when it captured an archive.
- The save-level `verifyMs` timing field is removed, because that step
no longer exists. The storage-level `verifyMs` stays.
- `doc/run-log-events.md` says that the encrypt step is the only full
read and where a mismatch fails.

Callers that relied on the early hash keep their behavior:
- Unchanged-snapshot reuse (`findUnchangedWorkspaceSnapshot`) compares
the sandbox receipt, not the local hash. When it reuses the parent, the
local copy is used only to list directories, and
`readRemoteWorkspaceArchiveDirectories` verifies the manifest digest.
- Reuse of an earlier upload attempt or an already uploaded snapshot
does not read the archive. Those objects were verified when they were
stored.
- The read-only disk-full fallback builds its archive on the server, so
its receipt always matches. Its behavior does not change.
- The error message and the failure path do not change, so the save
failure code, retry and fallback stay the same.

## Verification

- Red/green: `remote-workspace-transport.test.ts` "leaves the downloaded
archive's single full read to the encrypting upload" counts main-thread
read streams of the downloaded archive during capture. On master it is 1
(fails). With this change it is 0 (passes). The encrypt pass is the one
remaining read.
- New: digest mismatch and size mismatch in `uploadWorkspaceSnapshot`
reject with the transfer-mismatch error, do not call `putFile`, and
leave the row `pending` with no recorded attempt. A retry with the right
receipt saves.
- New: an end-to-end `saveRemoteWorkspace` with a transfer that flips
one byte (same size) fails with the transfer-mismatch error, stores
nothing, keeps the writer and the previous snapshot, and a retry saves
and restores the right content.
- New: a download whose size differs from the receipt fails in capture.
- `cd server && npx vitest run
src/services/remote-workspace-snapshot-storage.test.ts
src/services/remote-workspace-transport.test.ts
src/services/remote-workspaces.integration.test.ts
src/services/remote-workspace-transfer-timeout.test.ts
src/services/remote-workspace-readonly-capture-budget.test.ts
src/services/sandbox-disk-full-fault.test.ts` passes: 6 files, 360
tests. `src/services/remote-workspace-snapshot-reuse.test.ts` passes: 32
tests.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. A same-size corrupt download is now detected in the encrypt
pass instead of right after the download. It is still detected before
any attempt is recorded or any object is stored.
- When the sandbox reports an unchanged workspace, a corrupt local copy
no longer fails the save. The save reuses the verified parent snapshot,
which matches the sandbox's content. The local copy is used only for its
manifest, and the manifest digest is verified.
- The save-level `saveTiming.verifyMs` field is gone. No code reads it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:41:29 -06:00
DottaandClaude Opus 5.5 cf2a4221e3 perf(server): skip the per-request stale instance-admin purge when no row exists (#21680)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Cloud-hosted instances authenticate each request through trusted
Cloud headers in the actor middleware
> - That path deletes stale `instance_admin` rows for the user on every
request, static files and live-events upgrades included
> - On an idle hosted instance this delete ran about 1.2 times a second,
which is a write on an instance-wide table for each request
> - Almost always there is no row to delete, so the statement is wasted
work and a database round trip
> - This pull request finds out if a stale row exists inside the
membership read that the request already does, and deletes only when a
row exists
> - The benefit is one read and no write for each unchanged
authenticated request, with the same purge on the very next request when
a stale row appears

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Cloud trusted-header authentication in the actor middleware.

**Subsystem affected**
Server authentication (`server/src/middleware/auth.ts`).

**Current behavior**
`resolveCloudTenantActor` in `server/src/middleware/auth.ts` sends
`delete from instance_user_roles where user_id = $1 and role =
'instance_admin'` for every trusted-header request. The actor middleware
runs before the static UI, so asset requests run it too. Each
live-events WebSocket upgrade runs it again through `resolveCloudActor`.
The retry wrapper runs it again after a transient connection error.

**Proposed behavior**
An unchanged request sends no write. A stale `instance_admin` row is
still removed on the next request after it appears.

**Reason and benefit**
Idle hosted instances send about one delete for each request. This costs
a round trip and a write statement on the main database for no change.

**Breaking changes**
None. The resolved actor and the authorization decisions do not change.

## What Changed

- The Cloud trusted-header path reads the user's active memberships
together with an `exists` check for an `instance_admin` row, in one
statement.
- The purge runs only when that check finds a row, when the user has no
active membership row yet (first request), or when the read fails.
- The session path and the Cloud path share one membership filter
function, so both still resolve the same access set.
- New test `server/src/__tests__/cloud-tenant-actor-statements.test.ts`
meters the statements of one authenticated request on the embedded test
database.

## Verification

- `cd server && npx vitest run
src/__tests__/cloud-tenant-actor-statements.test.ts
src/middleware/cloud-tenant-actor.test.ts
src/__tests__/cloud-tenant-actor-round-trips.test.ts`: 41 passed.
- Red/green: with the `origin/master` version of `auth.ts`, the new test
fails. An unchanged member request sends 2 statements, and one of them
is the delete. With this change it sends 1 read and 0 writes.
- The test inserts a stale `instance_admin` row between two requests.
The next request deletes it, and the request after that sends no write
again.
- The test changes the stack role from member to admin and back. The
membership role and the experimental-settings access change on the very
next request each time.

## Risks

- Low risk. The actor that the middleware returns is the same.
Authorization decisions do not change.
- The purge now runs after the membership read, not before the tenant
sync. The sync does not touch `instance_user_roles`, so the order does
not matter. The request still waits for the purge before it continues.
- A race between the read and a new row insert existed before too
(delete, then insert). The next request removes such a row.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:34:30 -06:00
DottaandClaude Opus 5.5 dcba9628c7 perf(recovery): keep a finished task's settled attention settled and halve graph hint fan-out (#21676)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Issue recovery records "recovery actions" on tasks. A sweep resolves
the active action of a task that is already done or cancelled.
> - On an idle hosted instance, a debug-log run (from #21640) named a
ping-pong. A native worker re-surfaces a board attention for 13 finished
tasks. Recovery resolves it as `source_terminal`. The unique "one active
action" index then lets the worker insert it again, about every 26
seconds per task. That built 45,008 rows since 2026-10-21. Each insert
and each resolve also signals run dispatch and the task's recovery
graph, which wakes both sides again.
> - Separately, every task-graph recovery hint is published twice per
task queue: once as the graph hint and once as a `{companyId, backstop:
true}` twin. That is half of all idle signals, for a backstop that is
time-gated anyway.
> - This pull request breaks the ping-pong from the recovery side and
removes the backstop twin.
> - The benefit is that a finished task's settled attention stays
settled and wakes nothing. Each graph hint costs half the signals.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

With zero runs active, recovery and native workers kept re-creating and
re-resolving the same board attention on 13 done or cancelled tasks. The
rows are `active_run_watchdog`, owner `board`, cause
`native_restart_recovery_blocked`. The loop added about one row per task
every ~26 s and about 2.6 work-signal commits per second.

**Expected behavior**

Once recovery settles a finished task's attention, the same attention
identity is not surfaced again, and settling it wakes nothing that a
finished task cannot use.

**Steps to reproduce**

1. Create a cancelled task.
2. Surface the native "controller process space unverified" attention
for it (same cause and fingerprint each time), then run
`reconcileActiveRecoveryActions`.
3. Repeat step 2. On master each cycle inserts a new active row and
resolves it, and each resolve signals the task queues and run dispatch.

**Paperclip version or commit**

master at 36101ebcb9.

## What Changed

- `server/src/services/issue-recovery-actions.ts`:
- **No-match skip.** `resolveActiveForIssue` first reads whether any
action matches. When none does, it returns `null` with no transaction
and no signal. Before, `writeExecutionControl` always published
run-dispatch, graph-recovery and execution hints first.
- **Settled-attention guard.** `upsertSourceScoped` returns the existing
resolved action instead of inserting when ALL of these hold:
    - the task is done or cancelled;
    - it has no active action;
- an action with the same cause and fingerprint was already resolved
with `source_terminal`.
The guard writes no row and sends no signal, and the caller gets that
resolved action. It covers only the exact identity that recovery itself
settled because the task finished. A new fingerprint (renewed controller
authority), a different cause, or a reopened task still creates an
action.
- **Quiet terminal resolve.** A new `sourceTerminal` resolve mode. It
updates the row only while the task is still done or cancelled. It still
notifies native recovery notices, unsafe-workspace export recovery, and
a preserved native finalization waiter. It does NOT signal run dispatch,
the task's recovery graph, or the execution disposition/continuation
queues: a finished task has no run to dispatch and no assignment to
recover.
- `server/src/services/recovery/service.ts`: the `source_terminal`
branch of `reconcileActiveRecoveryActions` uses `sourceTerminal: true`.
- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery` publishes one graph hint per task queue and
no longer adds a `{companyId, backstop: true}` hint per queue. A new
`issueRecoveryBackstopHint(companyId)` builds that hint string.
- `server/src/services/recovery/work.ts`: when a task queue loads a
graph scope, it arms its own backstop in process
(`worker.invalidate(issueRecoveryBackstopHint(companyId))`). The #21552
guarantee stays: any graph activity arms the company backstop, which
runs at most once per backstop interval after the last full pass. It no
longer costs one signal per queue per hint.
- `server/src/__tests__/issue-recovery-actions.test.ts`: the fake-DB
unit tests accept the guard's extra read.
- `server/src/__tests__/issue-recovery-signal-fanout.test.ts` (new, real
PostgreSQL), six tests:
  - one graph hint per task queue and no backstop hint;
  - a resolve with no matching action returns null and signals nothing;
- an escalated action on a finished task is still resolved and still
signals;
- the ping-pong cycle: surface then reconcile, five times, leaves one
row and no stranded or active-recovery signals;
- settling a finished task's attention sends no task-queue, run-dispatch
or active-recovery signals;
- a renewed authority and a reopened task still get a new attention,
which stays active on a reopened task.

## Verification

- New test file:
  - master: 4 of 6 fail (red).
    - 8 task-queue hints per graph hint instead of 4.
    - A no-match resolve publishes 18 settled signals.
- Cycle 2 inserts a new active row; master builds a new row every cycle.
- Settling a finished task's attention signals the task queues and run
dispatch.
  - This branch: 6 of 6 pass (green).
- Regression on real PostgreSQL:
- heartbeat-native-restart-suspension, native-recovery-notice-work,
issue-recovery-deadlines (including the #21552 backstop tests),
assigned-work-invariant, execution-control-work, issue-recovery-actions,
issue-recovery-work-lifecycle, issue-recovery-assignments,
execution-recovery-resolution, automatic-run-brake-recovery-passes: all
pass (381 tests).
- Four unit tests in issue-recovery-actions use a hand-written fake DB.
They needed `innerJoin` on the fake query and an expected count of two
selects, because the guard adds one read.
  - recovery-deadline-acceptance: passes.
- `tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- The guard returns a resolved action to a caller that asked to surface
one. I checked the callers: the one that loops
(`surfaceNativeControllerNamespaceAttention`) ignores the return value.
The others only reach the guard for a finished task whose identical
attention recovery already settled, where re-surfacing it would be
resolved again within one sweep.
- The terminal resolve no longer wakes run dispatch or the task's graph.
Completing or cancelling a task already sends those signals itself.
- This does not stop the native worker from trying to surface attention
for finished tasks. That fix belongs in the native owner-release code,
which another change owns.
- The ~45k historical resolved rows on the hosted instance stay as they
are. A cleanup would need a migration or an operator script that deletes
resolved `native_restart_recovery_blocked` rows whose source is done or
cancelled, keeping the newest per (task, fingerprint) so the guard still
matches. That is not part of this change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21552, #21621, #21640.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:06:38 -06:00
DottaandClaude Opus 5.5 97a167093d fix(queue): send a message into a starting run without a cancel or recovery errors (#21661)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can queue a message while an agent works, then send it into
the running turn
> - A run that is still starting takes queued messages into its first
turn, and Interrupt hands off to them
> - Both rules only accepted a queue that a user started
> - A user's message can merge into an earlier automatic wake that waits
behind the task, and that wake keeps its system requester
> - Then the starting run did not take the message, Interrupt cancelled
the boot, and the task showed "Couldn't finish this attempt" with Retry
and "Waiting for execution recovery" under the composer until the
cancelled boot settled
> - This pull request makes both rules judge the messages and the
recorded Interrupt intent, not the wake's requester, and shows an
Interrupt hand-off as sent
> - The benefit is that sending a queued message feels immediate and
shows no error

## Linked Issues or Issue Description

**What happened?**
A run started from a resolved blocker. The user queued a message 20
seconds later, while the runner was still starting. The message merged
into an older automatic deferred wake. The queue offered Interrupt
instead of sending it. Interrupt cancelled the starting run. For about 7
seconds the task showed "Couldn't finish this attempt. Your message is
saved." with Retry, and the composer showed "Waiting for execution
recovery. Your message is saved." Then the message was delivered.

**Expected behavior**
The starting run takes the message into its first turn. When an
Interrupt does stop a run, the message shows as sent, with no error
banner, Retry, or recovery text.

**Steps to reproduce**
1. Have an automatic deferred wake for a task (for example from an
earlier blocked execution).
2. Start a run for the task with a Paperclip runner agent.
3. While the runner is still starting, queue a message and click the
send control.

## What Changed

- `deferredWakeJoinsFirstTurn`: a system or unattributed wake qualifies.
Each message must still be the responsible user's own
(`firstTurnEligibleComment`). A wake from another user or an agent keeps
its own delivery.
- Task interruption presentation: find the Interrupt queue by its
recorded `queuedCommentInterrupt` intent, not by `requestedByActorType =
user`. The hand-off now shows as working for a merged queue too.
- Queued-comment read: while a deferred queue carries an Interrupt
intent, report an `interrupt_handoff` wait instead of the recovery
message.
- UI (`TaskChatQueuedMessages`, `IssueDetail`): an `interrupt_handoff`
response counts as sent. The messages stay out of the queue and no
status or error line shows.

## Verification

Red/green tests. Each failed before the fix and passes after it:
- `server/src/__tests__/issue-queued-comments-routes.test.ts`: "gives a
booting run's first turn the user's message merged into an earlier
automatic wake"; the Interrupt fallback case now expects
`interrupt_handoff` on the response and on a poll. 144/144 pass.
- `server/src/__tests__/task-interruption-query.test.ts`: the hand-off
case runs for a user-started and a system-started queue. 32/32 pass.
- `ui/src/components/task-chat/TaskChatQueuedMessages.test.tsx`: "shows
an Interrupt as sent while the stopped turn hands off". 43/43 pass.
- `first-turn-queued-input.test.ts` 8/8, `issue-queued-comment-queue`
7/7. Server and UI typecheck pass.

Root cause from the staging wake diagnostics: the run started at
20:26:54, the message was coalesced at 20:27:17 into a deferred
`issue_execution_deferred` automation wake from the previous day, the
Interrupt at 20:27:20 recorded `execution_recovery`, and the
false-positive hold resolved at 20:27:26.

Not run: local runner E2E steering cells. The OpenCode chat-features
cells require the trusted CI host. The Codex and Claude
`revise-while-running` chat cells failed before steering on a fixture
workspace path (`workspaces/` vs `isolated-workspaces/`), which this
change does not touch. Staging verification follows the deploy.

## Risks

- Low. A wider set of deferred wakes can join a starting run's first
turn. Message eligibility, the claim receipt, and exactly-once delivery
are unchanged.
- An Interrupt intent that never delivers shows "sent" in the composer.
The task notice still shows a failure with Retry after its existing
grace period.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:06:15 -06:00
DottaandClaude Fable 5.2 ac114d9479 refactor: cut DEVELOPING.md to a setup and workflow guide (#21670)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `AGENTS.md` tells every contributor and agent to read
`doc/DEVELOPING.md` for setup, commands, testing, and worktrees
> - That file grew to 5,495 lines. Most of it was notes on how one
subsystem behaves, added one pull request at a time
> - Readers could not find the commands. Agents also read behavior notes
as rules
> - This pull request cuts `doc/DEVELOPING.md` to a 660-line setup and
workflow guide. It moves the subsystem notes to focused docs
> - The benefit is a guide that a person or an agent can read in full,
and a clear place for each kind of note

## Linked Issues or Issue Description

**Issue type**

Outdated or confusing documentation.

**Where is the issue?**

`doc/DEVELOPING.md`

**What's wrong?**

The file was 5,495 lines. It opened with 112 lines about case handoffs
before it reached Prerequisites. Unrelated sections sat below the
"OpenClaw Docker UI One-Command Script" heading. Statements about
current behavior read like policy.

**Suggested fix**

Keep setup, commands, and workflow in `doc/DEVELOPING.md`. Move each
subsystem note to a doc for that subsystem. Add an index.

## What Changed

- `doc/DEVELOPING.md` is now 660 lines: prerequisites, start dev, health
checks, local run, test drive, Docker, instance layout, database,
storage, secrets, worktrees, test commands, Storybook, CLI, and an index
of the other docs.
- 15 new docs hold the moved text without wording changes:
`ci-workflows`, `test-infrastructure`, `worktrees`,
`hot-restart-deploys`, `heartbeat-internals`, `runner-internals`,
`workspace-repositories`, `workspace-storage`,
`skills-and-teams-catalog`, `ui-internals`, `case-handoffs`,
`chat-connector-development`, `instance-configuration`,
`integration-api-notes`, `database-backups`.
- "Publish a branch Storybook" moved to `doc/STORYBOOK-DEPLOYMENT.md`.
The privacy rollout notes moved to `doc/ISSUE-PRIVACY.md`.
- Deleted about 280 lines: the story-by-story Storybook notes, "Agent
Artifact Uploads" (a copy of `doc/AGENT-ARTIFACTS.md`), two pointer-only
sections, "Workspace feature checks (2026-10-09)", and "Chat and task
navigation caching". The test-drive build-lock paragraph is now four
lines.
- Repointed every inbound link and every script or workflow comment that
named a moved section.
- `AGENTS.md` now says where a behavior note goes, so the guide does not
grow again.

## Verification

- A script split the file by line range. It asserts that each of the
5,495 source lines is kept, moved, or dropped exactly one time.
- A link check on every changed Markdown file found 0 missing files and
0 missing anchors.
- `node scripts/check-maps.mjs` passes.
- `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail.
- `node --check` passes on the three scripts with changed comments.
- `pnpm verify` was not run. This worktree has no installed
dependencies, and the change has no runtime code.

## Risks

- Low risk. The change is documentation and comments only.
- A link to a moved section from outside this repository will land on
the top of `doc/DEVELOPING.md`.
- The new docs are not in the audited prose list in
`.github/scripts/ci-inputs.mjs`. An edit to one of them runs full CI. An
edit to `doc/DEVELOPING.md` does not. This PR does not change that list.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool
use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 18:02:27 -06:00
DottaandClaude Opus 5.5 648b238be1 perf(native-runtime): stop re-finalizing native results whose finalization error is recorded (#21674)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime releases owners of failed runs. When the owner's
controller is in a process space that Paperclip cannot verify, it shows
controller namespace attention on the task instead.
> - On a hosted instance with zero active runs, this writer made about
21% of all work signals. It inserted a new active attention row for the
same task about every 26 seconds.
> - The tasks were already done or cancelled. Recovery resolved each row
as `source_terminal`. The active slot was then free, so the next
owner-release pass inserted a new row and published a graph recovery
signal again. The loop did not stop.
> - A live task had the same defect: after the fresh-continuation
fallback resolved its attention, the next pass made the same incident
again.
> - This pull request stops the writer from surfacing attention for
finished tasks. It also stops the writer from making an incident again
when that run authority was resolved after its lease expired. Those
owners are no longer release candidates and do not set the release
deadline.
> - The benefit is that an idle instance does not write or publish
anything for these runs. Attention still shows once for a live task.

## Linked Issues or Issue Description

**What happened?**
The failed-run owner release (`settleFailedRunOwner`, expired-owner
path) called `surfaceNativeControllerNamespaceAttention` for every
expired foreign owner that had no active attention. It did this even
when the source task was `done` or `cancelled`. Recovery resolved that
attention as `source_terminal`. The next pass found no active attention
and inserted a new board row. Each insert published
`notifyIssueGraphRecovery` and the other execution-control signals. The
stranded-task check then ran again for the task graph. A hosted instance
collected about 45,000 rows on 13 finished tasks in this way.

**Expected behavior**
A pass that changes nothing writes nothing and publishes nothing. A
finished task never gets controller namespace attention. A live task
gets one incident for each exact run authority. A resolved incident
stays resolved until the authority changes.

**Steps to reproduce**
1. Make a native run that failed, with an `observed` coordinator whose
foreign lease has expired and whose controller host is not recorded.
2. Mark the task done or cancelled, or let the fresh-continuation
fallback resolve the attention of a live task.
3. Run the execution-finalization worker with the issue recovery workers
for 30 minutes.
4. On master, each task gets about 40 to 60 new attention rows. The
writer publishes thousands of signals.

## What Changed

- `surfaceNativeControllerNamespaceAttention` (shared helper): before it
creates attention, it checks for a resolved incident for this run that
was resolved at or after the owner's current lease expiry. If one
exists, it writes nothing.
- New `settledNativeControllerNamespaceAttention` SQL predicate. It is
shared by the helper, the release candidate page, and the release
deadline query, so all three use the same rule.
- `settleFailedRunOwner` (expired-owner path) does not surface attention
when the task is `done` or `cancelled`. The restart-recovery claim path
has the same guard.
- `releaseAbandonedNativeOwners` and `nextAbandonedNativeOwnerReleaseAt`
skip foreign owners whose task is finished or whose incident is settled.
A pass that has nothing to do selects nothing, and no immediate deadline
is set again. Local owners keep the normal proof-based release.
- The owner release proofs did not change. A foreign owner is still
never released without proof.
- Tests:
`server/src/__tests__/native-failed-run-owner-release-idle.test.ts`
(real PostgreSQL, real coordinator and workers, fake clock, statement
and signal counters) and three new cases in
`native-failed-run-owner-release.test.ts`.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/native-failed-run-owner-release-idle.test.ts
src/services/native-runtime/native-failed-run-owner-release.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/services/native-runtime/native-restart-recovery-isolation.test.ts`:
80 tests pass.
- Red on master, green with this change. The test seeds one done task
and one cancelled task (each with an earlier resolved `source_terminal`
incident) and one live task, then runs 30 idle minutes:
- master: 126 attention inserts, 249 attention writes, 7,024 signals,
8,025 statements.
- this branch: 1 attention insert (the live task, surfaced once), 2
attention writes (that insert and the fallback's resolution), 36
signals, 350 statements. All of this is startup work. Nothing repeats.
- After the fresh continuation settles a live task's attention, six
unrelated finalization wakes over 30 minutes:
- master: 3 attention inserts, 90 signals, 551 statements, and the
release deadline stays due.
- this branch: 0 attention writes, 0 run writes, 6 signals (the test's
own wakes), 153 statements, and the release deadline is null.
- New unit cases: a done or cancelled task never surfaces, and surfaces
once when it is reopened. A settled incident stays settled over repeated
passes. A renewed authority (a new lease expiry) surfaces one new
incident.
- The three new unit cases fail on master and pass on this branch.
- `tsc --noEmit` for `server` passes.

## Risks

- Low risk. The change only removes writes. It does not weaken a release
proof, an authorization, a budget, a pause, or an approval check.
- An incident that a person resolved or dismissed after the lease
expired is not made again for the same authority. A renewed authority (a
later lease expiry) still makes a new incident.
- A superseded incident (status `cancelled`) still shows again when the
superseding action clears. This keeps the current behavior.
- The historical rows that the loop made are not removed. This change
only stops new rows.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21621, #21616, #21617

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:50:45 -06:00
DottaandClaude Opus 5.5 64c7122dae perf(native-runtime): stop re-surfacing settled controller namespace attention (#21675)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime releases owners of failed runs. When the owner's
controller is in a process space that Paperclip cannot verify, it shows
controller namespace attention on the task instead.
> - On a hosted instance with zero active runs, this writer made about
21% of all work signals. It inserted a new active attention row for the
same task about every 26 seconds.
> - The tasks were already done or cancelled. Recovery resolved each row
as `source_terminal`. The active slot was then free, so the next
owner-release pass inserted a new row and published a graph recovery
signal again. The loop did not stop.
> - A live task had the same defect: after the fresh-continuation
fallback resolved its attention, the next pass made the same incident
again.
> - This pull request stops the writer from surfacing attention for
finished tasks. It also stops the writer from making an incident again
when that run authority was resolved after its lease expired. Those
owners are no longer release candidates and do not set the release
deadline.
> - The benefit is that an idle instance does not write or publish
anything for these runs. Attention still shows once for a live task.

## Linked Issues or Issue Description

**What happened?**
The failed-run owner release (`settleFailedRunOwner`, expired-owner
path) called `surfaceNativeControllerNamespaceAttention` for every
expired foreign owner that had no active attention. It did this even
when the source task was `done` or `cancelled`. Recovery resolved that
attention as `source_terminal`. The next pass found no active attention
and inserted a new board row. Each insert published
`notifyIssueGraphRecovery` and the other execution-control signals. The
stranded-task check then ran again for the task graph. A hosted instance
collected about 45,000 rows on 13 finished tasks in this way.

**Expected behavior**
A pass that changes nothing writes nothing and publishes nothing. A
finished task never gets controller namespace attention. A live task
gets one incident for each exact run authority. A resolved incident
stays resolved until the authority changes.

**Steps to reproduce**
1. Make a native run that failed, with an `observed` coordinator whose
foreign lease has expired and whose controller host is not recorded.
2. Mark the task done or cancelled, or let the fresh-continuation
fallback resolve the attention of a live task.
3. Run the execution-finalization worker with the issue recovery workers
for 30 minutes.
4. On master, each task gets about 40 to 60 new attention rows. The
writer publishes thousands of signals.

## What Changed

- `surfaceNativeControllerNamespaceAttention` (shared helper): before it
creates attention, it checks for a resolved incident for this run that
was resolved at or after the owner's current lease expiry. If one
exists, it writes nothing.
- New `settledNativeControllerNamespaceAttention` SQL predicate. It is
shared by the helper, the release candidate page, and the release
deadline query, so all three use the same rule.
- `settleFailedRunOwner` (expired-owner path) does not surface attention
when the task is `done` or `cancelled`. The restart-recovery claim path
has the same guard.
- `releaseAbandonedNativeOwners` and `nextAbandonedNativeOwnerReleaseAt`
skip foreign owners whose task is finished or whose incident is settled.
A pass that has nothing to do selects nothing, and no immediate deadline
is set again. Local owners keep the normal proof-based release.
- The owner release proofs did not change. A foreign owner is still
never released without proof.
- Tests:
`server/src/__tests__/native-failed-run-owner-release-idle.test.ts`
(real PostgreSQL, real coordinator and workers, fake clock, statement
and signal counters) and three new cases in
`native-failed-run-owner-release.test.ts`.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/native-failed-run-owner-release-idle.test.ts
src/services/native-runtime/native-failed-run-owner-release.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/services/native-runtime/native-restart-recovery-isolation.test.ts`:
80 tests pass.
- Red on master, green with this change. The test seeds one done task
and one cancelled task (each with an earlier resolved `source_terminal`
incident) and one live task, then runs 30 idle minutes:
- master: 126 attention inserts, 249 attention writes, 7,024 signals,
8,025 statements.
- this branch: 1 attention insert (the live task, surfaced once), 2
attention writes (that insert and the fallback's resolution), 36
signals, 350 statements. All of this is startup work. Nothing repeats.
- After the fresh continuation settles a live task's attention, six
unrelated finalization wakes over 30 minutes:
- master: 3 attention inserts, 90 signals, 551 statements, and the
release deadline stays due.
- this branch: 0 attention writes, 0 run writes, 6 signals (the test's
own wakes), 153 statements, and the release deadline is null.
- New unit cases: a done or cancelled task never surfaces, and surfaces
once when it is reopened. A settled incident stays settled over repeated
passes. A renewed authority (a new lease expiry) surfaces one new
incident.
- The three new unit cases fail on master and pass on this branch.
- `tsc --noEmit` for `server` passes.

## Risks

- Low risk. The change only removes writes. It does not weaken a release
proof, an authorization, a budget, a pause, or an approval check.
- An incident that a person resolved or dismissed after the lease
expired is not made again for the same authority. A renewed authority (a
later lease expiry) still makes a new incident.
- A superseded incident (status `cancelled`) still shows again when the
superseding action clears. This keeps the current behavior.
- The historical rows that the loop made are not removed. This change
only stops new rows.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21621, #21616, #21617

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:45:02 -06:00
DottaandClaude Opus 5.5 172c267338 fix(runner): count returned validation rejections and give a copyable call in repeated-failure corrections (#21666)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents change Paperclip records through the `call_api` tool, and the
native runtime runs each tool call through a repeated-identical-failure
guard (#21360, #21398)
> - Some model hosts drop arguments. A model then sends the same failing
call again and again, and each attempt costs a model turn
> - In chat-eval cohort #14 (OpenCode, `deepseek-v4.1-flash`), two cases
sent a body-less `PATCH /api/issues/{id}` 7 and 10 times with one input
hash
> - The guard counted these refusals and stopped running the call after
the second failure. But its correction showed only a `"body": {...}`
placeholder, so the model had no concrete call to copy. Also, the guard
did not count a route's own 400/422 when it came back as an `ok: false`
result
> - This pull request puts a copyable call in each correction and counts
returned 400/422 rejections. It never counts transient failures
> - The benefit is that a model gets the exact call to send on the
second failure, and the third identical validation failure does not run,
whether the rejection is thrown or returned

## Linked Issues or Issue Description

Refs #21360, #21398

**What happened?**

In chat-eval cohort #14, `questions-and-approvals.review-plan` (run
470002dc) and `tasks.private-task` (run bb76a832) sent `call_api PATCH
/api/issues/{id}` with no `body` 7 and 10 times, all with the same input
hash. The pre-dispatch check (`assertBodyPresent`) refused each call and
returned HTTP 400. The guard corrected the second failure and refused
the rest. But the correction said only `Resend with a top-level body
argument: {"operationId": ..., "pathParams": {...}, "body": {...}}`. The
model kept sending the identical call, and each refusal cost another
model turn: 2 min 18 s in review-plan and 24 s in private-task. Also,
when the route itself returns a 400/422 validation response, `call_api`
gives back `{ ok: false, status: 422, ... }` as a result and does not
throw. The guard did not count that, so an identical invalid request
reached the route every time.

**Expected behavior**

On the second identical deterministic failure, the agent gets a
correction with a call it can copy that passes validation. On the third,
Paperclip refuses to run the call and tells the agent to change the
arguments or to stop and report. Transient failures (5xx, timeouts, rate
limits, conflicts) always run again.

**Steps to reproduce**

1. Run a native-runtime agent on a host that drops the `body` argument.
2. Have it update a task with `call_api PATCH /api/issues/{id}`.
3. Look at the tool results. They contain only a `{...}` placeholder. A
route's 422 returned as `ok: false` is never counted.

**Paperclip version or commit**

master d4498ae7ac (cohort #14), reproduced against ecdeed5e7e.

## What Changed

- `runner-api-client.ts`: Add `runnerApiCallExample`. It builds a
copyable call from the operation's catalog schema: the path parameters
the model sent, plus a body with the required fields. If no field is
required, the body has one free-text field, for example
`{"operationId":"PATCH /api/issues/{id}","pathParams":{"id":"<sent
id>"},"body":{"title":"<title>"}}`. The "requires body" correction and
the "body must be a JSON object, not a JSON-encoded string" correction
now include this example in place of the `{...}` placeholder.
- `repeated-tool-failure-guard.ts`: Count a result with `ok: false` and
HTTP 400/422 as a deterministic failure, in the same way as a thrown
400/422. The first returned rejection passes through unchanged. The
second gets a `guidance` note that says the call will not run again. The
third and later identical calls are refused without running, and the
refusal quotes the response.
- Never count transient results: 5xx, 408, 429, 409, 404, and a missing
status (timeouts or unknown outcome).
- A route can reject a request because of the state it found. So the
guard forgets a returned rejection after a different call succeeds. A
refusal before sending depends only on the arguments, and the guard
keeps it for the run, as before.
- Tests: the guard tests cover returned 422 (correct, then refuse), a
changed call that still runs, returned 500/503/429/408/409/404/no-status
and thrown 429/409/timeout that keep running, and a returned rejection
forgotten after a state change. The runner API tests check that the
example in each recorded body-less correction is sent when copied as is,
that the issue-update example passes `updateIssueSchema`, and that every
catalog write with a required body (more than 200) has an example its
request schema accepts.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/services/native-runtime/runner-api.test.ts`: 2190 passed.
- Red/green: with the master guard, the 2 returned-rejection tests fail.
With the master client, 6 example tests fail. With this change, all
pass.
- `node scripts/typecheck-layers.mjs` (server): no errors.
- Per-call cost (median of 5 × 20,000 calls, local M-series): guard on a
successful call 2.13 µs before, 2.19 µs after. Transient-failure path
2.12 µs. Building an example takes 0.67 µs and happens only on a refused
call. Nothing is added before the turn starts. The guard is created with
the run's tool authority, as before.
- Paid narrow run: see the comment below.

## Risks

- Low. The change affects only calls that fail validation twice with
identical arguments.
- A route that returns 400/422 because of state can succeed later with
the same arguments. To cover that, the guard forgets a returned
rejection after any different call succeeds. A model that retries with
no call in between is refused after two attempts and must change the
arguments or report.
- The example uses `<field>` placeholders for free text and for ids that
the model must look up. If a model copies a placeholder verbatim, the
route stores that text. The correction says to send the fields the model
means to change.
- This does not make a model stop resending a refused call. It only
gives better guidance and keeps the refused call from running.

## Model Used

- Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use,
code execution, and extended reasoning).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 15:55:09 -06:00
Nicky LeachandClaude Opus 5.5 c603f591dc fix(runner): bound ACPX instructions only where the sandbox delivers them (#21646)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner prepares a private sandbox for each ACPX agent before it
opens the provider session
> - `validateProviderPolicy` in that sandbox stage refuses system
instructions over 32 KiB for every ACPX agent
> - Only Pi gets the text from the sandbox stage. Claude, Codex, Grok
and Antigravity do not use it there. The host accepts 256 KiB
> - An agent with a large instruction bundle fails every run at
`session.open`. The error says only `classification=admission_sandbox`,
so the operator cannot see the cause
> - This pull request applies the 32 KiB sandbox bound only to Pi, and
gives every instructions rejection its own closed admission code
> - The benefit is that Claude and Codex agents with large instructions
can start, and an over-bound rejection names its real cause

## Linked Issues or Issue Description

Fixes: #20814

Refs #14430 (this pull request added the 32 KiB check, with no stated
reason for the other providers)

Related: #21644 (also fixes #20814, but only for Claude, and does not
change the classification)

**What happened?**
Every run of a Claude ACPX agent failed with
`provider_session_open_rejected`. The message was `ACPX sidecar command
session.open was rejected (retryable=false,
classification=admission_sandbox)`. A reproduction in a sandbox from the
same image showed that all the filesystem steps of
`prepareAcpxRuntimeSandbox` succeed. The real cause: the agent's
instructions entry file was 54,734 bytes. That is more than the 32 KiB
check in `validateProviderPolicy`. The host bound (`boundedInstructions`
in `runtime-host.ts`) is 256 KiB. Agents with smaller instructions (12.7
KB) work.

**Expected behavior**
The sandbox stage must refuse only the text that it delivers. #14430
added the check when only Pi consumed `systemInstructions` in the
sandbox (as the `PAPERCLIP_PI_SYSTEM_INSTRUCTIONS` env var). The check
runs before any per-provider branch, so it also refuses Claude, Codex,
Grok and Antigravity. Those providers get the text in the session
request. When a bound refuses the text, the error must name the
instructions, not the sandbox.

**Steps to reproduce**
1. Create a Claude ACPX agent whose composed instructions are more than
32 KiB (for example, an `AGENTS.md` of 54 KB).
2. Start a run of the agent on the Paperclip Runner.
3. See that `session.open` fails with
`classification=admission_sandbox`.

**Paperclip version/commit**
`master` at 163d9f323c.

**Deployment mode**
Paperclip Runner with ACPX providers (any deployment).

Where each provider consumes `systemInstructions` today:

| Provider | Delivery | Bound |
| --- | --- | --- |
| Pi | `PAPERCLIP_PI_SYSTEM_INSTRUCTIONS` env var (sandbox) | 32 KiB in
`validateProviderPolicy` (kept) |
| Cursor | `PAPERCLIP_CURSOR_INSTRUCTIONS` env var (sandbox) | 32 KiB in
`cursorInstructionBinding` |
| Kimi | `PAPERCLIP_KIMI_INSTRUCTIONS` env var (sandbox) | 32 KiB in
`kimiInstructionBinding` |
| Copilot | `copilot-instructions.md` (host, after the sandbox stage) |
32 KiB in `composeCopilotSystemInstructions` and
`refreshCopilotSystemInstructions` |
| Claude, Codex, Grok, Antigravity | session request
(`systemPrompt.append` or the provider's own session admission) | host
bound, 256 KiB |

## What Changed

- `runtime-sandbox.ts`: `validateProviderPolicy` gets the agent. It
applies the 32 KiB bound only to Pi. The type and NUL checks stay for
all agents. Cursor, Kimi and Copilot keep their own 32 KiB checks.
- `admission-diagnostics.ts`: new closed admission code
`ACPX_ADMISSION_INSTRUCTIONS`, and `acpxInstructionsAdmissionError`,
which tags the error with that code. The tag wins over the stage code
that the host records later.
- The instructions bound and NUL rejections in `runtime-sandbox.ts`,
`runtime-host.ts` (`boundedInstructions`), `cursor-instructions.ts`,
`kimi-instructions.ts` and `copilot-profile.ts` use the new tagged
error. The messages contain no instruction text and no provider output.
- `acpx_sidecar_transport.rs`: `ACPX_ADMISSION_INSTRUCTIONS` maps to
`classification=admission_instructions`. The classification set stays
closed.
- `antigravity-connection-probe.ts`: the closed classification pattern
accepts `admission_instructions`.
- `copilot-installation.test.ts`: the v70 identity test compared two
historical source hashes with the live files. Since #18325, Runner
source changes do not advance profile identities. The test now pins the
hashes that the v70 record holds.

## Verification

- Red-green: the new tests failed on `master` (11 failed) and pass with
the fix.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/acpx/runtime-sandbox.test.ts
src/drivers/acpx/runtime-host.test.ts
src/drivers/acpx/kimi-instructions.test.ts
src/drivers/acpx/cursor-instructions.test.ts
src/drivers/acpx/copilot-profile.test.ts
src/drivers/acpx/copilot-runtime-host.test.ts
src/drivers/acpx/pi-acp-runtime.test.ts src/cli`: 29 files, 489 tests
pass.
- New tests:
- The sandbox admits 54,734-byte instructions for Claude, Codex, Grok
and Antigravity, and does not put them in the launch environment.
- The host opens a Claude runtime with 54,734-byte instructions and
passes the full text to the provider.
- Pi and Cursor sandboxes refuse more than 32 KiB with
`ACPX_ADMISSION_INSTRUCTIONS`. NUL also gives that code.
- Through the host and the sidecar error mapping
(`acpxSidecarErrorCode`), oversized Pi (32 KiB + 1), Copilot (32 KiB +
1) and Claude (256 KiB + 1) instructions give
`ACPX_ADMISSION_INSTRUCTIONS` before the provider launches.
- The Rust classification test maps `ACPX_ADMISSION_INSTRUCTIONS` to
`admission_instructions`.
- The Antigravity probe logs `admission_instructions` as a closed
classification.
- `cargo test -p paperclip-runner-core --lib
classifies_only_allowlisted_internal_sidecar_failures` passes. `cargo
fmt --check` passes.
- `tsc -p packages/paperclip-runner/tsconfig.json --noEmit` passes.
- `cd server && npx vitest run
src/__tests__/antigravity-connection-probe.test.ts`: 11 tests pass.
- I did not do a live provider run. I did not run the full local suite.
CI supplies that coverage.

## Risks

- Low risk. The bound gets less strict only for providers that do not
get the text from the sandbox stage. Those providers stay under the host
bound (256 KiB). Pi, Cursor, Kimi and Copilot keep 32 KiB.
- An over-bound rejection now reports `admission_instructions` and not
`admission_sandbox` (or `admission_configuration` for Copilot). A
dashboard or alert that matches the old classification for this case
must also match the new one. Server code that checks a specific
classification does not check these values, except the Antigravity
probe, which this pull request updates.
- The NUL rejection message in `validateProviderPolicy` changes from
"exceed their bounded size" to "are invalid".

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:40:20 -08:00
Nicky LeachandClaude Fable 5.2 0ab73d4e30 feat(native-runtime): ask the Runner for provider-started turns on native ACPX Claude runs (#21456)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner can keep Claude while a background shell is running and
give the cycle of Claude to the controller as a turn that the provider
started. Core can wait for that turn. The transport can ask for it
(#21607), and a controller restart during it is safe (#21612).
> - No run does this yet, because Core does not ask the Runner for it.
> - This pull request makes the native runtime of Core ask, for ACPX
Claude. It is the only part that changes what a production run does.
> - The benefit is the goal of the whole work: a run with a background
command that is still running continues when the command finishes, and
does not fail with `native_finalization_missing`.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request turned the behavior on and had grown too large to
review. It is now three stacked pull requests:

1. #21607 (merged): the transport can ask the Runner for
provider-started turns. No caller asks.
2. #21612: the recovery after a controller restart during a wait or
during a provider-started turn.
3. **This pull request:** Core sets the option for native ACPX Claude
runs.

This is part 3. Its base branch is `fix/provider-turn-restart-recovery`.
Refs #19912 (design).

**What existing behavior does this improve?**

What a native ACPX Claude run does after a turn that ends with no result
while a background command is still running.

**Subsystem affected**

Core, the native runtime:
`server/src/services/native-runtime/native-session-executor.ts` and
`native-provider-turn-wait.ts`. ACPX Claude only.

**Current behavior**

- The run ends at the end of the turn. The Runner restarts Claude, the
background command stops, and the run fails with
`native_finalization_missing`.

**Proposed behavior**

- The native runtime sets `providerStartedTurns: true` in the transport
options for ACPX Claude, at the two places that build those options. One
function, `nativeProviderStartedTurns`, decides this and also decides if
the consumer gets the wait options, so the two cannot be different.
- A turn that ends with no result while a background command is still
running now waits, for a maximum of 10 minutes, for the turn that Claude
starts by itself. That turn is consumed as the next turn of the same
run, and its result settles the run.
- Other providers and other ACPX agents do the same as today.
- Limit of the Runner (see #21421): after a follow-up turn, the Runner
waits again only while a background command is still running. A command
that finishes while Claude reports on an earlier command gets no
follow-up.

**Reason and benefit**

- The opt-in is in the last part, so each earlier part can merge with no
change for a production run.

## What Changed

- `server/src/services/native-runtime/native-provider-turn-wait.ts` (5
lines added, 1 removed): `nativeProviderStartedTurns`.
- `server/src/services/native-runtime/native-session-executor.ts` (4
added, 2 removed): the option at the two places that build the transport
options, and the same function for the wait options.
- `server/src/services/native-runtime/native-session-executor.test.ts`
(4 added, 2 removed): the two native paths set the option for ACPX
Claude and not for other providers.

## State saved by the previous release

- No persisted state changes in this part.
- **A Runner that does not know the field** ignores it, and the run does
the same as today.
- **A deploy during a run:** a turn that started before the deploy has
no opt-in and ends as today. The next turn has the opt-in.
- **Rollback of Core:** the previous Core does not set the option, so
the Runner does not wait.

## Verification

The items below were run on the tree that was verified before the split
into three parts (commit `1b15ded1c4` has that tree, byte for byte). The
head of this branch (`23e2c13ad3`) is that commit on the new head of
#21612; its three server files are the same.

- `pnpm typecheck` and `node
scripts/verify-runner-vendor-dependencies.mjs` in `server`; `npx tsc
--noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`,
`pnpm run typecheck:typescript` and `pnpm run build:typescript` in
`packages/paperclip-runner`: all passed.
- `server/src/services/native-runtime/native-session-executor.test.ts`
and `native-runner-restart-recovery.integration.test.ts`: 1,732 passed,
21 skipped.
- The end-to-end behavior is tested in parts 1 and 2 with the real
`paperclip-runnerd`, where the test sets the option in the way that this
part does.
- Live run with a real Claude account, not in CI, at the adapter level
of the Runner (see #21316): the cycle of Claude was given as a turn with
its own usage, and each Paperclip tool call was admitted.

Not run: a live run through Core, `paperclip-runnerd` and the real
sidecar with a real Claude account (that path needs an API key, not a
local sign-in); a remote sandbox; the full suite. CI supplies the broad
coverage.

## Risks

- **This part turns the behavior on for each native ACPX Claude run.** A
run with no background command is not changed: the Runner waits only
when a shell is still running at the end of a turn that has no result.
- **This part must not merge before #21612.** Without #21612, a
controller restart during a wait or during a provider-started turn fails
the run.
- **A run can now stay open up to 10 minutes longer** when the
background command does not finish or Claude does not start a turn. The
agent computer stays active for that time.
- **A waiting run can show as "suspicious" after 5 minutes of silence**
in the run watchdog. The watchdog only labels a native run; it does not
stop it.
- **Follow-ups that the Runner cannot wait for are lost,** as the limit
above says.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 13:10:20 -08:00
DottaandClaude Opus 5.5 6ba46fd080 fix(github): read only the webhook secrets each delivery step uses (#21651)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents receive GitHub work through a GitHub App chat channel. GitHub
sends each event as a signed webhook delivery.
> - Each delivery passes three steps: durable staging, the durable
replay worker, and the replayed request's preflight under the credential
lease.
> - Each step resolved every stored GitHub App secret. A GitHub App
connection stores seven secrets, so each delivery decrypted and audited
about 21 secrets. The steps use only the signing secret, the bound
installation id and the App id.
> - On our staging instance, these reads are now the largest source of
secret reads and `secret_access_events` rows.
> - This pull request makes each webhook step read only its own keys,
and only when it needs them.
> - The benefit is fewer vault reads and audit inserts per delivery, and
the App private key and OAuth client secret stay sealed on the webhook
path.

## Linked Issues or Issue Description

Refs #21641 (fixed the other source: review-check polls now reuse cached
installation tokens).

**What happened?**
On our staging instance (serving 163d9f323c), from 2026-11-14 20:23:52Z
to 20:32Z, 32 `github_webhook_ingress` chat actions arrived. In the same
window, `secret_access_events` recorded 97 resolutions of
`credentials.webhookSecret`. It recorded the same count for
`privateKey`, `appId`, `installationId`, `clientId`, `clientSecret` and
`slug`. That is about 3 full credential resolutions per delivery, and
about 21 audit rows per delivery (roughly 70–80 rows per minute). In the
same window, 84 review-check polls caused no secret reads, because
#21641 fixed that source.

**What did you expect?**
A delivery reads only the secrets it uses. An unsigned or forged
delivery reads only the signing secret.

**Steps to reproduce**
1. Configure a GitHub App chat endpoint.
2. Send one signed `issue_comment` delivery for the bound installation.
3. Count the new `secret_access_events` rows for the connection. Before
this change, each stored key has 3 rows. A bad signature still reads
every key once.

The three call sites per delivery:
1. `server/src/services/chat-channels/github/webhook-ingress.ts`
`stageGitHubWebhookIngress` (`resolveCredentialRefs` with all refs).
2. `server/src/services/chat-channels/github/webhook-ingress.ts`
`processGitHubWebhookIngress` (`resolveCredentialRefs` with all refs, to
re-sign the retained body). This was the unattributed third resolution.
3. `server/src/services/chat-channels/github/webhook-authentication.ts`
preflight and ping (`resolveCredentials` with no key filter).

## What Changed

- Added `githubCredentialRefsFor(refs, keys)` in the GitHub webhook
ingress module. It selects the secret refs for named keys.
- Staging reads `webhookSecret` first. After the signature holds, it
reads `installationId` (only when the event names an installation and is
not an `installation` event) and `appId` (it scopes the recovery-receipt
denial query).
- The replay worker reads `webhookSecret` to re-sign the body. It reads
`appId`, `privateKey` and `installationId` only for a recovered delivery
that needs a source recheck.
- The preflight and ping read `webhookSecret` through the existing
`resolveCredentials` key filter. The preflight reads `installationId`
only after the signature holds and only for an event that names an
installation.
- The failed-delivery recovery scan reads only `appId`, `privateKey` and
`installationId`.
- Unchanged: the timing-safe compare, the credential-mutation lease, the
re-read of current endpoint state under the lease, the status and fence
checks, and the superseded-credential retry. No cache of decrypted
values.
- Per valid delivery, secret reads go from 7 keys × 3 (21 on staging) to
6: `webhookSecret` × 3, `installationId` × 2, `appId` × 1. A bad
signature goes from all keys to 1 read.
- `webhook-ingress.ts` was already above 1,000 lines. It grows by about
30 lines because the key selection belongs with the steps that use it.
`chat-channels.integration.test.ts` holds the GitHub webhook fixtures,
so the new test goes beside the existing signature tests.

## Verification

- Red on master, green with this change: `pnpm exec vitest run
server/src/__tests__/chat-channels.integration.test.ts -t "resolves only
the secrets a"`.
- `current`: before, `{webhookSecret: 3, installationId: 3, appId: 3,
privateKey: 3}` (the fixture stores 4 keys). After, `{webhookSecret: 3,
installationId: 2, appId: 1}`.
- `invalid_signature`: before, every key once. After, `{webhookSecret:
1}`.
- Regression: GitHub webhook, signature, rotation, ping, recovery,
installation, paused and revoked tests in
`chat-channels.integration.test.ts` (filter `-t
"[Ww]ebhook|signature|rotat|[Pp]ing|recovery|installation|GitHub
callback|revoked|paused"`): 304 passed, 1 failed. The failure is a Slack
test (`materializes direct external-chat finals once…`, matched by the
`ping` filter in "bookkeeping"). It passes when run alone, so it is a
load flake and not related to this change.
- `pnpm exec vitest run
server/src/services/chat-channels/github/webhook-ingress.test.ts
server/src/services/chat-github-provider-stress.test.ts
server/src/services/chat-github-webhook-config.test.ts`: 105 passed.
- `tsc --noEmit -p server/tsconfig.json`: passed.
- `pnpm verify` selects 2,726 of 3,554 Vitest files for this change (the
chat integration file pulls in a wide dependency set). I did not run
that broad set locally. Required CI supplies that coverage for the final
revision.

## Risks

- Low risk. Each step reads a subset of the same refs from the same
fenced record, so the values it uses do not change.
- A missing `webhookSecret` ref still fails the signature check as
before. A missing `installationId` ref still makes an
installation-bearing event "ignored" as before.
- The preflight now resolves `webhookSecret` and `installationId` in two
reads under the same credential lease. The lease blocks credential
mutation between them.
- Installation lifecycle events (`installation-lifecycle.ts`) still
resolve the full set, because they re-verify and re-persist all App
credentials. They are rare.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run in Claude Code
with tool use (shell, file edits, test execution) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 15:06:29 -06:00
DottaandClaude Opus 5.5 d4498ae7ac fix(runner): settle every recorded tool result and free sessions blocked by a closed turn's pending call (#21490)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call Paperclip tools through the Runner. The Runner records
each call and returns the controller's result to the provider (here
OpenCode)
> - In chat-eval cohort 13, a `list_workflows` result was recorded but
never reached OpenCode. OpenCode waited about 290 s, and the call stayed
pending
> - The next run's `run.attach` then failed with
`durable_pending_tool_calls`, so the person's follow-up question never
got an answer
> - This pull request fixes the result shape that caused the rejection,
makes the Runner settle every recorded result, and lets the next run
settle a call that a closed turn left pending
> - The benefit is that a tool result always reaches the agent, and a
stuck call can no longer block every later turn

## Linked Issues or Issue Description

**What happened?**
Chat-eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`). In
`cases.case-history` and `cases.create-case`, the CEO called
`list_documents` (or `list_projects`) and `list_workflows` (call id 5)
together. The server answered `GET /companies/{id}/workflows` with 200
in 584 ms. The OpenCode MCP bridge diagnostic for call 5 says `outcome:
"unanswered"`, `written: false`, `handlerMs: 289038` (290723 in
create-case). The sibling call was answered in about 1 s. The run ended
with `runner_settlement_incomplete`: `semanticTools.pending` held call 5
and its `semantic_tool.result` delivery command had status `failed`. The
next question started a run whose `run.attach` was refused: `run.attach
requires a settled Codex provider session: durable_pending_tool_calls`.
The run failed in 17 s, and the question was not answered.

Cause: `executeNativeWorkflowTool` returned the Runner API result
unchanged: `{ ok: true, status: 200, operationId: "GET
/api/companies/{companyId}/workflows", ... }`. Every other API-backed
tool renames that field to `apiOperationId`. The Rust tool bridge
(`semantic_response_value`) reads an object with a boolean `ok` and an
`operationId` as a semantic envelope that must name this tool and call.
The REST route did not match, so `apply_result` failed with `semantic
result envelope does not match its provider call`. Nothing was sent to
the provider, and the call stayed pending forever. The red test
reproduces this exact error. The parallel call is not the trigger: the
sibling call settled normally, and the OpenCode MCP bridge answered
other parallel pairs in the same runs. `list_workflows` first appears in
cohort 13, so cohorts 11 and 12 did not hit it. All four workflow tools
had this shape.

**Expected behavior**
Every recorded tool result reaches the provider. When a recorded result
cannot be delivered, the provider gets an error at once and the call
settles. A call that a closed turn left pending does not block the next
run.

**Steps to reproduce**
1. Run an OpenCode agent with Cases enabled on the durable Runner.
2. Ask a question that makes the agent call `list_workflows`.
3. See the call wait about 290 s. Then ask a follow-up question and see
`run.attach` fail with `durable_pending_tool_calls`.

**Paperclip version or commit**
`master` at `df1b12ad8e`.

## What Changed

- `server/src/services/native-runtime/native-workflow-tools.ts`:
workflow tool results name the route `apiOperationId`, like case tools
and `call_api`.
- `provider_bridge.rs`: add `settle_undeliverable_result`. When the
bridge rejects a recorded result for a call it still holds, the call
settles with a bounded, non-retryable runner error. Conflicting or
unknown results still fail as before.
- `provider_backend.rs` (`deliver_semantic_result`): use that
settlement, deliver the error to the provider, and record a
`semantic_tool_result_undeliverable` diagnostic with the call,
operation, reason and the rejected result's digest.
- `provider_bridge.rs` and `provider_backend.rs` (`attach_run`): add
`expire_closed_turn_calls`. When nothing is live (no active or ambiguous
turn, no provider request waiting), `run.attach` settles calls that a
closed turn left pending as an expired failure with an unknown outcome,
and records `semantic_tool_result_expired`. The provider stopped waiting
when its turn ended, and the new run's controller never delivers the old
run's results. A call that a live turn still waits on still blocks
attachment. This follows the existing `tool_call_interrupted_by_restart`
rule: never execute the operation again, and record an unknown outcome
when no result can ever arrive.
- Docs: `doc/run-log-events.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe both
diagnostics.

## Verification

- Red/green, Rust (`cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --test codex_provider --
a_rejected_recorded_result run_attach_expires`):
- Before:
`a_rejected_recorded_result_still_answers_the_provider_and_frees_the_session`
fails with `semantic result envelope does not match its provider call`
(the production error).
`run_attach_expires_a_call_left_pending_by_a_closed_turn` fails with
`run.attach requires a settled Codex provider session:
durable_pending_tool_calls` (the production refusal).
- After: both pass. The fake provider logs `tool-response:failure`, so
it got an answer at once. The next `run.attach` succeeds.
- Parallel calls (`cargo test -p paperclip-runner-core --lib --
provider_bridge`):
`a_rejected_result_in_a_parallel_pair_still_settles_its_own_call` sends
the cohort 13 pair (`list_documents` + `list_workflows`). The rejected
call settles, its sibling keeps its own result, and the next turn
starts. `only_a_closed_turn_expires_its_pending_calls` checks that an
open turn expires nothing.
- Upgrade and mid-run cutover:
`run_attach_expires_a_call_left_pending_by_a_closed_turn` saves state
with a pending call after the turn ends, stops that Runner, and attaches
with a new Runner process from the saved state.
`durable_backend_settles_pending_tools_when_recovery_finds_the_turn_ended`
(unchanged) still passes: a late result in the same run still settles as
`settled_after_turn`.
- Red/green, server (`cd server && npx vitest run
src/services/native-runtime/native-case-tools.integration.test.ts -t
workflows`): before, `expected { ok: true, status: 201, … } to not have
property "operationId"`; after, 2 passed.
- `cargo test -p paperclip-runner-core`: 643 lib tests pass. One lib
test (`fenced_receipt_output_keeps_prefix_when_stdout_never_closes`)
failed once with `EPERM` while signaling a process group under parallel
load. It passes alone 3 of 3 and does not touch this change. The
integration test binaries pass.
- Paid narrow run for `cases.case-history` and `cases.create-case`: see
the comment below.

## Risks

- Previous-release story: no persisted shape changes. State saved by the
previous Runner with a call pending after its turn closed carries
forward. The new Runner settles that call at the next `run.attach`, and
the session continues (tested from saved state with a new process). A
previous-release Runner that receives the corrected `apiOperationId`
result delivers it normally, because the result is no longer read as an
envelope.
- Behavior change: a call left by a closed turn is now settled at the
next run's attachment, not kept forever. Its outcome is recorded as
unknown, and the operation is never run again. A provider turn that is
still live keeps the old refusal (`durable_pending_tool_calls`,
`provider_side_effect_pending`), and the controller still never replaces
the session.
- A recorded result that the bridge rejects now becomes an error for the
agent, not an endless wait. The agent is told the action may have
completed and must check the current state before repeating it.
- Latency: nothing is added before turn start. The attach-time check
walks the in-memory pending calls (usually none) and adds no I/O beyond
the existing state save. The rejected-delivery path now answers in the
same command instead of after the provider's ~290 s client timeout, and
the follow-up run no longer fails after 17 s.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21398 (first release where `list_workflows` was called in this
eval), #19733 (the attach refusal), #20886 (earlier OpenCode delivery
fix).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:21 -06:00
DottaandClaude Opus 5.5 9b581d689b fix(chat): report an outcome once and create no run when an earlier reply gave it (#21489)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In agent chat, a finished delegated task or a declined tool action
wakes the chat agent so it tells the person
> - #21401 stopped a second completion report when an earlier reply
already gave it
> - But the suppressed completion still created a zero-second heartbeat
run, and declined tool actions still got a second report
> - Run accounting must count only real turns, and the person must get
each outcome once
> - This pull request settles an already-reported outcome before it
becomes a run, and applies the same rule to declined tool actions
> - The benefit is one report per outcome, no phantom run, and no extra
model call

## Linked Issues or Issue Description

Refs #21401

**What happened?**
Chat eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`)
showed three problems after #21401.
1. `questions-and-approvals.duplicate-answer`: the completion wake was
suppressed, but it still created a 0-second run (`runtimeMode: legacy`,
no usage). The bounded-native-runs check rejected that run.
2. `skills.studio-test`: the CEO's single reply reported the Studio
result after it read the test task done (0 passed, 1 failed, links
COM-2). The completion was then suppressed, but the eval waited for a
separate completion run and timed out after 240 s.
3. `questions-and-approvals.decline-tool`: the person declined the tool
review one second after the turn for their message started. That turn
read the declined action and said so. The decline continuation, which
waited behind that turn, then posted the same declined outcome again.

**Expected behavior**
Each finished task or declined tool action is reported once in the
conversation. An outcome that an earlier reply already reported creates
no heartbeat run.

**Steps to reproduce**
1. In chat, delegate a short task, or start a governed tool read that
needs approval.
2. While it waits, send a message that wakes the chat agent.
3. Let the task finish, or decline the tool review, while that turn
runs. The turn reads the outcome and reports it.
4. See the completion wake become a 0-second run, or see a second
decline report from the decline continuation.

**Paperclip version or commit**
master `df1b12ad8e`

## What Changed

- `server/src/services/chat-completion-delivery.ts`:
`settleReportedChatCompletions` marks each undispatched completion that
an earlier reply reported as delivered, with that reply as its response.
It returns true only when nothing else is owed. The outbox `deliver`
also checks this before it requests a wake, so no wake and no run exist
for an already-reported completion.
- `server/src/modules/wake-queue`: a new transaction port,
`skipAlreadyReportedChatWake`, runs in the release drain before a
deferred wake is promoted. When the earlier reply already reported every
outcome of the wake, the wake ends `skipped` ("Already reported by an
earlier reply in this conversation") and no run is created. Wakes that
carry queued comments or a failed-chat retry are never skipped.
- `server/src/services/chat-interaction-outcome-report.ts` (new): the
same structural rule for declined tool actions. A conversation agent run
that lists its interactions records `chat_interaction.declined_read` for
each declined tool action. The decline continuation is settled when the
same agent replied in the conversation after the decline, from a run
with that receipt, posted after the read. The rule covers only declined
tool-action confirmations. Answers, plan reviews and accepted actions
keep their own turn.
- `server/src/routes/issues.ts`: `GET /issues/:id/interactions` calls
the recorder for agent runs.
- The run-start check from #21401 stays as a fallback for a completion
turn that was admitted before the earlier reply landed.
- `tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.ts`:
the Studio oracle (v5) accepts either the completion turn's reply or an
earlier CEO reply as the delivery. The earlier reply counts only from
saved state: no completion run exists, the replying run has a
`chat_handoff.completed_read` receipt for the test task, the reply was
posted after that read, it is the only such reply, and it links the
saved result. No wording check is added.
- `doc/run-log-events.md`: documents both read receipts and the skipped
wake outcome.

## Verification

- `cd server && npx vitest run
src/__tests__/chat-completion-delivery.test.ts
src/modules/wake-queue/adapters/chat-declined-outcome-reported.test.ts
src/modules/wake-queue/application/use-cases.test.ts`: all pass.
- Red: with the drain check and the outbox check disabled, 5 tests fail:
"starts no completion wake or run ...", "settles a deferred completion
wake without a run ... (true)", "settles the decline continuation with
no run ...", and both "settles an already-reported chat wake without a
run" cases.
- Negative controls stay green: a linking reply without a done read, a
decline the earlier turn never read, a non-tool-action card, and an
accepted card all still run their continuation.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.test.ts`:
34 pass. Red on the old oracle: "accepts the earlier reply the server
took as the completion report" fails. New negative cases reject no done
read, a read of another task, a reply before the read, a reply from
another run, two replies, an extra completion run, and a missing task
link.
- A narrow paid eval run of `duplicate-answer`, `studio-test` and
`decline-tool` on this branch follows in a comment.

## Risks

- A completion turn admitted before the earlier reply landed (a narrow
race) still ends as a short succeeded run with no model call, as in
#21401.
- If an earlier reply read the outcome but did not mention it, the
person does not get a second message. The read receipt, the
reply-after-read order and, for completions, the task link make this
unlikely.
- The decline rule covers only declined tool-action confirmations. Other
interaction outcomes are unchanged.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:16 -06:00
DottaandClaude Opus 5.5 8a97037bd2 fix(runs): say who started a retry run in the issue run read (#21486)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat agent answers questions about failed runs by reading the
task's run list (`GET /api/issues/{id}/runs`)
> - For a retry run, that read returned only raw invocation fields
(`on_demand`, `manual`) and an owner id. It did not say who started the
retry
> - In chat-eval cohort 13, the agent told the person "Paperclip retried
automatically" about a run that the board started with Retry
> - This pull request adds a plain-words `retriedBy` field to each retry
run, and tells the chat agent to use it
> - The benefit is that people get a true account of who retried their
work, with a name and no raw ids

## Linked Issues or Issue Description

**What happened?**
Chat-eval cohort 13 ran OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017` on master `df1b12ad8e`.
In `runs-adapters.inspect-failed-run`, the reply said "Paperclip retried
automatically on a supported model". The successful run was a person's
manual Retry: `invocationSource: on_demand`, `triggerDetail: manual`,
wake reason `retry_failed_run`, and the responsible user was the board.
The agent's own reasoning hedged ("probably automatic"), because the run
read did not say who started the retry.

**Expected behavior**
The run reads that the agent uses say who started each retry: a person,
by display name, or Paperclip's automatic recovery, in plain words and
with no raw ids.

**Steps to reproduce**
1. Let a run fail on a task.
2. Press Retry on the failed run as a board user.
3. Read `GET /api/issues/{id}/runs`. Before this change, nothing in the
retry row says that a person pressed Retry.

## What Changed

- `runsForIssue` (`server/src/services/activity.ts`) reads the wake
request of each retry run. It adds `retriedBy` to each row:
- `{ kind: "person", personName, summary }` when a user wake with reason
`retry_failed_run` or `task_retry` started the run. The name comes from
the user record, or "Board" for the local board, with the same fallback
as activity actor names.
  - `{ kind: "automatic", summary }` for an `automation` retry run.
- `null` for a run that is not a retry, or a retry whose starter is not
recorded. The read does not guess. It does not use `responsibleUserId`,
because automatic retries keep the original responsible user.
- The OpenAPI description of `GET /api/issues/{id}/runs` and the chat
agent directive tell the agent to use `retriedBy` and not to call a
retry automatic unless `retriedBy` says so.
- The UI `RunForIssue` type has the new optional field.

## Verification

- `cd server && npx vitest run src/__tests__/activity-service.test.ts
src/__tests__/agent-conversations.test.ts
src/__tests__/activity-routes.test.ts
src/__tests__/openapi-routes.test.ts` passes.
- Red/green: the new test "says who started each retry run" fails
without the service change (`expected undefined to be null`) and passes
with it. It covers a board Retry, a named member's Retry, an automatic
retry that keeps the board as responsible user, a retry with no recorded
wake, and no raw ids in the output.
- `npx tsc --noEmit -p server` passes.
- A narrow paid run of `runs-adapters.inspect-failed-run` (with the two
sibling cohort-13 PRs) ran on a combined branch. The results are in a PR
comment.

## Risks

- Low risk. The change adds one read-only field and one bounded query by
primary key on the wake requests of retry runs in the page.
- No persisted state changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:04 -06:00
DottaandClaude Opus 5.5 95a6ab31e6 feat(db): attribute work signals by caller behind a debug flag (#21640)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Background queues wake from work signals. A transaction calls
`signalDatabaseWork(topic, hint)`, and the delivery coordinator wakes
the queue for that topic with that hint.
> - On a hosted instance with zero runs, the stranded-assignment
recovery queue re-checks tasks about 7 times a second. About 2.6 hinting
commits per second drive it. HTTP traffic, chat review polls and local
fixtures with all heartbeat workers do not explain the writer.
> - Statement statistics show what runs, but not who signalled it.
Nothing records the caller of a work signal or why a queue woke.
> - This pull request adds an opt-in attribution log for work signals
and recovery-queue wakes, off by default.
> - The benefit is that one short run on an affected instance names the
writer of any idle signal loop, without code changes.

## Linked Issues or Issue Description

No issue exists. This describes the gap with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

Observability of `signalDatabaseWork`
(`packages/db/src/work-signals.ts`) and of the issue recovery queues
(`server/src/services/recovery/work.ts`).

**Current behavior**

A queue that wakes several times a second at idle shows up only as
repeated statements. Nothing tells you which caller published the hint,
whether the wake came from another process, a reconnect, a direct
settled publish or a stored deadline, or which load and check it caused.

**Proposed behavior**

An environment variable, `PAPERCLIP_DEBUG_WORK_SIGNALS`, read once when
the module loads. It holds a comma-separated topic allowlist, and each
topic can carry `:N` for its stack sampling rate. When it is unset or
empty, the signal path only checks one boolean and does nothing else.

**Reason and benefit**

Several idle-load loops this week took hours of statement analysis
before the writer was found. Attribution by caller turns that into one
log query.

## What Changed

- `packages/db/src/work-signal-debug.ts` (new):
- `parseWorkSignalDebugSpec` reads `topic[:N],…`. An optional numeric
suffix is parsed from the right, so topics that contain colons (for
example `live:<companyId>`) work with or without `:N`. N defaults to 20;
malformed entries are ignored.
- `createWorkSignalDebug` returns `enabled`, `allows(topic)`,
`signal(topic, hint, entry)`, `event(topic, name, fields, stack, entry)`
and `flush()`. A disabled instance constructs no `Error`, captures no
stack, builds no line and starts no timer.
- Stack capture (`captureWorkSignalStack(entry)`) raises
`Error.stackTraceLimit` to 40 and restores it in a `finally`. It calls
`Error.captureStackTrace(holder, entry)`, so V8 cuts every frame above
the public entry (`signalDatabaseWork`, `publishDatabaseWorkSettled`, or
the recovery `wake()`).
- Frames are dropped by file (`node_modules`, `node:internal`, the
signal wrapper files) and by function name (`signalDatabaseWork`,
`notifyDeliveryWork`, …), so trimming works on the bundled `index.mjs`.
Positions are kept raw (`index.mjs:L:C`) for offline source-map lookup.
At most 10 frames are kept.
- Each (topic, event) pair has its own budget of 10 lines per second,
checked BEFORE any stack is captured. A `suppressed` line names the
topic, the dropped event and the count. Frequent `check` lines therefore
never drop `signal` or `direct_wake` lines.
- Stack sampling (1 in N) counts per (topic, event), so `check` and
`load` do not shift the phase of `signal` sampling. `direct_wake` always
captures its caller's stack.
- Lines go to an in-memory buffer that an unref'd 1-second timer flushes
to stderr. The signal path does no I/O and no await.
- `workSignalDebug` is the process instance, built from
`process.env.PAPERCLIP_DEBUG_WORK_SIGNALS` once at module load.
- `packages/db/src/work-signals.ts`: when enabled, it records:
  - `signal` in `signalDatabaseWork`;
  - `remote_notification` and `notification_reconnect`;
- `settled_publish` (with a sampled stack) for
`publishDatabaseWorkSettled`;
  - `uncertain_settled` when an uncertain transaction resolves.
- `server/src/services/recovery/work.ts`: under the same flag:
  - each task-queue `load` logs its scope key;
- each `check` logs the issue and company id, `dueAt`, and `wake:
"deadline"` or `"load"`. The marker rides on the queued entry itself,
and only while the flag is on, so nothing is tracked otherwise and
nothing outlives the entry;
- `wake()` logs ONE `direct_wake` line listing the queues it actually
invalidates (assigned-work only when registered), under the first
allowlisted one, with its caller's stack.
- **No request route is logged.** The server has no request async
context to read one synchronously, and this change does not add one.
- `docs/deploy/environment-variables.md`: documents the flag, internal
ids only, raw positions, and that enabling or changing it requires a
restart.
- Tests:
  - `packages/db/src/work-signal-debug.test.ts`, 12 tests:
- strict no-op when unset or empty (no `Error`, no `captureStackTrace`,
no stack capture, no sink);
    - the process instance is disabled without the env;
    - allowlist filtering, and spec parsing including colon topics;
    - per-(topic, event) sampling, and always-on direct-wake stacks;
- source-tree trimming, and a synthetic single-file bundle stack with
raw positions kept and wrappers dropped by name;
    - real capture below the entry with the limit raised and restored;
- checks at about 8/s alongside signals at about 3/s for 5 seconds, with
every signal line kept;
    - suppressed records never capture a stack;
    - buffered, timer-driven output.
- `packages/db/src/work-signals.test.ts`: on real PostgreSQL, the signal
and settled-publish paths never call the debug log while the flag is
off.
- `server/src/__tests__/issue-recovery-work-debug-attribution.test.ts`,
on the recovery fixture with real PostgreSQL:
    - no recovery log calls while the flag is off;
- with the flag on, a run graph hint logs its signal, the graph-scoped
load and the check;
- `wake()` logs exactly one `direct_wake` that lists the queues and
carries the caller's stack.

## Verification

- `packages/db`: `vitest run src/work-signal-debug.test.ts
src/work-signals.test.ts`: 39 passed. `tsc --noEmit` passes.
- `server`: `vitest run
src/__tests__/issue-recovery-work-debug-attribution.test.ts
src/__tests__/issue-recovery-deadlines.test.ts`: 17 passed. `tsc
--noEmit` passes.
- Staging values:
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:10`, then restart.
- Sample output from the synthetic single-file bundle test:

```
{"msg":"paperclip work signal debug","at":1000000,"event":"signal","topic":"stranded-assignments","hint":"{\"companyId\":\"c\",\"graphRunId\":\"r\"}","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","at async notifyRunDispatch (file:///app/server/dist-runtime/index.mjs:120501:3)","at async PostgresJsDatabase.connection.transaction [as transaction] (file:///app/server/dist-runtime/index.mjs:4021:22)","at async Object.settleRunProjection (file:///app/server/dist-runtime/index.mjs:300777:5)","at async file:///app/server/dist-runtime/index.mjs:310002:17"]}
{"msg":"paperclip work signal debug","at":1000000,"event":"check","topic":"stranded-assignments","issueId":"i","companyId":"c","wake":"load","dueAt":1}
{"msg":"paperclip work signal debug","at":1000000,"event":"direct_wake","topic":"stranded-assignments","queues":"stranded-assignments,orphan-blockers","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","…"]}
{"msg":"paperclip work signal debug","at":1001000,"event":"suppressed","topic":"stranded-assignments","suppressedEvent":"check","count":17}
```

## Risks

- Low risk. It is off by default. While off, each signal costs one
property read and one boolean check.
- While on, the log can show internal ids from hints (company, issue,
run). It holds no secrets and no user identifiers, and the topic
allowlist limits what is logged.
- Stack capture costs CPU. Sampling (default 1 in 20) and the 10 lines
per second cap bound it.
- The file owner of `packages/db/src/work-signals.ts` reviews this
before merge.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21621.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:54:00 -06:00
DottaandClaude Opus 5.5 163d9f323c perf(github): reuse repository installation tokens until shortly before expiry (#21641)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel lets an agent review pull requests and
answer comments through a GitHub App
> - Each GitHub call gets a repository-scoped installation token from
`githubBotRepositoryToken`
> - That function minted a new token on every call. To mint, it resolved
every App secret through the secret service, and each resolution wrote
one audit row
> - The review-check head poll calls it about every 5 minutes for each
waiting check. On staging this made the secret audit insert one of the
busiest statements in the database
> - This pull request keeps a minted token in memory until shortly
before it expires, and checks the connection's current state on every
call
> - The benefit is far fewer database writes and reads, fewer GitHub
token mints, and an audit log that records real secret use

## Linked Issues or Issue Description

**What happened?**
On paperclip.staging (commit a18479bbba, measured at 19:05Z on
2026-11-14), `secret_access_events` got about 285 rows in 60 s. These
were 44 resolutions each of the GitHub App connection's
`credentials.appId`, `privateKey`, `installationId`, `webhookSecret`,
`clientId` and `clientSecret` (+21 `slug`), with `consumer_type`
`tool_connection` and actor `system`. That is about 4.7 audit inserts/s.
The matching `secret_versions` and binding reads ran at about 3.6/s each
in Neon. These were among the busiest statements on the database.

**Expected behavior**
A GitHub installation token is valid for one hour. Paperclip should
reuse it and resolve the App secrets only when it mints a new token.

**Steps to reproduce**
1. Connect a GitHub App chat endpoint and get pull request reviews into
the `received` review-check state (staging had 84 `github_review_check`
actions).
2. Let the review-check worker poll the pull request head (23 polls/min
measured on staging).
3. Count `secret_access_events` rows per minute. Each poll adds one row
per App secret and mints a new token at GitHub.

**Paperclip version or commit**
master at a18479bbba (the commit serving paperclip.staging at the time
of measurement).

**Where in the code**
`githubBotRepositoryToken` in
`server/src/services/chat-github-client.ts`. Callers:
`chat-github-checks.ts` (review-check head poll),
`chat-github-response-comments.ts` and `chat-github-reviews.ts`.

## What Changed

- `githubBotRepositoryToken` keeps a minted token in a per-process,
in-memory cache (at most 512 entries, least recently used first out). It
never persists or logs tokens.
- The cache uses the token until 5 minutes before GitHub's `expires_at`
(capped at one hour). If GitHub does not give a usable expiry, the token
is not kept.
- The cache key holds the company, endpoint, repository id, the exact
permission set, and an authority revision. The revision holds the
connection id, the endpoint's runtime generation (pause, revoke and
availability changes advance it), and for each mint credential its
secret ref, version selector, secret status, current version,
`updatedAt`, and binding. The installation id is a secret, so its secret
ref and version identify it in the key.
- Each call reads the current authority from the database in two narrow
statements: one row from the endpoint and connection join (status,
enabled, refs, runtime generation), and the three mint secrets with
their bindings (status and version columns only, no secret values). The
endpoint and connection status checks run on every call, as before.
- If a status check fails, all cached tokens for that endpoint are
dropped. A token cannot survive a revoke, pause, disable or credential
change.
- Secrets are resolved only when a token is minted, and only the three
that minting uses (`appId`, `privateKey`, `installationId`). So the
audit log gets three rows per mint instead of one row per App secret per
call.
- New `githubBotRepositoryClient` sends repository requests. If GitHub
returns 401 for a cached token, it drops that token, mints a new one,
and sends the request once more. A 401 for a newly minted token is not
retried. The review-check, response-comment and review paths use this
client.
- The token policy stays in the GitHub chat client module
(`rules/module-boundaries.mdc`). The callers only change how they get a
request function.

## Verification

- New test file
`server/src/__tests__/github-repository-token-cache.test.ts` (embedded
Postgres, real secret service, mocked GitHub fetch):
- `mints once and resolves the App's secrets only when it mints`: red on
master (second call minted `installation-token-2`), green now. It also
checks that only the three mint secrets are audited.
- `refreshes a token shortly before GitHub's expiry`: reused at +54 min,
minted again at +56 min.
  - `never keeps a token GitHub issues for less than the safety margin`
- `mints again after a credential rotation and uses the new credential`:
rotates the installation id secret and checks that the new mint uses it.
  - `mints again after the connection's credential refs change`
- `never serves a cached token across a pause, revoke or disabled
connection`: includes a pause and resume with no call in between.
  - `mints a separate token for each repository`
- `replaces a cached token GitHub rejects and retries the request once`:
also checks that a 401 for a new token is not retried.
- On master, 5 of the 8 tests fail. All 8 pass with this change.
- Focused suites pass locally: every server test file that matches
`github`, plus `chat-sdk-runtime.adapters`, `chat-provider-inventory`,
`idle-completed-history` and `connection-management.integration` (41 of
42 files, 955 of 956 tests). The one failure,
`github-connection-enrollment-work.test.ts`, also fails on unchanged
master (`expected +0 to be 1`), so this change did not cause it.
- `pnpm --filter @paperclipai/server typecheck` passes.
- CI supplies the full typecheck, build and test coverage for the final
revision.

## Risks

- Low risk. A cached token is used only while the endpoint, connection
and mint secrets are unchanged, and the status checks still read current
database state on every call.
- A GitHub-side revoke of a token (for example, an App uninstall) makes
GitHub return 401. The client then mints again once. If the installation
is gone, the mint fails as before.
- The cache is per process. Each server process mints its own token, at
most once per hour per repository and permission set.
- Two concurrent first calls for the same key can both mint. This is the
same as before and is rare.
- No schema, migration or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:42:06 -06:00
DottaandClaude Opus 5.5 a18479bbba perf(workspaces): start a full terminal workspace pass only at startup and on the hourly backstop (#21630)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep walks every candidate workspace in a
full pass. On staging that is about 4,000 finished remote workspaces,
and each one costs a persistence probe and a work product read.
> - #21617 limited full passes to startup and an hourly backstop that
only activity arms. But three other paths in the worker still started a
full pass at once:
>   - a wake whose hints do not parse to workspace or task targets;
>   - more than 50 pending targets;
>   - any run that finds no targets.
> - Across one staging deploy the probe count rose by about 2.8 full
passes in 26 minutes. At 1,000 concurrent agents, more than 50
run-finish hints between passes would be normal, so the overflow path
alone would bring back passes proportional to history.
> - This pull request works through targets in batches of 50 and turns
hintless wakes into the backstop. It gives the one caller that needs a
broad look, a workspace provider plugin becoming ready, its own hint
that inspects only plugin workspaces.
> - The benefit is that a full pass happens only at startup and at most
once per backstop hour, whatever the activity.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On staging, the hourly steady state holds (one full pass per hour, no
probes between). But across a deploy the probe count rose by 11,351 in
under 26 minutes, about 2.8 full passes over the roughly 4,000 finished
remote workspaces.

**Steps to reproduce**

The new test file
`server/src/__tests__/terminal-workspace-rotation-limits.test.ts` runs
the real coordinator and worker against real PostgreSQL on a fake clock,
with finished remote workspaces that cannot be archived (the staging
state). These are the counts on master. The numbers are workspaces
checked by full passes; a pass over N workspaces checks N:

| Scenario | Full-pass checks on master | Expected |
| --- | --- | --- |
| A burst of 120 run-finish `issue:` hints, 300 workspaces | 600 (2
extra passes) | 0 |
| One hintless wake and one unparseable hint, 120 workspaces, 30 minutes
| 1,490 (about 12 passes) | 0, then 1 at the backstop |
| A workspace provider plugin becomes ready (hintless notify from
`plugin-registry`), 123 workspaces | 123 (1 pass) | 0; only the 3 plugin
workspaces |
| Restart: old instance drains while hints arrive, new instance starts
and gets the plugin hint and 120 run-finish hints, 202 workspaces, 30
minutes | 2,424 (12 passes) | 202 (the startup pass) |

**Cause of the passes across a deploy:**
- The new instance runs its startup pass.
- Each workspace provider plugin that turns `ready` sends a hintless
notification (`plugin-registry.ts`), which started a full pass.
- Run recovery at startup finishes many runs, and their `issue:` hints
overflow 50, which started another full pass.
- Any later wake without targets also started a full pass, including the
coordinator's 5-minute retry timer. In the fixture this produced two
back-to-back passes every 5 minutes.
- The draining old instance starts nothing, because the worker's
`canRun` is false during drain. The test confirms this.

**Expected behavior**

A full pass runs only at startup and when the activity-armed backstop is
due, at most once per hour. Targets are worked through in targeted
batches.

## What Changed

- `server/src/services/terminal-workspace-work.ts`:
- **Pending targets.** Workspace and task targets go into pending sets
with no cap. Each run inspects at most 50 of them, and the rest stay
queued for the next run, which is scheduled at once. A failed targeted
run puts its targets back.
  - **Hintless or unparseable wakes** only arm the hourly backstop.
  - **No targets, nothing due:** the run does nothing.
- **Due deadlines:** the due workspaces are queued as targets, including
isolated re-home and nested clone deadlines keyed by a workspace. A due
non-workspace deadline (the isolated source orphan sweep, task directory
work) runs a targeted sweep for its prelude and arms the backstop.
- **A failed plugin workspace listing** restores the plugin refresh
flag, so the coordinator's retry lists again.
- **Full passes** happen only at startup, when the backstop is due, and
on continuation pages of a pass already under way.
- **New hint `plugin-workspaces`** (`TERMINAL_WORKSPACE_PLUGIN_HINT`):
it queues the ids of plugin-provider candidate workspaces from
`listPluginTerminalWorkspaceIds`.
- The worker now always tells the service `targeted: true`, so a hint
during a pass no longer requests a rescan lap.
- `server/src/services/execution-workspaces.ts`:
- New `listPluginTerminalWorkspaceIds` reads the plugin-provider
candidates.
- A targeted sweep reads all matches of its at most 50 targets, with no
page limit and no rescan request.
- A failed targeted sweep no longer requests a rescan lap, because the
worker re-queues its targets.
- `server/src/services/plugin-registry.ts`: the two workspace-provider
notifications send `plugin-workspaces` instead of no hint.
- `server/src/__tests__/terminal-workspace-work.test.ts`: a new test,
`retries the plugin workspace listing after it fails instead of waiting
for the backstop`, fails without the retry fix. Three existing tests now
describe the new behaviour:
  - unscoped commits go to the backstop;
  - more than 50 hinted workspaces are inspected in batches of 50 and 1;
- a due deadline on a service without targeting still runs a full pass.
- `server/src/__tests__/terminal-workspace-rotation-limits.test.ts`:
four new tests, the scenarios above.

The startup pass stays. The 1,000-workspace test in
`terminal-workspace-deadlines.test.ts` shows that durable deadlines keep
an idle instance quiet after startup. But only the startup pass
reconciles workspaces whose hints were lost while no instance was
running, so removing it would need a stored "last full pass" marker
first.

## Verification

- The four new tests fail on master with the counts in the table and
pass after the change:
- 0 full-pass checks for the burst, worked as targeted batches of at
most 50 (120 checks in total);
- 0 full-pass checks for 30 minutes after hintless and unparseable
wakes, then 1 pass at the backstop;
  - only the 3 plugin workspaces are inspected after the plugin hint;
- exactly 1 full pass across the restart: none on the draining instance,
only the startup pass on the new one.
- Prompt archiving and the missed-hint backstop keep their existing
tests in `terminal-workspace-deadlines.test.ts`:
  - the release of a holding run inspects its workspace at once;
- a run made terminal outside `writeRunStatus` is archived within 1 hour
1 minute;
  - 1,000 finished remote workspaces stay unread at idle.
- `npx vitest run` in `server/` on `terminal-workspace-rotation-limits`,
`terminal-workspace-work`, `terminal-workspace-deadlines`,
`terminal-workspace-idle`, `terminal-workspace-page-batch`,
`execution-workspaces-service`, `plugin-workspaces.integration`,
`workspace-operations-reconciliation`, `execution-workspace-reopen`,
`isolated-task-directory-retention`, `git-workspace-expiration` and
`workspace-delivery-cleanup.integration`.
- `pnpm check:module-boundaries` passes.

## Risks

- **A notification that relied on an immediate broad pass now waits up
to an hour.** The only caller I found that sent such a notification on
purpose is the workspace provider readiness in `plugin-registry.ts`, and
it now has its own targeted hint. Other hintless notifications arm the
backstop.
- **The isolated source orphan sweep** runs only at the start of a full
pass. Its due deadline now arms the backstop instead of starting a pass
at once. This only matters when the isolated source re-home feature is
enabled.
- **A targeted sweep has no page limit.** Its at most 50 targets bound
the matched workspaces to those tasks, their ancestors and their
execution workspaces.
- **A burst of N hints costs N/50 targeted runs, back to back.** The
cost scales with activity, not with history.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 12:19:43 -06:00
DottaandClaude Opus 5.5 890dcbdd66 fix(native): wait for a live warm session owner during a deploy overlap (#21627)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap: the outgoing server still
serves while the incoming server starts new runs
> - A warm agent session stays open in the server process that ran the
last turn, after that run finished
> - The incoming server took over a warm session while the outgoing
server was alive and still held it, because the finished run's
finalization lease ends at commit
> - The takeover then failed, the task went blocked with a Retry, and in
one case a retry loop ran 27 times on the same sandbox
> - This pull request makes the incoming server wait while the server
that holds the warm session is alive, makes the outgoing server close
idle warm sessions as soon as it leaves new runs to the replacement, and
stops the retry loop when the promised fresh sandbox does not come
> - The benefit is that a follow-up during a deploy finishes without a
person, and no Runner ever gets two controllers

## Linked Issues or Issue Description

**What happened?**
A zero-downtime release-swap test (Gate B, wave B1, master aaec67b891)
found two failures with one cause.
Task 1: run A finished on the outgoing server at 15:13:56. Its warm
session stayed idle in that server. The incoming server claimed the
follow-up at 15:13:57. `inspectRemoteWarmSessionRecovery` asked
`evaluateNativeControllerTakeover`, which allowed the takeover with
`remote_controller_lease_released_or_expired`. That lease covers the
finished run, not the warm session. The outgoing server was alive until
15:18:18. `runner.prp.authenticate` failed after 62.3 s, the run failed
with `remote_warm_session_unproven`, the failure was saved as ambiguous
and owned by the board, and the task stayed blocked with a Retry.
Task 2: the same shape on a later swap started 27 automatic runs in 13
minutes. Fourteen of them said "The next attempt starts on a fresh
sandbox", but each one resumed the same sandbox and got the same refusal
(`journal_unsettled`).

**Expected behavior**
A follow-up claimed during an overlap waits for the warm session owner
to let go, then continues with no Retry banner and no second controller.
If the outgoing server dies with the session open, the takeover
continues after its ownership ends. A message that promises a fresh
sandbox is true, or the retries stop after a bounded number with a true
state.

**Steps to reproduce**
1. Run the restart-chaos release-swap preset with a warm (opencode or
codex) staging mix.
2. Let a run finish on the outgoing server after the traffic switch,
with a follow-up comment already queued.
3. See the follow-up fail with `remote_warm_session_unproven` and the
task go blocked.

**Paperclip version or commit**
master aaec67b891 and 45531b719e.

**Deployment mode**
Hosted deploy overlap (release swap) with Daytona sandboxes.

## What Changed

- `remote-warm-session-recovery.ts`: before live adoption, the run
proves that the server process that ran the finished turn is gone. It
uses the shared owner proof (`proveOwnerGone`): a PID check in the same
process space, or the `controller_instances` lease in another process
space. While that server is alive, the run defers with the existing
`native_controller_recovery_waiting_for_owner` wait and checks again
after 5 s. The wait ends when the server exits or is fenced, or when the
finished turn's warm window (idle timeout plus two minutes) ends.
- `native-session-executor.ts` and `run-start-handover.ts`: a server
that leaves new run starts to a replacement closes its idle warm
sessions after 1 s, through the existing idle-timeout close. A session
that becomes idle after that closes after 1 s too. This is the same
close that shutdown already does, done earlier.
- `predecessor-retirement-fallback.ts`: when a retry meets an unretired
owner on the same sandbox that its source already asked to replace, it
does not ask again. The run fails without another automatic retry, and
the task shows a Retry.
- `predecessor-retirement.ts`: the error no longer says "The next
attempt starts on a fresh sandbox", because the replacement is only a
request.
- `doc/execution-semantics.md`: documents the owner wait, the hand-off
at the swap, and the retry stop.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts`
- New: "waits while the outgoing server that ran the finished turn still
holds its warm session, then takes it over once that server exits" (red
on master: master returned a takeover while the owner was alive).
- New: "waits for a live outgoing server in another process space until
its controller lease ends" (red on master).
- New: "stops waiting for a live outgoing server once the finished
turn's warm window has closed" (bound check).
- `cd server && npx vitest run
src/__tests__/heartbeat-native-predispatch-retry.test.ts`
- New: "stops retrying with a truthful failure when the next attempt met
the same unretired sandbox instead of a fresh one" (red on master:
master scheduled another retry on the same sandbox).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"replacement server starts"`
- New: "closes a warm session promptly when a replacement server starts
this server's runs (before/after the turn ends)" (both red on master).
- Focused suites: native-session-executor, remote-warm-session-recovery,
native-controller-recovery, native-recovered-owner-followup,
native-runner-restart-recovery, heartbeat-native-predispatch-retry,
environment-runtime-reusable-resume-replacement,
task-interruption-presentation, sandbox-runner-port-holders.
- `npx tsc --noEmit -p server/tsconfig.json` passes. `pnpm verify` exits
2 (a doc is a global CI input), so required CI supplies broad coverage.
- Not run here: the restart-chaos release-swap wave. The maintainer runs
it after merge.

## Risks

- A follow-up can now wait up to the outgoing server's remaining overlap
when that server cannot close its idle session. Before, it failed and
needed a Retry. The wait is bounded by the owner's exit or the warm
window.
- The outgoing server closes idle warm sessions earlier during an
overlap. The next turn then resumes from a suspended Runner instead of a
live one, as it already does after a shutdown.
- A retry chain that meets the same unretired sandbox twice now stops
one attempt earlier than the shared cap. The task shows a Retry instead
of looping.
- Rows without a recorded process space keep the existing one-release
local proof, so they behave as before outside the same host.
- Previous-release state: this change adds no persisted field.
Finalization rows saved by the previous release already record the
controller boot, PID, process space and start time, so they carry
forward. A follow-up already deferred by the previous release resumes
through the same wait code.
- Runtime cost: the owner proof adds one `ps` start-time read (same
process space) or one `controller_instances` row read (other process
space) per inspection. It runs once per attempt, at most every 5 s while
a live owner holds the session.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:59:17 -06:00
DottaandClaude Opus 5.5 45531b719e perf(recovery): stop re-asserting a failed legacy run whose decision is recorded (#21621)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Stranded-work recovery checks tasks that are assigned to an agent
but have no live run. It runs as a scoped-deadline queue that a change
notification or a stored deadline wakes.
> - For a failed legacy run, the check calls
`terminalizeLegacyExecution` with the run's current status. When the
run's recovery decision is already recorded (for example "continue in a
fresh run from saved work"), that call changes nothing. But it still
updated the run row and notified about 18 delivery queues, including the
run's own recovery graph.
> - The graph notification re-checked the same task, which re-asserted
the same run again. The loop ran about once a second per waiting task,
forever, with zero runs active. On a hosted instance this shows up as
constant idle transactions.
> - This pull request makes a re-assertion that changes nothing return
before it writes or notifies.
> - The benefit is that an idle instance does no stranded-recovery work
for tasks that wait on a recorded decision. A real change (another run
fails, a comment, a status change) still notifies and recovers at once.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

With zero runs active, the server runs many transactions per second. A
CPU profile groups most samples under `reconcileStrandedAssignedIssues`
(`server/src/services/recovery/service.ts`). A local fixture on real
PostgreSQL reproduces it: a task in `in_progress`, assigned to an agent,
whose latest legacy run failed and whose recovery decision is recorded.

Each check reaches the branch that terminalizes a failed legacy run that
needs reconciliation (`service.ts`, the
`legacyExecutionNeedsReconciliationWithEvidence` branch, and the same
call in `enqueueStrandedIssueRecovery`). `terminalizeLegacyExecution`
then:
- updates the run with the same status (only `updated_at` changes);
- calls `notifyExecutionRunWork`, `notifyInstructionCollection` and
`notifyUnsafeWorkspaceExportRecovery`, which hint the run graph of the
task and about 18 queues;
- finds the recorded decision and returns.

The graph hint re-checks the same task after the queue's 1-second pass
interval, so the loop never ends.

**Expected behavior**

A check that finds nothing to change stores nothing and wakes nothing.
Only a real change re-checks the task.

**Steps to reproduce**

1. Create a task in `in_progress` assigned to an agent, with a failed
legacy run (`adapter_failed`).
2. Start the issue-state recovery work. The boot pass records a
fresh-continuation decision for the run.
3. Leave the instance idle. On master: 591,045 statements in 30 minutes
for 10 such tasks (fake clock, real PostgreSQL), and one no-op `update
"heartbeat_runs"` per task per second.

**Paperclip version or commit**

master at 7db92ea319.

## What Changed

- `server/src/services/legacy-execution-recovery.ts`:
- `terminalizeLegacyExecution` returns the unchanged run, without a
write or a notification, when all of these are true: the call re-asserts
the current status with no patch; the run has no workspace restore
failure or required workspace recovery; the task's execution and
checkout locks do not point at the run; and either the run needs no
reconciliation (`reconcileIfNeeded`), no assignee or reviewer owns a
decision, or the decision is already recorded.
- Why the short-circuit is safe: stranded-work and quota checks
re-assert the run's unchanged terminal status each time they revisit its
task. Once the decision is recorded, that re-assertion changes nothing.
If it wrote or notified anyway, the notifications would re-check the
same task, which would re-assert the run again, about once a second for
as long as the task waits.
- `findRecordedLegacyDecision` returns the resolved decision already
recorded for the run (a reconciliation for this run, an explicit user
continuation with a real successor, or a no-replay hold).
`decisionOwner` says whether the task's assignee or current reviewer
owns a decision about the run.
- The recorded-decision query moves into `findRecordedLegacyDecision`,
and the owner condition into `decisionOwner`. The full path uses both
helpers, so its behavior does not change.
- `server/src/__tests__/issue-recovery-recorded-decision-idle.test.ts`:
two new tests on real PostgreSQL. They use the real coordinator, the
real issue-state recovery work and the production 5-minute backstop on a
fake clock.

## Verification

- `stays quiet for half an hour once each failed run has its decision`:
10 waiting tasks, zero runs. The test first checks that each failed run
has a recorded decision, then counts statements over 30 idle minutes.
  - master: 591,045 statements (red).
- this branch: 0 statements (green). No run is re-written or started,
and no task moves.
- `still recovers promptly when another run of a waiting task fails`:
after 10 idle minutes, a new legacy run of one task fails and notifies
like a finished run.
- this branch: within 5 seconds, recovery creates the run's
`legacy-execution:<run>` hold. Within the next 6 minutes, the
fresh-continuation fallback resolves that hold with
`executionReconciliation.runId` set to the new run. Over the next 30
minutes there is no `update "heartbeat_runs"` and fewer than 5,000
statements (the assigned-work invariant keeps its own spaced re-checks).
- master: 18,000 no-op `update "heartbeat_runs"` statements in the same
window (red).
- Neighbouring suites pass locally: `legacy-workspace-restore-recovery`,
`legacy-workspace-restore-lock-contention`,
`legacy-continuation-authority`, `heartbeat-process-recovery`,
`issue-recovery-assignments`, `recovery-deadline-ordinary-bootstrap`,
`issue-recovery-deadlines` (600 of 601 in one parallel run; `resolves a
run's task graph through index lookups` failed once under machine load
on a planner row count, and passes alone; it does not use the changed
code).
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The short-circuit applies only when the full path would also
write nothing except `updated_at`. Every other case (status change,
patch, workspace restore evidence, an owned lock, no recorded decision)
takes the unchanged full path.
- A re-assertion no longer re-sends notifications. Queues that relied on
that repeated hint as a side effect lose it. The first terminalization
of a run still sends all notifications, and run, comment and status
writes still send their own.
- No schema or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21552, #21616, #21617.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:30:03 -06:00
DottaandClaude Opus 5.5 7db92ea319 perf(workspaces): read a sweep page's task trees and active runs in two statements (#21618)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep inspects candidate workspaces 50 per
page. For each workspace it read the task tree with its own recursive
query. For many of them it also ran one or two queries to check for an
active run.
> - The cost of a page therefore grew with the page: about 50 to 110
statements. On the hosted instance, building the SQL text of these
statements costs about as much main-thread time as parsing their
results.
> - The archive statement re-checks the terminal tree and the absence of
a live run in the same statement, and the remote and plugin cleanups
re-check active runs under the lifecycle lock. So these early reads are
only pre-filters.
> - This pull request reads the task trees and the active-run workspaces
of a whole page in two fixed-text statements.
> - The benefit is that a page costs a constant number of these reads,
whatever its size.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The per-page reads of the terminal workspace sweep
(`sweepTerminalWorkspaces` in
`server/src/services/execution-workspaces.ts`).

**Current behavior**

Each candidate calls `listWorkspaceIssueTree`, one recursive query. A
candidate that reaches the active-run checks calls
`workspaceHasActiveRun`: one query for the linked tasks' run ids, then
one for live runs.

**Proposed behavior**

- `readPageIssueTrees` reads the tree rows of every root on the page in
one statement. Its SQL text is fixed: the roots are one `jsonb`
parameter. The trees are then built in memory and keep the same row
shape.
- `readPageActiveRunWorkspaces` returns, in one fixed-text statement,
the page workspaces that have a `queued` or `running` run on a linked
task.
- The two pre-filter checks in the loop use these results. A workspace
whose source task changed earlier in the same pass (rehome) still reads
its own tree.
- The authoritative checks are unchanged: the archive `UPDATE` with its
terminal-tree and live-run predicates, the active-run check inside the
remote and plugin cleanup under the lifecycle lock, and the
per-workspace check before a stale reopen flag is cleared.

**Reason and benefit**

One rotation pass should issue a bounded number of statements per page.
For a page of 50 workspaces (40 with open trees, 10 held by a run), the
tree reads fall from 50 to 1 and the active-run reads from 20 to 1.

## What Changed

- `server/src/services/execution-workspaces.ts`: add
`readPageIssueTrees` and `readPageActiveRunWorkspaces`, and use them for
the per-page pre-filter.
- `server/src/__tests__/terminal-workspace-page-batch.test.ts`: two new
tests on real PostgreSQL.

## Verification

- `reads the task trees and active runs of a page in a fixed number of
statements`: 2 companies, 50 workspaces, each with a child task, 10 held
by a `running` run.
  - Before the change it fails: 50 tree reads.
- After the change: one tree read, one active-run read, fewer than 10
statements for the page. The results are the same (`checked: 50`,
`skippedNonTerminalTree: 40`, `skippedActiveRun: 10`).
- `does not archive a workspace whose run starts after the page was
read`:
- The page is read first. Then, during that workspace's close-readiness
check, a `running` run is inserted and set as its task's checkout run.
- The sweep counts the workspace `eligible: 1`, `archived: 0`,
`skippedRace: 1`, and it stays `active`.
  - After the run succeeds, the next sweep archives it (`archived: 1`).
- `npx vitest run src/__tests__/terminal-workspace-page-batch.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/git-workspace-expiration.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/nested-clone-migration.test.ts
src/__tests__/execution-workspace-compaction.test.ts
src/__tests__/git-restore-point-expiration.test.ts
src/__tests__/plugin-workspaces.integration.test.ts` (in `server/`).
- `pnpm check:module-boundaries` passes.

## Risks

- **The pre-filter reads happen once, at the start of the page.** A page
can take long, because cleanups do remote and Git I/O. A run that starts
mid-page on a workspace the pre-filter saw as idle is still caught by a
check made at the destructive step itself, in the same statement or
under a lock. Line numbers are on this head:
- **Archive** of an active workspace: the archive `UPDATE` in
`sweepTerminalWorkspaces` carries `NOT EXISTS (… live_run.status IN
('queued', 'running'))`
(`server/src/services/execution-workspaces.ts:4291`). It changes 0 rows
and the workspace counts as `skippedRace` (`:4340`).
- **Remote cleanup** (destroying the sandbox):
`runTerminalWorkspaceCleanup` re-runs `workspaceHasActiveRun` under the
lifecycle lock before `cleanupRemoteWorkspaceAllocation` (`:2702`).
- **Plugin workspace removal**: the same check under the lifecycle lock
before `removePluginWorkspace` (`:2706`).
- **Git cache expiration** and **isolated task directory expiry**:
`isGitWorkspaceExpirationEligible` re-reads the tree and refuses any run
that is not finished, inside the locked transaction
(`server/src/services/git-workspace-expiration.ts:311` and `:416`,
`server/src/services/isolated-task-directory-retention.ts:244` and
`:290`).
- A run that ends mid-page leaves its workspace for the run-release hint
or the next inspection.
- **Tree rows are now deduplicated** (`UNION` instead of `UNION ALL`).
Parent cycles therefore cannot repeat rows.
- Low risk otherwise. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 08:51:28 -06:00
DottaandClaude Opus 5.5 081dacba1d perf(workspaces): inspect only due workspaces and wait for a holding run's release (#21617)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep stores one deadline per workspace:
cooldown end, compaction, cleanup retry, Git idle.
> - When any deadline was due, the worker ran a full rotation over every
open candidate workspace. That costs a task tree read and more for each
one.
> - A workspace held by a `running` run kept a past deadline, so the
5-minute deadline rotation came back for it forever. A run that never
finishes would keep the instance awake indefinitely.
> - This pull request makes a due deadline inspect only the due
workspaces. A held workspace waits for its run instead of for a timer.
> - The benefit is that an idle instance does work only for workspaces
whose deadline really arrived. The cost no longer grows with the number
of open workspaces.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

How the terminal workspace worker
(`server/src/services/terminal-workspace-work.ts`) acts on due deadlines
and on workspaces held by an active run.

**Current behavior**

`hasPending` reads `nextTerminalWorkspaceSweepAt`. When the earliest
deadline is past, the next run starts a full keyset rotation, 50
workspaces per page with 1 second between pages, over every candidate. A
workspace whose issue still has a `queued` or `running` run is skipped,
but the archive deadline set earlier in the same pass stays. It is
already past, so the rotation repeats every 5 minutes.

**Proposed behavior**

- When the run fires for a due deadline, the worker takes the due
workspace ids from the service (`takeDueTerminalWorkspaceTargets`) and
runs a targeted inspection of only those workspaces. The service first
clears the deadlines of the targeted workspaces; the inspection sets
fresh ones.
- If more than 50 workspaces are due, or a due deadline belongs to the
isolated source, nested clone or task directory work, the worker still
runs a full rotation. Startup still runs a full rotation.
- A workspace skipped because a run holds it drops its deadline.
- `writeRunStatus` (`server/src/services/heartbeat/run-lifecycle.ts`)
publishes `issue:<runIssueId>` to the terminal workspace queue when a
run reaches a terminal status. That targeted inspection reaches the
workspace the run held.
- **Backstop.** About 200 code paths write `heartbeat_runs` directly
(reapers, recovery terminalization, bulk cancels), and they do not pass
through `writeRunStatus`. So the worker keeps a full rotation as a
backstop, armed only by activity, like the recovery backstop from
#21552:
  - Any terminal-workspace hint arms it.
- Once armed, a full rotation runs at most once per hour
(`TERMINAL_WORKSPACE_BACKSTOP_MS`) after the previous full rotation.
  - An idle instance with no hint schedules nothing.
- A missed release hint therefore delays archiving by at most one hour
after activity, without a restart.
- The hour is a compromise. On master today, the 5-minute rotation
re-reads every candidate. On the hosted instance that is about 1,000
finished remote workspaces, roughly 3,000 statements per lap. The hourly
backstop re-reads them at most once per hour while there is activity,
about 1 statement per second averaged.

**Reason and benefit**

Background work must be driven by deadlines and change notifications,
and its cost must not grow with the backlog. Every guarantee keeps a
named trigger:
- cooldown end, compaction, retry backoff and Git idle each have a
stored deadline, and a due deadline inspects exactly that workspace;
- the release of a holding run publishes a hint;
- task status changes and workspace writes still publish hints;
- startup still reconciles everything.

## What Changed

- `server/src/services/execution-workspaces.ts`:
  - New `takeDueTerminalWorkspaceTargets()`.
- A targeted inspection clears the deadlines of its targets before it
inspects them.
  - A workspace skipped for an active run drops its deadline.
- `server/src/services/terminal-workspace-work.ts`: a due run uses the
due targets instead of a full rotation, unless they overflow or a
non-workspace deadline is due. A due run that finds nothing due does
nothing. Any hint arms the hourly backstop rotation.
- `server/src/services/heartbeat/run-lifecycle.ts`: a terminal run
status hints the run's task to the terminal workspace queue.
- `server/src/__tests__/terminal-workspace-deadlines.test.ts`: four new
tests on real PostgreSQL. They use the real coordinator and worker on a
fake clock.
- `server/src/__tests__/terminal-workspace-idle.test.ts` (from #21616):
its idle window now bounds the number of sweeps instead of requiring
zero targeted sweeps, because a due deadline is now a targeted
inspection.

## Verification

- `inspects only the workspace whose cooldown ends, however many trees
stay open`: 60 workspaces with open task trees and 1 done workspace
whose 1-day cooldown ends 1 hour later.
- Before the change: after startup, every due deadline starts a full
rotation (27 full sweeps in 2 hours).
- After the change: no full rotation; the done workspace is inspected
alone (`checked: 1`) once its cooldown ends; fewer than 200 statements
in 2 hours.
- `waits for the release of the run that holds a workspace instead of
rechecking it`: a done workspace held by a `running` run.
- Before the change: a full rotation every 5 minutes (22 sweeps in 2
hours).
- After the change: 0 statements in 2 idle hours. When the run is set to
`succeeded` through the lifecycle, the workspace is inspected (`checked:
1`, `skippedActiveRun: 0`).
- `leaves a thousand finished remote workspaces unread at idle`. This is
the hosted state: 1,000 `active` remote workspaces with a location, a
committed snapshot and a task done 5 days ago, and zero runs.
- After startup and 2 idle hours: no full rotation. No remote workspace
is read again; the only inspections are of the one due local workspace.
Statements stay within 10 per inspection.
  - On master, every due deadline re-reads all 1,000.
- `archives a workspace whose run ended outside the run lifecycle within
the activity-armed backstop`:
- A workspace held by a `running` run. The run is set to `failed` by a
direct `UPDATE`, so no release hint is published, and one unrelated hint
arrives.
- The workspace is archived within one hour and one minute, without a
restart.
  - With the backstop scheduling disabled, it stays `active`.
- `npx vitest run src/__tests__/terminal-workspace-deadlines.test.ts
src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/git-workspace-expiration.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/nested-clone-migration.test.ts
src/__tests__/execution-workspace-compaction.test.ts
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 333
tests pass.
- `pnpm check:module-boundaries` passes.

Local fixture with #21616 applied as the base, 30 idle minutes on a fake
clock, real PostgreSQL. Each fixture holds open trees, a stranded
`running` holder, cooldown, done local workspaces with missing files,
closed remote workspaces with and without a committed snapshot:

| Workspaces | Before: sweeps | Before: statements | After: sweeps |
After: statements |
| --- | --- | --- | --- | --- |
| 140 | 20 (full rotations) | 1,371 | 21 (targeted only) | 866 |
| 420 | 84 (full rotations) | 6,597 | 33 (targeted only) | 3,272 |

After the change, no sweep reads the task tree of a workspace whose tree
is still open (`skippedNonTerminalTree` falls from 1,440 to 0 at 420
workspaces). What remains comes only from workspaces with a due retry or
inspection backoff, so it grows with the number of stuck workspaces, not
open ones. Per inspected workspace a pass issues the same 3–6 statements
as before; batching them per page is a separate pull request.

## Risks

- **A run that leaves `running` without `writeRunStatus`** does not
hint. The activity-armed backstop catches it within one hour of
activity, and startup reconciles. This pull request does not decide that
a stranded run is gone; it waits for a status change.
- **A workspace whose deadline is reset to "now"** (a reopen-pending
flag within its grace period, task directory budget overflow) is
inspected alone on each due run. The inspection backoff in
`nextTerminalWorkspaceSweepAt` still spaces these runs.
- **Workspaces without a stored deadline** (for example an undelivered
workspace whose deadline was deleted) are inspected again only when a
hint or the startup rotation reaches them. Before this change they were
also reached by the rotation that a different workspace's deadline
started.

## Open decision

**Should a terminal workspace whose local files are gone be archived, or
should it wait?** This pull request does not change the behaviour; it
leaves the decision to the owner.

Today such a workspace fails its close-readiness inspection and is
reported as `skippedUndelivered`. A non-remote workspace keeps its past
deadline, so the 5-minute inspection backoff re-inspects it again and
again. That costs about 3–4 statements per stuck workspace every 5
minutes (from the local fixture: 255 inspections, 866 statements), and
it never ends while the files stay missing. The cause is typically a
replaced container.

The two options:
- **Archive it.** Nothing remains on disk to keep.
- **Store a "files missing" state.** It is then inspected again only
when a write announces a change.

Either option removes this cost. Archiving also removes the row from
later rotations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 08:23:49 -06:00
DottaandClaude Opus 5.5 20ccf03676 fix(workspaces): settle closed remote workspaces after one confirmed destroy (#21616)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server cleans up closed execution workspaces in a background
sweep (`terminal-workspaces`). For a remote workspace, cleanup destroys
the sandbox lease after it confirms that the saved snapshot is
committed.
> - A closed remote workspace stays a sweep candidate while
`metadata.remoteWorkspace.location` is an object. A confirmed destroy
did not clear that location.
> - The cleanup write goes through the lifecycle gateway, and the
gateway published a `workspace:<id>` hint for the same workspace. That
hint started a new inspection, which cleaned the workspace again and
published again.
> - On a hosted instance with no runs, this loop kept three statements
at about 15–24 per second each and a dedicated lock connection per
inspection. Every pass also asked the sandbox provider to destroy a
sandbox that was already destroyed.
> - This pull request clears the location after a confirmed destroy,
skips the provider call when the stored lease already has a destroy
receipt, and stops the sweep from hinting itself for its own writes.
> - The benefit is that an idle instance with closed remote workspaces
settles: each workspace is cleaned once, and the provider is called at
most once per lease.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

With no runs, the terminal workspace sweep never became idle. On the
hosted instance these statements ran all the time:
`workspaceHasActiveRun` (22.8/s), the task tree read (22.6/s) and the
snapshot persistence read in `inspectRemoteWorkspacePersistence`
(14.8/s). `SET application_name` for the
`paperclip-workspace-finalization-lock` connections ran at 9.6/s.

**Steps to reproduce**

1. Create closed (`archived`) remote workspaces that have a committed
snapshot, a clean authority and a `location`.
2. Start the terminal workspace worker with no other activity.
3. Watch the sweeps. Each cleanup reports `archived`, the gateway
publishes `workspace:<id>`, and the next targeted inspection cleans the
same workspace again.

Locally, a fixture with 140 workspaces (40 of them in this state) ran
568 sweeps, 136,888 statements and 23,880 lease destroy calls in 30 idle
minutes. The destroys always hit the same 40 workspaces.

**Expected behavior**

A closed remote workspace is cleaned once. After the destroy is
confirmed it leaves the candidate set, and an idle instance issues no
statements between real deadlines.

## What Changed

- `server/src/services/remote-workspace-lifecycle.ts`
(`cleanupRemoteWorkspaceAllocation`):
- Read the stored lease first. When it already carries a destroy receipt
for the same environment, lease and provider lease, do not call
`destroyLease` again.
- After a confirmed destroy, set `metadata.remoteWorkspace.location` to
`null`. One fixed-text `jsonb_set` update does this, and only when the
stored location is still the location that was destroyed.
- `server/src/services/execution-workspaces.ts`: the gateway takes
`sweepOwned`. The sweep's own cleanup, cleanup-failure and reopen-flag
writes set it, and then the gateway does not publish a terminal
workspace hint for that workspace. All other gateway callers still
publish.
- `server/src/__tests__/terminal-workspace-idle.test.ts`: two new tests
on real PostgreSQL.

## Verification

- `settles after one cleanup of each closed remote workspace and
inspects nothing on its own writes`:
- Fixture: 40 workspaces (open trees, a stranded `running` holder,
cooldown, done, closed remote with a committed snapshot, closed remote
without a snapshot).
- Runs the real coordinator and worker on a fake clock for 2 minutes,
then 30 idle minutes.
- Asserts: each closed remote workspace is destroyed exactly once; no
targeted inspection in the idle window; fewer than 5,000 statements in
30 minutes; every cleaned workspace has `location: null`.
- Before the change it fails: 850 destroy calls instead of 10 in the
warm-up alone. After the change it passes.
- `clears the location of an already destroyed lease without calling the
provider again`: the stored lease has a destroy receipt. Before the
change, the sweep calls `destroyLease`. After the change the provider
mock is never called, the location is cleared, and the next inspection
finds no candidate.
- `npx vitest run src/__tests__/terminal-workspace-idle.test.ts
src/services/remote-workspace-lifecycle.test.ts
src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts` (in
`server/`).

Local fixture, 30 idle minutes, 140 workspaces (40 closed remote with a
committed snapshot):

| | Before | After |
| --- | --- | --- |
| Sweeps | 568 (all targeted, continuous) | 20 (one deadline rotation
every 5 minutes) |
| Statements | 136,888 | 1,371 |
| Lease destroy calls | 23,880 | 40 (once per workspace) |
| `workspaceHasActiveRun` reads | 45,440 | 260 |
| Snapshot persistence reads | 22,720 | 40 |

The remaining rotation every 5 minutes comes from deadlines of
workspaces that cannot make progress (a stranded active-run holder,
cleanup backoff). A separate pull request handles it.

## Risks

- **Who reads `remoteWorkspace.location`.** For a closed, clean
workspace with a committed snapshot and a destroyed lease, none of these
readers needs the location:
- the live run binding checks (`remote-workspace-lifecycle`,
`fresh-continuation-fallback`, `workspace-recovery-fallback`,
`workspace-recovery-source`) need a current writer or a dirty run
reference;
- the abandonment and hold paths (`remote-workspace-epoch-abandonment`,
`remote-workspace-abandoned-files-hold`,
`remote-workspace-unchanged-proof`) need a dirty or held allocation;
- `reusable-sandbox-resume-replacement` treats `null` as no conflicting
sandbox;
- `remote-workspace-files` reads the saved snapshot when the location is
`null`. With a destroyed location it would fail as unavailable.
- The lease-release path
(`clearRemoteWorkspaceLocationForReleasedLease`) already writes the same
`null` for the same state. This change makes the sweep stop depending on
that side effect.
- **Provider calls.** Before this change every loop pass re-claimed the
lease and called the driver's `destroyRunLease` again, through
`destroyClaimedSandboxLease` in
`server/src/services/environment-runtime.ts`. After it, a lease with a
stored destroy receipt causes no provider call.
- **Existing backlog after deploy.** The startup rotation and the next
deadline rotation visit each closed remote workspace that still has a
location once, 50 per page with 1 second between pages. A lease that
already has a destroy receipt costs no provider call, and its location
is cleared. Only leases without a receipt get one destroy call. The
workspaces then leave the candidate set.
- **No self-hint.** The sweep already handles the workspace it writes,
and an inspection's deadline bookkeeping is unchanged. Other writers
still publish hints. The git restore point and task directory paths
depend on task or explicit hints, not on these writes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 07:23:35 -06:00
DottaandClaude Opus 5.5 c1451814c1 perf(chat): scope GitHub review check run hints to the issue and throttle company-wide rebuilds (#21610)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel keeps each unresolved "Paperclip Review"
check in step with its task and run. A background queue
(`chat-github-review-checks`) rebuilds its checks when a hint arrives
> - Each run milestone (`notifyChatRunMilestones`) sends this queue a
company-wide hint, also when the milestone knows the issue. Each
company-wide hint rebuilds every unfinished check of the company and
reads its full state again. Passes are limited to one a second
> - So in a company with a steady stream of run milestones, each
unresolved check reads its whole state about once a second, although
nothing about it changed
> - On a hosted test database the five state statements of a review
check ran 10.3 times a second each. #21608 removes the endless 5-second
head poll. This is the second cause
> - This pull request sends the issue with the hint, rebuilds only the
checks of that issue, and rebuilds for a company-wide hint at most once
every 30 seconds
> - The benefit is that run activity no longer makes every unresolved
check of the company read its state each second. A run change of the
check's own task still publishes at once

## Linked Issues or Issue Description

Refs #21608 (backs off the head poll of an unresolved check; the other
cause of the same hosted load). Refs #21580 (the same 30-second rule for
the GitHub response sweeps). Refs #18384 (the same rule for external
objects).

**What happened?**

On a hosted test database with zero active runs, the five statements of
`githubReviewCheckState` (`chat_endpoints`, `chat_github_reviews`,
`chat_message_links`, the `heartbeat_runs` `wakeCommentId` lookup and
`chat_deliveries`) ran 10.3 times a second each. One source is the
rebuild of the review check queue for company-wide hints.

**Current behavior**

`notifyChatRunMilestones` sends `{"companyId": ...}` to
`chat-github-review-checks` for every milestone, also when its scope has
an issue. The queue scope has no issue field. So each milestone rebuilds
every received and retryable check of the company, and calls
`githubReviewCheckState` for each received check.

**Proposed behavior**

A milestone with an issue sends `{"companyId", "issueId"}`. The queue
matches a check to an issue through the issue of its latest review for
the pull request head. This is the same review that
`githubReviewCheckState` reads. A company-wide hint (company ID only)
rebuilds at most once every 30 seconds. All narrower hints, the startup
scope (`*`) and saved deadlines are not changed.

**Steps to reproduce**

1. Admit a GitHub review check that stays unresolved (a review with no
run that the check can find).
2. Send a run milestone for the company once a second (for example,
status changes of other runs).
3. Count statements. Each hint rebuilds the check and reads its full
state.

## What Changed

- `review-check-work.ts`: the scope has an optional `issueId`. `page()`
reads the issue of the latest review for the check head
(`reviewIssueId`, through the existing `chat_github_reviews_pull_idx`
index) and filters on it for an issue scope. `matches()` uses the same
value.
- `review-check-work.ts`: set `reconstructionIntervalMs` to
`githubReviewCheckReconstructionIntervalMs`: 30 seconds for a
company-only scope, 0 for all other scopes.
- `chat-run-milestone-notifications.ts`: send the issue with the review
check hint when the milestone has one.
- `chat-channels.integration.test.ts`: two regression tests on the real
queue, the real `dispatch` and embedded PostgreSQL with a controlled
clock. The test helper now counts only the state read of
`chat_github_reviews`, not the new subquery in `page()`.

## Verification

Red/green regression tests:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "company-wide hints
at most once per interval|at once from its issue milestone"`
- Red (without the fix): `expected [ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 ] to
deeply equal [ 1, +0, +0, +0, +0, +0, +0, +0, …(2) ]`. Each of 10
company-wide hints read the check state. The milestone hint was
`{"companyId"}` only.
- Green: 2 of 2 pass. The first company-wide hint reads the state once.
The next 9 read nothing. After 30 seconds the waiting hint reads it
once. A hint for the check's issue reads it at once each time. A hint
for another issue or another company reads nothing. A real
`notifyChatRunMilestones` with the check's issue publishes `in_progress`
on the next pass and resets the head poll interval.

Statement counts for one unresolved check with one company-wide hint
each second (120 simulated seconds, real queue and `dispatch`, embedded
PostgreSQL):

| | Statements a second | Check state reads a second |
| --- | --- | --- |
| Before | 6.0 | 1.0 |
| After | 0.2 | 0.033 |

In the real-coordinator reproduction of #21608, one hint each second
raised one stuck check from 0.70 to 1.67 state reads a second and from
8.4 to 15.2 statements a second.

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: passed,
including all review check queue tests.
- `cd server && npx vitest run
src/__tests__/native-status-reassessment-work.test.ts
src/__tests__/chat-run-milestone-work.test.ts
src/services/native-runtime/external-chat-wait.integration.test.ts
src/services/chat-github-review-repair.test.ts
src/__tests__/idle-completed-history.test.ts
src/__tests__/chat-admission-condition.test.ts
src/__tests__/app-private-hostname-gate.test.ts`: 518 passed.
- `cd server && node ../scripts/run-tsc.mjs` and `node
scripts/typecheck-layers.mjs` (with test layers): passed.
- `pnpm verify --explain` retains 2,718 test files conservatively, so I
did not run it locally. Required CI supplies that coverage.

## Risks

- A change that only a company-wide hint reports can take up to 30
seconds to reach a check. Company-wide hints now come from milestones
without an issue (a run with no task, a comment merged into an active
run) and from company authority changes. A run with no task cannot be
the run of a review check. For a merged comment, the run's next status
change sends an issue hint. The saved head poll deadline still applies.
Each dispatch reads the full state again before it publishes.
- An issue hint finds a check through the issue of its latest review for
the head. If a run of another issue were the check's run, its change
would wait for the next company-wide rebuild (30 seconds at most) or the
next head poll. Run status and run identity milestones use the run's
task issue, which is the review's issue for a GitHub review task.
- The deadline reconciliation snapshot reports the queue as not
reconciled while a company-wide hint waits (up to 30 seconds). This is
the same as the GitHub response sweeps and the external-object refresh
sweep.
- `page()` reads one more indexed subquery for each row. It uses
`chat_github_reviews_pull_idx`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local embedded PostgreSQL
reproductions).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 05:46:48 -06:00
DottaandClaude Opus 5.5 a7453eba60 perf(chat): back off GitHub review check head polls while the check cannot change (#21608)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel publishes a "Paperclip Review" check for
each admitted pull request head. A background queue
(`chat-github-review-checks`) keeps each unresolved check in step with
its task and run
> - While a check is queued or running, the queue polls GitHub for the
pull request head every 5 seconds. This finds a push whose webhook was
lost
> - The poll never stops while the check is unresolved. A check whose
run never comes, or whose run the check cannot find, stays queued
forever. It re-reads its whole state and calls GitHub every 5 seconds,
also when nothing changes
> - On a hosted test database with zero active runs, the five state
statements of this check ran 10.3 times a second each
> - This pull request doubles the head poll interval for each poll that
finds the same state, up to 5 minutes. A state change resets it to 5
seconds
> - The benefit is that an idle unresolved check costs about one poll
each 5 minutes, not one each 5 seconds. Local changes still publish at
once

## Linked Issues or Issue Description

Refs #17662 (added the 5-second head reconciliation for unresolved
checks). Refs #21580 (the same kind of fix for the GitHub response
sweeps).

**What happened?**

On a hosted test database (Neon, 488-second window, zero running or
queued runs, 2 runs created in 30 minutes), these five statements ran at
exactly 10.3 a second each: the `chat_endpoints` source read, the
`chat_github_reviews` read, the `chat_message_links` comment read, the
`heartbeat_runs` `wakeCommentId` lookup and the `chat_deliveries` state
read. All five come from `githubReviewCheckState`
(`server/src/services/chat-channels/github/review-check-state.ts`).

**Expected behavior**

A check whose state cannot change makes no recurring database reads or
GitHub calls until something that can change it happens: a local hint, a
lease expiry, or a saved deadline. The missed-webhook head check stays,
at a bounded cost.

**Steps to reproduce**

1. Admit a GitHub review check. Bind a task review to it, but give it no
run that the check can find (for example, the run was never created, or
its wake comment is not the delivery's comment).
2. Start the review check queue and leave the instance idle.
3. Count statements. The check is due again 5 seconds after each poll,
for ever. Each poll evaluates the check state about 3 to 4 times (before
dispatch, in dispatch, after dispatch, and in the rebuild that the
publication lease release starts). Each poll also asks GitHub for an
installation token and the pull request.

**Cause**

`dispatch` (`server/src/services/chat-github-checks.ts`) saved `retryAt
= now + 5s` after each poll. `entry()` in `review-check-work.ts` uses
that deadline when the state is unchanged. So an unresolved check was
due every 5 seconds without end.

## What Changed

- `chat-github-checks.ts`: save `unchangedPolls` in the check result. A
poll that finds the same state as the saved state adds one (up to 6). A
changed state, a first publication and a recovered failure set it to 0.
The next head poll is due after `min(5 minutes, 5 seconds ×
2^unchangedPolls)`.
- Export `GITHUB_REVIEW_CHECK_HEAD_POLL_MS` (5 seconds) and
`GITHUB_REVIEW_CHECK_MAX_HEAD_POLL_MS` (5 minutes).
- `chat-channels.integration.test.ts`: two regression tests. They use
the real review check queue, the real `dispatch`, embedded PostgreSQL, a
controlled clock and one pass each second. The fixture has the hosted
state: a review with no run, a delivery with a linked comment, and no
run for that comment.

## Verification

Reproduction with the real delivery coordinator, the real queue and
embedded PostgreSQL. One queued check with no findable run, 30 seconds
of real time, before this change:

| Company-wide hints | Statements a second | Check state reads a second
| GitHub pull reads a second |
| --- | --- | --- | --- |
| None | 8.4 | 0.70 | 0.17 |
| One each second (`notifyChatRunMilestones`) | 15.2 | 1.67 | 0.17 |

So one stuck check costs about 0.7 state reads a second from the head
poll alone. The hosted rate (10.3 a second) is consistent with about 15
such checks, or fewer checks with frequent company-wide hints.
Company-wide hints are a separate cause and a separate pull request.

Red/green regression test, 600 simulated seconds with one pass each
second:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "unresolved GitHub
review check cannot change|publishes a local change at once"`
- Red (without the fix): `expected 120 to be less than or equal to 7`.
Over 600 seconds: 5,160 statements, 360 check state reads, 120 GitHub
pull reads.
- Green: 258 statements, 18 check state reads, 6 GitHub pull reads (8.6
→ 0.43 statements a second, a 95% drop). The saved interval stops at 5
minutes.
- Correctness (green): after the back-off, a restarted queue makes no
early GitHub call. A run bound to the review publishes `in_progress` at
the next pass and resets the interval. A pushed head is still found
within 5 minutes, the check is cancelled and the review is superseded.

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed,
including all 12 review check queue tests.
- `cd server && npx vitest run
src/services/chat-github-review-repair.test.ts
src/__tests__/idle-completed-history.test.ts
src/__tests__/chat-admission-condition.test.ts
src/__tests__/app-private-hostname-gate.test.ts`: 109 passed.
- `cd server && node ../scripts/run-tsc.mjs` and `node
scripts/typecheck-layers.mjs` (with test layers): passed.
- `pnpm verify --explain` retains 2,718 test files conservatively, so I
did not run it locally. Required CI supplies that coverage.

## Risks

- A push whose webhook is lost is found later: after up to 5 minutes,
not 5 seconds, when the check has not changed for a while. The new
head's own webhook still creates its new check at once. Only the old
head's check stays queued or running on GitHub a little longer before it
is cancelled.
- Local changes are not delayed. `entry()` still makes a check due at
once when its state differs from the saved state, and run, review,
enqueue and lease hints still rebuild it.
- Saved actions from an older release have no `unchangedPolls`. They
start at 0. An invalid saved value also starts at 0. No migration.
- Restart recovery is unchanged: the deadline is saved on the action,
and startup rebuilds all unfinished checks.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local embedded PostgreSQL
reproductions).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 05:22:35 -06:00
DottaandPaperclip 7c0c3c4f31 fix(mcp): preserve Tasks sleep authority and re-enable recovery (#21603)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected apps can retain unfinished provider work through MCP
Tasks.
> - Managed defaults can enable Tasks without a stored experimental
flag.
> - The sleep check used the stored flag and could miss an active
provider task.
> - Enabling Tasks also failed to notify an owner that had become idle.
> - This pull request uses the effective setting and wakes the existing
owner after the setting commits.
> - Pending work keeps its recovery path. Completed history can become
quiet.

## Linked Issues or Issue Description

Refs #21263, which added durable provider Tasks. The current roadmap
covers the governed MCP Tool Gateway. This change repairs two existing
paths.

**What happened?**

The gateway could enable MCP Tasks through managed defaults while the
idle check reported no pending work. After an operator disabled and
enabled Tasks, an existing task could stay idle until a separate task
write woke its owner.

**Expected behavior**

The gateway and sleep check must use the same effective setting.
Enabling Tasks must notify the existing owner. It must retain the task's
original deadline and backoff.

**Steps to reproduce**

1. Retain a task with `next_poll_at`. Use managed Tasks defaults without
a stored true flag. Read the normal sleep projection.
2. Disable Tasks, retain the same task, then enable Tasks through the
settings service without writing the task again.

## What Changed

- Read the effective Tasks flag inside the same read-only sleep
transaction. Bind that value to the existing pending-task query.
- Register the existing Tasks queue notification before the settings
transaction writes. The normal commit protocol owns the wake.
- Add two real-PostgreSQL regressions to the existing nine-case suite.
Extend eight mock transactions with the read projection they now need.
Keep their assertions unchanged.
- Publish the Tailscale enrollment fixture state by rename so its CLI
watcher cannot parse a truncated JSON file. Existing enrollment
assertions and time limits remain unchanged.

## Verification

The two regressions fail against the preceding current-master products:
the pending-work projection reports `none`, and re-enable does not rearm
the owner within the original two-second oracle. After the fix, all nine
PostgreSQL cases pass at the recorded source, including completed
history, disabled Tasks, credential debt and fail-closed query errors.

At current `72bc32f`, the normal DB build and migrations, all nine
PostgreSQL cases, 155 affected mock cases, and the complete normal
Server typecheck/planner pass. The mock run includes three selected
safety families and six complete files; it explicitly excludes 196
existing durable-PG cases. All original case deadlines and assertions
remain. The startup file's earlier 133-case pass is limited to its
unchanged mock contract; a new hot-restart implementation is not
relabeled as tested. Root independently verified all current outputs and
process closure. The MCP product and regression files remain
byte-identical at `321c4b2ae2f3`; its additional change is the Tailscale
test fixture repair below. Exact current-head CI passes: 55 successful
checks, four explicit skips and one neutral result; Snyk succeeds. Fresh
Greptile is 5/5 for this exact head. The complete latest edited summary
and all paginated threads were inspected, with no open findings.

An earlier additional whole-eight-file run reached its 180-second
aggregate watchdog while the existing mixed PostgreSQL suite was still
running. It produced no individual failing assertion. Its startup file
passed 133 cases. The original failure remains recorded. The follow-up
selects the three affected mock families and runs the other affected
mock files whole; it does not claim that the interrupted PostgreSQL file
passed. No test assertion or case deadline changed. Full required CI
remains a merge gate.

The first remote run at `72bc32f` passed 51 checks but failed one
Tailscale enrollment case and an interrupted browser shard; two
aggregate gates then failed. The browser log records a runner shutdown
signal; its underlying cause is unknown, and one specific-job recovery
was requested. The Tailscale case failed its status CLI on both original
CI attempts. A local control then held publication until the original
file watcher observed an empty state: the CLI threw `Unexpected end of
JSON input`, and the existing enrollment assertion failed. Publishing
the state atomically passed the same control. All nine original
enrollment cases now pass in 4.393 seconds with the fixture repair. The
recorded remote interleaving remains unknown; the original logs and the
initial local non-reproduction are retained. No retry count, assertion,
timeout or product behavior was weakened. The later exact-head run
passes all required checks. A subsequent server shard was interrupted by
a runner shutdown and later recovered at the same source; the
interruption and unknown underlying cause remain recorded.

Use Node24/pnpm9, one worker, an owned external PostgreSQL test database
and the existing fixture/admin URL options. The two new regressions are
in `server/src/__tests__/idle-sleep-diagnostics-postgres.test.ts`.

These checks do not establish deployed suspension or a live provider's
availability.

## Risks

- The read adds the normal effective-settings projection to the existing
bounded snapshot. Query errors must still block sleep.
- A settings update now notifies the existing task owner. Transaction
rollback must not publish that notification.
- This adds no schema, provider protocol, timer or replacement task
framework. Existing active task polling remains required until its work
settles.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI `gpt-6.1-sol` implemented the regressions and advised the fix.
GPT-6-based Codex also reviewed and integrated it. The host does not
expose every supervising model ID or context-window size. The agents
used source editing, Git, code execution and regression tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (the focused scope above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (the
existing pending-task contract remains accurate)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-14 05:00:50 -06:00
DottaandClaude Opus 5.5 aaec67b891 fix(native): close idle warm sessions in parallel across agents at shutdown (#21597)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - At shutdown, the server closes its idle warm Runner sessions before
it prepares the hot-restart snapshot
> - It closed them one at a time, and each full close (checkpoint, then
two stopped-copy collections) takes 3-5 s
> - In a zero-downtime swap test, five closes used the whole 20 s
pre-drain budget. Hot-restart preparation timed out, a checkpoint was
cut mid-way, and shutdown logged "hot-restart shutdown preparation
failed"
> - The same test logged 34 "Delivery reconciliation failed" errors, all
from directory cleanup retries that found the agent directory lock busy
> - This pull request closes idle warm sessions in parallel across
agents, one at a time within an agent, and treats a busy directory lock
as a quiet retry
> - The benefit is that shutdown preparation fits its budget with
several warm sessions, and expected lock contention no longer looks like
a failure

## Linked Issues or Issue Description

**What happened?**
`closeIdleWarmNativeSessions` awaited each warm session's full close in
turn. In swap 2 of a release-swap test, five idle sessions closed at
22:28:42, :46, :51, :56 and :59 (one "transport_close_requested"
diagnostic each). The `hot_restart_prepare` step then timed out at 19.9
s. Separately, `recoverDirectoryRelease` threw
`AgentDirectoryOperationBusyError` when another close held the agent
directory lock. The delivery coordinator logged each one at ERROR level.

I first checked if these sessions should have taken the cheap
released-session detach. They could not: `entry.released` is set only
after a successful `session.release()` through a release/resume-capable
plugin Runner task. The Runner state files from all sandboxes in the
test recorded only `run.prepare`, `session.open`, `turn.start`,
`runner.drain` and `runner.suspend` commands, and no `runner.release`.
The Daytona daemon logs show only `/process/execute` and
`/process/session` calls, and no Runner-task API. So no session in that
test was ever released, and a full close was the only correct shutdown
for them.

**Expected behavior**
Idle warm sessions of different agents close at the same time. Two
sessions of the same agent never close together, because they collide on
the agent directory advisory lock. A cleanup retry that finds that lock
busy stays pending until its claimed retry time and logs no error.

**Steps to reproduce**
1. Keep five or more idle warm sessions on remote sandboxes.
2. Send SIGTERM.
3. The closes run one after another and `hot_restart_prepare` times out.

**Paperclip version or commit**
`master` at bbed32450c.

## What Changed

- `closeIdleWarmNativeSessions` groups sessions by company and agent. Up
to four agents close at once, and each agent's sessions close one at a
time. The per-session close logic is unchanged.
- Idle-sleep retirement (`prepareIdleWarmNativeSessionsForSleep`) keeps
a concurrency of one, so it still checks drain authority before each
retirement.
- `recoverDirectoryRelease` catches `AgentDirectoryOperationBusyError`,
keeps the deferred retry time, and returns the pending entry instead of
throwing.

## Verification

- Red, then green (each failed against `master` source and passes with
the fix):
- `closes idle warm sessions of different agents in parallel and one at
a time within an agent` in
`server/src/services/native-runtime/native-session-executor.test.ts`.
Before the fix, the second agent's close was never started while the
first agent's close was pending.
- `keeps a cleanup pending without an error while another server holds
the agent directory lock` in
`server/src/__tests__/agent-directory-working-copies.test.ts`. It holds
the real Postgres advisory lock from a second connection. Before the
fix, the coordinator received an `AggregateError` with
`AgentDirectoryOperationBusyError`.
- `native-session-executor.test.ts` (1502 passed) and
`agent-directory-working-copies.test.ts` pass in full.
- `pnpm typecheck` in `server` passes.

## Risks

- Up to four warm closes now run at once, so shutdown makes more Runner
and database calls at the same time. The cap keeps this small.
- The order of entries in `failures` can change when several closes
fail.
- A busy release now waits for its deferred retry silently. A lock that
stays busy for a long time shows as a stuck pending cleanup, not as
repeated errors.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 04:15:19 -06:00
DottaandClaude Opus 5.5 12ac7cfd9f perf(recovery): read only the run keys that active execution recovery checks use (#21602)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Live recovery checks each running run when recovery work wakes it.
The check is `recoverActiveExecutions` in
`server/src/services/heartbeat/recovery.ts`.
> - Each check read the full `heartbeat_runs` row and the agent's
`adapter_config`. That includes `context_snapshot` and
`runner_profile_json`, which are large JSON columns.
> - Most checks only defer the run to its lease deadline. They use a few
JSON keys and never use the rest of the row.
> - Recovery wakes come from runner checkpoints, run events and company
work signals. Each running run is checked about 4.5 times per minute
under that load, so this read grows with the number of running runs.
> - This pull request reads only the JSON keys that the check gates use.
A run that recovery settles as lost reads its full context, runner
profile and adapter config before the settle.
> - The benefit is about 97% fewer bytes per check, with the same
recovery decisions.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The read that live recovery does for each running run
(`recoverActiveExecutions`).

**Subsystem affected**
Server: heartbeat run recovery
(`server/src/services/heartbeat/recovery.ts`).

**Current behavior**
Each check selects the whole `heartbeat_runs` row plus
`agents.adapter_config`. With a realistic row (`context_snapshot` 115 KB
of text, 23 KB stored; `runner_profile_json` 48 KB of text, 10 KB
stored), one check receives 175 KB from Postgres. A server with 20
running runs did 90 checks per minute, which is 15.7 MB per minute from
this one statement. That was 99% of all bytes in the measured window.

**Proposed behavior**
The check selects all scalar run columns, a small JSON object with only
the keys its gates read, and no output excerpts or adapter config. The
process-lost settle path reads the full `context_snapshot`,
`runner_profile_json` and `adapter_config` for that one run before it
decides on a retry.

**Reason and benefit**
The staging server spends 21-24% of main-thread CPU on postgres.js
result parsing at about 26 KB per statement. Whole-row reads in
recurring loops cause that overfetch. This change removes about 171 KB
from each check.

**Breaking changes**
None. No schema, API or contract change.

**Additional context**
I examined the `+1 s` re-arm (when revocation finds a child process that
the inventory snapshot did not have). I kept it. It only fires when the
revocation sees a changed row. The revocation also writes a new
60-second lease for a random boot ID, so the next pass defers on that
live lease. It does not loop on unchanged state. In the 60-second
measurement it did not fire.

## What Changed

- `recoverActiveExecutions` selects `activeRecoveryRunColumns`. This is
all `heartbeat_runs` columns, except that `context_snapshot` holds only
`wakeReason`, `issueId` and `nativeQuestionCancellation`,
`runner_profile_json` holds only `nativeStartupContext`,
`nativeWorkspaceSync` and `remoteWorkspace`, and the stdout/stderr
excerpts are null. It does not select `agents.adapter_config`.
- New `readActiveRecoverySettlement` reads the full `context_snapshot`,
`runner_profile_json` and `adapter_config` of one run. The process-lost
settle uses that data for the conversation-adapter check, the retry
budget, the process-loss diagnostic, the stop metadata and the
first-heartbeat flag.
- New tests in `server/src/services/heartbeat/recovery.test.ts`:
- A regression test counts the bytes Postgres sends during one deferred
check through a TCP proxy. It must be below 32 KB.
- A settle test (two cases) gives a lost run its retry budget,
first-heartbeat flag and conversation adapter only in the JSON keys
outside the narrow read. The test checks that the settle still uses
them.
- A test checks that a run with `nativeQuestionCancellation` in its
context is still skipped.

## Verification

- Red/green: the byte test fails on `origin/master` (`expected 454426 to
be less than 32768`) and passes with this change. The other three new
tests pass both before and after.
- Focused local runs pass: `recovery.test.ts`,
`live-recovery-work.test.ts`, `native-startup-cleanup-recovery.test.ts`,
`ownership-reconciliation-report.test.ts` (4 files, 255 tests).
- `pnpm verify` selected 2,715 of 3,541 files (conservative retention).
On this machine 1,689 files passed and 32 failed. The failures were
`afterAll` hook timeouts (30 s) and unrelated tool-access, email and
runner artifact suites under local load. The recovery suites above pass
when run alone. Required CI gives the full coverage.
- Before/after measurement (local embedded Postgres, a wire proxy that
assigns each server response to its statement, the real live-recovery
scheduler, 60 s window, 20 running runs: 10 native runs held by a live
controller and 10 legacy runs with a live peer lease; a run recovery
signal every 5 s for each run, a company work signal every 10 s, legacy
lease renewal every 20 s):

| | Before | After |
| --- | --- | --- |
| Checks per minute | 90 | 90 |
| Bytes per check (inventory statement) | 174,954 | 4,470 |
| Inventory bytes per minute | 15,745,819 | 402,298 |
| All bytes per minute in the window | 15,848,137 | 504,640 |

## Risks

- Low risk. The narrow read keeps the keys that the gates read. The
settle path reads the full row data before it uses it.
- The settle path now reads `context_snapshot`, `runner_profile_json`
and `adapter_config` after the lease revocation, not at the start of the
check. The values can be some milliseconds newer. The revocation already
uses the committed row state, so this agrees with the rest of the
settle.
- The settle path has one more query. It only runs when a run is settled
as lost.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as an agent in
Claude Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:58:47 -06:00
DottaandClaude Opus 5.5 30fc64e212 fix(instructions): collect a waiting working copy as soon as its stop proof is written (#21596)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run gets a private working copy of the agent's instructions or
agent files. A queue collects the copy after the provider is proven to
have stopped.
> - #21586 stopped this queue from re-reading copies forever: a copy
that waits for proof of the stop is now checked again after half of its
wait, capped at 6 hours.
> - With only that change, a copy whose stop is recorded T after its run
finished is collected up to T/2 later (at most 6 h). Before #21586 this
took at most 30 seconds. A sandbox that takes minutes to stop therefore
delays the save of an agent's instruction changes, and copies of
stranded runs wait up to 6 h after their stop is finally recorded.
> - This pull request makes the write that records the proof bring the
run's waiting copy forward, so the queue checks it on its next pass.
> - The benefit: collection is prompt again when the proof arrives,
without polling copies that are still waiting.

## Linked Issues or Issue Description

Follow-up to #21586. No public issue exists. Bug description:

**What happened?**
After #21586, a working copy waiting for proof of the provider stop
keeps its saved backoff deadline when the proof arrives. Proof is a
`native.local_process_stopped` event or a remote lease termination
receipt. A copy whose run finished 30 days ago is collected up to 6
hours after its stop is recorded.

**Expected behavior**
When a stop proof for a run is written, that run's waiting copy is
checked on the next queue pass. A write for a run without a waiting copy
costs one indexed no-op UPDATE and sends no work signal.

**Steps to reproduce**
1. Create a working copy for a native run that finished 30 days ago and
has no stop event, then let the collection queue check it once. Its next
check is about 6 h later.
2. Record the local stop (`recordNativeLocalProcessStopInTransaction`),
or release the run's remote lease with a termination receipt.
3. On master the copy stays in `pending_collection` until the old
deadline.

**Paperclip version or commit**
`origin/master` at 633dec7e43.

## What Changed

- New `notifyInstructionCopyStopEvidence(tx, { companyId, runId })` in
`instruction-collection-notifications.ts`.
- It reads the run's copy by its primary key (`run_id`, scoped by
`company_id`), and only if that copy waits in the collection queue or
the agent-files cleanup queue.
  - If there is no such copy, it stops: one SELECT, no write, no signal.
- If there is one, it first registers the run-scoped hint for both the
collection and the cleanup queue, and then runs one UPDATE by the
primary key. The UPDATE sets `next_attempt_at` to `now()` (or keeps an
earlier value) and sets a new `updated_at`. All of this happens in the
producer's transaction.
- The collection claim now compares `updated_at` with the value the
worker read (millisecond precision). If a stop proof is written between
the worker's evidence read and its backoff write, the claim fails and
the worker returns the copy as due, so the next pass collects it.
Without this fence, the worker could save a 6 h deadline over the proof.
- The writes that record stop proof call it in their own transaction:
- `native.local_process_stopped` receipts: `appendLocalProcessStop`
(used by `recordNativeLocalProcessStop` and its transaction form),
`recordUnidentifiedLocalRunnerStop`, and the operator confirmation in
`confirmUnidentifiedLocalRunnerStopped`.
- Remote lease termination receipts: `notifyEnvironmentCleanup` when it
has a run scope. Every lease release (`releaseLease`) and every
`writeEnvironmentCleanup` write calls it, which covers the receipt
writes in `environment-runtime`, `environment-task-runtime`,
`native-workspace-export-resume`, `sandbox-stop-and-retain` and
pending-cleanup recovery. The one remaining receipt write (sandbox
absence in `sandbox-stop-and-retain`) calls it directly.
- Orchestration's own stop path already calls `collectStopped` directly
after it joins the provider, so it collects without the queue and needs
no change.

## Verification

- `server/src/__tests__/agent-instruction-copy-retry-backoff.test.ts`, 3
new tests (embedded PostgreSQL, real queue registration):
- A local copy whose run finished 30 days ago (next check about 6 h
away) is collected within 10 seconds after
`recordNativeLocalProcessStopInTransaction` records the stop.
- A remote copy with the same backoff is collected within 10 seconds
after `environmentService.releaseLease` writes a termination receipt.
- For a run without a waiting copy, the transaction contains one SELECT
and nothing else, and no hint is sent. For a run with a waiting copy,
the only hint is `{"companyId":…,"runId":…}` (run-scoped, not
company-wide).
- A stop recorded between the worker's evidence read and its claim
(injected in the claim's transaction hook) is still collected on the
next pass.
- Red first: with the producer changes reverted, the two collection
tests fail (the copies stay in `pending_collection` until the old
deadline). Without the `updated_at` fence, the race test fails.
- Existing suites, one at a time: see the PR comment for the list and
results.
- `pnpm check:module-boundaries` passes.

## Risks

- Every lease cleanup write with a run scope now runs one extra
primary-key SELECT on `agent_instruction_working_copies`, and an UPDATE
only when a waiting copy exists.
- For agent-files cleanup (scratch files only, the saved data is not at
risk), the same race is not fenced. A missed advance there waits for the
cleanup backoff.
- A lease cleanup write that does not carry a termination receipt can
also bring a waiting copy forward. The queue then checks the stop
evidence once more (3–5 statements) and moves the copy back to its
backoff if the stop is still not proven.
- A release written with only a lease scope (no run) does not bring a
copy forward. That copy is still collected at its saved deadline.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:57:36 -06:00
DottaandClaude Opus 5.5 009c8884a4 perf(attention): read blocker retention state once per page (#21595)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `attention-retention` queue archives old attention items. One
kind is `blocker_attention`: a blocked task whose terminal blocker needs
a person.
> - When the queue loads a page of blocked tasks, it reads `keep` and
`archived_at` from `decision_retention` for each of them, one full-row
`select` per task.
> - A company hint walks every page. On a hosted instance with 2,112
blocker attention items and one active run, these per-task reads ran at
about 16 per second.
> - This pull request reads the retention state for the whole page in
one narrow statement. It also scopes three remaining run-path attention
hints to the task graph.
> - The benefit is one retention read per page instead of one per
blocked task, with the same decisions.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The `blocker_attention` eligibility read in
`server/src/services/attention-retention.ts` (`eligible()`), and three
run-path producers of company-wide attention hints.

**Subsystem affected**

Server background retention (`attention-retention.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-review-participant.ts`,
`native-runtime/native-workspace-reconstruction.ts`).

**Current behavior**

- For each blocked task whose blocker attention is terminal (and for the
owner variant), `eligible()` ran `select <all columns> from
decision_retention where company_id = $1 and source_kind = $2 and
source_id = $3`. It did this during `load`, once per task, on every
company hint.
- The hosted statement is identified by that column prefix: `id,
company_id, source_kind, source_id, source_activity_at, …`.
- I checked every other full-row reader of `decision_retention`. None
fires per run event:
- `syncItems`: attention summary and list requests, which a board calls
once per load; retention `check` and the sweep, which run for due
entries only; Keep and Archive actions.
  - `getState`: user actions only.
  - `archiveReviewedManifest`: proposal acceptance only.
  - `decisions.list`: only when archive proposals exist.
- `native-provider-capacity-retry`, `native-review-participant` and
`native-workspace-reconstruction` sent `{ companyId }` attention hints,
which make the queue walk every page.

**Proposed behavior**

- `pageEvidence` reads `source_id`, `keep` and `archived_at` for every
blocker root and every terminal blocker on the page in one statement
(ids passed as one JSON parameter). `eligible()` uses that map when it
has page evidence.
- A missing row means "not kept, not archived", exactly as the per-task
read returning no row did.
- The per-entry `check` path still reads one row, now only `keep` and
`archived_at`.
- The three producers send `notifyIssueAttentionRetentionWork` with
their task (and run). That is the task, its parents and the tasks it
blocks, with a company fallback past 100.
  - A capacity retry changes the failed-run items of its task.
  - A review participant claim changes the review of its task.
- A workspace reconstruction resolves the recovery action of its task,
which can also change the blocker attention of the tasks it blocks.
- The configuration, lifecycle, import and fallback producers keep their
company hints.

**Reason and benefit**

Background work must not grow with the backlog. A page now costs one
retention read instead of up to 100.

## What Changed

- `server/src/services/attention-retention.ts`: batched and narrowed
blocker retention read; `PageEvidence.blockerRetention`.
-
`server/src/services/native-runtime/native-provider-capacity-retry.ts`,
`native-review-participant.ts`, `native-workspace-reconstruction.ts`:
scoped attention hints.
- Tests in `server/src/__tests__/attention-retention-work.test.ts` and
`server/src/services/native-runtime/native-workspace-reconstruction.test.ts`.

## Verification

- `reads blocker retention state once per page, not once per blocked
task` (I saw it fail first). 160 blocked tasks, each behind its own
unassigned blocker; the blockers' retention rows are kept (40), archived
(40), plain (40) or missing (40). The full-size response capture and the
statement log measure the load of both pages:
- before: one `decision_retention` read per task (160), 26,240 result
bytes;
- after: 2 reads (one per page), 5,480 bytes, none of them selecting
`source_activity_at`.
  - The same tasks are listed: only plain and missing.
  - `check` still drops the kept and archived ones.
- `scopes the attention retention hint of a reconstruction to the task
graph` (I saw it fail first): before, the reconstruction sent `{
companyId }`; after, it sends a hint that names the task.
- `batches 100 blocker graphs with exact terminal timestamps and fresh
terminal visibility` pinned 200 retention reads per page (one per root
and variant). It now expects 1; the rest of that test is unchanged and
passes.
- Not separately tested: the capacity-retry and review-participant hint
changes. Each is one line that passes the task the function already
holds, the same pattern as the reconstruction test.
- `attention-retention-work` (58), `attention-retention-candidates`,
`native-workspace-reconstruction`, `native-review-participant` and
`heartbeat-run-dispatch-work` pass, each file run alone (in `server/`,
against dedicated local `_test` databases).
- `pnpm check:module-boundaries` passes.

## Risks

- The page read happens once, before the per-entry decisions of that
page. Before, each decision read its row just in time. A Keep or Archive
that lands between the page read and an entry's decision is seen by the
next load or by `check`, which re-reads under a row lock, as before.
- A run-path hint that changes more than its task graph would now miss
the wider effect. The 90-day archive delay makes a missed reload cheap.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:05:59 -06:00
DottaandClaude Opus 5.5 cf09024292 fix(heartbeat): count an early native hand-off only after its suspend commits (#21572)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each live Runner run to the next server, so
the agent keeps working without a retry
> - At shutdown start, the outgoing server hands off native runs early,
in parallel with the rest of shutdown
> - The early hand-off put a run in the "handed off" set before its
suspend finished, and the drain counted every run in that set as
suspended without waiting
> - The early hand-off shares the 20 s pre-drain budget, and one Runner
detach can take 20-30 s, so the process could exit with the suspend
unfinished
> - Then the finalization lease stays live, the next server gets no
hand-over signal, and the run waits for lease expiry or the reaper
> - This pull request makes the drain wait for each in-flight early
hand-off, and fall back to its own hand-off when the early one failed or
did not finish
> - The benefit is that a run counts as handed over only after its
hand-over is committed, so the next server can always reach it at once

## Linked Issues or Issue Description

**What happened?**
A passing zero-downtime swap test showed a latent race in the shutdown
hand-off. `handOffNativeRunsForShutdown` added a run to
`nativeRunsHandedOffEarly` before `suspendNativeRunForShutdown`
finished. `drainRunningRunsForShutdown` then reported every run in that
set as `restartSuspended` without awaiting it. Because the early
hand-off shares the 20 s pre-drain budget and a Runner detach can wait
20-30 s, the process can exit before the lease release commits. Also, a
timed-out `prepareHotRestartShutdown` could still write the hot-restart
snapshot and detach sessions after the fallback drain started. In the
same test, swap 1's early hand-off took 2.08 s, and no log showed which
phase was slow.

**Expected behavior**
A run counts as handed over only after its suspend commits. If the early
suspend fails or does not finish, the drain hands the run off itself. A
timed-out hot-restart preparation does no further work after the drain
starts. The hand-off logs its phase timings.

**Steps to reproduce**
1. Start a native Runner run on a remote sandbox.
2. Send SIGTERM while the Runner detach is slow (or fails).
3. The drain reports the run as restart-suspended while its suspend is
still pending, and the process can exit before the lease release.

**Paperclip version or commit**
`master` at bbed32450c.

**Deployment mode**
Hosted, zero-downtime release swap.

## What Changed

- `nativeRunsHandedOffEarly` now holds each run's in-flight suspend
promise, not only its id.
- The shutdown drain awaits that promise (bounded at 30 s). If it fails,
returns false or does not finish, the drain runs its own suspend, as it
does for a run without an early hand-off.
- `runBoundedShutdownStep` and the pre-drain budget pass an
`AbortSignal` to each step and abort it on timeout.
- `prepareHotRestartShutdown` takes that signal and stops before the
snapshot write, the run events, and the session detach when the step
timed out.
- `handOffNativeRunsForShutdown` logs one structured line, "native run
shutdown hand-off phase timings": intent read, controller identity,
select, max per-run detach, max per-run transaction, and total.

## Verification

- Red, then green (each failed against `master` source and passes with
the fix), all through the public shutdown functions in
`server/src/services/heartbeat/recovery.test.ts`:
- `does not count an early hand-off as suspended while its detach is
still pending at the shutdown drain`
- `hands off a run at the shutdown drain when its pending early hand-off
fails`
- `writes no hot-restart snapshot and detaches nothing after a timed-out
preparation hands over to the drain`
- New test in `server/src/shutdown.test.ts`: `aborts the signal of a
step that outlives its bound, and only that step`.
- `pnpm vitest run src/shutdown.test.ts
src/services/heartbeat/recovery.test.ts
src/__tests__/heartbeat-native-early-hand-off.test.ts` passes (145
tests).
- `tsc --noEmit` for `server` passes. `pnpm verify` passes for the
selected tests.

## Risks

- When an early hand-off does not finish within 30 s, the drain starts a
second suspend. The second detach joins the first, because the session
memoizes its restart detach. The run can then get two "suspended" run
events. That is preferred to reporting a hand-over that did not happen.
- The drain can now wait up to 30 s for a slow early hand-off before the
process exits. Before, it exited without the hand-over.
- The abort signal only stops work between steps of the preparation. A
snapshot write that already started still completes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:02:25 -06:00
DottaandClaude Opus 5.5 633dec7e43 fix(instructions): back off working copies that cannot make progress (#21586)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run gets a private working copy of the agent's instructions or
agent files. Background queues collect the copy after the provider stops
and clean up its scratch directory.
> - When the queue cannot prove that the provider stopped, or cannot
finish a cleanup, it re-arms the same copy every 30 seconds with no
limit. The attempt count does not increase on the "stop not proven"
branch.
> - Every re-arm is a transaction that sends work signals to its own
queue, to the stranded-agent-status queue and to the skill-retention
queue. A self-signal also reloads the copy's scope. On a hosted instance
with only a few live runs, these copies were one of the largest sources
of SQL calls and kept the main thread busy.
> - This pull request makes a copy that cannot progress wait on a
growing, capped backoff. It also removes the work signals from a write
that only moves a deadline later, and reads the copy once per check.
> - The benefit: a copy that waits for proof of the provider stop costs
about 1/350 of the old steady-state load, and a copy that becomes
collectable is still collected automatically.

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened?**
A hosted instance had only a handful of live agent runs, but its server
main thread was about 95% busy. An analysis of its most-called SQL found
the instruction working-copy queues re-reading the same unresolved
copies forever, at about 25–30 statements/s. A copy whose run is
terminal but whose provider stop is not proven was re-armed every 30 s.
A completed agent-files copy whose remote scratch cleanup cannot finish
(its lease is released but not destroyed, or the lease row is gone) was
also re-armed every 30 s. Each re-arm was a transaction with work
signals that woke three queues.

**Expected behavior**
A copy that cannot progress is not re-read until a growing, capped
backoff elapses. A write that only moves a deadline later does not wake
other queues. A copy that becomes collectable is still collected
automatically, and no data is lost.

**Steps to reproduce**
1. Create working copies of terminal native runs that have no
`native.local_process_stopped` event, and completed agent-files copies
with `cleanupPending: true` whose remote lease does not exist.
2. Register the instruction-collection and directory-cleanup queues.
3. Count statements while time advances. With 300 + 300 such copies,
master sends 252 statements/s, 38.7 transactions/s and 38.7 work
signals/s.

**Paperclip version or commit**
`origin/master` at 3c3e6c2d75.

## What Changed

- `instructionCopyRetryAt(waitingSince)` sets the next check to half of
the time the copy has waited. The minimum is `INSTRUCTION_COPY_RETRY_MS`
(30 s) and the maximum is `INSTRUCTION_COPY_MAX_RETRY_MS` (6 h). An
unproven stop measures the wait from the run's `finished_at`. A cleanup
measures it from `process_stopped_at`. The deadline stays in the
existing `next_attempt_at` column, so a restart keeps it. There is no
schema change.
- The collection check now reads the copy once, checks the stop
evidence, and then claims. For an unproven stop, the claim writes the
backoff deadline (and the move from `prepared` to `pending_collection`)
in the same UPDATE. Before, the check was two reads plus two signalling
transactions.
- `tryPatch` skips all work signals when a write only moves deadlines
later and keeps the copy in the same queues (collection, cleanup, commit
and the stranded-agent predicate). A write that changes queue membership
or moves a deadline earlier still signals as before. The write stays in
its own transaction, so the existing claim and race tests keep their
hook.
- A cleanup that cannot finish gets the same capped backoff after
`release` re-arms it. The saved copy and its cleanup receipt are kept.
The claim does not change `attempts`, because a concurrent foreground
collection uses `attempts` in its compare-and-set.
- A remote agent-files copy that stopped but was not destroyed, and an
unavailable agent-files copy that waits for a destroy receipt, also get
the capped backoff instead of a 30 s re-arm.
- No copy is parked or dropped. A copy with an unproven stop stays in
the queue with the 6 h cap. When proof of the stop arrives later (a
`native.local_process_stopped` event, a lease termination receipt, or a
recovery that records that the host is gone), the next backoff check
collects it automatically. No manual call is necessary.

## Verification

- New suite
`server/src/__tests__/agent-instruction-copy-retry-backoff.test.ts`
(embedded PostgreSQL, real queue registration, statement meter):
- The backoff constants and their values at 0 s, 30 min, 2 h, 12 h and
30 days.
- A copy that has waited 2 days: two checks in 12 h, 5 statements and 1
transaction per check, and zero work signals.
- A copy owned by a live older server (run finished 5 minutes ago): it
is checked every few minutes. When the older server records the stop,
the copy is collected within 10 minutes, without a manual call.
- A copy that has waited 30 days: it is collected within one 6 h cap
after the stop is recorded.
- A backlog of 300 copies that have waited 30 days: one pass drains it
with zero work signals, and the hour after it is silent.
- A write that moves the deadline earlier, and a write that changes
queue membership, both still send the signal.
- A cleanup that cannot finish, with release returning and with release
throwing: at most 32 deadline writes in the first 6 h, at most 4 in the
next 6 h, zero work signals, and the copy is still saved with its
cleanup pending. Before the fix, the throwing case made 74 writes in 6
h.
- Red first: on master this suite fails 5 of 7 tests (the 30 s re-arm,
150 statements in the drain window, extra signals, a cleanup checked
every 30 s). The two "collects after stop" tests pass on master and
still pass: the positive behavior is kept.
- Steady state with 300 unproven stops plus 300 stuck cleanups, each
waiting for 1 day (10-minute window after warm-up):

| | statements/s | transactions/s | work signals/s | stop checks/s |
cleanup checks/s |
  | --- | --- | --- | --- | --- | --- |
  | master | 251.6 | 38.7 | 38.7 | 9.7 | 9.6 |
  | this PR | 0.45 | 0.10 | 0 | 0 | 0.03 |

- Per check: an unproven stop goes from 12 statements, 2 transactions
and 2 work signals to 5 statements, 1 transaction and 0 work signals. A
stuck cleanup goes from about 14 statements, 2 transactions and 2 work
signals to about 15 statements, 3 transactions and 0 work signals.
Cleanup checks are exponentially less frequent.
- Backlog drain: queue batches stay at 20 per pass, with one pass per
second at most. An overdue copy moves to its long backoff on its first
check. The first drain of N stuck copies therefore takes about N/20
seconds. For example, 17,000 copies drain in about 15 minutes with no
work signals.
- Existing suites, one at a time: `agent-instruction-working-copies`
(50/50), `agent-directory-working-copies` (120/120),
`heartbeat-instruction-working-copy` (4/4),
`heartbeat-agent-directory-busy` (8/8), `agent-file-checkpoint-locking`
(9 passed, 1 skipped), `agent-file-durability` (17/17),
`idle-sleep-safety` (226/226), `heartbeat-process-recovery` (459/459),
`agent-instructions-routes` (26/26), `services/heartbeat/agent-status`
(42/42).
- `pnpm check:module-boundaries` passes.

## Risks

- Product-visible choice: the 6 h cap. A copy that has waited for a long
time is collected at most 6 h after its stop proof arrives (half of the
wait while the wait is shorter than 12 h). During that time, the
uncollected candidate shows in the agent's instruction candidates
(`pending_collection`, `INSTRUCTION_STOP_UNCONFIRMED`) for legacy
copies, and in the run's agent-files diagnostics for agent-files copies.
Its bytes are kept on disk. An explicit `collectStopped` from
orchestration still collects immediately.
- There is no seven-day park. A park at `attempts = 3` would leave the
queue predicate, and a later stop proof would not reach the copy without
changes in files that another fix owns. The copy therefore stays in the
queue with the 6 h cap. The steady cost is about 5 statements per copy
every 6 h.
- During the 40-minute deploy overlap, a copy owned by a live older
server waited only a short time, so its backoff is still 30 s to a few
minutes. The test covers this case.
- A write that only moves a deadline later no longer wakes the
stranded-agent-status and skill-retention queues. The stranded-agent
predicate is compared before and after the write, and any change still
signals.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:47:06 -06:00
DottaandClaude Opus 5.5 c3f9a0c040 fix(retention): evaluate native retention blockers for many sessions in one statement (#21588)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep runner checkpoints and harness backups so that a
session can resume. Retention expires them after the session can no
longer resume.
> - A retention blocker decides whether a session can still resume: a
live or recent run, an open or recently closed task, an unsettled
finalization, or a live lease.
> - The startup checkpoint pass evaluated that blocker in its own
transaction, once for each retained checkpoint, with three reads. A
session that stays blocked (for example, a long-lived open task) was
evaluated again for each of its runs on every server start. On a hosted
instance with frequent deploys and a 40-minute overlap of two servers,
this pass ran almost continuously at about 7 checkpoints/s, about 21
statements/s.
> - Backup reconciliation also used four reads and a transaction for
each run hint that reached a backup.
> - This pull request evaluates the blocker for many sessions in one
fixed-text statement. Only candidates that are not blocked enter the
existing locked transaction.
> - The benefit: a restart costs 2 statements for each page of 200
blocked checkpoints instead of 5 statements and 1 transaction for each
checkpoint, and a backup hint costs 2 statements with no transaction.

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened?**
A hosted instance had only a handful of live agent runs, but its server
main thread was about 95% busy. An analysis of its most-called SQL found
three retention-blocker reads (`company_id, agent_id from
heartbeat_runs`; the session's runs; the session's issues) at exactly
7.0/s each, each in its own transaction. The startup checkpoint pass
sent them once for each retained checkpoint older than 7 days whose
session was still blocked, and it repeated the full pass after every
start.

**Expected behavior**
A checkpoint or backup that is blocked from expiring costs no
transaction and no statements of its own, because one statement
evaluates its session together with the rest of its page. It expires as
soon as its blocker clears: checkpoints on the next startup pass (the
current design), and backups at their exact deadline.

**Steps to reproduce**
1. Create 100 native sessions with 3 finished runs each (finished 10
days ago), settled finalizations, a checkpoint header for each run, and
an open task.
2. Register the startup checkpoint expiration and count statements. Then
register it again, as a restart.
3. On master each start costs 1,502 statements and 300 transactions, and
nothing expires.

**Paperclip version or commit**
`origin/master` at 3c3e6c2d75.

## What Changed

- `SESSION_RETENTION_BLOCKERS_SQL` (new module
`native-runtime/native-session-retention-blockers.ts`) is one fixed-text
statement, sent through `$client.unsafe`. It takes a JSON list of
sessions and returns, for each one, the same blocker reasons as
`sessionRetentionBlocker`, in the same order: missing run, live run,
incomplete native recovery, recent run (checkpoint mode), unknown, open,
executing or recently closed task, pending finalization, live lease. It
also returns the session's resume pointer, run, task and lease
dependencies, and the latest close time.
- `expireCheckpoints` and `expireDurableBackups` evaluate each page of
candidates with this statement. They skip blocked candidates and
candidates whose session still has a resume pointer. A candidate that is
not blocked still enters the existing transaction. That transaction
checks the blocker again under row locks before it retires anything.
- `reconcileBackup` uses the statement for its dependencies and its
deadline. It replaces three dependency reads plus a blocker transaction.
The lease references use one index-backed arm for each reference kind
(run, backup stamp, task, identity lease).
- The unused `lock: false` path of `sessionRetentionBlocker` and the
`backupDependencies` helper are removed.
- The existing index test now checks the plan of the new statement.
Every arm that reads `environment_leases` uses an index in both the
custom and the generic plan.

## Verification

- New suite
`server/src/__tests__/native-retention-blocker-batch.test.ts` (embedded
PostgreSQL, real coordinator registration, statement meter):
- 300 blocked checkpoints: a start costs 4 statements and 0
transactions, and a restart costs the same. On master it is 1,502
statements and 300 transactions for each start. The checkpoints are
kept.
- When a session's task is closed long enough ago, its checkpoints
expire on the next startup pass. The other sessions are not touched.
- A session with a resume pointer keeps its checkpoints for 2 statements
in total.
- Two servers that run the startup pass at the same time expire every
checkpoint exactly once with no failures, because the locked transaction
checks again and the retirement header is compared before deletion.
- 50 run hints that reach 50 blocked backups: at most 2 statements for
each hint (the scope lookup and one blocker statement) and no
transaction. On master it is 9 statements and 1 transaction for each
hint. The backups are kept.
- Existing suites, one at a time: `native-harness-backup-expiration`
(109/109, including the backup expiry flows for task close, reopen and
lease release), `native-harness-backup-expiration-config` (7/7),
`native-harness-backup-storage.integration` (73/73),
`native-harness-backup-publication` (4/4),
`native-retired-backup-source.integration` (38/38).
- `pnpm check:module-boundaries` passes.

## Risks

- The startup pass still runs on every server start. It also runs on
both servers during the deploy overlap. Its new cost is 2 statements for
each page of 200 candidates (the candidate page and one blocker
statement). A page of candidates that are not blocked still uses one
locked transaction for each candidate, as before.
- Two servers can run the pass at the same time. They cannot retire the
same session twice or retire a blocked one: the unlocked statement only
filters, and the existing transaction locks the finalizations, checks
the blocker again, and compares the retirement header before it deletes.
The concurrency test covers this.
- If the SQL blocker and the TypeScript blocker ever disagree in the
"not blocked" direction, the locked transaction still refuses. If they
disagree in the "blocked" direction, retention is only delayed. When
more than one task blocks a session, the reported reason can differ from
the TypeScript loop, which reads the tasks in no fixed order.
- Checkpoints keep their current design: they are considered again on
the next startup, not at a deadline. Backups keep their exact deadline
after a task closes.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:47:01 -06:00
DottaandClaude Opus 5.5 f452b449fd fix(recovery): stop re-checking settled native runs that cannot be released (#21590)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server releases the process identity (`process_pid`,
`process_group_id`, `process_started_at`) of a finished native run when
it can prove that the runner process stopped (#17374)
> - A background queue (`settledNativeProcesses`) finds these runs and
checks them. A check that cannot finish ("skipped" or "pending") writes
nothing, so the run stays in the queue and every recovery hint checks it
again
> - A finished run that a replaced container recorded can never pass the
check, because only the host that recorded the process can prove that it
stopped. Each deploy adds more of these runs
> - On one hosted instance this loop ran about 10 checks per second and
about 55 SQL statements per second, and the server main thread was about
95% busy with only a few live runs
> - This pull request moves each "cannot make progress" reason into the
queue's SQL predicate, adds a growing backoff for "pending" runs, and
makes one check cost one locking statement
> - The benefit is that runs which cannot progress cost nothing until
something that can change the answer happens, and a run that becomes
releasable is still released at once

## Linked Issues or Issue Description

No public issue exists. Bug report:

**What happened?**
The settled native process recovery queue checks the same finished
native runs again on every recovery hint and every 30 seconds, forever.
With 300 such runs and one company-wide hint per second it ran 9 checks
per second and 84 SQL statements per second (9.4 statements per check,
2.2 ms of server CPU per check) on embedded PostgreSQL. A hosted
instance showed the same rates. The runs that never finish are: runs
recorded by a host that is gone (each redeploy starts a new container
with a new hostname), runs whose coordinator phase is not settled, runs
with no checkpoint and no provider evidence, and runs whose local
process is still alive (they defer 30 seconds with no limit). Startup
discovery (`discoverSettledNativeRunProcesses`) also opens one
transaction for every such run on every boot.

**Expected behavior**
A run that cannot make progress is not read again until something that
can change the answer happens. A run that becomes releasable (its phase
settles, provider evidence arrives, its own host checks it, its sandbox
receipt arrives, its process exits) is released promptly.

**Steps to reproduce**
1. Create a failed native run with a recorded local process identity
from another host (or an unsettled coordinator phase, or no checkpoint
and no provider evidence, or a live process).
2. Register the residual recovery work and send company-wide
`notifyLiveRunRecovery` hints.
3. Count the run's `for update` checks: on `master` every hint checks
the run again.

`server/src/__tests__/settled-native-process-rechecks.test.ts` automates
these steps.

**Paperclip version or commit**
`master` at 3c3e6c2d75.

**Deployment mode**
Any. The cost grows with every deploy of a hosted instance.

## What Changed

- `server/src/services/native-local-process-stop.ts`
- New `settledNativeProcessCandidates`: one fixed-text statement that
finds releasable runs. It reads only runs recorded in this server's PID
space (`process_host_hash`), sandbox runs (`remote`, with a non-local
lease), and rows saved before the hash (their latest identity event must
name this host). It excludes runs with an unsettled coordinator phase
and runs with no checkpoint and no provider evidence. Each PID-space
branch reads its own range of the new partial index.
- `releaseSettledNativeRunProcess` now reads the run, phase, evidence,
lease facts and identity match in one `for update` statement (it was 6
statements). It reads the lease rows again only for a sandbox run. The
new `inspectSettledNativeRunProcess` also returns whether a remote stop
acknowledgement can apply.
- Startup discovery uses the same candidate statement, so it opens no
transaction for an excluded run.
- `server/src/services/heartbeat/live-recovery-work.ts` (settled kind
only)
- The load runs the candidate statement and a separate indexed statement
for cancelled `cursor_cloud_api` runs whose Stop is still `requested`.
- The check uses the loaded entry. It no longer runs the load again for
each entry.
- A "pending" run backs off 30 s, 60 s, 120 s, up to 15 min. A
run-scoped hint (identity event, lease cleanup, stop receipt) resets the
backoff.
- A "skipped" run that the predicate cannot exclude (for example a birth
time mismatch on this host) waits for a run-scoped hint or a restart.
- `server/src/services/heartbeat/recovery.ts` (the settled handler
only): returns the outcome. It calls `acknowledgeRemoteStop` only when
the run has a non-local lease or is a cancelled `cursor_cloud_api` run.
For any other run that function reads two rows and writes nothing.
- `packages/db`: migration 0487 builds
`heartbeat_runs_settled_process_identity_idx` and
`heartbeat_runs_pending_api_stop_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. The migration is `online_compatible` and is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. `doc/DATABASE.md` describes both
indexes.

End state for each reason that a check cannot finish:

| Reason | Before | After | What re-admits the run |
| --- | --- | --- | --- |
| Recorded by another PID space | checked on every hint | not read |
only a server in that PID space reads it (a live older server in a
deploy overlap still releases its own runs) |
| Coordinator phase not settled | checked on every hint | not read | the
phase write; the finalizer sends a company hint in the same transaction
|
| No checkpoint and no provider evidence | checked on every hint | not
read | the next hint after a provider or turn event |
| Live local process, unreleased lease, missing sandbox receipt | every
hint and every 30 s | backoff 30 s to 15 min | a run-scoped hint resets
the backoff |
| Other same-host skip | checked on every hint | parked | a run-scoped
hint (an identity event sends one) or a restart |

## Verification

- `pnpm exec vitest run
src/__tests__/settled-native-process-rechecks.test.ts` (new, 12 tests,
real PostgreSQL, real queue and coordinator, fake clock):
- each exclusion: replaced container, unsettled phase, no evidence: 0
checks over 4 hints (red on `master`: 3 to 5 checks);
- a live older server and a stopped older server (different hash): the
new server never reads the run and writes no receipt; this server's own
run is released at start;
- re-admission: phase settles, late provider evidence, legacy rows with
no hash (this host released, another host not read), sandbox run
released from any server with its receipt;
- live process: at most 4 checks over 8 hints (red: 17), released on the
run-scoped hint after the process exits;
- one check is BEGIN, one `for update` statement and COMMIT (3
statements); the next hints do not check it; a run-scoped hint checks it
once more (red: 5 statements and a check on every hint);
- a restart costs one check for each pending run; startup discovery with
50 excluded runs sends 1 statement (red: 302).
  - On `master` 11 of the 12 tests fail.
- Measurement on embedded PostgreSQL, 300 stuck runs, one company hint
per second for 300 s (fake clock), same harness before and after:
- before: 9.0 checks/s, 84.3 statements/s, 9.4 statements per check, 5.9
s CPU (2.2 ms per check);
- after: 0 checks/s, 3.2 statements/s for all seven residual queues
together, 0.16 s CPU.
- Load plan with 12,000 excluded finished runs (10% without a hash) in a
31,000-row `heartbeat_runs`: the candidate statement reads only index
entries for this server's hash, `remote` and `unknown`, and a bitmap
scan of the 1,200 legacy rows: 2.7 ms, 4,416 buffers. Without the index
the same exclusion read all 31,000 rows (31,127 buffers).
- Existing suites, one file at a time: all pass.
`live-recovery-work.test.ts` 51/51 (an earlier run under machine load
hit the `afterAll` hook timeout after all tests passed; the rerun passed
clean, and `native-safe-replacement.test.ts` hits the same hook timeout
on the base commit), `native-settled-process-release.test.ts` 31/31,
`native-local-process-stop.test.ts` 5/5, `heartbeat/recovery.test.ts`
107/107, `native-runner-restart-recovery.integration.test.ts` 231 passed
/ 14 skipped, `native-restart-recovery.test.ts` 49/49,
`native-controller-recovery.integration.test.ts` 30/30,
`retained-native-session-cleanup-work.test.ts` 29/29,
`native-provider-failure.integration.test.ts` 16/16,
`native-safe-replacement.test.ts` 70/70 (with the `afterAll` hook
timeout that also occurs on the base commit),
`native-remote-retry-admission.test.ts` 93/93.
- `pnpm check:module-boundaries`, `pnpm check:migrations` (packages/db),
server typecheck: pass.
- Not run: the full suite and browser suites.

## Risks

- Unresolved by design: a finished run whose server was replaced keeps
its process fields indefinitely and is never released, because only the
host that recorded a process can prove that it stopped, and the shared
prover refuses a controller-lease proof for a run with a process group.
This pull request only stops paying for it. These runs keep no stop
receipt (`native.local_process_stopped`), so what waits for that receipt
still waits: accounting close of the run's usage source, retained native
session cleanup, and instruction or directory working copies that wait
for proof that the run stopped. A follow-up must decide the ownership
rule for a replaced container.
- Rows saved before `process_host_hash` exist are read through the
legacy branch, which checks the latest identity event for each of them
on each load. On an instance with many such rows the load costs one
index lookup per row. The count can be measured with `select count(*)
from heartbeat_runs where process_host_hash is null and process_pid is
not null and status in
('failed','cancelled','timed_out','interrupted')`.
- A run that this server recorded but whose `process_host_hash` changed
(a new PID namespace on the same host, for example a container restart)
is no longer read here. Before, it was read and skipped, so no release
is lost.
- The backoff and parked state live in memory. A restart checks each
pending run once.
- The migration builds two partial indexes concurrently on
`heartbeat_runs`. It holds no long lock.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:29:23 -06:00
DottaandClaude Opus 5.5 e8c79235ca perf(attention): scope run lifecycle retention hints and read only used columns (#21579)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `attention-retention` queue archives old attention items (failed
runs, reviews, questions, recovery actions, blocked tasks). A company
hint makes it reload every candidate of the company, 100 per page, with
eligibility reads for each page.
> - Several run lifecycle producers sent that company hint: every agent
status change after a run, the execution-control writes and legacy run
terminalization when they name only a run, scheduled retries, and the
Runner's task monitor tool.
> - Each page also read whole rows it did not use: full `heartbeat_runs`
rows (recent ones carry tens of KB of `context_snapshot` and
`runner_profile_json`), full `issues` rows (descriptions), and the whole
agent roster with its configuration.
> - This pull request names the task graph or the source kind in those
hints, and reads only the columns each eligibility decision uses.
> - The benefit is fewer company reloads per run event and far fewer
result bytes per reconstruction, with the same decisions.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`. Stacked on #21552 (it builds
on that pull request's change to `execution-control-notifications.ts`);
review that one first.

**What existing behavior does this improve?**

The attention retention queue
(`server/src/services/attention-retention.ts`) and the producers that
wake it on run lifecycle paths.

**Subsystem affected**

Server background retention (`attention-retention.ts`,
`retention-work-notifications.ts`, `execution-control-notifications.ts`,
`heartbeat/agent-status.ts`, `heartbeat/retries.ts`,
`native-runtime/paperclip-runner-tool-authority.ts`).

**Current behavior**

- `heartbeat/agent-status.ts` sends `{ companyId }` on every agent
status change after a run (two places). `heartbeat/retries.ts` sends it
for each scheduled retry. `writeExecutionControl` and
`notifyExecutionRunWork` send it when the scope names only a run. The
Runner's `set_task_monitor` tool sends it for one task.
- On a local fixture with 1,500 open tasks and run churn, the queue
loaded 47 company candidate rows per second from these hints.
- `pageEvidence` read `select *` from `heartbeat_runs`,
`issue_recovery_actions` and (through `getTableColumns`) `issues`;
`reviewEligibility` read `select *` from `issues`;
`interactionEligibility` read the full agent roster; the per-entry
`check` repeated full-row reads.

**Proposed behavior**

- Agent status changes send `{ companyId, sourceKind:
"agent_error_alert" }`. The status moves between `running`, `idle` and
`error`. All three are invokable for blocker attention
(`BLOCKER_ATTENTION_INVOKABLE_AGENT_STATUSES`) and for questions
(`agent-eligibility.ts`). Failed-run attention only excludes terminated
agents, and this path never sets that. So only agent error alerts can
change.
- Run-scoped execution-control writes, legacy run terminalization and
scheduled retries use `notifyIssueAttentionRetentionWork` with the run
id (the run's task, its parents and the tasks it blocks; company
fallback past 100). This is the same helper as #21521. The task monitor
tool scopes to its task.
- `notifyIssueAttentionRetentionWork` matched the run's saved task id
with `issue.id::text = …`. It now validates and casts the id, so the
primary key is used, as #21552 does for recovery.
- Narrow reads, same decisions:
- runs: `id`, `agent_id`, `created_at` and the saved task id computed in
SQL with the same rules as `readRunIssueId` (`issueId` unless it is
absent or JSON null, then `taskId`; only non-empty strings).
  - recovery actions: `id`, `source_issue_id`.
- issues for evidence and `visibleIssue`: the blocker-attention input
columns plus `unblock_descriptor` and `blocked_transition_at`.
- review issues: `id`, `company_id`, `status`, `execution_state`,
`assignee_user_id`.
- agent roster for question addressees: `id`, `company_id`, `name`,
`reports_to`, `status` (the `AgentOrgRow` that
`evaluateAgentInvokability` takes).

**Reason and benefit**

A run event no longer reloads every attention candidate of the company.
A reconstruction no longer transfers whole run, task and agent rows.

## What Changed

- `server/src/services/attention-retention.ts`: column lists for runs,
recovery actions, issues and the roster; the run probe type replaces
whole rows.
- `server/src/services/heartbeat/agent-status.ts`,
`heartbeat/retries.ts`, `execution-control-notifications.ts`,
`native-runtime/paperclip-runner-tool-authority.ts`: scoped attention
hints.
- `server/src/services/retention-work-notifications.ts`: indexed lookup
of a run's saved task.
- `server/src/__tests__/attention-retention-work.test.ts`: two tests.

## Verification

- `reads only the fields its decisions use from large runs, reviews and
agents` (I saw it fail first): 50 failed runs, 50 tasks in review with
large descriptions, 10 questions to a paused agent, about 40 KB of
payload in each run's snapshot, profile and stdout and in the agents'
configuration. It loads the three kinds with the full-size response
capture (`full-database-response-capture.ts`). Before: `heartbeat_runs`
6,317,700 bytes, `agents` 84,336 bytes, `issues` 4,235,810 bytes. After:
under the test bounds of 20,000, 5,000 and 60,000 bytes. The decisions
are the same before and after: 50 failed runs, 40 reviews and 10
questions.
- `scopes run lifecycle attention hints to the run's task graph and
agent status to agent alerts` (I saw it fail first):
`writeExecutionControl` and `notifyExecutionRunWork` with a run id send
the hint for the task and its parent. `finalizeAgentStatus` sends the
agent-alert hint. Before, all three sent `{ companyId }`.
- `attention-retention-work` (57 tests),
`attention-retention-candidates`, `attention-service`,
`execution-control-work` and `heartbeat-agent-status-recovery` pass,
each file run alone (in `server/`, against a dedicated local `_test`
database).
- `pnpm check:module-boundaries` passes.

Not done in this pull request, and why:

- I did not reproduce `syncItems` re-syncing unchanged rows from the
retention `check`. On my fixture (300 stalled blocked tasks, 4,500 old
recovery-action and blocker retention rows, run churn), `syncItems` ran
0 times in a 180-second window and `decision_retention` reads were 0 per
second. `check` calls `syncItems` only for entries that are already due
(older than the 90-day archive delay), and a due entry is archived and
leaves the candidates. So I did not batch `syncItems` or skip checks of
unchanged rows.
- `readIssues` still selects the `issues` columns that blocker attention
needs. I did not narrow `issueService.listBlockerAttentionBatch`.

## Risks

- A producer that changes more than the run's task graph would now miss
the wider effect. Company-level changes (agent configuration, company
status, budgets) keep their company hints. The 90-day archive delay
makes a missed reload cheap: the item is reloaded on the next hint that
covers it.
- The saved task id now comes from SQL. The SQL mirrors `readRunIssueId`
for strings, absent keys and JSON null. A non-string `issueId` value now
yields no task, as before.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:17:21 -06:00
DottaandClaude Opus 5.5 a1c37b5cc4 perf(chat): rebuild GitHub response sweeps for company-wide hints at most every 30 seconds (#21580)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel has two background sweeps for request
response comments: acknowledgements, and terminal responses. Each sweep
rebuilds its candidate set when a hint arrives, and it reads the latest
run of each candidate request (`readGitHubResponseRun`) on each rebuild
> - A run milestone for a run with no task sends a company-wide hint to
both sweeps. Examples: each status change of a run without a task
(`setRunStatus`), and a comment merged into an active run in the wake
queue. In a busy company these hints arrive more than once a second
> - So both sweeps rebuild the whole company about once a second (their
minimum pass interval). Each rebuild reads every parked request again,
although nothing about those requests changed. On a hosted test
database, `readGitHubResponseRun` ran 6.7 times a second
> - This pull request rebuilds a GitHub response sweep for a
company-wide hint at most once every 30 seconds. Issue, request,
conversation, endpoint and lease hints still rebuild immediately.
`external-objects.ts` uses the same rule for its refresh sweep
> - The benefit is that a steady stream of company-wide hints no longer
makes each sweep read every parked request every second

## Linked Issues or Issue Description

Refs #21577 (makes each `readGitHubResponseRun` call cheap; this pull
request reduces how often the sweeps call it). Refs #18384 (the same
reconstruction interval for company-wide external-object hints).

**What existing behavior does this improve?**
The candidate rebuild of `registerGitHubAcknowledgementWork`
(`acknowledgement-work.ts`) and `registerGitHubTerminalResponseWork`
(`terminal-response-work.ts`) for company-wide hints.

**Subsystem affected**
GitHub chat channel: request acknowledgement and terminal-response
recovery sweeps.

**Current behavior**
`notifyChatRunMilestones` calls `notifyChatResponseAvailability` with
its scope. When the scope has no issue (only a company, or a company and
a run), the hint for both GitHub response queues is company-wide
(`{"companyId": ...}`), because the response scope has no run field.
Each company-wide hint makes each sweep load every candidate of the
company and call `readGitHubResponseRun` for each one. Passes are
limited to one a second, so with a steady hint stream the cost is about
(parked candidates) reads a second.

**Proposed behavior**
Both sweeps set `reconstructionIntervalMs`. A company-wide scope
(company ID only) waits 30 seconds after its last rebuild before it
rebuilds again. All narrower scopes, the startup scope (`*`), and saved
deadlines are not changed. Entries that are already loaded stay
available for dispatch while a company-wide hint waits, and each check
still reads the request again from the database.

**Reason and benefit**
On the hosted test database, `readGitHubResponseRun` ran 6.7 times a
second for no change in state. In a local reproduction, the call rate
drops from about one call for each parked request each second to one
call for each parked request each 30 seconds.

**Breaking changes**
None.

## What Changed

- Add `response-sweep-scope.ts` with the shared scope type of the two
GitHub response sweeps and `githubResponseReconstructionIntervalMs`: 30
seconds for a company-wide scope, 0 for all other scopes.
- Use it as `reconstructionIntervalMs` in the acknowledgement sweep and
the terminal-response sweep. Both files now import the shared scope type
instead of declaring it.
- Add `response-sweep-scope.test.ts`. It registers both real sweeps on a
real database with 6 parked requests (3 for each sweep) and a controlled
clock. It sends company-wide hints and issue hints, and counts
`readGitHubResponseRun` statements on each pass.

## Verification

Cause, from a local reproduction with the real delivery coordinator and
both real sweeps on embedded PostgreSQL 18.1, with 8 parked requests (4
for each sweep), each with a succeeded native run that waits for its
presentation:

| Hints | Hints a second | `readGitHubResponseRun` calls a second |
| --- | --- | --- |
| None | 0 | 0 |
| Company-wide run milestone (`{ companyId, runId }`) | 9.7 | 8.75 |
| Company-wide run milestone | 1.0 | 7.98 |
| Run milestone for an unrelated issue | 9.7 | 0 |

So the call rate is about one call for each parked request each second,
from company-wide hints only. The hosted rate (6.7 a second) is
consistent with about 7 parked GitHub requests in one company.

Red/green regression test:

- `cd server && npx vitest run
src/services/chat-channels/github/response-sweep-scope.test.ts`
- Red (without the two `reconstructionIntervalMs` lines): `expected [ 6,
6, 6, 6, 6 ] to deeply equal [ 6, +0, +0, +0, +0 ]`. Each of 5
company-wide hints rebuilt all 6 parked requests.
- Green: 2 of 2 pass. The first company-wide hint rebuilds once (6
reads). The next 4 hints read nothing. After 30 seconds, the waiting
hint rebuilds (6 reads). Issue hints still read exactly the one request
of that issue at once, and a hint for another company reads nothing. No
acknowledgement is sent for a parked request.

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed
(all GitHub chat-channel integration tests, including the
acknowledgement and terminal-response sweeps).
- `cd server && npx tsc --noEmit`: passed.
- After the rebase onto #21577: both new test files pass (4 of 4), and
the server typecheck passes.
- `pnpm verify` selected most of the server suite, because the sweep
modules are imported widely. I stopped that broad local run. Required CI
supplies that coverage.

## Risks

- A change that only a company-wide hint reports can take up to 30
seconds to rebuild a GitHub response sweep. I checked the callers.
Changes to one request use issue, delivery, conversation or lease hints:
run status changes with a task, native presentation commits
(`status-decision-committer.ts`, `native-run-finalizer.ts`,
`native-chat-review-presentation.ts`), publications, fast responses, and
the publication lease. Those stay immediate. Company-wide hints come
from runs with no task, from a comment merged into an active run (the
wake queue sends only the run ID), and from company authority changes. A
run with no task cannot change the latest run of a GitHub request's
task. For a merged comment, the active run's next status change sends an
issue hint, and staging the request's response sends a delivery hint;
both rebuild at once. Only a mid-run ownership update can wait up to 30
seconds. For an authority change, each check reads the request and its
authority again before it acts.
- The deadline reconciliation snapshot reports these two sweeps as not
reconciled while a company-wide hint waits (up to 30 seconds). This is
the same as the external-object refresh sweep.
- The startup rebuild (`*`) and failure retries do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local PostgreSQL
reproductions).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:05:45 -06:00
DottaandClaude Opus 5.5 340157d1d6 perf(recovery): scope run lifecycle recovery hints to the run's task graph (#21552)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues in the background. Two of
them, `stranded-assignments` and `assigned-work-invariant`, find
assigned tasks that have no live run and restart them.
> - Every run dispatch and every run state change sent a recovery hint
for the whole company. On each hint these queues reload every eligible
open task of the company and check each one again, at most every 30
seconds per queue.
> - With live runs the hint comes all the time, so the queues never
stop. Their cost is (run activity) x (open-task backlog). On a hosted
instance with about 1,400 open tasks and 5 live runs, the server main
thread stayed at 87-92% of a core.
> - A run event can only change the tasks of its own task graph. This
pull request makes run and task producers name that graph, and keeps one
slow full pass as a backstop.
> - The benefit is that a run event costs a few checks, whatever the
size of the backlog, and a missed hint still delays recovery by at most
five minutes.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The issue recovery queues in `server/src/services/recovery/work.ts`
(`stranded-assignments`, `assigned-work-invariant`,
`resolved-dependencies`, `stale-issue-locks`) and the producers that
wake them on run dispatch, run start, run state change, run finalization
and accounting settlement.

**Subsystem affected**

Server background recovery (`server/src/services/recovery/`,
`server/src/services/scoped-deadline-work.ts`, the notification helpers
in `server/src/services/*-notifications.ts`).

**Current behavior**

`notifyRunDispatch`, `notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` sent `notifyIssueRecovery(tx, {
companyId })`. Most run lifecycle paths call one of them, often two or
three in the same transaction (for example `notifyIssueReviewState`).
Some other run paths sent the company hint directly: the wake status
update, the execution-control deadline writes, legacy run
terminalization, scheduled retries, accounting settlement, task checkout
and task document writes. Each company hint makes each task queue reload
all eligible open tasks of the company, 100 per page, and check each
task again. A company hint is reconstructed at most every 30 seconds per
queue, so one run event per 30 seconds is enough to keep a full pass
running all the time. `issue-recovery-deadlines.test.ts` pinned this: a
company hint re-checks every eligible task.

**Proposed behavior**

Design. For each task queue, a run event of run R on task T can change:

| Queue | What the check reads | What one run event can change |
| --- | --- | --- |
| `stranded-assignments` | T's own runs, wakes, handoffs, questions,
recovery actions, goal, assignee eligibility | T (its runs). The parent
of T and the tasks T blocks when the event also writes T's status. |
| `assigned-work-invariant` | the same plus open blockers and open
children | T, the parent of T (open children), the tasks T blocks (open
blockers) |
| `resolved-dependencies` | blocked tasks whose blockers are all done |
the tasks T blocks |
| `stale-issue-locks` | tasks whose checkout or execution run is R | T |

The assignee's agent status (`running`, `idle`, `error`) does not change
whether the agent is invokable
(`packages/shared/src/agent-eligibility.ts`), and a freed concurrency
slot starts the agent's next queued run through the
`queued-run-dispatch` queue, which keeps its company hint. So the
agent's other tasks are not in the set.

- `notifyIssueGraphRecovery(tx, { companyId, runId?, issueId? })` sends
`{ companyId, graphRunId?, graphIssueId? }` to the four task queues. It
runs no query in the producer transaction.
- When a task queue loads that hint, it resolves the graph with one
bounded, indexed query (`resolveIssueGraph`): the task of the run
(`issue_id`, `native_issue_id`, or the saved `issueId`/`taskId`,
validated and cast so the primary key is used), its parent, and the
tasks it blocks. The four queues share the result for 2 seconds. With no
task or more than 100 tasks, the queue turns the hint into the plain
company hint, so task-less runs coalesce under its 30-second gate
instead of each starting its own full pass. The lookup runs after
commit, so it sees the written state. Graph hints do not wait for the
30-second reconstruction gate (they load at most 100 tasks); hints for
the same run coalesce while dirty, and the lookup is shared across the
four queues, so it runs about once per pass instead of once per producer
call. An earlier round ran the lookup in every producer transaction: 34
lookups/s at 1,500 open tasks and 70/s at 5,000.
- The three company-level queues (`orphan-blockers`,
`active-recovery-actions`, `task-watchdogs`) keep the company hint. They
hold one entry per company or per watchdog, not per open task.
- `notifyRunDispatch`, `notifyScheduledRetryDispatch`,
`notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` use it when the scope names a run
or a task. A scope with only a company keeps the company hint.
- Run paths that knew the run or the task but sent only the company now
pass it. This includes three `writeExecutionControl` deadline writes,
legacy run terminalization (`notifyExecutionRunWork`) and native safe
replacement. Those helpers also forward the scope to
`notifyLiveRunRecovery` and `notifyLiveResourceRecovery`, so their hints
to the live-run queues now carry the run id.
- Recovery action changes (`issue-recovery-actions.ts`: create,
supersede, update, exhaust, resolve) name their source task. A recovery
action is a fact of its source task for these queues. The company-level
`active-recovery-actions` queue still gets the company hint. In the
fixture, failing runs create and resolve recovery actions every few
seconds, and that alone kept a full pass running.
- `writeExecutionControl` and `notifyExecutionRunWork` send the
attention retention hint for a task through
`notifyIssueAttentionRetentionWork` (the task, its parents and the tasks
it blocks), a superset of the single-task hint they sent before. A
recovery action of a blocker can change the blocker attention of its
dependents, so the narrower recovery action scope needs this.
- `promoteDeferredWorkIntents` scopes `lock_released` with a task to
that task, and `accounting_settled` to the run when the company has no
active hard-stop budget policy. Only a reached hard stop can refuse work
because of accounting (`evaluateAdmissionInTransaction`), so without
such a policy the settlement of one run cannot admit another task. The
other hold releases keep the company hint.

Backstop. Each scoped hint also sends `{ companyId, backstop: true }` to
the same four queues. Nothing else arms it. The queue runs it as a full
company pass, but not before 5 minutes after the last full pass of that
queue for that company or of the startup pass. It is a scheduler
deadline (`nextWakeAt` reports it), but it holds no idle intent and does
not mark the queue unreconciled. So:

- Startup still reconstructs every company.
- A failed scoped load retries with the existing back-off (30 seconds to
5 minutes). The backstop is pending in any case.
- A graph with no task or more than 100 tasks loads the whole company at
once.
- A task that a scoped hint missed is checked at most
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes) after the event. While runs
continue, each queue runs at most one full pass per company per 5
minutes. With no activity, nothing is scheduled.

Why arming on activity only cannot leave a stranded task undetected when
the instance then goes quiet. A task becomes stranded only through a
write or through a run that stops without a write:

1. The write names the task's graph (run and task producers). The scoped
pass checks the task at once.
2. The write sends a company hint (pause, budget, agent and company
configuration, tree holds, the producers listed below). The company pass
checks it within the 30-second gate, as before.
3. The write is a run event whose graph does not contain the task. That
same event armed the backstop. The backstop deadline stays scheduled
while the instance is quiet and checks the task at most 5 minutes later.
4. A run stops without any write (process lost, controller gone). The
existing deadline paths find it: `stale-issue-locks` waits for the
stored controller lease, and `active-run-silence` and
`orphan-active-leases` act on their own deadlines. Each of them
terminalizes the run, which is a run write and falls under 1 or 3. The
new test `restarts a task whose run vanished without a final event once
its lease lapses` covers this with the production backstop.
5. The process restarts or a quiet instance is woken by a restart. The
startup pass reconstructs every company.
6. A check that found a task waiting (grace, back-off, monitor) returns
its own deadline. That entry keeps its deadline whatever hints arrive.

Five minutes is the existing upper bound of recovery back-off in this
area: the maximum retry delay of a failed scoped-deadline check
(`scoped-deadline-work.ts`), and `DEFERRED_WAKE_PENDING_BOUND_MS`, the
time `assigned-work-sweep.ts` already trusts a deferred wake before it
calls a task stranded.

Producers that keep the company hint, and why. They can change
eligibility of tasks outside one task graph, or I could not show that
their effect stays in the graph. None of them fires once per run event
in the fixture.

- Eligibility for many tasks: company status (`companies.ts`), budgets
(`budgets.ts`), agent lifecycle and configuration (`agent-lifecycle.ts`,
`agent-configuration.ts`), instance settings (`instance-settings.ts`,
`*`), tree holds (`issue-tree-control.ts`), company deletion and import
(`company-deletion.ts`, `company-portability.ts`), and the hold releases
`pause_lifted`, `tree_hold_released`, `scheduling_resumed` and
`reconciler_backstop`. `accounting_settled` also keeps it when the
company has an active hard-stop budget policy.
- Task writes that can move a task between graphs or change eligibility:
task creation and a task update that changes the parent in `issues.ts`
(other task updates and checkout releases now name the task), document
delete and restore paths in `documents.ts`, task watchdogs
(`task-watchdogs.ts`), system and workspace recovery
(`system-recovery.ts`, `task-retry.ts`,
`workspace-recovery-capture-policy.ts`,
`workspace-recovery-stop-proof.ts`).
- Conversation and chat paths: `agent-conversations.ts`, the daily-cap
release in `heartbeat/queue.ts`, the Telegram attachment failure,
`chat-attachment-reuse.ts`, `native-runner-file-handoff.ts`,
`notifyChatRunMilestones` without a task or run (company authority
change), and `notifyNativeStatusReassessment` from `access.ts`.
- `writeExecutionControl` and `notifyExecutionRunWork` callers that pass
only a company.

**Reason and benefit**

Background work must not grow with the backlog of open tasks. The checks
a run event causes now depend on its task graph, not on the company.

## What Changed

- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery`, `resolveIssueGraph`, the scope matcher and
decoder, `ISSUE_RECOVERY_MAX_ISSUE_IDS` (100) and
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes).
- `server/src/services/recovery/work.ts`: the task queues resolve a
graph hint when they load it (shared for 2 seconds), remember the last
full pass per company, and gate the backstop hint on it.
`registerIssueRecoveryWork` takes an optional `backstopMs`.
- `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects`
accepts `issueIds`.
- `server/src/services/scoped-deadline-work.ts`: an optional
`backstop(scope)` that returns the earliest time of a backstop hint. A
pending backstop hint keeps its deadline but does not count as pending
work or as unreconciled.
- Producers: `issue-recovery-actions.ts`,
`execution-control-notifications.ts`, `run-dispatch-notifications.ts`,
`native-status-reassessment-notifications.ts`,
`external-conversation-settlement-notifications.ts`,
`work-intent-outbox.ts`, `run-cost-accounting.ts`,
`heartbeat/run-lifecycle.ts`, `heartbeat/runtime-execution.ts`,
`heartbeat/retries.ts`, `heartbeat.ts`, `legacy-execution-recovery.ts`,
`issues.ts` (checkout), `documents.ts` (task document writes),
`modules/wake-queue/adapters/postgres.ts` (recovery runs for a task),
`heartbeat/queue.ts` (wake enqueue), `recovery/service.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-safe-replacement.ts`,
`modules/run-dispatch/adapters/workspace-busy-wait.ts` (scheduled
retries).
- `bounds subsidiary scans for100 assignment candidates`: a resolved
recovery action now adds one scoped load per task queue (bound 8 to 11),
and the coalescing check sends one company hint before the burst of 100,
because the first full pass after the resolve is now a backstop pass
with its own key.
- `checks only blocked tasks whose blockers are done`: uses the shared
recording harness and runs out the backstop before its quiet check,
because the blocker's status change is now a graph hint (its 300
dependents overflow to a company load).
- Tests. `issue-recovery.test-fixture.ts` and
`live-recovery-work.test.ts` register the queues with `backstopMs: 0`,
so their quiet checks keep the earlier timing. The new tests use the
production value.

## Verification

New tests. I saw each red test fail before the change.

- `issue-recovery-deadlines.test.ts` `re-checks only the tasks a run
lifecycle event can affect, whatever the backlog` (red before): a run of
the agent on a task with a parent and a dependent, 1,000 backlog tasks
of another agent, and a second open task of the same agent. Seven real
producers (`notifyNativeStatusReassessment`, `notifyExecutionRunWork`,
`notifyRunDispatch`, `promoteDeferredWorkIntents` lock release and
accounting settlement, recovery action create and resolve) check exactly
the 3 tasks of the graph. The recovery action steps were red before the
third commit. With 4,000 more backlog tasks the event issues the same
number of statements. Before the change it checked 1,004 tasks.
- `resolves a run's task graph through index lookups, not the backlog`
(red before the second commit, from a Greptile finding): with 2,000
backlog tasks the graph lookup read 2,005 issue rows
(`pg_stat_xact_user_tables`) because it compared `id::text`. It now
validates and casts the saved id, reads fewer than 50 rows, and returns
exactly the run's graph.
- `finds a stranded task outside the affected set within the backstop
bound` (red before): a task inserted without a hint is not checked by
the run event, and is checked at most 5 minutes after the last full
pass. After 20 quiet minutes, a run event checks a new unrelated task
within 5 seconds, because the last full pass is older than 5 minutes.
Then the queues are quiet.
- `keeps the company pass for hold releases that can admit any task`
(guard, passes before and after): accounting settlement with a hard-stop
policy and a lifted pause still check every task.
- `falls back to the company pass when a run affects more tasks than a
hint names` (guard): a run task that blocks 100 more tasks checks the
whole company.
- `limits an ordinary comment to its issue among one thousand open
assignments`: the old final assertion (a company hint checks all 1,000
tasks) stays, and the test now also runs out the backstop and ends
quiet.
- `issue-recovery-assignments.test.ts`, real heartbeat, real producers,
production backstop (guards; they also pass with the old company hint):
a task whose run failed, a task whose run the system cancelled, a run
that vanished without a final event and is found when its controller
lease lapses, and an agent's queued task that starts when its previous
run ends. Each one recovers within 15 seconds, well before the backstop.
- `scoped-deadline-work.test.ts` `keeps a backstop hint's deadline
without holding the worker busy` (red before).

Suites, each file alone, in `server/` (issue recovery, live recovery,
recovery action and attention suites against dedicated local `_test`
databases): `issue-recovery-deadlines` (13/13),
`issue-recovery-assignments` (7/7), `issue-recovery-work-lifecycle`
(14/14), `scoped-deadline-work` (24/24), `live-recovery-work` (51/51),
`assigned-work-invariant` (32/32), `assigned-work-sweep-round-trips`
(2/2), `saved-work-recovery-overdue` (4/4),
`issue-recovery-dependencies` (5/5), `issue-recovery-conversations`
(3/3), `issue-recovery-actions` (112/112), `attention-retention-work`
(55/55). `recovery-deadline-acceptance` skips without its external
database. `pnpm check:module-boundaries` passes.

Local benchmark. The real server ran against PostgreSQL with
`log_statement=all` and instrumented counters (not committed). Fixture:
1 company, 60 agents, open tasks at 150 / 1,500 / 5,000 (52% in review,
39% blocked behind open blockers, 3% in progress, 2% to do), with
process agents that fail at once so recovery re-dispatches runs all the
time. The tables are analyzed after loading. Each window is 180 s,
starting 150 s after the server is ready, so each "after" window
contains one backstop pass. Base is `master` at 54c3616ad9; after is
this branch at dcd170e29f. The last commit (graph hints without the
30-second gate, task updates and checkout releases scoped) is covered by
the tests below but not re-measured. The machine was shared (load
average 9-16, one base window at 45). Run churn differs between runs, so
the table gives counts per run event (runs created in the window).

| Open tasks | Runs created (/s) base → after | Stranded checks per run
base → after | Assigned-work checks per run base → after |
`agent_task_sessions` reads (/s, 4 shapes, stranded check) | Statements
per run event base → after |
| --- | --- | --- | --- | --- | --- |
| 150 | 0.08 → 0.14 | 45.1 → 6.2 | 2.00 → 0.96 | 3.75 → 1.27 | 2,260 →
1,257 |
| 1,500 | 1.09 → 1.00 | 22.0 → 3.9 | 1.24 → 1.53 | 26.5 → 7.2 | 1,313 →
1,060 |
| 5,000 | 1.12 → 1.99 | 20.4 → 1.6 | 4.10 → 0.91 | 29.3 → 8.1 | 1,400 →
829 |

New statements: the task graph lookup (`with seed(id) ...`) at 0.8 / 9.3
/ 18.5 per second, and scoped loads of the task queues (`select "id",
"company_id" from "issues" ...` grew from 0.7 / 6.6 / 7.8 to 2.9 / 20.6
/ 34.4 per second, each a lookup of at most 100 ids). Stale-lock checks
grew from 0 / 50 / 52 to 22 / 190 / 445 per window: each run event now
checks the locks of its own task at once instead of waiting for the next
full pass. In an earlier round the graph lookup ran in every producer
transaction: 34/s at 1,500 and 70/s at 5,000.

With no activity the server issues no statements before and after this
change (quiet checks in `issue-recovery-deadlines`,
`issue-recovery-assignments`, `live-recovery-work`).

## Risks

- A task outside the graph of the run whose state the event changes is
now found by the backstop, at most 5 minutes later, instead of within 30
seconds. One known case: from #21427, `assigned-work-invariant` counts a
held recovery wake as pending while its agent has queued recovery runs.
When the last one starts, the continuation queue delivers the held wake;
the invariant check of that task waits for the backstop.
- A producer that names a task but changes something wider would now
miss the wider effect. Producers that change eligibility for many tasks
(pause, budget, agent and company configuration, tree holds) still send
the company hint.
- The graph is resolved after commit, from the written state. A parent
change is not a run event and keeps its company hint.
- The live-recovery and issue-recovery suites opt out of the backstop
deadline with `backstopMs: 0`.
- The execution-control deadline writes and legacy run terminalization
now pass the run id. `writeExecutionControl` also forwards that scope to
the live-run queues (`active-run-silence`, `native-process-sessions` and
others), so those hints now name the run instead of the company. Those
queues already handle run-scoped hints from native finalization.
- Test-only option: `registerIssueRecoveryWork(..., { backstopMs })`.
The shared recovery fixtures pass `0` so their quiet checks keep the
earlier timing; the new tests use the production value.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:29:22 -06:00
DottaandClaude Opus 5.5 c1d3f90b37 fix(recovery): keep the resolved-dependency probes per task whatever the statistics (#21574)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `resolved-dependencies` recovery queue loads the blocked tasks
whose blockers are all done (#21517). It runs this load on recovery
hints.
> - The load has two correlated subqueries on `issue_relations`.
PostgreSQL turns them into a semi join and an anti join, and lets the
statistics choose how.
> - After a bulk change in a company that the statistics have not seen
(a large import, a restore, a big cleanup), the planner estimates one
row for the company. It then picks nested loops whose inner side scans
every relation of the company and filters by task id. On a local copy
with 5,000 open tasks the query ran for more than 20 seconds (once 19
minutes) until the table was analyzed; after `ANALYZE` it takes 3.7 ms.
> - This pull request keeps both subqueries as per-task subplans, so
each one is an index probe by (company, task id) whatever the statistics
say.
> - The benefit is that the load stays proportional to the blocked tasks
it reads, also right after bulk changes. This is prevention: the hosted
instance does not show this query as slow today.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a local database that had been analyzed with a smaller dataset and
then loaded with 5,000 open tasks in a new company, the
`resolved-dependencies` load query did not finish within 20 seconds; in
a benchmark run it stayed active for 19 minutes after its server
stopped.

**Steps to reproduce**

1. Load a company with about 1,500 tasks and run `ANALYZE issues,
issue_relations`.
2. With autovacuum off for these tables, load another company with 5,000
tasks, 1,900 of them blocked with `blocks` relations.
3. Run the load query of `resolved-dependencies` for the new company.

Plan with stale statistics (abridged):

```
Limit
  ->  Nested Loop Semi Join  (rows=1)
        Join Filter: (issues.id = r.related_issue_id)
        ->  Nested Loop Anti Join  (rows=1)
              Join Filter: (r_1.related_issue_id = issues.id)
              ->  Index Scan using issues_company_id_uq on issues  Index Cond: (company_id = ...)
              ->  Nested Loop
                    ->  Index Scan using issue_relations_company_related_issue_idx on issue_relations r_1
                          Index Cond: (company_id = ...)
                    ->  Index Scan using issues_company_id_uq on issues blocker  Index Cond: (company_id = ...)
        ->  Index Scan using issue_relations_company_related_issue_idx on issue_relations r  Index Cond: (company_id = ...)
```

The index condition holds only `company_id`. The task id is a join
filter, so each blocked task rescans every relation (and every task, for
the blocker) of the company.

**Expected behavior**

Each blocked task probes its own relations through
`issue_relations_company_related_issue_idx` (company_id,
related_issue_id) and its blockers through `issues_company_id_uq`
(company_id, id).

## What Changed

- `server/src/services/recovery/work.ts`: the candidate predicate is a
named export, `resolvedDependencyCandidate`, and both subqueries end
with `offset 0`. PostgreSQL does not pull a subquery with `OFFSET` up
into a join, so it stays a correlated subplan. Its index condition then
includes the outer task id. The predicate and its results are otherwise
unchanged.
- `server/src/__tests__/resolved-dependency-candidates.test.ts`: two
tests.

Indexes that the subplans use (all existing):
`issue_relations_company_related_issue_idx` (company_id,
related_issue_id) for both relation probes, and `issues_company_id_uq`
(company_id, id) for the blocker status.
`issue_relations_company_issue_idx` (company_id, issue_id) is not needed
here.

## Verification

- `selects the same tasks as the earlier predicate`: 40 blocked tasks
with a done blocker, an open blocker, a done and a cancelled blocker, or
no blocker. The new predicate returns the same 10 tasks as the earlier
one.
- `probes the relations of each task, whatever the planner statistics
say`: analyzes a small company, then loads 3,000 blocked tasks in a new
company. The plan (`EXPLAIN (FORMAT JSON)`) has two subplans and no semi
or anti join, and the load returns 100 rows within a 5-second statement
timeout. I saw it fail first with the earlier predicate: the plan had a
semi join and an anti join.
- Local plan on the stale database: EXPLAIN ANALYZE shows `SubPlan 1`
and `SubPlan 2` with `Index Cond: ((company_id = issues.company_id) AND
(related_issue_id = issues.id))`, 1,933 probes each, 4.9 ms.
- `issue-recovery-deadlines.test.ts` and
`issue-recovery-dependencies.test.ts` pass (14 tests). `pnpm
check:module-boundaries` passes.

## Risks

- With fresh statistics the planner could use a merge or hash join for a
large page; the subplan form gives up that choice. Each blocked task now
costs two index probes, which is the same order as the fresh-statistics
plan (3.7 ms before, 4.9 ms after on 1,933 blocked tasks, measured once
each on a loaded machine).
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:28:55 -06:00
DottaandClaude Opus 5.5 6b6c94a4ea perf(chat): read the GitHub response run by issue run indexes (#21577)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel keeps one response comment for each GitHub
request. Two background sweeps (acknowledgements and terminal responses)
and the acknowledgement step read the latest run of the assigned agent
for the request's issue with `readGitHubResponseRun`
> - That function sends two statements. Both match runs with
`coalesce(native_issue_id::text, context_snapshot->>'issueId') = <issue
id>`. No index has that expression, so each statement reads every run of
the agent and decompresses the large `context_snapshot` of each legacy
run
> - On a hosted test database, these two statements were the two most
expensive statements by database time. In a 701 s window, each ran 6.7
times a second, for 203 s and 137 s of database time (about 30 ms and 20
ms for each call). Together they kept about half of one database session
busy
> - This pull request reads the latest run with two index lookups
(native binding, and legacy context without a native binding), the same
way #20671 changed the terminal-response sweep. It decides request
ownership on that one run in the same statement
> - The benefit is that each call reads a few index entries in one
statement instead of the agent's whole run history in two statements.
The results do not change, and no migration is necessary

## Linked Issues or Issue Description

Refs #20671 (the same rewrite for the terminal-response discovery page;
its Risks section names this function as the remaining case).

**What existing behavior does this improve?**
`readGitHubResponseRun` in
`server/src/services/chat-channels/github/response-run-state.ts`. The
GitHub acknowledgement sweep calls it once for each candidate request.
The terminal-response sweep and the acknowledgement step call it through
`githubResponseExecution`.

**Subsystem affected**
GitHub chat channel: request acknowledgement and terminal-response
recovery.

**Current behavior**
The function sends two statements:

1. The latest run of the agent for the issue that woke for the request:
`heartbeat_runs` joined to the inbound `chat_message_links` row of the
delivery on `wakeCommentId` or `wakeCommentIds`, ordered by `created_at
desc`, limit 1.
2. The latest run of the agent for the issue, with the
`responseRunColumns` projection, ordered by `created_at desc, id desc`,
limit 1.

Both statements filter on `coalesce(native_issue_id::text,
context_snapshot->>'issueId') = $issueId`. The planner can only use
`heartbeat_runs_company_agent_started_idx` or
`heartbeat_runs_company_created_at_desc_idx`. It reads all runs of the
agent and evaluates the `coalesce` on each row, which decompresses the
context of each legacy run. On the hosted test database,
`heartbeat_runs` has about 31,000 rows and 1.96 GB, because
`context_snapshot` is large.

**Proposed behavior**
One statement:

- The latest run ID comes from two disjoint branches, each `order by
created_at desc, id desc limit 1`: runs with `native_issue_id =
$issueId` (through
`heartbeat_runs_company_native_issue_contract_id_uq`), and runs with
`native_issue_id is null and context_snapshot->>'issueId' = $issueId`
(through `heartbeat_runs_company_ctx_issue_created_idx`). Together the
branches are exactly the rows where the old `coalesce` equals the issue
ID.
- The statement reads that one run by primary key with the same
`responseRunColumns` projection.
- An `exists` subquery on the inbound `chat_message_links` row of the
delivery decides whether that run woke for the request. It uses the same
`wakeCommentId` / `wakeCommentIds` match as before, but on one run only.
- When the issue ID is null, the function returns no run without a
query. The old statements could not match a null issue ID either.

**Reason and benefit**
These were the two most expensive statements on the hosted test
database. On a local fixture with the hosted shape, the median call goes
from 111.6 ms to 1.03 ms (about 108 times faster), and the buffers for
one call go from about 73,000 to 98.

**Breaking changes**
None. The function signature, the returned run, and the ownership result
do not change.

## What Changed

- `readGitHubResponseRun` reads the latest run of the agent for the
issue through the two index-backed branches, and decides request
ownership on that run in the same statement. Two statements become one.
- Add `response-run-state.test.ts`. It runs the old two-statement
function (kept in the test as the oracle) and the new function on the
same inputs, and it requires equal results. It also requires a plan
bound: one call must read fewer than 150 `heartbeat_runs` rows while the
agent has 1,500 unrelated runs.
- Add `scripts/benchmark-github-response-run-state.ts`. It loads the
baseline module from a given commit, compares the complete output of
both versions for every input, measures both, and prints the plans.

## Verification

Why the old statements were slow, and the thesis: each statement filters
the agent's whole run history with an expression that no index has, and
detoasts the context of every legacy run. If each precedence branch uses
its own issue index and ownership is checked on the one latest run, the
cost no longer depends on the agent's run count, and the results stay
identical.

Red/green regression test:

- `cd server && npx vitest run
src/services/chat-channels/github/response-run-state.test.ts`
- Red (test file on the old `response-run-state.ts`): the plan bound
fails with `expected 3597 to be less than 150`. The equivalence test
passes, because the oracle is the old code.
- Green (this change): 2 of 2 pass.

The equivalence fixture has two companies, two agents each, 60 issues
each, and one or two GitHub requests for each issue. Runs cover these
cases: native binding, legacy context without a native binding, a native
binding to another issue while the legacy context names this issue,
another agent, another company, `wakeCommentId`, `wakeCommentIds` as an
array, `wakeCommentIds` as a JSON string, a link to the outbound message
of the delivery, a link to another company's comment, a link to the
sibling request of the same issue, a request with no inbound comment,
ties on `created_at`, and a null issue ID. It also calls with another
company's endpoint. The test requires more than 20 results in each
outcome (owned, latest run not owned, no run) and both binding kinds.

Local benchmark with the hosted shape:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-github-response-run-state.ts --baseline-ref=3c3e6c2d75
```

Fixture (synthetic): one company, 4 agents with about 7,750 runs each
(31,000 runs), 2,000 issues, 300 GitHub requests. Each run has about 60
KiB of incompressible context, and `heartbeat_runs` is 1,932 MB. Runs
are native or legacy at random, and about half link to the request
comment. Embedded PostgreSQL 18.1 on an Apple laptop. The hosted
database is Neon, so absolute times are local measurements only. The
benchmark compares the complete output for all 600 inputs (262 owned,
329 latest run not owned, 9 no run): equal. It then times 20 inputs with
runs (two agents), with 2 warmups and 7 alternating samples.

| | Statements per call | Median ms per call (range) | Buffers for one
call |
| --- | --- | --- | --- |
| Before | 2 | 111.6 (100.9–240.5) | 18,358 hit + 26,498 read, then
14,997 hit + 13,489 read |
| After | 1 | 1.03 (0.81–1.39) | 80 hit + 18 read |

Plans for one call. Before: statement 1 is a `Bitmap Heap Scan` on
`heartbeat_runs_company_agent_started_idx` that removes 7,674 rows by
filter (75.7 ms). Statement 2 is an `Index Scan` on
`heartbeat_runs_company_created_at_desc_idx` that removes 19,801 rows by
filter (60.2 ms). After: one `Index Scan` on `heartbeat_runs_pkey`. Its
`InitPlan` is a `Merge Append` of two `Limit` index scans on
`heartbeat_runs_company_native_issue_contract_id_uq` and
`heartbeat_runs_company_ctx_issue_created_idx`, and its `SubPlan` is one
`Index Scan` on `chat_message_links_delivery_idx` (0.128 ms).

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed
(all GitHub chat-channel integration tests, including the
acknowledgement and terminal-response sweeps).
- `cd server && npx tsc --noEmit`: passed.
- `pnpm verify` exited 2 because a file under `scripts/` changed, which
requires the broad CI coverage. Required CI covers the rest.

Why the function runs 6.7 times a second. The acknowledgement sweep and
the terminal-response sweep rebuild their candidate set when a hint
arrives, and they call this function once for each candidate on each
rebuild. A run milestone for a run with no issue (for example, every
status change of a run without a task in `setRunStatus`, or a comment
merged into an active run in the wake queue) sends a company-wide hint
to both sweeps. So in a busy company, both sweeps rebuild about once a
second (their minimum pass interval), and each rebuild reads every
parked candidate again, although nothing about those candidates changed.
In a local reproduction with 8 parked candidates (4 for each sweep) and
the real coordinator and workers, 10 company-wide milestone hints a
second gave 8.75 calls a second; 1 hint a second gave 7.98 calls a
second; 10 hints a second for an unrelated issue gave 0 calls; and no
hints gave 0 calls. So the hosted rate is consistent with about 7 parked
GitHub requests in one company and a steady stream of company-wide run
milestones. This pull request makes each call cheap. A separate pull
request will address the repeated rebuilds.

## Risks

- Low risk. The change is one function. The run match is the same set of
runs: `native_issue_id::text = <uuid text>` is equal to `native_issue_id
= <uuid>` for the canonical UUID text that callers pass (the issue IDs
come from UUID columns), and a null `native_issue_id` makes the old
`coalesce` use the legacy context.
- Ownership: the old code compared the latest run that woke for the
request (ordered by `created_at` only) with the latest run (ordered by
`created_at, id`). That is true exactly when the latest run woke for the
request, except when two runs that both woke for the request tie on
`created_at` with the latest run. In that case the old result depended
on the physical row order. The new code always returns the deterministic
answer (the latest run woke for the request).
- The number of calls does not change in this pull request.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 00:59:25 -06:00
Nicky LeachandClaude Opus 5.5 e977c3b922 feat(native-runtime): end the wait for the provider's own turn when a person comments (#21455)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner part of this work is complete in #21316 and the pull
requests below it: when a controller sends `providerStartedTurns` on
`turn.start`, the Runner keeps Claude while a background shell is
running, and gives the cycle of Claude to the controller as a turn that
the provider started.
> - Part 2 gave the consumer a wait with a limit and two inputs: a hook
that ends the wait early, and a start time.
> - This pull request gives both from the server for a native ACPX
Claude run. A new comment from a person ends the wait. The start time
comes from an event that is already saved, so a restart of Core does not
make the wait longer.
> - The benefit is that a person who comments does not wait up to 10
minutes for the agent to see it.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one of four parts on the Core side. They are
stacked on #21316:

1. The driver admits a turn that the provider started.
2. The consumer waits for that turn, for a limited time.
3. The server ends the wait when a person comments, and keeps the clock
across a restart.
4. The transport sends the opt-in. Only this part changes what a
production run does.

This is part 3. Its base branch is `feat/core-provider-turn-consumer`.
Refs #19912 (design).

**What existing behavior does this improve?**

Nothing visible yet. What Core gives to the consumer of a native ACPX
Claude run.

**Subsystem affected**

Server: `server/src/services/native-runtime`.

**Current behavior**

- A comment from a person during a run is queued as a deferred wake. The
next run delivers it. This does not change.
- Core has no wait for a provider-started turn.

**Proposed behavior**

- For a native ACPX Claude run, Core gives `executeNativeSession` the
option `providerTurnWait`.
- One delivery worker does the person-comment check for all runs that
wait. The server registers it at startup with the shared delivery work
coordinator, as it registers the cost accounting worker. The worker
listens to the existing notification for deferred wakes. On each notice
it runs one query for each run that waits: a comment from a person on
this issue, made after the start of the run, that a deferred wake of
this agent names. A comment that the query finds ends the wait of that
run. The run then takes the path of today, and the next wake delivers
the comment as today.
- If a database read fails, that run stays owed a check, and the
coordinator runs the worker again after 5 seconds, as for each other
delivery worker. A read that fails for one run does not stop the check
of the other runs. When no run is owed a check, the worker has no
deadline and does no query.
- The start time of the wait is the time of the first saved
`turn.completed` event of the run that has `awaitingProviderTurn`. With
no such event it is empty.
- Each query has the company id in its condition.

**Reason and benefit**

- The comment window starts at the start of the run, not at the start of
the wait. A comment made during the prompt turn must not wait for the
limit.

## What Changed

- `server/src/services/native-runtime/native-provider-turn-wait.ts`
(new, 109 lines) and 4 lines in `native-session-executor.ts`.
- `server/src/index.ts` (4 lines): the registration of the worker, its
stop at server close, and its place in the list of delivery workers that
must be ready. `delivery-work-notifications.ts` (1 line): the queue key
of the worker. The key needs no database change, because each topic uses
the one shared channel.
- `server/src/vendor/paperclip-runner/index.ts`: one type export.
- Tests: `native-provider-turn-wait.test.ts` (new, 252 lines) and one
case in `native-usage-accounting.test.ts` (25 lines).

## State saved by the previous release

- No new persisted state and no schema change. The start time is read
from `heartbeat_run_events`.
- **A run that the previous release started:** it has no waiting
terminal, so the start time is empty.
- **Core restarts during a wait:** the new process reads the start time,
so the limit is the original one. The consumer then continues the wait
(#21454), and the worker checks the comments again from the start of the
run, so a comment that was made before the restart still ends the wait.
- **Rollback of Core during a wait:** the previous Core stops at the
waiting terminal and finalizes the run as `native_finalization_missing`,
as today.

## Verification

At the head of this branch:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json` and `pnpm run build:typescript` in
`packages/paperclip-runner`; `node
scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in
`server`; `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed.
- `native-provider-turn-wait.test.ts`, 10 tests on embedded Postgres
with the real coordinator: a comment from a person in a deferred wake
ends the wait; a comment from an agent, a comment before the run, and a
comment of a different company do not; the start time comes from the
first waiting terminal, and a run of the previous release has none.
- The 6 new tests: a read that fails one time is tried again after 5
seconds and finds a comment that is already saved, with no new notice; a
notice during a read that fails gives a new check; a read that fails for
one run does not hold back a second run, and each run ends only for its
own comment; a run that starts to wait during a check is checked; a wait
that is aborted is removed, nothing checks it again and no deadline
stays; with no run owed a check there is no deadline and no query in one
hour of test time.
- The test helper now registers the wake intent before it writes the
deferred wake, in the same transaction. This is the order of each
production writer (`modules/wake-queue/adapters/postgres.ts` and
`services/heartbeat/queue.ts`). Production code did not change for this.
- Mutation checks: a read that fails clears the owed check (3 tests
fail); an aborted wait is not removed (the abort test fails).
- Also run, unchanged and green: `delivery-work-coordinator.test.ts`,
`budget-enforcement-worker.test.ts`,
`server-startup-feedback-export.test.ts`, `startup-timeline.test.ts`.
The 7 files together: 1,699 passed, 7 skipped.
- `native-usage-accounting.test.ts`: a prompt turn and the
provider-started turn after it are each counted one time, with the
events as the executor receives them. The cost of the provider-started
turn is the session total in the cumulative channel, and it is not added
two times. The accounting code is not changed.

Not run: a run through the executor with a database; the full suite. CI
supplies the broad coverage.

## Risks

- **One more query at each start of a native ACPX Claude run,** on
`heartbeat_run_events` by company and run.
- **The worker is registered in each server process, and it is idle with
no wait.** A notice with no run that waits does no query. No wait starts
before part 4.
- **After an abort during a retry, the 5 second retry of the coordinator
runs one more time.** It finds nothing owed and does no query. The
coordinator has no call to cancel the retry of one worker.
- **A process that does not register the worker does no comment check.**
The server registers it with no condition at startup, and native runs
execute only in that process. Only a unit test can be in this state.
- **A comment ends the wait but does not interrupt a turn that is
running.** That is the rule of part 2.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:55:11 -08:00
DottaandPaperclip 2102492806 fix(discord): refresh due command registration after task drains (#21573)
## Thinking Path

> - Paperclip runs agents and connects their work to chat services.
> - Discord command registration uses a shared worker and saved
deadlines.
> - A task drain can hold that worker while registration is already due.
> - The coordinator can keep a 30-second fallback after the drain ends.
> - This change refreshes the command owner on drain release or observed
expiry.
> - Normal admission, earliest-time and provider retry checks still
apply.

## Linked Issues or Issue Description

Refs: #17557, #21562. Related: #15532 checks credential reads when
registration is not due. This change handles resumption of due work in
the separate command worker.

**What happened?**

A due command registration can wait for the coordinator's fallback after
a task drain ends. The worker retains the request but has no
drain-release subscription. This delays command work and its return to
an idle state.

**Expected behavior**

Resume due command work when admission returns. Preserve future earliest
times and provider failure backoffs. Leave an empty or stopped worker
idle.

**Steps to reproduce**

1. Register the normal command worker while an idle task drain is
active.
2. Retain a due registration and capture the coordinator's fallback.
3. Release the drain, or observe its expiry, before that fallback.
4. Check that the due endpoint is checked before the fallback. Both
regression cases fail before this change.

**Paperclip version or commit**

Main base `3c15acd3fc1b200ef0e36ac904a03cd5fcba00c0`.

**Deployment mode**

Built from source. The focused tests use the real coordinator and
command owner with controlled registration operations. They do not
contact Discord.

## What Changed

- Subscribe the command owner to normal drain changes and observed
expiry.
- Wake it only when current admission allows work and its existing next
time is due.
- Remove both subscriptions through the normal worker stop callback.
- Add eight controls for release, expiry, another closed gate, future
earliest time, provider backoff, empty state and both stop paths.

## Verification

Local checks used the integrated branch that exposed this contract gap.
This main port contains the same two tested file afterimages. Its
admission, scheduler and fairness modules match those local inputs.
Main's coordinator and DB modules have other changes. Fresh CI on
`aec6ecc65717aa6372bad526e044ec4aa3888bec` now qualifies this main
context.

- Both release and expiry cases fail at the intended missing
endpoint-check assertion before the product change. No setup or timeout
failure is used as causal proof.
- All eight focused cases pass after the change. No case is excluded.
- The complete normal Server typecheck and affected-file program checks
pass on the integrated branch.
- Normal committed Source SDK verification passes on this main branch.
- Exact-head main CI passes, including typecheck, build, the general and
serialized suites, and all eight new controls (142 ms) in server shard
2/12. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/43565579653).
A full local repository build and suite were not run in this checkout.
- Greptile reviewed the exact head at 5/5 with no actionable findings.
The complete edited summary and all paginated review threads were
inspected; no unresolved threads remain.

Run the focused controls with:

```sh
pnpm --filter @paperclipai/server exec vitest run src/services/chat-channels/discord/command-work.test.ts --maxWorkers=1 --no-file-parallelism --retry=0
```

The existing admission-drain contract remains accurate. This fix adds no
configuration or command, so no documentation change is required. Full
transport sleep acceptance and deployed suspension are separate claims.

## Risks

A release event can request an earlier pass for already due work. The
existing `canRun`, earliest-time, batching and per-item backoff checks
remain active. Both worker stop and coordinator stop must remove the
subscriptions; the controls cover each path. No scheduler policy, claim
or provider deadline changes.

## Model Used

OpenAI `gpt-6.1-sol` assisted with implementation and independent
review. The Codex managing agent performed technical review and
verification. The host does not expose its exact runtime model ID or
context-window size. Work used source inspection, code edits and local
test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (not
required: the existing contract remains accurate)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-14 00:46:49 -06:00
DottaandClaude Opus 5.5 3c3e6c2d75 fix(retention): stop rescanning unchanged pinned workspace snapshots (#21571)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote workspaces keep saved versions in the
`execution_workspace_snapshots` ledger, and a retention sweep retires
versions that no reference holds
> - Every execution notification wakes this sweep, and the sweep
examined every pinned version again on each pass
> - Some pins never change. A migration cohort supersession leaves
failed operations with `lastFailure` and no `retiredSave`, and these
hold their snapshots until a separate review releases them
> - On a busy server the sweep ran four statements per pinned row, many
times each second, on a main thread that was already saturated
> - This pull request records the reference that pins a version and
leaves that version out of the candidate query until that reference
releases
> - The benefit is that an execution notification costs one candidate
query, not one transaction per pinned row

## Linked Issues or Issue Description

No public issue exists. Bug report:

**What happened?**
A hosted instance ran four statements at 13.4 calls per second each: the
workspace read, the importing-parent read and the operation read in
`pinned()`, and the locked ledger row read. The sweep in
`server/src/services/remote-workspace-snapshot-retention.ts` caused
them. A pinned superseded row keeps `status = 'superseded'` with
`delete_after = null`, so the candidate query selects it on every pass.
Each pass examines up to 50 rows. A truncated page runs again after one
second. Each run-work notification (`notifyExecutionRunWork`) sets
`rescanRequested`, which starts the scan again. The sequence stopped at
the operation read because a failed operation with `lastFailure` and no
`retiredSave` pinned these rows.
`supersedeExpiredRemoteWorkspaceMigrationCohort` writes such operations,
and they hold their snapshots permanently by design (see
`doc/DATABASE.md`, "Workspace cohort admission proofs").

**Expected behavior**
A pinned version is examined again only when the reference that pins it
can have changed. An idle or busy server does not repeat reads for
unchanged pinned history. A released version is retired on its normal
deadline.

**Steps to reproduce**
1. Create 300 superseded snapshots. Pin each with a failed
`remote_workspace_migration` operation that has `lastFailure` and no
`retiredSave`.
2. Register the retention worker on a delivery coordinator and let the
first pass finish.
3. Send `notifyExecutionRunWork` about 13 times each second for 60
seconds.
4. Count statements with `pg_stat_statements`.

**Paperclip version or commit**
`master` at 6dd0349d52

**Additional context**
The local reproduction showed the same four statements at equal call
counts, as on the hosted instance (see Verification).

## What Changed

- `pinned()` returns the first reference that holds the version (current
pointer, importing parent, operation, unfinished child, run, or recovery
action) and its ID, not a boolean.
- The sweep stores that reference in `metadata.retentionPin` when it
pins a version. It writes again only when the reference changes or a
deadline must be cleared.
- The candidate query leaves out a version with no deadline while its
recorded reference still holds. Each pin condition is one SQL builder.
`pinned()` and the candidate query use the same builder, so the
exclusion can never hold a version that `pinned()` would release.
- When the recorded reference releases, the version comes back, and
`pinned()` checks every reference again. It records the next reference
that holds, or schedules or retires the version.
- The record is removed when the sweep schedules or claims a version.
- The worker coalesces wakes for up to one second
(`REMOTE_WORKSPACE_SNAPSHOT_COALESCE_MS`). Retention deadlines are days,
so this delay has no effect on retirement.
- `doc/DATABASE.md` describes the record.
- No schema migration. The record is in the existing `metadata` JSONB
column.

## Verification

Red/green regression tests in
`server/src/services/remote-workspace-snapshot-retention.test.ts`:
- `leaves %s-pinned history out of later passes, across restarts, until
that pin is released`. This runs once for each of the six pin kinds. It
pins the version and checks that a new service instance (a restart)
examines zero rows. Then it releases the pin and checks that the version
is deleted.
- `re-examines a pinned snapshot when its recorded pin is released but
another still holds it`.
- `revisits a failed save's recorded pin once its abandoned upload is
settled`.
- `does not re-examine unchanged pinned history on execution
notifications`. This uses the coordinator and `notifyExecutionRunWork`.

On the parent commit, the first eight new tests fail (6 pin kinds, the
second-pin case, and the notification case). With the fix, all 31 tests
pass.

Related suites pass locally (212 tests):
`remote-workspaces.integration.test.ts`,
`remote-workspace-snapshot-reuse.test.ts`,
`remote-workspace-cohort-migration.test.ts`,
`snapshot-attempt-upload-abort.test.ts`, and
`remote-workspace-snapshot-retention.test.ts`. `tsc --noEmit -p
server/tsconfig.json` is clean.

Load measurement. Local PostgreSQL 18 with `pg_stat_statements`. 300
pinned rows. About 13 run-work notifications each second for 60 seconds,
after the first pass finished:

| | Before | After |
| --- | --- | --- |
| Pinned rows examined again | 30,250 – 39,000 / min | 0 / min |
| Sweep statements | ~151,000 – 196,000 / min | 88 / min |
| All statements, including the test's own notification transactions |
153,327 – 197,205 / min | 1,647 / min |
| Sweep database execution time | ~1,285 ms / min | ~52 ms / min |
| Candidate query | 0.07 ms, ~10 / min | 0.88 ms, 58 / min |

Before: `begin`, the locked row read, the workspace read, the
importing-parent read and the operation read each ran 30,250 times in 60
seconds. This is the same signature as on the hosted instance. After:
the sweep ran 58 candidate queries and 30 deadline reads. With
per-notification wakes and no coalescing, the sweep ran 776 candidate
queries and 776 deadline reads (1,552 / min).

I did not verify the change on a hosted instance.

## Risks

- A recorded pin is checked against only its own reference. The check
uses the same SQL condition as `pinned()`, so a version cannot stay
excluded after that reference releases. The parametrized test covers
each pin kind.
- A version can be pinned by more than one reference. When the recorded
reference releases, the sweep checks all of them again and records the
next one.
- A producer that rewrites the row's metadata and drops the record
causes one extra examination. A producer that sets a deadline makes the
record inactive, because the exclusion applies only when `delete_after`
is null.
- Versions that have no record yet, which includes all existing pinned
rows, are examined once after deploy. The examination writes the record.
- The candidate query does one primary-key lookup for each pinned row.
It takes about 0.9 ms with 300 pinned rows, and the time grows linearly
with the number of pinned rows.
- Pinned history from failed migration cohorts stays pinned. This change
does not alter that hold.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, local PostgreSQL)

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 00:05:32 -06:00
DottaandClaude Opus 5.5 3c15acd3fc perf(recovery): look up held runs by key so the fresh-continuation deadline stops scanning run history (#21568)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is cut off, the fresh-continuation fallback continues the
task in a new run. The execution-disposition worker schedules its next
pass with `nextFreshContinuationCheckAt` and lists due work with
`heldExecutionsAwaitingFreshContinuation`
> - Both queries share two predicates (`heldExecutionWhere` and
`recentInterruptionSourceSql`). These predicates matched the run as
`source.id::text = <evidence text>` and the task's runs as
`native_issue_id::text = <issue id>::text`. A cast on an indexed column
prevents an index lookup
> - So for each held record, PostgreSQL read the company's whole run
history (two times), and it decoded each run's wide context JSON. On
staging, the deadline query took about 13.5 s of database time in each
27 s worker cycle. It was the largest statement by time
> - This pull request compares the uuid columns directly. It turns the
recorded run id into a uuid only when the text is the canonical
lowercase form, which is exactly when `id::text = text` can be true
> - The benefit is that each held record costs a few index lookups. The
same records are selected, and the worker cycle no longer keeps a
database session busy

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The fresh-continuation scheduler queries: `nextFreshContinuationCheckAt`
(the disposition worker's deadline),
`heldExecutionsAwaitingFreshContinuation` (the backlog listing, also
probed by the schedule), and the "left alone" count. They are in
`server/src/services/fresh-continuation-fallback.ts`.

**Subsystem affected**
Server: recovery and the execution-disposition worker
(`executionControlSchedule().disposition` in
`execution-control-work.ts`).

**Current behavior**
Neon statement statistics on staging (window of 160 s on 2026-11-14):
the `select
min((issue_recovery_actions.evidence#>>'{freshContinuationCheck,nextCheckAt}')::bigint)
…` statement ran 6 times at about 13.5 s each. That is 81 s of database
time in 160 s, about half of one busy session. All other statements in
the window used less than 35 s together. Staging has 31,399 runs (1.96
GB, wide `context_snapshot`), 75,285 recovery records, and 282 records
with a check deadline.

The local plan shows the cause. For each candidate record:
- `source.id::text = coalesce(evidence->>'runId',
evidence->>'sourceRunId')` cannot use the primary key. PostgreSQL
filters the company's failed runs (6,767 rows removed in each loop, and
6,798 again in the outer semi-join).
- In the recent-interruption check, `max(held.finished_at)` matches runs
of the task with `native_issue_id::text = source_issue_id::text or
issue_id::text = … or context_snapshot->>'issueId' = …`. The casts
cannot use the task-link indexes, so PostgreSQL filters the company's
run history and decodes each run's context JSON (29,937 rows removed in
each loop).

**Proposed behavior**
The queries look up the source run by its primary key and the task's
runs by the existing `(company_id, native_issue_id, …)`, `(company_id,
issue_id, …)` and `(company_id, context_snapshot->>'issueId', …)`
indexes. The results do not change.

**Reason and benefit**
On a staging-shaped local fixture, the deadline query goes from 2,330 ms
to 27 ms (87x), and the backlog listing and count go from 3.7-5.7 s to
19-27 ms. Buffer accesses go from 2.15 M to 28 k.

**Breaking changes**
None. No schema change and no migration.

## What Changed

- `server/src/services/fresh-continuation-fallback.ts`:
- Add `sourceRunKey`. It compares the recorded run id with
`heartbeat_runs.id` through the existing `canonicalExecutionUuid` guard
(cast only when the text matches `^[0-9a-f]{8}-…-[0-9a-f]{12}$`).
`uuid::text` always gives that lowercase form, so this matches exactly
the rows `id::text = text` matched. Uppercase, braced, padded, numeric
and invalid ids still match nothing, and no cast error is possible.
- Use it in `heldExecutionWhere`, `interruptionEligibility` and the
listing's order expression.
- `runOfTask` compares `native_issue_id` and `issue_id` with the issue
id as a uuid. Both callers pass a uuid: the `source_issue_id` column, or
the id of a task row read in the same transaction.
- Export `heldExecutionWhere` and `recentInterruptionSourceSql` so the
regression test can compare them with the previous predicates row by
row.
- Add `server/src/services/fresh-continuation-schedule-queries.test.ts`
(regression test, see Verification).
- Add `scripts/benchmark-fresh-continuation-deadline.ts`. It seeds a
staging-shaped fixture, captures the SQL that the baseline revision and
the working tree send, compares complete positional rows, and reports
timings and plans.

## Verification

- Red/green: `fresh-continuation-schedule-queries.test.ts` "finds the
deadline by run-key and task-link lookups, not a scan of run history per
record" counts the `heartbeat_runs` rows that the deadline query's plan
reads (loops x (rows + rows removed)). With the previous predicates it
fails: 549,166 rows read against a bound of 9,442 (two times the run
count). With this change it passes.
- Equivalence: the same file seeds 900 held records with a seeded random
generator. The records cover these cases: `runId`, `sourceRunId`, both
keys, uppercase, braced, padded, numeric, invalid and unknown ids; a
source run in another company; source statuses (failed, timed_out,
interrupted, cancelled, succeeded, running) with and without
`finished_at`; the assignee and other agents; newer runs linked by
native id, legacy issue id, or context only; unrelated runs;
cancelled-at-dispatch runs; live (queued, running, scheduled_retry)
runs; earlier and later creation times; retries of the source; person
Retry wakes; old and recent finish times; restart, person-owned and
sandbox-resume error codes; and issue statuses and assignees. For every
record it compares `heldExecutionWhere()`, `heldExecutionWhere({
includeActive: true })` and `recentInterruptionSourceSql(now)` with
verbatim copies of the previous predicates. It also compares
`nextFreshContinuationCheckAt` and the full ordered
`heldExecutionsAwaitingFreshContinuation` rows with the previous
queries. The test checks that each predicate is true and false for more
than 5 records, so the comparison is meaningful.
- Focused suites pass locally: `fresh-continuation-fallback.test.ts` and
`fresh-continuation-schedule-queries.test.ts` (82 tests). Related
recovery suites pass: `fresh-continuation-retry-delivery`, `task-retry`,
`fresh-continuation-move-forward-offer`,
`execution-recovery-dead-controller`,
`automatic-run-brake-recovery-passes`, `execution-control-work`,
`recovery-wake-herd` (with the new test: 8 files, 252 tests). `node
server/scripts/typecheck-layers.mjs` passes.
- `pnpm verify` exits 2 (the new benchmark script is a global CI input),
so CI supplies the broad coverage.
- Benchmark: local embedded PostgreSQL 18.1, `jit=on`,
`shared_buffers=128MB`, macOS arm64, a shared machine with other work.
Baseline `67652580ee`. Fixture: 5,669 issues, 31,351 runs with 60 KiB
incompressible context JSON (1,962 MB), 75,285 recovery records (282
with a check deadline, a third overdue). One warm-up and 5 alternating
timed pairs. Both versions return identical positional rows for every
query.

| Query | Baseline median (samples, ms) | This PR median (samples, ms) |
Speedup | Shared buffers |
  | --- | --- | --- | --- | --- |
| `nextFreshContinuationCheckAt` | 2,330 (1,962-2,421) | 27 (21-28) |
87x | 2,149,802 -> 28,227 |
| `heldExecutionsAwaitingFreshContinuation` limit 5 | 3,692
(3,296-4,014) | 19 (18-24) | 194x | 2,831,539 -> 23,970 |
| same, limit 1 (schedule probe) | 3,554 (3,210-4,861) | 20 (18-24) |
179x | 2,831,539 -> 23,970 |
| left-alone count | 5,746 (5,090-5,992) | 27 (25-28) | 213x | 5,435,614
-> 40,953 |

With 1 KiB contexts (rows stored inline) the deadline query goes from
4,096 ms to 19 ms. Staging measured 13.5 s for the deadline query. The
local fixture reproduces the same plan shape, but the staging data
distribution and Neon storage are not reproduced exactly, so the staging
gain is not verified yet.

  ```sh
pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-fresh-continuation-deadline.ts \
--baseline-ref=67652580ee --context-kib=60 --samples=5 --warmups=1
--print-plans
  ```

- Plan after the change: the source run is an `Index Scan using
heartbeat_runs_pkey` (0 rows removed), and each task-run probe is a
`BitmapOr` over the native-id, issue-id and context-issue indexes. The
newer-run `not exists` already used that `BitmapOr` before. It does not
change.

## Risks

- Low risk. The SQL change is limited to how two keys are compared. The
regression test compares the old and new predicates for every record of
a varied fixture.
- `runOfTask` now casts its issue id to uuid. A caller that passes a
non-uuid text would get an error instead of no match. The two callers
pass a uuid column and a task row's id.
- Follow-up, not changed here: the disposition schedule calls
`deadline()`, which returns "retry in 15 s" when the earliest check is
already due. An active hold that the disposition pass does not pick up
(for example, one that records its run only as `sourceRunId`, which
`unresolvedExecutionDispositions` does not read) keeps the deadline
overdue, so the worker runs again every 15 s with nothing to do. A local
probe confirmed this. This explains a fixed cadence of 15 s plus the
query time (about 27 s on staging). After this change each such pass
costs milliseconds, not seconds.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, local PostgreSQL benchmarks and
tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
behavior or command change; no doc update needed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:37:19 -06:00
DottaandClaude Opus 5.5 3ae7aba643 perf(db): reuse dedicated connections instead of opening one per call (#21567)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server uses `withDedicatedDbConnection` (packages/db) when an
action must hold one database session outside the main pool: advisory
transaction locks, lock keepalive, and `pg_settings` reads for workspace
lifecycle, finalization, source locks, skill caches and status cards
> - Each call built a new postgres.js client, opened a new physical
connection (TCP, TLS and startup through the pooler), built a new
drizzle client with the full schema, and then ended all of it
> - On staging the dedicated application name shows about 21 connection
setups per second, and the main thread is pinned (event-loop utilization
0.965)
> - This pull request keeps a small set of idle single-connection
clients per owner and reuses them
> - The benefit is that a dedicated action costs about 0.2 ms of
main-thread time instead of 4 to 8 ms, with the same exclusive-session
guarantee

## Linked Issues or Issue Description

**What existing behavior does this improve?**

`withDedicatedDbConnection` opens and closes one new database client and
one new physical connection for every call.

**Subsystem affected**

Database client (`packages/db`) and the server services that take
dedicated advisory-lock sessions.

**Current behavior**

Each call builds a new postgres.js client, opens a new connection (TCP,
TLS and startup through the pooler), and builds a new drizzle client
with the full schema. On staging (Neon statement statistics, 326 s
window) `SET application_name='paperclip-workspace-finalization-lock'`
ran 21.4 times per second. Only this factory uses that application name.
In a 30 s staging CPU profile, drizzle `relations.ts` was 2.2%
inclusive, `n:net` 5.8% and postgres.js construction 0.3%. Part of that
is this per-call setup.

**Proposed behavior**

A caller still gets exclusive use of one session for the whole action.
The session and its drizzle client are reused by later actions, so the
per-call cost is the action's own queries, not a full connection setup.

**Reason and benefit**

The server main thread is saturated on staging (event-loop utilization
0.965). Reuse removes about 5 ms of main-thread work per call.

**Breaking changes**

None. The exported function keeps its signature.
`closeDedicatedDbConnections` and `DEDICATED_CONNECTION_IDLE_MAX` are
new exports.

## What Changed

- `packages/db/src/client.ts`: each `createDb` owner now has one set of
dedicated single-connection clients, shared with every private client
derived from that owner (so nested dedicated actions use the same set).
- An action takes an idle client (last in, first out) or opens a new
one. It never waits. Callers hold advisory locks and nest dedicated
actions, so a waiting queue could deadlock.
- On success the client returns to the idle set if fewer than
`DEDICATED_CONNECTION_IDLE_MAX` (4) are idle. Otherwise it is closed.
- When the action throws, the client is closed and never reused. This
covers a failed or cancelled transaction, a broken socket, and an
interrupted lock wait.
- When an action ends the client itself (the source-lock cancel path
calls `$client.end`), the client is never reused.
- Idle clients keep the owner's idle timeout and max lifetime, so
postgres.js still closes idle physical connections after
`DATABASE_IDLE_TIMEOUT_SECONDS`.
- New `closeDedicatedDbConnections(db, timeoutSeconds)` closes the idle
set and stops retention. Ending the owner `$client` does the same. A
later action still works with a connection that closes on release.
- `server/src/index.ts`: shutdown closes the dedicated set next to the
other private pools.
- `doc/DATABASE.md`: documents the idle cap, the "not an admission
limit" rule, and the transaction-scoped session state contract.
- Session state audit: every current caller uses `pg_advisory_xact_lock`
/ `pg_try_advisory_xact_lock` and `set_config(..., true)`. No caller
uses session advisory locks, `SET` without `LOCAL`, `LISTEN`, or
temporary tables. A reset round trip per call is therefore not added.

## Verification

- New `packages/db/src/dedicated-connection-pool.test.ts` (embedded
PostgreSQL, 10 tests).
- Red on `origin/master`: 200 sequential calls used 200 distinct
sessions (`expected 200 to be 1`). All 10 tests failed.
- Green with this change, 3 runs in a row: 200 sequential calls use 1
session; 7 concurrent callers each hold a different session for the
whole transaction, and afterwards at most 4 stay open and later calls
reuse them; a failed transaction's connection is closed; a terminated
socket during the action is not reused; an idle connection terminated by
the server reconnects; a client the action ended is not reused;
transaction-local settings do not leak; shutdown and owner `$client.end`
close every retained connection; nested actions do not block.
- Focused suites pass: packages/db `work-signals`,
`prepared-queries-postgres`, `response-capture-postgres`,
`checkpoint-pool`, `client`, `client-options`,
`client-teardown-registry`, `postgres-connection-recovery`,
`client-disconnect-replay` (132 tests). Server suites that use dedicated
connections: `skill-package-store`, `status-card-evaluation-work`
(updated: it now proves advisory locks are released, since sessions are
reused), `agent-instruction-working-copies`,
`workspace-recovery-retry-copyback`, `native-execution-lease-batch`,
`company-skills-empty-runtime`, `decisions-service`,
`legacy-controller-lease`, `server-startup-feedback-export` (467 tests).
- `pnpm verify --explain` selects 2779 files for a
`packages/db/src/client.ts` change. Per AGENTS.md I ran the focused
checks above and let CI give the broad coverage.
- Benchmark: 200 calls of `withDedicatedDbConnection(db, d =>
d.transaction(tx => tx.execute(select pg_try_advisory_xact_lock(42))))`
against local embedded PostgreSQL 18, 10 warm-up calls, 5 alternating
before/after process pairs per mode. Main-thread time is
`eventLoopUtilization().active`.

| Mode | Variant | Wall ms/call (median, range) | Main-thread ms/call
(median, range) |
| --- | --- | --- | --- |
| TLS | before | 35.55 (32.93–37.52) | 5.18 (4.68–7.95) |
| TLS | after | 0.19 (0.17–0.21) | 0.18 (0.17–0.20) |
| Plain | before | 18.39 (17.91–20.53) | 3.65 (3.58–4.30) |
| Plain | after | 0.17 (0.13–0.18) | 0.13 (0.12–0.14) |

At the staging rate of about 21 calls per second, this removes roughly
100 ms of main-thread work per second (TLS figures; staging CPUs may be
slower).

## Risks

- A retained connection that the server or network drops while idle:
postgres.js reconnects on the next query when it has seen the close
(tested). If the drop and the reuse race, the first query of that action
fails like a main-pool query would. Before this change, a dedicated
action always started on a fresh connection.
- Through a transaction-mode pooler, `SET application_name=...` is a
server-side parameter switch, so its count may not drop as much as the
client-side connection count. Measure the deployed effect with pooler
client connections and main-thread CPU, not only that statement.
- Concurrency above 4 opens extra connections exactly as before; they
close on release. Total retained connections per owner grow by at most 4
idle clients, which still close after the idle timeout.
- An action that sets session-level state would now leak it to the next
action. The contract is documented on `withDedicatedDbConnection` and in
`doc/DATABASE.md`; no current caller does this.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:37:04 -06:00
DottaandClaude Opus 5.5 c36b96ae00 fix(recovery): hold the start of queued automatic runs while a company's automatic-run brake is on (#21561)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The recovery system starts automatic runs: continuations, retries,
and sweep replacements
> - #21443 added a per-company automatic-run brake. It stops recovery
from creating new automatic runs while a company's runs keep failing
before they start
> - The brake did not hold automatic runs that were already queued. The
dispatcher still started them, so a burst of queued recovery runs kept
failing in the same way
> - This pull request adds the brake's start gate to the queued-run
dispatch pass, and wakes a held company at its next try without a
polling loop
> - The benefit is that a failing company stops spending runs on
automatic work, people's work still starts at once, and held work starts
again on its own when the brake releases

## Linked Issues or Issue Description

Refs #21443
Refs #21433
Refs #21427
Refs #21505

**What happened?**
While the automatic-run brake was open for a company, its already-queued
automatic runs (continuations and retries) still started. Each one
failed before it started, for the same cause that opened the brake.

**Expected behavior**
A queued automatic run must stay queued while the brake is open. Only
the brake's single probe, and then the half-open release rate (2, 4, 8
per window), can start one. A person's work (a Retry, or a continuation
of a person's request) must never be held. Held runs must start again on
their own, with nobody acting.

**Steps to reproduce**
1. Make 20 automatic runs in one company fail before they start, so the
brake opens.
2. Queue automatic continuations for that company.
3. Run a dispatch pass. Before this change, the continuations start.

## What Changed

- `server/src/services/heartbeat/queue.ts`: the dispatch pass asks
`reserveAutomaticRunStart` after the tried/backoff skip and the
claim-attempt budget check, and before the budget increment and the full
row read. A held run spends no claim attempt, does no row read, and does
not enter the declined-claim backoff. If the brake grants the start (its
probe or a half-open release start) but the claim then returns null,
throws, or finds the row gone, the pass gives the start back, so another
automatic run can use it in the same pass. The dispatcher's
person-origin answer is passed as is, and it is `undefined` only when
the dispatcher did not check origin (an agent with one queued run). Then
the brake's own cached person check decides.
- `server/src/services/heartbeat/queued-run-line.ts`:
`queuedRunDispatchFacts` also returns `personOriginChecked`, so the
queue knows when the dispatcher's negative answer is real.
- `server/src/services/heartbeat/dispatch-work.ts`: the queued-run
dispatch deadline worker adds companies whose next try is due to its
scopes, and includes the next try time in its next wake. A held company
wakes for its next try without a polling loop. If the brake's bounded
wake list had to drop a company, the worker wakes all companies at the
earliest dropped time.
- `server/src/services/recovery/automatic-run-brake.ts`: when the brake
holds a run, it records the company's next try time (probe time or the
next release window) in a bounded map. A probe or half-open release
grant returns a release that gives the start back with a compare-and-set
on its own run or window. It exports `reserveAutomaticRunStart`,
`takeDueAutomaticRunWakes`, and `nextAutomaticRunWakeAt`. The brake now
uses `isRecoveryRun`, `isPersonOwnedFailureCode`, and
`isRestartClassFailureCode` from `continuation-cause.ts` directly. The
rules are the same as before; the lazy loader keeps only the imports
that would make an import cycle.
- `server/src/__tests__/automatic-run-start-gate.test.ts` (new, 11 tests
through the real heartbeat queue): bounded queries for 250 held runs
over 100 passes, and for 1,500 held runs (past the 1,024-run cache) over
3 passes; a declined probe claim gives the probe back and another
automatic run starts in the same pass, and the probe stays with that run
on a later pass; an unused half-open start goes back to its window; with
276 held companies (past the 256-company wake list) every company is
still woken; a work run behind 250 held runs starts in the same pass; a
person's Retry is never held; the only queued run is not held when it
continues a person's Retry; exactly one probe starts across two server
instances; the release gives 2, 6, 14 cumulative starts over three
windows; held runs are not in the declined-claim backoff after the
pause; the dispatch worker wakes the company at the next try; and an
end-to-end test that a held automatic run stays queued while the brake
is open, then starts and succeeds on its own at the next try.
- Two small test updates in `automatic-run-brake.test.ts` and
`automatic-run-brake-import.test.ts` for the new cache size field and
the moved rule.

- `server/src/__tests__/automatic-run-brake-recovery-passes.test.ts`:
one test for the sweep path from #21505 (continue a blocked task behind
a run refused at dispatch). The sweep only records the continuation (no
wake, no new run), and `deliverReconciledExecutions`, which the brake
already gates, creates the run only after the pause. No second gate is
needed on that path.

## Verification

- `npx vitest run src/__tests__/automatic-run-start-gate.test.ts`: 11
passed.
- Red before for the review fixes: on the first PR head, the 1,500-run
cost test made 3,848 queries (bound 130), the declined-probe test
started 0 runs (expected 1), and the 276-company wake test missed 20
companies.
- Red before: with the `queue.ts` gate line removed, 5 of the 8 tests
fail (bounded pass, single probe, release rate, backoff, end-to-end).
- Red before for the person case: with the dispatcher's `false` passed
through as final, "never holds the only queued run when it continues a
person's Retry" fails (0 started). It passes after the change.
- `automatic-run-brake.test.ts` 19 passed,
`automatic-run-brake-import.test.ts` 1 passed,
`automatic-run-brake-recovery-passes.test.ts` 28 passed,
`failed-continuation-continues-again.test.ts` 32 passed (after merging
master with #21505).
- Nearby queue tests (on the final head): `heartbeat-run-dispatch-work`
15, `queued-run-dispatch-stacking` 6,
`heartbeat-queued-run-claim-isolation` 10,
`assigned-work-stalled-queued-run` 12, all passed.
- `server` `tsc --noEmit` is clean.

## Risks

- Behavior change only while a company's brake is open or half-open.
When the brake is closed, the gate returns from its cache with no query.
- The first time the gate sees a non-person automatic run while the
brake is on, the brake does one person-origin query for that run. The
result is cached per run (bounded to 1,024 runs).
- No migration. No API or UI change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:33:53 -06:00
DottaandClaude Opus 5.5 84cb28f9b7 fix(connections): keep machine sign-in secrets and settings in one locked commit (#21566)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21473 added machine sign-in (MCP OAuth client credentials) for
organization connections
> - Auto-merge squashed #21473 from an earlier head, so its last review
fixes did not reach master
> - Those fixes close a P1 race and two smaller setup gaps that Greptile
found on #21473
> - This pull request applies that last commit to current master, with
its tests
> - The benefit is that a client ID and its secret always stay together,
and key switching and app selection work as intended

## Linked Issues or Issue Description

Refs #21473

**What existing behavior does this improve?**
Machine sign-in setup and rotation, added in #21473.

**Problem**
- A connection test wrote a replacement client secret before checking
whether the connection changed. If the save was then refused, the secret
stayed and could pair with another client ID, so the next token request
failed.
- Switching a client-secret connection to a Paperclip key always created
an RS256 key, even when the sign-in service accepts only ES256.
- Machine sign-in dropped the setup's app ID and created a separate app.

**Proposed improvement**
Write the secret and the settings in one locked transaction after the
change check. Record the accepted signing algorithms and use them. Pass
the app ID through.

## What Changed

- `client-credentials-service.ts`: `saveConnection` locks the row,
checks that the sign-in method and credential references are unchanged
and that the connection is not removed, then writes any new credential
and the settings in the same transaction. Test, secret rotation and key
staging all use it.
- Setup records `tokenEndpointAuthSigningAlgValuesSupported`. Switching
from a secret to a key uses `chooseAssertionAlgorithm` on it.
- UI: `MachineSignInSetup` takes `applicationId`, and the custom-server
flow passes `prefill.applicationId`.
- Tests: two new server tests and one UI expectation.

## Verification

- `server/src/__tests__/tool-access-client-credentials.test.ts`: 25/25
pass. The 2 new tests fail on master's service code ("never leaves a
refused test's secret behind with another client ID", "creates the kind
of key the sign-in service accepts when switching from a secret").
- `MachineSignInSetup.test.tsx` and
`GenericMcpConnectStep.machine.test.tsx`: 8/8. The app ID expectation
fails on master's component.
- Server `tsc --noEmit` clean. UI `tsc` clean except the pre-existing
generated Storybook catalog imports.

## Risks

Low. The change is limited to machine sign-in. A concurrent change now
returns 409 `client_credentials_changed` instead of saving partly. No
migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 05:30:14 +00:00
DottaandClaude Opus 5.5 bbed32450c perf(load-model): model recorded native run rates, the recorded event mix and coalesced delta batches (#21549)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `scripts/benchmark-agent-scaling-load.ts` (#21418) estimates how
many concurrent agents one server process can carry
> - It assumed 20 text deltas/s and 1 other event/s per agent, and every
other event was an `item.completed`
> - Recorded hosted native runs show about 0.2 deltas/s and 0.5 other
events/s, with a mix in which 39% of the other events are server trace
spans that take a cheaper path
> - This pull request makes the recorded rates and mix the model's
default, keeps the old rates as an explicit stress profile, adds a burst
profile, and mirrors the delta batching of #21520
> - The benefit is a model that measures what production does, so the
next optimizations target the real per-agent cost

## Linked Issues or Issue Description

Refs #21418 (load model and plan). Refs #21520 (delta batching that the
model now mirrors).

**What existing behavior does this improve?**
The load model's default rates, its event mix and its delta batching
rule, and the plan document's measurements.

**Subsystem affected**
`scripts/benchmark-agent-scaling-load.ts`, its test
`server/src/__tests__/agent-scaling-load-model.test.ts`, and
`doc/plans/2026-11-13-thousand-concurrent-agents.md`.

**Current behavior**
The model defaults to 20 deltas/s and 1 other event/s per agent. Every
other event is a native `item.completed` with a chat-progress wake and a
checkpoint save. It cuts a delta batch on the timer even while the
previous batch commits.

**Proposed behavior**
- `PROFILE=recorded` (default): 0.2 deltas/s and 0.5 other events/s per
agent. `PROFILE=burst`: 2.5 and 3, a run's busiest minute.
`PROFILE=stress`: the earlier 20 and 1. API requests stay at 0.3/s. Each
rate can still be set on its own.
- `EVENT_MIX=recorded` (default) sends other events in the recorded
proportions. A `run.performance.span` is an ordinary append with no wake
and no checkpoint, as the server's run trace writes it.
`tool.execution.*` and `item.completed` are native appends with a
chat-progress wake, then a checkpoint save. `EVENT_MIX=<type>` sends one
type only.
- Delta batching follows #21520: deltas that arrive while a batch
commits all join one next batch.
- The report adds `profile` and `eventsIssuedByType`.

**Reason and benefit**
At the recorded rates the per-agent cost is the non-delta runner events
and their checkpoint saves, not deltas. The plan and the next
optimizations need that picture.

**Breaking changes**
None for the product. Anyone who runs the model with no options now gets
the recorded profile; `PROFILE=stress` gives the earlier load.

## What Changed

- Profiles, the recorded event mix (a 100-slot cycle per agent, offset
by agent so the fleet sends the mix at every moment), the span append
path, and the coalesced delta batches.
- The model test now checks the profile in the report and that both
server trace spans and runner events with a progress wake are in the
mix, for the recorded profile and for the stress profile with one
transaction per delta.
- The plan has a new section "Load model at recorded rates": the
recorded source (40 recent native runs on one hosted workspace, as
aggregates), the profiles, the mix and its server paths, fresh runs at
500, 1,000 and 2,000 agents plus burst and stress, and what they show.
It records the 3-connection checkpoint pool as a computed per-process
limit on a hosted database, and that SQL building in Drizzle costs about
as much main-thread time as result parsing on the hosted instance. A new
section "Considered and not done" records why `run.performance.span`
rows stay (about 3% of the statements per agent, with readers) and why
checkpoint saves per critical event only are not done now.

## Verification

- `server/src/__tests__/agent-scaling-load-model.test.ts` passes (2
tests). The new assertions fail against the previous script: it reports
no `profile` and no `eventsIssuedByType`.
- `server/src/__tests__/agent-scaling-load-model.test.ts` also checks
each profile's rates with no overrides, and that a rate set on its own
wins.
- Load model runs, recorded in the plan (60 s, main pool of 10, embedded
PostgreSQL on the same machine, each run started below a 1-minute load
average of 15): recorded rates at 1,000 agents (event-loop utilization
0.61) and 2,000 (0.85) finish every operation with no failure; 500
agents all in their busiest minute run at 0.95 with p99 lateness under
0.4 s. About 4.4 statements per agent per second at recorded rates, 35%
of them on the checkpoint pool.
- Earlier runs at load averages of 20–55 had shown one process falling
behind at 500 burst agents and at 2,000 recorded; the quieter runs after
the model fixes (event order, spans off the runner stream) do not, and
the plan says so. The checkpoint pool as a per-process limit is now
stated as a computed limit for a hosted round trip (about 400 agents in
a busy minute at 1 ms), not as measured.
- Greptile findings fixed: a waiting delta batch no longer takes deltas
sent after the next runner event, takes every delta queued when it
starts, and spans no longer join the runner event stream.

## Risks

- Low: a benchmark script, its test and a plan document. No product code
changes.
- The machine that measured the plan's table was shared; the plan marks
the rows that the machine distorted.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:13:07 -06:00
DottaandClaude Opus 5.5 351b69941c perf(checkpoint): write a routine controller checkpoint save in one statement (#21553)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote runner's controller saves a checkpoint after each runner
event it commits, on a 3-connection pool, before it acknowledges the
event
> - Each save is a transaction of 4 statements (5 where commit
notifications are on): BEGIN, the owner lock with the transaction id,
the write, COMMIT
> - What forces the transaction is the live-run recovery wake that every
save registers. Recovery does not react to a routine save
> - This pull request writes a routine save (new or changed event,
command and cursor records) with one statement that checks the owner and
writes, and keeps today's transaction for every other save
> - The benefit is one statement instead of four for most checkpoint
saves, on the pool that limits one process in a busy minute

## Linked Issues or Issue Description

Refs #21528 (owner lock returns the transaction id). Refs #21549 (load
model at recorded rates; the plan records the checkpoint pool as a
per-process limit).

**What existing behavior does this improve?**
`nativeRunnerCheckpointWriter` in
`server/src/services/native-runtime/native-runner-checkpoint.ts`.

**Subsystem affected**
Controller checkpoint persistence for remote native runs.

**Current behavior**
Every save locks the coordinator row (`native_run_finalizations` `for
update`), checks the owner, calls `notifyLiveRunRecovery` (6 recovery
queues), and upserts the changed parts, in one transaction.

**Proposed behavior**
A routine save is one statement: a CTE locks the coordinator row only
while the lease owner, boot id, generation and an unexpired lease
(`clock_timestamp()`) all match, and the upsert writes only through that
row. It registers no recovery wake. If the owner does not match, it
writes nothing and the save takes today's full path, which runs the
complete owner check (successor rules included) and fails with
`native_runner_checkpoint_owner_lost` when it should.

A save is routine when it has changed parts, no reset, no removed parts,
no header change, no predecessor, and the run is its own authority.
Every other save (the first save of a controller, which is a reset, a
save after a failed save, header or retirement changes, removals,
source-run consumers) is unchanged.

**Reason and benefit**
At recorded rates about a third of one process's statements are
checkpoint saves. In a busy minute the 3-connection pool cannot keep up
at 500 agents (#21549).

**Breaking changes**
None. No schema change.

## What Changed

- `ROUTINE_CHECKPOINT_SAVE_SQL` and the routine branch in
`nativeRunnerCheckpointWriter`.

## Why a routine save needs no recovery wake

I read the six consumers of `notifyLiveRunRecovery`:
- `nativeProcessSessions`, `nativeQuestionCancellations`,
`activeExecutionRecovery`, `settledNativeProcesses`
(`live-recovery-work.ts`): each selects candidates from `heartbeat_runs`
status, runtime mode, `native_run_finalizations` phase, lease and retry
times, the context snapshot and process identity. None reads checkpoint
rows to decide eligibility.
- `strandedAgentStatus` (same file, `agents` rows) and
`activeRunSilence` (`active-run-watchdog/adapters/deadline-work.ts`,
`heartbeat_runs.last_output_at`): neither reads checkpoint rows.
- The checkpoint rows matter to recovery in two places:
`listExpiredNativeRunnerAuthorities` requires that a checkpoint exists,
and the remote bootstrap proof reads its parts. A checkpoint first
appears with a reset save, which keeps its wake. A routine save only
adds event and command parts, which can only take a run further from the
bootstrap proof; recovery reads the current rows when its own deadline
(lease expiry) fires.
- The intent of the wake also fenced a recovery pass against a save in
flight. The routine save is one atomic statement and holds the
coordinator row lock while it writes, so a recovery claim that takes
that row waits for it, and a reader sees the save whole or not at all.

## Verification

- Red/green on embedded PostgreSQL
(`native-runner-checkpoint-save.test.ts`); the new tests failed against
`origin/master` first:
- A routine save is one statement (`with owner as ... for update ...
insert into native_runner_checkpoints`) and registers no recovery wake
(before: 4 statements and a wake).
- A reset save keeps BEGIN, the owner lock with the transaction id, and
COMMIT, and still registers its recovery wake.
- A routine save that waits while another connection takes over the
lease (new owner and generation) falls back to the full path and fails
with `native_runner_checkpoint_owner_lost`; the checkpoint is unchanged.
- The same race where the new owner also resets the checkpoint (deletes
every part and writes its own header): the save fails the same way and
the checkpoint holds only the new owner's header.
- A lease that expires while the routine save waits for the coordinator
row lock (the holder does not change the row) writes nothing: the
statement judges the lease after the lock, as
`lockNativeRunnerCheckpointOwner` does. My first version checked the
lease in the locking CTE itself, and PostgreSQL does not re-check a
condition after a lock wait on an unchanged row; the existing
`native-runner-restart-recovery.integration.test.ts` test "fences
checkpoint delivery when ownership expires during an unchanged row-lock
wait" failed on it, and so does the new test. Both pass now.
- The existing test "refuses a save after the owner's lease expired and
writes nothing" passes through the new fallback.
- Mutation checks: with the owner conditions removed from the statement,
both race tests and the expired-lease test fail; with the lease check
moved back into the locking CTE, the lock-wait expiry test fails.
- Other focused suites, one file at a time:
`native-runner-restart-recovery.integration` (231 passed, 14 skipped),
`native-runner-restart-checkpoint-cost.integration` (1; it greps for
`insert into "native_runner_checkpoints"`, so the statement quotes the
table name), `native-recovered-owner-followup` (97),
`native-controller-recovery.integration` (30),
`native-runner-checkpoint-authority.integration` (1),
`native-restart-recovery-isolation` (1), the port suite with conformance
(105), `heartbeat-native-restart-suspension` (115), `live-recovery-work`
(51 tests pass; on this busy machine its teardown hook timed out, as it
has before on master). `pnpm --filter @paperclipai/server run typecheck`
passes.
- Statements per routine save: 4 → 1 (5 → 1 where commit notifications
are on). I did not rerun the load model for this change: the machine is
too busy for a fair timing comparison today, and the statement count is
exact.

- CPU per operation (a local script, not committed: 600 routine saves of
one event part and the cursor for one run on embedded PostgreSQL, one
connection, three rounds, `origin/master` code against this branch):
main-thread CPU per save 0.38–0.56 ms before and 0.10–0.12 ms after
(encryption included in both); wall time per save 0.45–0.73 ms before
and 0.22–0.30 ms after.

## Risks

- A routine save no longer wakes the six recovery queues. Their own
deadlines and the 30 s reconstruction still run; see the reading above
for why no decision depends on a routine save.
- The owner check of the routine statement accepts only the exact active
owner. Any other state goes to the unchanged full path, so a successor
or a finished result keeps today's rules.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:58:06 -06:00
DottaandClaude Opus 5.5 67652580ee perf(tool-access): index enabled policies so each decision checks only candidates (#21563)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool access policies decide whether an agent can call each tool, and
an MCP tools/list response decides access once for each listed tool
> - Each decision matched the raw selector of every enabled policy two
times, so one listing cost tools x policies selector checks on the
server's main thread
> - On staging, a company with 363 enabled policies kept the main thread
at 95-98% of a core: policy matching was 22% of CPU time, with single 80
ms runs inside the selector match
> - This pull request compiles the enabled policies once for each
decision cache and indexes them by the selector key that rejects most
(catalog entry, then tool name)
> - The benefit is that each decision checks only the policies that can
match its tool, with the same decisions, so a listing no longer blocks
the event loop

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Tool access decisions for MCP tool listings
(`toolAccessPolicyService.decide`, called by `decideToolsForListing` in
the tool gateway).

**Subsystem affected**
Server: tool access policy evaluation
(`server/src/services/tool-access-policy.ts`).

**Current behavior**
Each decision loops over all enabled policies two times. The first loop
matches the selector and schema-checks the conditions of every
conditioned policy. The second loop evaluates conditions and matches the
selector of every policy. The selector match reads and allocates every
selector key in a fixed order, so a catalog entry mismatch is found only
at the end. A listing of 800 tools with 365 policies does 305,600
selector evaluations. On staging this was 22% of main-thread time
(event-loop utilization 0.965, delay p99 144 ms, max 710 ms).

**Proposed behavior**
The decision cache holds the enabled policies compiled once: parsed
selectors, an index by catalog entry and tool name, and a memoized
conditions schema check. Each decision evaluates only the candidate
policies, in the original priority and creation order.

**Reason and benefit**
Listing cost becomes proportional to the policies that can match each
tool. In the benchmark, one listing goes from about 164 ms to 12 ms of
main-thread time, and the longest event-loop block goes from 84 ms to 4
ms.

**Breaking changes**
None. The policy data model, API, and listing contract do not change.

## What Changed

- Add `server/src/services/tool-access/policy-selector-index.ts`. It
compiles a policy selector into a matcher with the same semantics as the
raw `selectorMatches`, and it indexes policies by `catalogEntryId`, then
by `toolName` (matched against the tool name and the upstream tool name)
for policies without a catalog entry selector. Policies with neither key
are candidates for every call. Candidates keep their original order.
- `decideInternal` caches the compiled index under the same
`enabled-policies:${companyId}` key. The new `matchEnabledToolPolicies`
keeps the two-step semantics: the first selected policy with unsupported
conditions denies the call before any match applies, then the selected
policies with matching conditions go to the unchanged decision loop
(block, rate limit, trust rule, require approval, allow, explanations).
- Decisions without a cache, and decisions that consume a rate limit
(which read fresh rows), compile the rows they read in the same way.
- The conditions schema check runs once per policy and cache, not once
per decision.
- Conditions are now evaluated only for policies whose selector matches.
Before, a malformed legacy condition on an unrelated policy could throw
in every decision. A selected policy with such conditions still gets the
unsupported-conditions deny first.
- Add `server/scripts/benchmark-tool-policy-matching.ts` and a section
in `doc/mcp-discovery-performance.md`.

## Verification

- Red/green regression:
`server/src/__tests__/tool-access-policy-index.test.ts` "reads policy
selectors once per decision batch, however many tools it decides". It
decides 40 tools with one cache against a staging-shaped set of 365
policies and counts reads of the raw policy selectors. On
`origin/master` it fails: 308,110 reads for 40 tools against 7,711 for
the first tool. With this change the count stays at the first-tool
count. The test also checks that cached and uncached decisions are
equal.
- Equivalence: the same file compares the compiled selector with the raw
`selectorMatches` for 404 selectors x 300 contexts (including empty,
whitespace, non-string, string-plural, duplicate and contradictory
values). It compares the indexed policy selection with the old unindexed
two-step selection (copied into the test) for 5 random sets of 400
policies x 400 contexts, and for the staging-shaped set over 800 tools.
The sets include catalog-entry-only, catalog-entry+connection,
multi-key, tool-name, unindexed, conditioned, unsupported-condition,
block, rate_limit, trust_rule, allow and legacy policy types with mixed
priorities.
- Index edge cases: an earlier tool-name policy (matched through the
upstream name), an unindexed policy and an empty catalog-entry selector
apply before a catalog-entry block; an unsupported-condition policy in
another bucket still denies. A DB-backed test checks the same through
`decide`, with and without a cache.
- Focused suites pass locally: `tool-access-policy-index`,
`tool-access-policy-service`, `tool-gateway-listing-memory`,
`connector-tool-policy`, `tool-gateway`, `tool-gateway-service` (6
files, 406 tests). `node server/scripts/typecheck-layers.mjs` passes.
- `pnpm verify` exits 2 (the new benchmark script is a broad input), so
CI supplies the broad coverage.
- Benchmark, local embedded PostgreSQL, macOS arm64, Node 24. One
listing of 800 tools, 365 enabled policies (staging shape plus the 2
fixture policies), 16 concurrent decisions with one cache as in the
gateway. Three runs of 9 alternating rounds after 2 warm-up rounds. Run
3 uses the event-loop delay histogram that also covers the end of the
listing. Both versions return identical decisions.

| Per listing | `origin/master` median (min-max) | This PR median
(min-max) |
  | --- | --- | --- |
  | Elapsed, run 1 | 157.5 ms (146.1-192.8) | 11.4 ms (10.3-15.3) |
  | Elapsed, run 2 | 165.3 ms (161.7-185.4) | 11.9 ms (10.1-14.0) |
  | Process CPU, run 1 | 160.5 ms (149.7-188.9) | 13.7 ms (11.1-21.0) |
  | Process CPU, run 2 | 162.8 ms (157.2-192.7) | 12.3 ms (9.6-20.5) |
  | Elapsed, run 3 | 163.9 ms (151.6-185.3) | 12.4 ms (10.4-18.5) |
  | Process CPU, run 3 | 165.4 ms (155.7-214.9) | 14.5 ms (10.6-20.2) |
| Longest event-loop block, run 3 | 84.1 ms (76.9-95.1) | 3.7 ms
(3.0-5.0) |
  | Selector evaluations | 305,600 | 1,164 |
  | Raw selector property reads | 6,155,830 | 9,490 (compile only) |

  ```sh
cd server && pnpm exec tsx scripts/benchmark-tool-policy-matching.ts
--baseline-ref origin/master --tools 800 --rounds 9
  ```

- Not verified: the hosted staging profile after deploy.

## Risks

- Low risk. The decision loop is unchanged; only the set of policies
that reach it is narrowed, and every left-out policy has a selector that
rejects the call. The equivalence tests compare against the old
selection.
- The index lives only in the per-listing decision cache (or for one
uncached decision). There is no cross-request cache, so policy edits
apply on the next listing as before.
- The cached index memoizes the conditions schema check on shared
entries. The check is pure, so concurrent decisions in one listing get
the same result.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test and benchmark runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:51:43 -06:00
Dotta ca7d42ba71 fix(heartbeat): refresh queued dispatch when task drains end (#21562)
## Thinking Path

> - Paperclip runs agent work for a company.
> - A task drain holds new run starts while maintenance or idle handoff
is in progress.
> - A queued dispatcher can keep an unreconciled request and a 30-second
retry while the drain is active.
> - Ending the drain can resume domain work without refreshing that
dispatcher's saved deadline.
> - This change refreshes the pending queued owner when a drain ends or
its expiry is observed.
> - Normal admission checks and real retry deadlines still apply.

## Linked Issues or Issue Description

Related: #20640. That change refreshes deferred receipt workers. This
change refreshes the separate queued-run dispatcher.

**What happened?**

A queued-work notification received during a drain could leave a
30-second fallback in the dispatcher after the drain ended. The owner
stayed unreconciled after useful work had settled. This delayed its next
pass and the instance's idle state.

**Expected behavior**

Refresh an unreconciled queued owner when normal dispatch becomes
allowed. Keep other drain holds, admission rules and scheduled retry
deadlines.

**Steps to reproduce**

1. Let the normal dispatcher complete its first pass.
2. Start an idle task drain and commit a company-scoped queued-work
notification.
3. Stop the drain, or observe its expiry, before the saved fallback is
due.
4. Check that the scoped queued pass runs before that fallback. The two
regression cases fail on the unchanged owner and pass with the fix.

**Paperclip version or commit**

The main-specific patch is based on
`f7b1651799d8c0c660a76266878c420a12f0beca`. The bug was found in
assembled sleep acceptance. The affected queued-dispatch, drain and
notification modules match main.

## What Changed

- Listen for normal drain changes and observed expiry in the queued-run
owner.
- Wake that owner only when dispatch is allowed and its reconciliation
is pending.
- Forward the deadline worker's optional stop callback after its stop
fence.
- Remove both subscriptions when the owner or coordinator stops.
- Add release, expiry, remaining-gate, shutdown, retry-deadline and idle
controls.

## Verification

Local checks used the integrated feature branch that exposed the issue.
The main-specific port preserves main's existing `canRun` logic and
excludes the private branch's extra cancellation hook.

- Both new release and expiry cases failed at the intended scoped-resume
assertion before the product change.
- Six dispatch cases pass with the normal package import.
- Fourteen existing deadline controls and nineteen queue-order controls
pass. Their inputs did not change during the test-import correction.
- The complete normal Server typecheck and its three affected-file
program checks pass.
- The first typecheck caught a test import outside the Server root. The
import now uses the existing public package wildcard. The corrected
six-case suite and normal typecheck pass.
- Normal Source SDK input verification passes.
- Exact-head CI at `fd623d3ba9345e5ec8f252c55932ff797cd57e8f` completed:
54 passing checks, four expected skips and one neutral result; Snyk
passed. Greptile reviewed this exact head at 5/5 with no actionable
findings. All review threads were read; none remain open. A full local
repository build and suite were not run in this checkout.

The existing admission-drain contract in `doc/execution-semantics.md`
already requires prompt admission after drain release. This fix restores
that contract. No configuration or command documentation changes are
needed.

## Risks

Drain events can request an earlier pass for a pending owner. The normal
permission, reconciliation, coalescing and earliest-time checks remain
active. The scheduled retry owner has no new listener. Shutdown must
remove both listeners; the coordinator-stop control covers that path.

This PR does not claim deployed suspension or complete transport sleep
acceptance.

## Model Used

OpenAI `gpt-6.1-sol` assisted with implementation and independent
review. The Codex managing agent performed technical review and
verification. The host does not expose its exact runtime model ID or
context-window size. Tools used include source inspection, code edits
and local test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (not
required: the existing contract remains accurate)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-13 22:45:26 -06:00
DottaandClaude Opus 5.5 9824277f65 perf(run-events): append events whose only wake is chat progress in one statement (#21559)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Most runner events that are not text deltas
(`tool.execution.started/progressed/completed`, `item.completed`) wake
the chat progress projection
> - That wake is registered inside the append's transaction, so each
such event takes 4 statements (BEGIN, the run lock read, the append,
COMMIT), where an event without a wake takes 1 (#21511)
> - The chat progress projection rebuilds itself from the stored events;
the wake only tells it when to look
> - This pull request writes an event whose only wake is chat progress
with the one-statement append, and sends the wake after that statement
commits
> - The benefit is 1 statement instead of 4 for most non-delta runner
events, with response, reassessment, retention and recovery wakes
unchanged

## Linked Issues or Issue Description

Refs #21511 (one-statement append). Refs #21528 (lock read returns the
transaction id). Refs #21549 (load model at recorded rates).

**What existing behavior does this improve?**
`appendHeartbeatRunEvent` for native source events whose only wake is
chat progress.

**Subsystem affected**
Server run event ingest (`heartbeat-run-events.ts`), the chat progress
notification helper, and two additive exports in
`packages/db/src/work-signals.ts`.

**Current behavior**
A progress event opens a transaction, reads the run lock with
`runtime_mode`, the bound task and the transaction id, probes for a
replay, registers the chat-progress wake as a tracked intent, appends,
and commits.

**Proposed behavior**
When chat progress is the event's only wake, the append takes the
one-statement path of #21511. That statement also returns the run's
`runtime_mode` and bound task id from the row it locks, and, where
commit notifications are on, calls `pg_notify` inside the same statement
for runs that are native and bound to a task, so other server processes
get the wake at commit. After the statement returns, this process
publishes the wake as settled to its own listeners. A replay, a gap, a
changed run or any other fallback takes the existing transaction, which
registers the wake as before.

**Reason and benefit**
At recorded rates these events are about 61% of the non-delta events
(#21549), so this cuts about 3 statements per such event.

**Breaking changes**
None. No schema change.

## What Changed

- `appendHeartbeatRunEvent`: `progressOnly` (progress set, every other
wake clear) also qualifies for the one-statement path.
- `APPEND_NEW_NATIVE_SOURCE_EVENTS_WITH_PROGRESS_SQL`: the one-statement
append, plus `runtime_mode` and the bound task id (`boundRunIssueIdSql`,
rendered once) from the locked row, plus a conditional `pg_notify`.
- `chat-native-progress-notifications.ts`:
`chatNativeProgressCommitNotification` and
`publishChatNativeProgressSettled`.
- `work-signals.ts` (additive): `databaseWorkCommitNotification(owner,
topic)` returns the cross-process payload when notifications are on;
`publishDatabaseWorkSettled(owner, topic, hint)` publishes a settled
wake to this process's listeners.

## Verification

- Red/green on embedded PostgreSQL. Each new test failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "appends an event whose
only wake is chat progress in one statement and wakes progress after it
commits": 1 statement (before: a transaction), the wake is published
once as settled with no intent, and a replay publishes nothing.
- "keeps the transaction for the last events of a turn and a run, and
one statement for progress-only events":
`tool.execution.started/progressed/completed` and a non-final
`item.completed` take one statement; a final `item.completed`,
`run.result.accepted`, `turn.completed`, `turn.failed`, `turn.cancelled`
and `turn.interrupted` keep the transaction.
- "sends the progress wake to other server processes with the one
statement's commit": with commit notifications on, another connection
that listens on the work channel receives exactly one chat-progress
notification, and a replay sends none.
- `chat-native-progress-work.test.ts` "wakes the projection from a
one-statement progress event after the statement commits": the real chat
progress worker publishes the progress for an event appended in one
statement.
- Mutation check: without the settled publish after the statement, that
test fails.
- Lost-wake repair (condition b): an event row stored without any wake
models a lost hint.
- "repairs a lost progress wake with the run's next progress event":
nothing is published for the stored event; the run's next progress event
publishes the progress.
- "repairs a lost progress wake with the startup reconcile": a worker
that starts after the stored event publishes it.
- The existing response-wake test now uses a final reply and still shows
the intent before the write, and the race test of #21528 uses a final
reply so it still covers the transaction path.
- Other focused suites, one file at a time: the port suite with
conformance (105), `native-observer-replay` (1),
`paperclip-semantic-conformance` (1), `native-interaction-bridge` (7),
`heartbeat-run-event-sequencing` (5), `native-event-boundary-golden`
(1), `heartbeat-native-restart-suspension` (115), `live-recovery-work`
(51), `work-signals` (25), atomic-append (31),
`chat-native-progress-work` (38). `pnpm --filter @paperclipai/server run
typecheck` and `pnpm check:module-boundaries` pass.

- CPU per operation (a local script, not committed: 800
`tool.execution.progressed` appends to one native task run on embedded
PostgreSQL, one connection, three rounds, `origin/master` code against
this branch): main-thread CPU per event 0.28–0.50 ms before and 0.10 ms
after in the warm rounds; wall time per event 0.84–1.04 ms before and
0.52–0.53 ms after.
- Inside a caller's transaction: new test "keeps a progress-only event
inside a caller's transaction on the locked path, and settles its wake
only when that transaction ends". `publishDatabaseWorkSettled` publishes
at once only outside a transaction; inside one it waits for the
outermost commit and drops the wake on rollback (new
`work-signals.test.ts` test, which failed with the unguarded publish).
The one-statement path is only taken when the owner is not a transaction
(`!is(db, PgTransaction)`, #21511), so the two guards agree.

## Risks

- **Lost wake.** If the statement's outcome is unknown (the connection
drops after the server ran it), this process sends no wake for that
event. In a chat channel (Slack, Telegram) the progress line for that
task can then stay one step behind (for example it does not move from
"working" to the latest tool step) until the run's next runner event, a
run milestone, or a server restart. A turn normally ends with an event
that keeps the transaction
(`turn.completed/failed/cancelled/interrupted`), and the run's
completion notifies chat run milestones, which include chat progress
(`notifyChatRunMilestones` in `run-completion.ts` and
`native-run-finalizer.ts`). So progress cannot stay stale after a run
ends; in the worst case it lags one step for the rest of a turn. The
projection rebuilds from stored events, so nothing is lost or
duplicated.
- The chat-progress wake no longer publishes an intent before the write.
Intents let the coordinator fence a pass against an in-flight write; the
settled wake after the commit makes the projection run again after the
row is visible.
- `packages/db/src/work-signals.ts` gets two additive exports and no
change to existing behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:37:35 -06:00