Commit Graph
8021 Commits
Author SHA1 Message Date
DottaandPaperclip 7c0c3c4f31 fix(mcp): preserve Tasks sleep authority and re-enable recovery (#21603)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected apps can retain unfinished provider work through MCP
Tasks.
> - Managed defaults can enable Tasks without a stored experimental
flag.
> - The sleep check used the stored flag and could miss an active
provider task.
> - Enabling Tasks also failed to notify an owner that had become idle.
> - This pull request uses the effective setting and wakes the existing
owner after the setting commits.
> - Pending work keeps its recovery path. Completed history can become
quiet.

## Linked Issues or Issue Description

Refs #21263, which added durable provider Tasks. The current roadmap
covers the governed MCP Tool Gateway. This change repairs two existing
paths.

**What happened?**

The gateway could enable MCP Tasks through managed defaults while the
idle check reported no pending work. After an operator disabled and
enabled Tasks, an existing task could stay idle until a separate task
write woke its owner.

**Expected behavior**

The gateway and sleep check must use the same effective setting.
Enabling Tasks must notify the existing owner. It must retain the task's
original deadline and backoff.

**Steps to reproduce**

1. Retain a task with `next_poll_at`. Use managed Tasks defaults without
a stored true flag. Read the normal sleep projection.
2. Disable Tasks, retain the same task, then enable Tasks through the
settings service without writing the task again.

## What Changed

- Read the effective Tasks flag inside the same read-only sleep
transaction. Bind that value to the existing pending-task query.
- Register the existing Tasks queue notification before the settings
transaction writes. The normal commit protocol owns the wake.
- Add two real-PostgreSQL regressions to the existing nine-case suite.
Extend eight mock transactions with the read projection they now need.
Keep their assertions unchanged.
- Publish the Tailscale enrollment fixture state by rename so its CLI
watcher cannot parse a truncated JSON file. Existing enrollment
assertions and time limits remain unchanged.

## Verification

The two regressions fail against the preceding current-master products:
the pending-work projection reports `none`, and re-enable does not rearm
the owner within the original two-second oracle. After the fix, all nine
PostgreSQL cases pass at the recorded source, including completed
history, disabled Tasks, credential debt and fail-closed query errors.

At current `72bc32f`, the normal DB build and migrations, all nine
PostgreSQL cases, 155 affected mock cases, and the complete normal
Server typecheck/planner pass. The mock run includes three selected
safety families and six complete files; it explicitly excludes 196
existing durable-PG cases. All original case deadlines and assertions
remain. The startup file's earlier 133-case pass is limited to its
unchanged mock contract; a new hot-restart implementation is not
relabeled as tested. Root independently verified all current outputs and
process closure. The MCP product and regression files remain
byte-identical at `321c4b2ae2f3`; its additional change is the Tailscale
test fixture repair below. Exact current-head CI passes: 55 successful
checks, four explicit skips and one neutral result; Snyk succeeds. Fresh
Greptile is 5/5 for this exact head. The complete latest edited summary
and all paginated threads were inspected, with no open findings.

An earlier additional whole-eight-file run reached its 180-second
aggregate watchdog while the existing mixed PostgreSQL suite was still
running. It produced no individual failing assertion. Its startup file
passed 133 cases. The original failure remains recorded. The follow-up
selects the three affected mock families and runs the other affected
mock files whole; it does not claim that the interrupted PostgreSQL file
passed. No test assertion or case deadline changed. Full required CI
remains a merge gate.

The first remote run at `72bc32f` passed 51 checks but failed one
Tailscale enrollment case and an interrupted browser shard; two
aggregate gates then failed. The browser log records a runner shutdown
signal; its underlying cause is unknown, and one specific-job recovery
was requested. The Tailscale case failed its status CLI on both original
CI attempts. A local control then held publication until the original
file watcher observed an empty state: the CLI threw `Unexpected end of
JSON input`, and the existing enrollment assertion failed. Publishing
the state atomically passed the same control. All nine original
enrollment cases now pass in 4.393 seconds with the fixture repair. The
recorded remote interleaving remains unknown; the original logs and the
initial local non-reproduction are retained. No retry count, assertion,
timeout or product behavior was weakened. The later exact-head run
passes all required checks. A subsequent server shard was interrupted by
a runner shutdown and later recovered at the same source; the
interruption and unknown underlying cause remain recorded.

Use Node24/pnpm9, one worker, an owned external PostgreSQL test database
and the existing fixture/admin URL options. The two new regressions are
in `server/src/__tests__/idle-sleep-diagnostics-postgres.test.ts`.

These checks do not establish deployed suspension or a live provider's
availability.

## Risks

- The read adds the normal effective-settings projection to the existing
bounded snapshot. Query errors must still block sleep.
- A settings update now notifies the existing task owner. Transaction
rollback must not publish that notification.
- This adds no schema, provider protocol, timer or replacement task
framework. Existing active task polling remains required until its work
settles.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI `gpt-6.1-sol` implemented the regressions and advised the fix.
GPT-6-based Codex also reviewed and integrated it. The host does not
expose every supervising model ID or context-window size. The agents
used source editing, Git, code execution and regression tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (the focused scope above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (the
existing pending-task contract remains accurate)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-14 05:00:50 -06:00
DottaandClaude Opus 5.5 aaec67b891 fix(native): close idle warm sessions in parallel across agents at shutdown (#21597)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - At shutdown, the server closes its idle warm Runner sessions before
it prepares the hot-restart snapshot
> - It closed them one at a time, and each full close (checkpoint, then
two stopped-copy collections) takes 3-5 s
> - In a zero-downtime swap test, five closes used the whole 20 s
pre-drain budget. Hot-restart preparation timed out, a checkpoint was
cut mid-way, and shutdown logged "hot-restart shutdown preparation
failed"
> - The same test logged 34 "Delivery reconciliation failed" errors, all
from directory cleanup retries that found the agent directory lock busy
> - This pull request closes idle warm sessions in parallel across
agents, one at a time within an agent, and treats a busy directory lock
as a quiet retry
> - The benefit is that shutdown preparation fits its budget with
several warm sessions, and expected lock contention no longer looks like
a failure

## Linked Issues or Issue Description

**What happened?**
`closeIdleWarmNativeSessions` awaited each warm session's full close in
turn. In swap 2 of a release-swap test, five idle sessions closed at
22:28:42, :46, :51, :56 and :59 (one "transport_close_requested"
diagnostic each). The `hot_restart_prepare` step then timed out at 19.9
s. Separately, `recoverDirectoryRelease` threw
`AgentDirectoryOperationBusyError` when another close held the agent
directory lock. The delivery coordinator logged each one at ERROR level.

I first checked if these sessions should have taken the cheap
released-session detach. They could not: `entry.released` is set only
after a successful `session.release()` through a release/resume-capable
plugin Runner task. The Runner state files from all sandboxes in the
test recorded only `run.prepare`, `session.open`, `turn.start`,
`runner.drain` and `runner.suspend` commands, and no `runner.release`.
The Daytona daemon logs show only `/process/execute` and
`/process/session` calls, and no Runner-task API. So no session in that
test was ever released, and a full close was the only correct shutdown
for them.

**Expected behavior**
Idle warm sessions of different agents close at the same time. Two
sessions of the same agent never close together, because they collide on
the agent directory advisory lock. A cleanup retry that finds that lock
busy stays pending until its claimed retry time and logs no error.

**Steps to reproduce**
1. Keep five or more idle warm sessions on remote sandboxes.
2. Send SIGTERM.
3. The closes run one after another and `hot_restart_prepare` times out.

**Paperclip version or commit**
`master` at bbed32450c.

## What Changed

- `closeIdleWarmNativeSessions` groups sessions by company and agent. Up
to four agents close at once, and each agent's sessions close one at a
time. The per-session close logic is unchanged.
- Idle-sleep retirement (`prepareIdleWarmNativeSessionsForSleep`) keeps
a concurrency of one, so it still checks drain authority before each
retirement.
- `recoverDirectoryRelease` catches `AgentDirectoryOperationBusyError`,
keeps the deferred retry time, and returns the pending entry instead of
throwing.

## Verification

- Red, then green (each failed against `master` source and passes with
the fix):
- `closes idle warm sessions of different agents in parallel and one at
a time within an agent` in
`server/src/services/native-runtime/native-session-executor.test.ts`.
Before the fix, the second agent's close was never started while the
first agent's close was pending.
- `keeps a cleanup pending without an error while another server holds
the agent directory lock` in
`server/src/__tests__/agent-directory-working-copies.test.ts`. It holds
the real Postgres advisory lock from a second connection. Before the
fix, the coordinator received an `AggregateError` with
`AgentDirectoryOperationBusyError`.
- `native-session-executor.test.ts` (1502 passed) and
`agent-directory-working-copies.test.ts` pass in full.
- `pnpm typecheck` in `server` passes.

## Risks

- Up to four warm closes now run at once, so shutdown makes more Runner
and database calls at the same time. The cap keeps this small.
- The order of entries in `failures` can change when several closes
fail.
- A busy release now waits for its deferred retry silently. A lock that
stays busy for a long time shows as a stuck pending cleanup, not as
repeated errors.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.7
2026-11-14 04:15:19 -06:00
DottaandClaude Opus 5.5 12ac7cfd9f perf(recovery): read only the run keys that active execution recovery checks use (#21602)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Live recovery checks each running run when recovery work wakes it.
The check is `recoverActiveExecutions` in
`server/src/services/heartbeat/recovery.ts`.
> - Each check read the full `heartbeat_runs` row and the agent's
`adapter_config`. That includes `context_snapshot` and
`runner_profile_json`, which are large JSON columns.
> - Most checks only defer the run to its lease deadline. They use a few
JSON keys and never use the rest of the row.
> - Recovery wakes come from runner checkpoints, run events and company
work signals. Each running run is checked about 4.5 times per minute
under that load, so this read grows with the number of running runs.
> - This pull request reads only the JSON keys that the check gates use.
A run that recovery settles as lost reads its full context, runner
profile and adapter config before the settle.
> - The benefit is about 97% fewer bytes per check, with the same
recovery decisions.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The read that live recovery does for each running run
(`recoverActiveExecutions`).

**Subsystem affected**
Server: heartbeat run recovery
(`server/src/services/heartbeat/recovery.ts`).

**Current behavior**
Each check selects the whole `heartbeat_runs` row plus
`agents.adapter_config`. With a realistic row (`context_snapshot` 115 KB
of text, 23 KB stored; `runner_profile_json` 48 KB of text, 10 KB
stored), one check receives 175 KB from Postgres. A server with 20
running runs did 90 checks per minute, which is 15.7 MB per minute from
this one statement. That was 99% of all bytes in the measured window.

**Proposed behavior**
The check selects all scalar run columns, a small JSON object with only
the keys its gates read, and no output excerpts or adapter config. The
process-lost settle path reads the full `context_snapshot`,
`runner_profile_json` and `adapter_config` for that one run before it
decides on a retry.

**Reason and benefit**
The staging server spends 21-24% of main-thread CPU on postgres.js
result parsing at about 26 KB per statement. Whole-row reads in
recurring loops cause that overfetch. This change removes about 171 KB
from each check.

**Breaking changes**
None. No schema, API or contract change.

**Additional context**
I examined the `+1 s` re-arm (when revocation finds a child process that
the inventory snapshot did not have). I kept it. It only fires when the
revocation sees a changed row. The revocation also writes a new
60-second lease for a random boot ID, so the next pass defers on that
live lease. It does not loop on unchanged state. In the 60-second
measurement it did not fire.

## What Changed

- `recoverActiveExecutions` selects `activeRecoveryRunColumns`. This is
all `heartbeat_runs` columns, except that `context_snapshot` holds only
`wakeReason`, `issueId` and `nativeQuestionCancellation`,
`runner_profile_json` holds only `nativeStartupContext`,
`nativeWorkspaceSync` and `remoteWorkspace`, and the stdout/stderr
excerpts are null. It does not select `agents.adapter_config`.
- New `readActiveRecoverySettlement` reads the full `context_snapshot`,
`runner_profile_json` and `adapter_config` of one run. The process-lost
settle uses that data for the conversation-adapter check, the retry
budget, the process-loss diagnostic, the stop metadata and the
first-heartbeat flag.
- New tests in `server/src/services/heartbeat/recovery.test.ts`:
- A regression test counts the bytes Postgres sends during one deferred
check through a TCP proxy. It must be below 32 KB.
- A settle test (two cases) gives a lost run its retry budget,
first-heartbeat flag and conversation adapter only in the JSON keys
outside the narrow read. The test checks that the settle still uses
them.
- A test checks that a run with `nativeQuestionCancellation` in its
context is still skipped.

## Verification

- Red/green: the byte test fails on `origin/master` (`expected 454426 to
be less than 32768`) and passes with this change. The other three new
tests pass both before and after.
- Focused local runs pass: `recovery.test.ts`,
`live-recovery-work.test.ts`, `native-startup-cleanup-recovery.test.ts`,
`ownership-reconciliation-report.test.ts` (4 files, 255 tests).
- `pnpm verify` selected 2,715 of 3,541 files (conservative retention).
On this machine 1,689 files passed and 32 failed. The failures were
`afterAll` hook timeouts (30 s) and unrelated tool-access, email and
runner artifact suites under local load. The recovery suites above pass
when run alone. Required CI gives the full coverage.
- Before/after measurement (local embedded Postgres, a wire proxy that
assigns each server response to its statement, the real live-recovery
scheduler, 60 s window, 20 running runs: 10 native runs held by a live
controller and 10 legacy runs with a live peer lease; a run recovery
signal every 5 s for each run, a company work signal every 10 s, legacy
lease renewal every 20 s):

| | Before | After |
| --- | --- | --- |
| Checks per minute | 90 | 90 |
| Bytes per check (inventory statement) | 174,954 | 4,470 |
| Inventory bytes per minute | 15,745,819 | 402,298 |
| All bytes per minute in the window | 15,848,137 | 504,640 |

## Risks

- Low risk. The narrow read keeps the keys that the gates read. The
settle path reads the full row data before it uses it.
- The settle path now reads `context_snapshot`, `runner_profile_json`
and `adapter_config` after the lease revocation, not at the start of the
check. The values can be some milliseconds newer. The revocation already
uses the committed row state, so this agrees with the rest of the
settle.
- The settle path has one more query. It only runs when a run is settled
as lost.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as an agent in
Claude Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:58:47 -06:00
DottaandClaude Opus 5.5 30fc64e212 fix(instructions): collect a waiting working copy as soon as its stop proof is written (#21596)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run gets a private working copy of the agent's instructions or
agent files. A queue collects the copy after the provider is proven to
have stopped.
> - #21586 stopped this queue from re-reading copies forever: a copy
that waits for proof of the stop is now checked again after half of its
wait, capped at 6 hours.
> - With only that change, a copy whose stop is recorded T after its run
finished is collected up to T/2 later (at most 6 h). Before #21586 this
took at most 30 seconds. A sandbox that takes minutes to stop therefore
delays the save of an agent's instruction changes, and copies of
stranded runs wait up to 6 h after their stop is finally recorded.
> - This pull request makes the write that records the proof bring the
run's waiting copy forward, so the queue checks it on its next pass.
> - The benefit: collection is prompt again when the proof arrives,
without polling copies that are still waiting.

## Linked Issues or Issue Description

Follow-up to #21586. No public issue exists. Bug description:

**What happened?**
After #21586, a working copy waiting for proof of the provider stop
keeps its saved backoff deadline when the proof arrives. Proof is a
`native.local_process_stopped` event or a remote lease termination
receipt. A copy whose run finished 30 days ago is collected up to 6
hours after its stop is recorded.

**Expected behavior**
When a stop proof for a run is written, that run's waiting copy is
checked on the next queue pass. A write for a run without a waiting copy
costs one indexed no-op UPDATE and sends no work signal.

**Steps to reproduce**
1. Create a working copy for a native run that finished 30 days ago and
has no stop event, then let the collection queue check it once. Its next
check is about 6 h later.
2. Record the local stop (`recordNativeLocalProcessStopInTransaction`),
or release the run's remote lease with a termination receipt.
3. On master the copy stays in `pending_collection` until the old
deadline.

**Paperclip version or commit**
`origin/master` at 633dec7e43.

## What Changed

- New `notifyInstructionCopyStopEvidence(tx, { companyId, runId })` in
`instruction-collection-notifications.ts`.
- It reads the run's copy by its primary key (`run_id`, scoped by
`company_id`), and only if that copy waits in the collection queue or
the agent-files cleanup queue.
  - If there is no such copy, it stops: one SELECT, no write, no signal.
- If there is one, it first registers the run-scoped hint for both the
collection and the cleanup queue, and then runs one UPDATE by the
primary key. The UPDATE sets `next_attempt_at` to `now()` (or keeps an
earlier value) and sets a new `updated_at`. All of this happens in the
producer's transaction.
- The collection claim now compares `updated_at` with the value the
worker read (millisecond precision). If a stop proof is written between
the worker's evidence read and its backoff write, the claim fails and
the worker returns the copy as due, so the next pass collects it.
Without this fence, the worker could save a 6 h deadline over the proof.
- The writes that record stop proof call it in their own transaction:
- `native.local_process_stopped` receipts: `appendLocalProcessStop`
(used by `recordNativeLocalProcessStop` and its transaction form),
`recordUnidentifiedLocalRunnerStop`, and the operator confirmation in
`confirmUnidentifiedLocalRunnerStopped`.
- Remote lease termination receipts: `notifyEnvironmentCleanup` when it
has a run scope. Every lease release (`releaseLease`) and every
`writeEnvironmentCleanup` write calls it, which covers the receipt
writes in `environment-runtime`, `environment-task-runtime`,
`native-workspace-export-resume`, `sandbox-stop-and-retain` and
pending-cleanup recovery. The one remaining receipt write (sandbox
absence in `sandbox-stop-and-retain`) calls it directly.
- Orchestration's own stop path already calls `collectStopped` directly
after it joins the provider, so it collects without the queue and needs
no change.

## Verification

- `server/src/__tests__/agent-instruction-copy-retry-backoff.test.ts`, 3
new tests (embedded PostgreSQL, real queue registration):
- A local copy whose run finished 30 days ago (next check about 6 h
away) is collected within 10 seconds after
`recordNativeLocalProcessStopInTransaction` records the stop.
- A remote copy with the same backoff is collected within 10 seconds
after `environmentService.releaseLease` writes a termination receipt.
- For a run without a waiting copy, the transaction contains one SELECT
and nothing else, and no hint is sent. For a run with a waiting copy,
the only hint is `{"companyId":…,"runId":…}` (run-scoped, not
company-wide).
- A stop recorded between the worker's evidence read and its claim
(injected in the claim's transaction hook) is still collected on the
next pass.
- Red first: with the producer changes reverted, the two collection
tests fail (the copies stay in `pending_collection` until the old
deadline). Without the `updated_at` fence, the race test fails.
- Existing suites, one at a time: see the PR comment for the list and
results.
- `pnpm check:module-boundaries` passes.

## Risks

- Every lease cleanup write with a run scope now runs one extra
primary-key SELECT on `agent_instruction_working_copies`, and an UPDATE
only when a waiting copy exists.
- For agent-files cleanup (scratch files only, the saved data is not at
risk), the same race is not fenced. A missed advance there waits for the
cleanup backoff.
- A lease cleanup write that does not carry a termination receipt can
also bring a waiting copy forward. The queue then checks the stop
evidence once more (3–5 statements) and moves the copy back to its
backoff if the stop is still not proven.
- A release written with only a lease scope (no run) does not bring a
copy forward. That copy is still collected at its saved deadline.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:57:36 -06:00
DottaandClaude Opus 5.5 0ab63cf4d9 fix(restart-chaos): ignore a done the server rejected as incomplete (#21598)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness runs staging-mix waves and grades each
task for duplicated side effects and extra turns
> - The grader counts a run as a completer when its last accepted result
reports `done`
> - The server can reject that `done` as incomplete. The status arbiter
then queues a same-agent continuation wake with key
`native-completion-incomplete`
> - The grader ignores the rejection. It reports a false duplicate
completion and anchors completion on the rejected run, so the
continuation run that finished the work shows as an extra turn
> - This pull request marks a `done` as rejected when a
`native-completion-incomplete` wake for the issue follows it, and
ignores that `done` unless the same run posted the DONE comment
> - The benefit is that wave results stop reporting false duplicates and
false extra turns, so real regressions stay visible

## Linked Issues or Issue Description

**What happened?**
In wave `w-20261114T081656-s7105`, task t005, run d80d3ab3 reported
`done` with no DONE comment. The server rejected it as incomplete and
queued a `native-completion-incomplete` continuation wake. Run a777038c
then posted the only DONE comment and completed the issue. The grader
reported "duplicate completion: runs d80d3ab3, a777038c" and listed
a777038c as an extra turn.

**Expected behavior**
The grader reports one completer (a777038c), anchors completion on
a777038c, and lists no extra turn.

**Steps to reproduce**
1. Build wake evidence where run A has an accepted `done` result and no
DONE comment.
2. Add a continuation wake with payload `continuationIdempotencyKey:
"native-completion-incomplete"` created after A's result.
3. Add run B from that wake. B posts the DONE comment and reports
`done`.
4. Grade the task with `gradeStagingMix`, `completedAtAnchor` and
`runsAfterDone`.

**Additional context**
This is a test-harness change only. It changes no product code.

## What Changed

- `tests/restart-chaos/follow-up-delivery.ts`: add `rejectedDoneRuns`.
It attributes each `native-completion-incomplete` wake to the latest run
whose accepted `done` came at or before the wake.
- `tests/restart-chaos/wake-evidence.ts` (`completionEvidence`) and
`tests/restart-chaos/staging-mix-retry.ts` (`evidenceCompletion`): set a
new optional `doneRejected` flag on each completion run.
- `tests/restart-chaos/staging-mix.ts`: `completedAtAnchor` and
`completionRunIds` ignore a rejected `done`, unless that run posted the
DONE comment.
- `tests/restart-chaos/staging-mix-retry.test.ts`: add the t005
red/green test, a test for a rejected run that posted DONE itself, and a
G5c t004 guard.
- `tests/restart-chaos/README.md`: document the rule.

## Verification

- Red on master, green with the fix: `a done the server rejected as
incomplete > anchors on the run that finished the work, with one
completer and no extra turn (w-20261114T081656-s7105 t005)`. On master
the anchor is the rejected run's result time (1794646168026) instead of
the finisher's (1794646218506).
- Guard, green on master and with the fix: `a done the server rejected
as incomplete > keeps a run woken by recovery after a real completion as
an extra turn and a duplicate completion (G5c t004)`.
- `npx vitest run --config tests/restart-chaos/vitest.config.ts
--maxWorkers=3`: 71 files, 816 tests pass.
- `tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean.
- The real t005 evidence from the wave now grades as one completer
(a777038c) with no extra turns.
- `pnpm verify` exits 2 because `follow-up-delivery.ts` is a global CI
input. Required CI supplies the broad coverage.

## Risks

- Low risk. The change affects only the restart-chaos grader.
- The grader trusts the wake key. If the server queued
`native-completion-incomplete` for another reason, the grader would skip
the latest `done` before it. The server queues this key only from the
incomplete branch of `arbitrateNativeStatus`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:56:51 -06:00
github-actions[bot]andlockfile-bot 0b2b7aa737 chore(lockfile): refresh pnpm lockfiles (#21604)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.
- Resync the SDK public source proof's lock pins from the refreshed
lock.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when every added
package version is at least 3 days old on npm. Otherwise the PR waits
for a maintainer, and the daily refresh tries again. The bot merges only
the exact commit this workflow generated from manifests already reviewed
on master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
canary/v2026.1114.0-canary.6
2026-11-14 09:49:06 +00:00
DottaandClaude Opus 5.5 d25667f370 perf(db): keep postgres.js connection queues proportional to live connections (#21601)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - The driver keeps its pool connections in small FIFO queues (`open`,
`busy`, `full` and others) in `src/queue.js`
> - The `open` queue keeps an empty slot for every connection it hands
out until it is completely empty, and `remove` scans every slot
> - A busy pool always has some idle connections, so `open` never
empties. Each query then scans one slot for every query the process has
already run
> - The main thread of a long-running server therefore spends more time
on each query as the process ages
> - This pull request compacts the queue once empty slots outnumber
queued items, and makes `remove` search only the queued items
> - The benefit is that query dispatch cost stays flat for the life of
the process instead of growing linearly

## Linked Issues or Issue Description

**What happened?**
A hosted Paperclip server spends more of its main thread in the
postgres.js dispatcher the longer it runs. A CPU profile 14 minutes
after start showed the driver's `handler` (`src/index.js:350`) at 9.6%
self time, plus 3.8% more in the same inlined queue work. The server had
about 117 open sockets, a pool maximum of 500, and about 1,000
statements per second.

The cause is in `src/queue.js` (and `cjs/src/queue.js`). `shift` resets
its backing array only when the queue is completely empty. Otherwise it
leaves an `undefined` slot. `handler` takes a connection with
`open.shift()`. Then `move` calls `open.remove(c)` for a connection that
`shift` already took out, so `indexOf` scans the whole backing array and
finds nothing. When the query finishes, `move(c, open)` pushes the
connection back. The backing array grows by about one slot per query.

**Steps to reproduce**
1. Open a pool with more connections than the number of concurrent
queries (for example 24 connections and 8 concurrent queries).
2. Run 200,000 `select 1` queries.
3. Measure main-thread CPU per query for each block of 20,000 queries.
It grows from 30.6 µs to 48.7 µs.
4. Or run the same queue operations without a database: with 117
connections and 8 in flight, the cost per query is 16 µs at 200,000
queries and 151 µs at 1,000,000 queries.

**Expected behavior**
The cost to dispatch a query depends on the number of connections in the
queues, not on how many queries the process has run.

**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18, hosted and local servers.

## What Changed

- `patches/postgres@3.4.9.patch`: adds one hunk each to `src/queue.js`
(ESM, which the server loads) and `cjs/src/queue.js` (CommonJS). All
existing hunks are unchanged. The added hunk is:

  ```diff
       remove: (x) => {
  -      const index = xs.indexOf(x)
  -      return index === -1
  +      const at = xs.indexOf(x, index)
  +      return at === -1
           ? null
  -        : (xs.splice(index, 1), x)
  +        : (xs.splice(at, 1), x)
       },
  ...
         if (index === xs.length) {
           index = 0
           xs = []
  +      } else if (index > 1024 && index * 2 > xs.length) {
  +        xs = xs.slice(index)
  +        index = 0
         } else {
           xs[index - 1] = undefined
         }
  ```

The compaction copies only the queued items, and only after more than
1,024 shifts and after empty slots outnumber queued items. Its cost is
therefore amortized O(1) per `shift`. FIFO order, `length`, `push`,
`shift` (including `shift` on an empty queue) and `remove` (including
`null` for an absent item) keep the same results.
- `packages/db/src/postgres-queue.test.ts`: for both the ESM and
CommonJS builds, checks FIFO order, `length` and `remove` results. It
compares the queue with a plain array over 300,000 random operations
that cross the compaction threshold. It also runs 1,000,000 queries in
the pattern where `open` never empties, and checks that the slots
scanned by `remove` and the backing array size stay bounded by the live
connections.
- `packages/db/src/__fixtures__/postgres-queue-pool.mjs` and
`packages/db/src/postgres-queue-pool.test.ts`: run a real pool (24
connections, 8 concurrent queries, 20,000 queries) against embedded
PostgreSQL for both builds. They check that a dispatch scans no more
slots than there are connections.
- `packages/db/scripts/benchmark-postgres-queue.ts`: reports per-query
cost for the queue simulation (1,000,000 queries) and for the real pool
(200,000 queries, 10 blocks).
- `doc/DATABASE.md`: describes the fix, the tests and the benchmark
command.
- `pnpm-lock.yaml` is not changed. The lockfile refresh workflow updates
the patch hash, as it did after #21432.

## Verification

- Red before the fix: `pnpm --filter @paperclipai/db exec vitest run
src/postgres-queue.test.ts src/postgres-queue-pool.test.ts` failed 4
tests. The unit test scanned 13,340 slots (limit 125). The real pool
scanned 5,017 slots in its first 5,000 queries (limit 24). The semantic
tests passed before and after the fix.
- Green after the fix: the same command, plus
`src/postgres-receive-buffer.test.ts` and
`src/postgres-connection-recovery.test.ts`, passes 32 of 32 tests.
- `pnpm --filter @paperclipai/db exec tsc --noEmit` passes.
- `pnpm verify` exits 2 because the patch is a global CI input. Required
CI supplies the broad coverage.
- Benchmark: `pnpm --filter @paperclipai/db exec tsx
scripts/benchmark-postgres-queue.ts` on Apple Silicon, Node.js 26.4,
embedded PostgreSQL 18:

  | Measurement | Before | After |
  | --- | --- | --- |
| Simulation, µs per query at 200k / 400k / 600k / 800k / 1M queries |
16.0 / 50.4 / 81.3 / 128.4 / 150.7 | 0.070 / 0.070 / 0.071 / 0.073 /
0.072 |
| Real pool, main-thread CPU µs per query, first vs last 20k of 200k |
30.6 vs 48.7 (+59%) | 22.8 vs 14.6 (first block includes warm-up) |
| Real pool, CPU µs per query, blocks 2 to 10 | 29.6 rising to 48.7 |
13.3 to 15.9, flat |

## Risks

- Low risk. The change is limited to the driver's internal queue. The
results of every queue operation are the same, and the semantic test
compares them with a plain array over 300,000 random operations in both
builds.
- `remove` now searches from the first queued item. Slots before it are
always empty, so it cannot miss a queued item. Before this change,
`remove(undefined)` could remove an empty slot. The driver never calls
it that way.
- Compaction allocates a new array with only the queued items. It runs
at most once per 1,024 shifts, so memory use falls.
- The lockfile refresh workflow updates the patch hash in
`pnpm-lock.yaml` after merge, as it did after #21432.
- A possible follow-up is to offer the same fix to the upstream
postgres.js project. This pull request does not do that.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:46:41 -06:00
DottaandClaude Opus 5.5 009c8884a4 perf(attention): read blocker retention state once per page (#21595)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `attention-retention` queue archives old attention items. One
kind is `blocker_attention`: a blocked task whose terminal blocker needs
a person.
> - When the queue loads a page of blocked tasks, it reads `keep` and
`archived_at` from `decision_retention` for each of them, one full-row
`select` per task.
> - A company hint walks every page. On a hosted instance with 2,112
blocker attention items and one active run, these per-task reads ran at
about 16 per second.
> - This pull request reads the retention state for the whole page in
one narrow statement. It also scopes three remaining run-path attention
hints to the task graph.
> - The benefit is one retention read per page instead of one per
blocked task, with the same decisions.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The `blocker_attention` eligibility read in
`server/src/services/attention-retention.ts` (`eligible()`), and three
run-path producers of company-wide attention hints.

**Subsystem affected**

Server background retention (`attention-retention.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-review-participant.ts`,
`native-runtime/native-workspace-reconstruction.ts`).

**Current behavior**

- For each blocked task whose blocker attention is terminal (and for the
owner variant), `eligible()` ran `select <all columns> from
decision_retention where company_id = $1 and source_kind = $2 and
source_id = $3`. It did this during `load`, once per task, on every
company hint.
- The hosted statement is identified by that column prefix: `id,
company_id, source_kind, source_id, source_activity_at, …`.
- I checked every other full-row reader of `decision_retention`. None
fires per run event:
- `syncItems`: attention summary and list requests, which a board calls
once per load; retention `check` and the sweep, which run for due
entries only; Keep and Archive actions.
  - `getState`: user actions only.
  - `archiveReviewedManifest`: proposal acceptance only.
  - `decisions.list`: only when archive proposals exist.
- `native-provider-capacity-retry`, `native-review-participant` and
`native-workspace-reconstruction` sent `{ companyId }` attention hints,
which make the queue walk every page.

**Proposed behavior**

- `pageEvidence` reads `source_id`, `keep` and `archived_at` for every
blocker root and every terminal blocker on the page in one statement
(ids passed as one JSON parameter). `eligible()` uses that map when it
has page evidence.
- A missing row means "not kept, not archived", exactly as the per-task
read returning no row did.
- The per-entry `check` path still reads one row, now only `keep` and
`archived_at`.
- The three producers send `notifyIssueAttentionRetentionWork` with
their task (and run). That is the task, its parents and the tasks it
blocks, with a company fallback past 100.
  - A capacity retry changes the failed-run items of its task.
  - A review participant claim changes the review of its task.
- A workspace reconstruction resolves the recovery action of its task,
which can also change the blocker attention of the tasks it blocks.
- The configuration, lifecycle, import and fallback producers keep their
company hints.

**Reason and benefit**

Background work must not grow with the backlog. A page now costs one
retention read instead of up to 100.

## What Changed

- `server/src/services/attention-retention.ts`: batched and narrowed
blocker retention read; `PageEvidence.blockerRetention`.
-
`server/src/services/native-runtime/native-provider-capacity-retry.ts`,
`native-review-participant.ts`, `native-workspace-reconstruction.ts`:
scoped attention hints.
- Tests in `server/src/__tests__/attention-retention-work.test.ts` and
`server/src/services/native-runtime/native-workspace-reconstruction.test.ts`.

## Verification

- `reads blocker retention state once per page, not once per blocked
task` (I saw it fail first). 160 blocked tasks, each behind its own
unassigned blocker; the blockers' retention rows are kept (40), archived
(40), plain (40) or missing (40). The full-size response capture and the
statement log measure the load of both pages:
- before: one `decision_retention` read per task (160), 26,240 result
bytes;
- after: 2 reads (one per page), 5,480 bytes, none of them selecting
`source_activity_at`.
  - The same tasks are listed: only plain and missing.
  - `check` still drops the kept and archived ones.
- `scopes the attention retention hint of a reconstruction to the task
graph` (I saw it fail first): before, the reconstruction sent `{
companyId }`; after, it sends a hint that names the task.
- `batches 100 blocker graphs with exact terminal timestamps and fresh
terminal visibility` pinned 200 retention reads per page (one per root
and variant). It now expects 1; the rest of that test is unchanged and
passes.
- Not separately tested: the capacity-retry and review-participant hint
changes. Each is one line that passes the task the function already
holds, the same pattern as the reconstruction test.
- `attention-retention-work` (58), `attention-retention-candidates`,
`native-workspace-reconstruction`, `native-review-participant` and
`heartbeat-run-dispatch-work` pass, each file run alone (in `server/`,
against dedicated local `_test` databases).
- `pnpm check:module-boundaries` passes.

## Risks

- The page read happens once, before the per-entry decisions of that
page. Before, each decision read its row just in time. A Keep or Archive
that lands between the page read and an entry's decision is seen by the
next load or by `check`, which re-reads under a row lock, as before.
- A run-path hint that changes more than its task graph would now miss
the wider effect. The 90-day archive delay makes a missed reload cheap.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:05:59 -06:00
DottaandClaude Opus 5.5 cf09024292 fix(heartbeat): count an early native hand-off only after its suspend commits (#21572)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each live Runner run to the next server, so
the agent keeps working without a retry
> - At shutdown start, the outgoing server hands off native runs early,
in parallel with the rest of shutdown
> - The early hand-off put a run in the "handed off" set before its
suspend finished, and the drain counted every run in that set as
suspended without waiting
> - The early hand-off shares the 20 s pre-drain budget, and one Runner
detach can take 20-30 s, so the process could exit with the suspend
unfinished
> - Then the finalization lease stays live, the next server gets no
hand-over signal, and the run waits for lease expiry or the reaper
> - This pull request makes the drain wait for each in-flight early
hand-off, and fall back to its own hand-off when the early one failed or
did not finish
> - The benefit is that a run counts as handed over only after its
hand-over is committed, so the next server can always reach it at once

## Linked Issues or Issue Description

**What happened?**
A passing zero-downtime swap test showed a latent race in the shutdown
hand-off. `handOffNativeRunsForShutdown` added a run to
`nativeRunsHandedOffEarly` before `suspendNativeRunForShutdown`
finished. `drainRunningRunsForShutdown` then reported every run in that
set as `restartSuspended` without awaiting it. Because the early
hand-off shares the 20 s pre-drain budget and a Runner detach can wait
20-30 s, the process can exit before the lease release commits. Also, a
timed-out `prepareHotRestartShutdown` could still write the hot-restart
snapshot and detach sessions after the fallback drain started. In the
same test, swap 1's early hand-off took 2.08 s, and no log showed which
phase was slow.

**Expected behavior**
A run counts as handed over only after its suspend commits. If the early
suspend fails or does not finish, the drain hands the run off itself. A
timed-out hot-restart preparation does no further work after the drain
starts. The hand-off logs its phase timings.

**Steps to reproduce**
1. Start a native Runner run on a remote sandbox.
2. Send SIGTERM while the Runner detach is slow (or fails).
3. The drain reports the run as restart-suspended while its suspend is
still pending, and the process can exit before the lease release.

**Paperclip version or commit**
`master` at bbed32450c.

**Deployment mode**
Hosted, zero-downtime release swap.

## What Changed

- `nativeRunsHandedOffEarly` now holds each run's in-flight suspend
promise, not only its id.
- The shutdown drain awaits that promise (bounded at 30 s). If it fails,
returns false or does not finish, the drain runs its own suspend, as it
does for a run without an early hand-off.
- `runBoundedShutdownStep` and the pre-drain budget pass an
`AbortSignal` to each step and abort it on timeout.
- `prepareHotRestartShutdown` takes that signal and stops before the
snapshot write, the run events, and the session detach when the step
timed out.
- `handOffNativeRunsForShutdown` logs one structured line, "native run
shutdown hand-off phase timings": intent read, controller identity,
select, max per-run detach, max per-run transaction, and total.

## Verification

- Red, then green (each failed against `master` source and passes with
the fix), all through the public shutdown functions in
`server/src/services/heartbeat/recovery.test.ts`:
- `does not count an early hand-off as suspended while its detach is
still pending at the shutdown drain`
- `hands off a run at the shutdown drain when its pending early hand-off
fails`
- `writes no hot-restart snapshot and detaches nothing after a timed-out
preparation hands over to the drain`
- New test in `server/src/shutdown.test.ts`: `aborts the signal of a
step that outlives its bound, and only that step`.
- `pnpm vitest run src/shutdown.test.ts
src/services/heartbeat/recovery.test.ts
src/__tests__/heartbeat-native-early-hand-off.test.ts` passes (145
tests).
- `tsc --noEmit` for `server` passes. `pnpm verify` passes for the
selected tests.

## Risks

- When an early hand-off does not finish within 30 s, the drain starts a
second suspend. The second detach joins the first, because the session
memoizes its restart detach. The run can then get two "suspended" run
events. That is preferred to reporting a hand-over that did not happen.
- The drain can now wait up to 30 s for a slow early hand-off before the
process exits. Before, it exited without the hand-over.
- The abort signal only stops work between steps of the preparation. A
snapshot write that already started still completes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:02:25 -06:00
DottaandClaude Opus 5.5 633dec7e43 fix(instructions): back off working copies that cannot make progress (#21586)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run gets a private working copy of the agent's instructions or
agent files. Background queues collect the copy after the provider stops
and clean up its scratch directory.
> - When the queue cannot prove that the provider stopped, or cannot
finish a cleanup, it re-arms the same copy every 30 seconds with no
limit. The attempt count does not increase on the "stop not proven"
branch.
> - Every re-arm is a transaction that sends work signals to its own
queue, to the stranded-agent-status queue and to the skill-retention
queue. A self-signal also reloads the copy's scope. On a hosted instance
with only a few live runs, these copies were one of the largest sources
of SQL calls and kept the main thread busy.
> - This pull request makes a copy that cannot progress wait on a
growing, capped backoff. It also removes the work signals from a write
that only moves a deadline later, and reads the copy once per check.
> - The benefit: a copy that waits for proof of the provider stop costs
about 1/350 of the old steady-state load, and a copy that becomes
collectable is still collected automatically.

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened?**
A hosted instance had only a handful of live agent runs, but its server
main thread was about 95% busy. An analysis of its most-called SQL found
the instruction working-copy queues re-reading the same unresolved
copies forever, at about 25–30 statements/s. A copy whose run is
terminal but whose provider stop is not proven was re-armed every 30 s.
A completed agent-files copy whose remote scratch cleanup cannot finish
(its lease is released but not destroyed, or the lease row is gone) was
also re-armed every 30 s. Each re-arm was a transaction with work
signals that woke three queues.

**Expected behavior**
A copy that cannot progress is not re-read until a growing, capped
backoff elapses. A write that only moves a deadline later does not wake
other queues. A copy that becomes collectable is still collected
automatically, and no data is lost.

**Steps to reproduce**
1. Create working copies of terminal native runs that have no
`native.local_process_stopped` event, and completed agent-files copies
with `cleanupPending: true` whose remote lease does not exist.
2. Register the instruction-collection and directory-cleanup queues.
3. Count statements while time advances. With 300 + 300 such copies,
master sends 252 statements/s, 38.7 transactions/s and 38.7 work
signals/s.

**Paperclip version or commit**
`origin/master` at 3c3e6c2d75.

## What Changed

- `instructionCopyRetryAt(waitingSince)` sets the next check to half of
the time the copy has waited. The minimum is `INSTRUCTION_COPY_RETRY_MS`
(30 s) and the maximum is `INSTRUCTION_COPY_MAX_RETRY_MS` (6 h). An
unproven stop measures the wait from the run's `finished_at`. A cleanup
measures it from `process_stopped_at`. The deadline stays in the
existing `next_attempt_at` column, so a restart keeps it. There is no
schema change.
- The collection check now reads the copy once, checks the stop
evidence, and then claims. For an unproven stop, the claim writes the
backoff deadline (and the move from `prepared` to `pending_collection`)
in the same UPDATE. Before, the check was two reads plus two signalling
transactions.
- `tryPatch` skips all work signals when a write only moves deadlines
later and keeps the copy in the same queues (collection, cleanup, commit
and the stranded-agent predicate). A write that changes queue membership
or moves a deadline earlier still signals as before. The write stays in
its own transaction, so the existing claim and race tests keep their
hook.
- A cleanup that cannot finish gets the same capped backoff after
`release` re-arms it. The saved copy and its cleanup receipt are kept.
The claim does not change `attempts`, because a concurrent foreground
collection uses `attempts` in its compare-and-set.
- A remote agent-files copy that stopped but was not destroyed, and an
unavailable agent-files copy that waits for a destroy receipt, also get
the capped backoff instead of a 30 s re-arm.
- No copy is parked or dropped. A copy with an unproven stop stays in
the queue with the 6 h cap. When proof of the stop arrives later (a
`native.local_process_stopped` event, a lease termination receipt, or a
recovery that records that the host is gone), the next backoff check
collects it automatically. No manual call is necessary.

## Verification

- New suite
`server/src/__tests__/agent-instruction-copy-retry-backoff.test.ts`
(embedded PostgreSQL, real queue registration, statement meter):
- The backoff constants and their values at 0 s, 30 min, 2 h, 12 h and
30 days.
- A copy that has waited 2 days: two checks in 12 h, 5 statements and 1
transaction per check, and zero work signals.
- A copy owned by a live older server (run finished 5 minutes ago): it
is checked every few minutes. When the older server records the stop,
the copy is collected within 10 minutes, without a manual call.
- A copy that has waited 30 days: it is collected within one 6 h cap
after the stop is recorded.
- A backlog of 300 copies that have waited 30 days: one pass drains it
with zero work signals, and the hour after it is silent.
- A write that moves the deadline earlier, and a write that changes
queue membership, both still send the signal.
- A cleanup that cannot finish, with release returning and with release
throwing: at most 32 deadline writes in the first 6 h, at most 4 in the
next 6 h, zero work signals, and the copy is still saved with its
cleanup pending. Before the fix, the throwing case made 74 writes in 6
h.
- Red first: on master this suite fails 5 of 7 tests (the 30 s re-arm,
150 statements in the drain window, extra signals, a cleanup checked
every 30 s). The two "collects after stop" tests pass on master and
still pass: the positive behavior is kept.
- Steady state with 300 unproven stops plus 300 stuck cleanups, each
waiting for 1 day (10-minute window after warm-up):

| | statements/s | transactions/s | work signals/s | stop checks/s |
cleanup checks/s |
  | --- | --- | --- | --- | --- | --- |
  | master | 251.6 | 38.7 | 38.7 | 9.7 | 9.6 |
  | this PR | 0.45 | 0.10 | 0 | 0 | 0.03 |

- Per check: an unproven stop goes from 12 statements, 2 transactions
and 2 work signals to 5 statements, 1 transaction and 0 work signals. A
stuck cleanup goes from about 14 statements, 2 transactions and 2 work
signals to about 15 statements, 3 transactions and 0 work signals.
Cleanup checks are exponentially less frequent.
- Backlog drain: queue batches stay at 20 per pass, with one pass per
second at most. An overdue copy moves to its long backoff on its first
check. The first drain of N stuck copies therefore takes about N/20
seconds. For example, 17,000 copies drain in about 15 minutes with no
work signals.
- Existing suites, one at a time: `agent-instruction-working-copies`
(50/50), `agent-directory-working-copies` (120/120),
`heartbeat-instruction-working-copy` (4/4),
`heartbeat-agent-directory-busy` (8/8), `agent-file-checkpoint-locking`
(9 passed, 1 skipped), `agent-file-durability` (17/17),
`idle-sleep-safety` (226/226), `heartbeat-process-recovery` (459/459),
`agent-instructions-routes` (26/26), `services/heartbeat/agent-status`
(42/42).
- `pnpm check:module-boundaries` passes.

## Risks

- Product-visible choice: the 6 h cap. A copy that has waited for a long
time is collected at most 6 h after its stop proof arrives (half of the
wait while the wait is shorter than 12 h). During that time, the
uncollected candidate shows in the agent's instruction candidates
(`pending_collection`, `INSTRUCTION_STOP_UNCONFIRMED`) for legacy
copies, and in the run's agent-files diagnostics for agent-files copies.
Its bytes are kept on disk. An explicit `collectStopped` from
orchestration still collects immediately.
- There is no seven-day park. A park at `attempts = 3` would leave the
queue predicate, and a later stop proof would not reach the copy without
changes in files that another fix owns. The copy therefore stays in the
queue with the 6 h cap. The steady cost is about 5 statements per copy
every 6 h.
- During the 40-minute deploy overlap, a copy owned by a live older
server waited only a short time, so its backoff is still 30 s to a few
minutes. The test covers this case.
- A write that only moves a deadline later no longer wakes the
stranded-agent-status and skill-retention queues. The stranded-agent
predicate is compared before and after the write, and any change still
signals.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.5
2026-11-14 02:47:06 -06:00
DottaandClaude Opus 5.5 c3f9a0c040 fix(retention): evaluate native retention blockers for many sessions in one statement (#21588)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep runner checkpoints and harness backups so that a
session can resume. Retention expires them after the session can no
longer resume.
> - A retention blocker decides whether a session can still resume: a
live or recent run, an open or recently closed task, an unsettled
finalization, or a live lease.
> - The startup checkpoint pass evaluated that blocker in its own
transaction, once for each retained checkpoint, with three reads. A
session that stays blocked (for example, a long-lived open task) was
evaluated again for each of its runs on every server start. On a hosted
instance with frequent deploys and a 40-minute overlap of two servers,
this pass ran almost continuously at about 7 checkpoints/s, about 21
statements/s.
> - Backup reconciliation also used four reads and a transaction for
each run hint that reached a backup.
> - This pull request evaluates the blocker for many sessions in one
fixed-text statement. Only candidates that are not blocked enter the
existing locked transaction.
> - The benefit: a restart costs 2 statements for each page of 200
blocked checkpoints instead of 5 statements and 1 transaction for each
checkpoint, and a backup hint costs 2 statements with no transaction.

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened?**
A hosted instance had only a handful of live agent runs, but its server
main thread was about 95% busy. An analysis of its most-called SQL found
three retention-blocker reads (`company_id, agent_id from
heartbeat_runs`; the session's runs; the session's issues) at exactly
7.0/s each, each in its own transaction. The startup checkpoint pass
sent them once for each retained checkpoint older than 7 days whose
session was still blocked, and it repeated the full pass after every
start.

**Expected behavior**
A checkpoint or backup that is blocked from expiring costs no
transaction and no statements of its own, because one statement
evaluates its session together with the rest of its page. It expires as
soon as its blocker clears: checkpoints on the next startup pass (the
current design), and backups at their exact deadline.

**Steps to reproduce**
1. Create 100 native sessions with 3 finished runs each (finished 10
days ago), settled finalizations, a checkpoint header for each run, and
an open task.
2. Register the startup checkpoint expiration and count statements. Then
register it again, as a restart.
3. On master each start costs 1,502 statements and 300 transactions, and
nothing expires.

**Paperclip version or commit**
`origin/master` at 3c3e6c2d75.

## What Changed

- `SESSION_RETENTION_BLOCKERS_SQL` (new module
`native-runtime/native-session-retention-blockers.ts`) is one fixed-text
statement, sent through `$client.unsafe`. It takes a JSON list of
sessions and returns, for each one, the same blocker reasons as
`sessionRetentionBlocker`, in the same order: missing run, live run,
incomplete native recovery, recent run (checkpoint mode), unknown, open,
executing or recently closed task, pending finalization, live lease. It
also returns the session's resume pointer, run, task and lease
dependencies, and the latest close time.
- `expireCheckpoints` and `expireDurableBackups` evaluate each page of
candidates with this statement. They skip blocked candidates and
candidates whose session still has a resume pointer. A candidate that is
not blocked still enters the existing transaction. That transaction
checks the blocker again under row locks before it retires anything.
- `reconcileBackup` uses the statement for its dependencies and its
deadline. It replaces three dependency reads plus a blocker transaction.
The lease references use one index-backed arm for each reference kind
(run, backup stamp, task, identity lease).
- The unused `lock: false` path of `sessionRetentionBlocker` and the
`backupDependencies` helper are removed.
- The existing index test now checks the plan of the new statement.
Every arm that reads `environment_leases` uses an index in both the
custom and the generic plan.

## Verification

- New suite
`server/src/__tests__/native-retention-blocker-batch.test.ts` (embedded
PostgreSQL, real coordinator registration, statement meter):
- 300 blocked checkpoints: a start costs 4 statements and 0
transactions, and a restart costs the same. On master it is 1,502
statements and 300 transactions for each start. The checkpoints are
kept.
- When a session's task is closed long enough ago, its checkpoints
expire on the next startup pass. The other sessions are not touched.
- A session with a resume pointer keeps its checkpoints for 2 statements
in total.
- Two servers that run the startup pass at the same time expire every
checkpoint exactly once with no failures, because the locked transaction
checks again and the retirement header is compared before deletion.
- 50 run hints that reach 50 blocked backups: at most 2 statements for
each hint (the scope lookup and one blocker statement) and no
transaction. On master it is 9 statements and 1 transaction for each
hint. The backups are kept.
- Existing suites, one at a time: `native-harness-backup-expiration`
(109/109, including the backup expiry flows for task close, reopen and
lease release), `native-harness-backup-expiration-config` (7/7),
`native-harness-backup-storage.integration` (73/73),
`native-harness-backup-publication` (4/4),
`native-retired-backup-source.integration` (38/38).
- `pnpm check:module-boundaries` passes.

## Risks

- The startup pass still runs on every server start. It also runs on
both servers during the deploy overlap. Its new cost is 2 statements for
each page of 200 candidates (the candidate page and one blocker
statement). A page of candidates that are not blocked still uses one
locked transaction for each candidate, as before.
- Two servers can run the pass at the same time. They cannot retire the
same session twice or retire a blocked one: the unlocked statement only
filters, and the existing transaction locks the finalizations, checks
the blocker again, and compares the retirement header before it deletes.
The concurrency test covers this.
- If the SQL blocker and the TypeScript blocker ever disagree in the
"not blocked" direction, the locked transaction still refuses. If they
disagree in the "blocked" direction, retention is only delayed. When
more than one task blocks a session, the reported reason can differ from
the TypeScript loop, which reads the tasks in no fixed order.
- Checkpoints keep their current design: they are considered again on
the next startup, not at a deadline. Backups keep their exact deadline
after a task closes.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:47:01 -06:00
DottaandClaude Opus 5.5 f452b449fd fix(recovery): stop re-checking settled native runs that cannot be released (#21590)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server releases the process identity (`process_pid`,
`process_group_id`, `process_started_at`) of a finished native run when
it can prove that the runner process stopped (#17374)
> - A background queue (`settledNativeProcesses`) finds these runs and
checks them. A check that cannot finish ("skipped" or "pending") writes
nothing, so the run stays in the queue and every recovery hint checks it
again
> - A finished run that a replaced container recorded can never pass the
check, because only the host that recorded the process can prove that it
stopped. Each deploy adds more of these runs
> - On one hosted instance this loop ran about 10 checks per second and
about 55 SQL statements per second, and the server main thread was about
95% busy with only a few live runs
> - This pull request moves each "cannot make progress" reason into the
queue's SQL predicate, adds a growing backoff for "pending" runs, and
makes one check cost one locking statement
> - The benefit is that runs which cannot progress cost nothing until
something that can change the answer happens, and a run that becomes
releasable is still released at once

## Linked Issues or Issue Description

No public issue exists. Bug report:

**What happened?**
The settled native process recovery queue checks the same finished
native runs again on every recovery hint and every 30 seconds, forever.
With 300 such runs and one company-wide hint per second it ran 9 checks
per second and 84 SQL statements per second (9.4 statements per check,
2.2 ms of server CPU per check) on embedded PostgreSQL. A hosted
instance showed the same rates. The runs that never finish are: runs
recorded by a host that is gone (each redeploy starts a new container
with a new hostname), runs whose coordinator phase is not settled, runs
with no checkpoint and no provider evidence, and runs whose local
process is still alive (they defer 30 seconds with no limit). Startup
discovery (`discoverSettledNativeRunProcesses`) also opens one
transaction for every such run on every boot.

**Expected behavior**
A run that cannot make progress is not read again until something that
can change the answer happens. A run that becomes releasable (its phase
settles, provider evidence arrives, its own host checks it, its sandbox
receipt arrives, its process exits) is released promptly.

**Steps to reproduce**
1. Create a failed native run with a recorded local process identity
from another host (or an unsettled coordinator phase, or no checkpoint
and no provider evidence, or a live process).
2. Register the residual recovery work and send company-wide
`notifyLiveRunRecovery` hints.
3. Count the run's `for update` checks: on `master` every hint checks
the run again.

`server/src/__tests__/settled-native-process-rechecks.test.ts` automates
these steps.

**Paperclip version or commit**
`master` at 3c3e6c2d75.

**Deployment mode**
Any. The cost grows with every deploy of a hosted instance.

## What Changed

- `server/src/services/native-local-process-stop.ts`
- New `settledNativeProcessCandidates`: one fixed-text statement that
finds releasable runs. It reads only runs recorded in this server's PID
space (`process_host_hash`), sandbox runs (`remote`, with a non-local
lease), and rows saved before the hash (their latest identity event must
name this host). It excludes runs with an unsettled coordinator phase
and runs with no checkpoint and no provider evidence. Each PID-space
branch reads its own range of the new partial index.
- `releaseSettledNativeRunProcess` now reads the run, phase, evidence,
lease facts and identity match in one `for update` statement (it was 6
statements). It reads the lease rows again only for a sandbox run. The
new `inspectSettledNativeRunProcess` also returns whether a remote stop
acknowledgement can apply.
- Startup discovery uses the same candidate statement, so it opens no
transaction for an excluded run.
- `server/src/services/heartbeat/live-recovery-work.ts` (settled kind
only)
- The load runs the candidate statement and a separate indexed statement
for cancelled `cursor_cloud_api` runs whose Stop is still `requested`.
- The check uses the loaded entry. It no longer runs the load again for
each entry.
- A "pending" run backs off 30 s, 60 s, 120 s, up to 15 min. A
run-scoped hint (identity event, lease cleanup, stop receipt) resets the
backoff.
- A "skipped" run that the predicate cannot exclude (for example a birth
time mismatch on this host) waits for a run-scoped hint or a restart.
- `server/src/services/heartbeat/recovery.ts` (the settled handler
only): returns the outcome. It calls `acknowledgeRemoteStop` only when
the run has a non-local lease or is a cancelled `cursor_cloud_api` run.
For any other run that function reads two rows and writes nothing.
- `packages/db`: migration 0487 builds
`heartbeat_runs_settled_process_identity_idx` and
`heartbeat_runs_pending_api_stop_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. The migration is `online_compatible` and is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. `doc/DATABASE.md` describes both
indexes.

End state for each reason that a check cannot finish:

| Reason | Before | After | What re-admits the run |
| --- | --- | --- | --- |
| Recorded by another PID space | checked on every hint | not read |
only a server in that PID space reads it (a live older server in a
deploy overlap still releases its own runs) |
| Coordinator phase not settled | checked on every hint | not read | the
phase write; the finalizer sends a company hint in the same transaction
|
| No checkpoint and no provider evidence | checked on every hint | not
read | the next hint after a provider or turn event |
| Live local process, unreleased lease, missing sandbox receipt | every
hint and every 30 s | backoff 30 s to 15 min | a run-scoped hint resets
the backoff |
| Other same-host skip | checked on every hint | parked | a run-scoped
hint (an identity event sends one) or a restart |

## Verification

- `pnpm exec vitest run
src/__tests__/settled-native-process-rechecks.test.ts` (new, 12 tests,
real PostgreSQL, real queue and coordinator, fake clock):
- each exclusion: replaced container, unsettled phase, no evidence: 0
checks over 4 hints (red on `master`: 3 to 5 checks);
- a live older server and a stopped older server (different hash): the
new server never reads the run and writes no receipt; this server's own
run is released at start;
- re-admission: phase settles, late provider evidence, legacy rows with
no hash (this host released, another host not read), sandbox run
released from any server with its receipt;
- live process: at most 4 checks over 8 hints (red: 17), released on the
run-scoped hint after the process exits;
- one check is BEGIN, one `for update` statement and COMMIT (3
statements); the next hints do not check it; a run-scoped hint checks it
once more (red: 5 statements and a check on every hint);
- a restart costs one check for each pending run; startup discovery with
50 excluded runs sends 1 statement (red: 302).
  - On `master` 11 of the 12 tests fail.
- Measurement on embedded PostgreSQL, 300 stuck runs, one company hint
per second for 300 s (fake clock), same harness before and after:
- before: 9.0 checks/s, 84.3 statements/s, 9.4 statements per check, 5.9
s CPU (2.2 ms per check);
- after: 0 checks/s, 3.2 statements/s for all seven residual queues
together, 0.16 s CPU.
- Load plan with 12,000 excluded finished runs (10% without a hash) in a
31,000-row `heartbeat_runs`: the candidate statement reads only index
entries for this server's hash, `remote` and `unknown`, and a bitmap
scan of the 1,200 legacy rows: 2.7 ms, 4,416 buffers. Without the index
the same exclusion read all 31,000 rows (31,127 buffers).
- Existing suites, one file at a time: all pass.
`live-recovery-work.test.ts` 51/51 (an earlier run under machine load
hit the `afterAll` hook timeout after all tests passed; the rerun passed
clean, and `native-safe-replacement.test.ts` hits the same hook timeout
on the base commit), `native-settled-process-release.test.ts` 31/31,
`native-local-process-stop.test.ts` 5/5, `heartbeat/recovery.test.ts`
107/107, `native-runner-restart-recovery.integration.test.ts` 231 passed
/ 14 skipped, `native-restart-recovery.test.ts` 49/49,
`native-controller-recovery.integration.test.ts` 30/30,
`retained-native-session-cleanup-work.test.ts` 29/29,
`native-provider-failure.integration.test.ts` 16/16,
`native-safe-replacement.test.ts` 70/70 (with the `afterAll` hook
timeout that also occurs on the base commit),
`native-remote-retry-admission.test.ts` 93/93.
- `pnpm check:module-boundaries`, `pnpm check:migrations` (packages/db),
server typecheck: pass.
- Not run: the full suite and browser suites.

## Risks

- Unresolved by design: a finished run whose server was replaced keeps
its process fields indefinitely and is never released, because only the
host that recorded a process can prove that it stopped, and the shared
prover refuses a controller-lease proof for a run with a process group.
This pull request only stops paying for it. These runs keep no stop
receipt (`native.local_process_stopped`), so what waits for that receipt
still waits: accounting close of the run's usage source, retained native
session cleanup, and instruction or directory working copies that wait
for proof that the run stopped. A follow-up must decide the ownership
rule for a replaced container.
- Rows saved before `process_host_hash` exist are read through the
legacy branch, which checks the latest identity event for each of them
on each load. On an instance with many such rows the load costs one
index lookup per row. The count can be measured with `select count(*)
from heartbeat_runs where process_host_hash is null and process_pid is
not null and status in
('failed','cancelled','timed_out','interrupted')`.
- A run that this server recorded but whose `process_host_hash` changed
(a new PID namespace on the same host, for example a container restart)
is no longer read here. Before, it was read and skipped, so no release
is lost.
- The backoff and parked state live in memory. A restart checks each
pending run once.
- The migration builds two partial indexes concurrently on
`heartbeat_runs`. It holds no long lock.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:29:23 -06:00
DottaandClaude Opus 5.5 e8c79235ca perf(attention): scope run lifecycle retention hints and read only used columns (#21579)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `attention-retention` queue archives old attention items (failed
runs, reviews, questions, recovery actions, blocked tasks). A company
hint makes it reload every candidate of the company, 100 per page, with
eligibility reads for each page.
> - Several run lifecycle producers sent that company hint: every agent
status change after a run, the execution-control writes and legacy run
terminalization when they name only a run, scheduled retries, and the
Runner's task monitor tool.
> - Each page also read whole rows it did not use: full `heartbeat_runs`
rows (recent ones carry tens of KB of `context_snapshot` and
`runner_profile_json`), full `issues` rows (descriptions), and the whole
agent roster with its configuration.
> - This pull request names the task graph or the source kind in those
hints, and reads only the columns each eligibility decision uses.
> - The benefit is fewer company reloads per run event and far fewer
result bytes per reconstruction, with the same decisions.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`. Stacked on #21552 (it builds
on that pull request's change to `execution-control-notifications.ts`);
review that one first.

**What existing behavior does this improve?**

The attention retention queue
(`server/src/services/attention-retention.ts`) and the producers that
wake it on run lifecycle paths.

**Subsystem affected**

Server background retention (`attention-retention.ts`,
`retention-work-notifications.ts`, `execution-control-notifications.ts`,
`heartbeat/agent-status.ts`, `heartbeat/retries.ts`,
`native-runtime/paperclip-runner-tool-authority.ts`).

**Current behavior**

- `heartbeat/agent-status.ts` sends `{ companyId }` on every agent
status change after a run (two places). `heartbeat/retries.ts` sends it
for each scheduled retry. `writeExecutionControl` and
`notifyExecutionRunWork` send it when the scope names only a run. The
Runner's `set_task_monitor` tool sends it for one task.
- On a local fixture with 1,500 open tasks and run churn, the queue
loaded 47 company candidate rows per second from these hints.
- `pageEvidence` read `select *` from `heartbeat_runs`,
`issue_recovery_actions` and (through `getTableColumns`) `issues`;
`reviewEligibility` read `select *` from `issues`;
`interactionEligibility` read the full agent roster; the per-entry
`check` repeated full-row reads.

**Proposed behavior**

- Agent status changes send `{ companyId, sourceKind:
"agent_error_alert" }`. The status moves between `running`, `idle` and
`error`. All three are invokable for blocker attention
(`BLOCKER_ATTENTION_INVOKABLE_AGENT_STATUSES`) and for questions
(`agent-eligibility.ts`). Failed-run attention only excludes terminated
agents, and this path never sets that. So only agent error alerts can
change.
- Run-scoped execution-control writes, legacy run terminalization and
scheduled retries use `notifyIssueAttentionRetentionWork` with the run
id (the run's task, its parents and the tasks it blocks; company
fallback past 100). This is the same helper as #21521. The task monitor
tool scopes to its task.
- `notifyIssueAttentionRetentionWork` matched the run's saved task id
with `issue.id::text = …`. It now validates and casts the id, so the
primary key is used, as #21552 does for recovery.
- Narrow reads, same decisions:
- runs: `id`, `agent_id`, `created_at` and the saved task id computed in
SQL with the same rules as `readRunIssueId` (`issueId` unless it is
absent or JSON null, then `taskId`; only non-empty strings).
  - recovery actions: `id`, `source_issue_id`.
- issues for evidence and `visibleIssue`: the blocker-attention input
columns plus `unblock_descriptor` and `blocked_transition_at`.
- review issues: `id`, `company_id`, `status`, `execution_state`,
`assignee_user_id`.
- agent roster for question addressees: `id`, `company_id`, `name`,
`reports_to`, `status` (the `AgentOrgRow` that
`evaluateAgentInvokability` takes).

**Reason and benefit**

A run event no longer reloads every attention candidate of the company.
A reconstruction no longer transfers whole run, task and agent rows.

## What Changed

- `server/src/services/attention-retention.ts`: column lists for runs,
recovery actions, issues and the roster; the run probe type replaces
whole rows.
- `server/src/services/heartbeat/agent-status.ts`,
`heartbeat/retries.ts`, `execution-control-notifications.ts`,
`native-runtime/paperclip-runner-tool-authority.ts`: scoped attention
hints.
- `server/src/services/retention-work-notifications.ts`: indexed lookup
of a run's saved task.
- `server/src/__tests__/attention-retention-work.test.ts`: two tests.

## Verification

- `reads only the fields its decisions use from large runs, reviews and
agents` (I saw it fail first): 50 failed runs, 50 tasks in review with
large descriptions, 10 questions to a paused agent, about 40 KB of
payload in each run's snapshot, profile and stdout and in the agents'
configuration. It loads the three kinds with the full-size response
capture (`full-database-response-capture.ts`). Before: `heartbeat_runs`
6,317,700 bytes, `agents` 84,336 bytes, `issues` 4,235,810 bytes. After:
under the test bounds of 20,000, 5,000 and 60,000 bytes. The decisions
are the same before and after: 50 failed runs, 40 reviews and 10
questions.
- `scopes run lifecycle attention hints to the run's task graph and
agent status to agent alerts` (I saw it fail first):
`writeExecutionControl` and `notifyExecutionRunWork` with a run id send
the hint for the task and its parent. `finalizeAgentStatus` sends the
agent-alert hint. Before, all three sent `{ companyId }`.
- `attention-retention-work` (57 tests),
`attention-retention-candidates`, `attention-service`,
`execution-control-work` and `heartbeat-agent-status-recovery` pass,
each file run alone (in `server/`, against a dedicated local `_test`
database).
- `pnpm check:module-boundaries` passes.

Not done in this pull request, and why:

- I did not reproduce `syncItems` re-syncing unchanged rows from the
retention `check`. On my fixture (300 stalled blocked tasks, 4,500 old
recovery-action and blocker retention rows, run churn), `syncItems` ran
0 times in a 180-second window and `decision_retention` reads were 0 per
second. `check` calls `syncItems` only for entries that are already due
(older than the 90-day archive delay), and a due entry is archived and
leaves the candidates. So I did not batch `syncItems` or skip checks of
unchanged rows.
- `readIssues` still selects the `issues` columns that blocker attention
needs. I did not narrow `issueService.listBlockerAttentionBatch`.

## Risks

- A producer that changes more than the run's task graph would now miss
the wider effect. Company-level changes (agent configuration, company
status, budgets) keep their company hints. The 90-day archive delay
makes a missed reload cheap: the item is reloaded on the next hint that
covers it.
- The saved task id now comes from SQL. The SQL mirrors `readRunIssueId`
for strings, absent keys and JSON null. A non-string `issueId` value now
yields no task, as before.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:17:21 -06:00
DottaandClaude Opus 5.5 a1c37b5cc4 perf(chat): rebuild GitHub response sweeps for company-wide hints at most every 30 seconds (#21580)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel has two background sweeps for request
response comments: acknowledgements, and terminal responses. Each sweep
rebuilds its candidate set when a hint arrives, and it reads the latest
run of each candidate request (`readGitHubResponseRun`) on each rebuild
> - A run milestone for a run with no task sends a company-wide hint to
both sweeps. Examples: each status change of a run without a task
(`setRunStatus`), and a comment merged into an active run in the wake
queue. In a busy company these hints arrive more than once a second
> - So both sweeps rebuild the whole company about once a second (their
minimum pass interval). Each rebuild reads every parked request again,
although nothing about those requests changed. On a hosted test
database, `readGitHubResponseRun` ran 6.7 times a second
> - This pull request rebuilds a GitHub response sweep for a
company-wide hint at most once every 30 seconds. Issue, request,
conversation, endpoint and lease hints still rebuild immediately.
`external-objects.ts` uses the same rule for its refresh sweep
> - The benefit is that a steady stream of company-wide hints no longer
makes each sweep read every parked request every second

## Linked Issues or Issue Description

Refs #21577 (makes each `readGitHubResponseRun` call cheap; this pull
request reduces how often the sweeps call it). Refs #18384 (the same
reconstruction interval for company-wide external-object hints).

**What existing behavior does this improve?**
The candidate rebuild of `registerGitHubAcknowledgementWork`
(`acknowledgement-work.ts`) and `registerGitHubTerminalResponseWork`
(`terminal-response-work.ts`) for company-wide hints.

**Subsystem affected**
GitHub chat channel: request acknowledgement and terminal-response
recovery sweeps.

**Current behavior**
`notifyChatRunMilestones` calls `notifyChatResponseAvailability` with
its scope. When the scope has no issue (only a company, or a company and
a run), the hint for both GitHub response queues is company-wide
(`{"companyId": ...}`), because the response scope has no run field.
Each company-wide hint makes each sweep load every candidate of the
company and call `readGitHubResponseRun` for each one. Passes are
limited to one a second, so with a steady hint stream the cost is about
(parked candidates) reads a second.

**Proposed behavior**
Both sweeps set `reconstructionIntervalMs`. A company-wide scope
(company ID only) waits 30 seconds after its last rebuild before it
rebuilds again. All narrower scopes, the startup scope (`*`), and saved
deadlines are not changed. Entries that are already loaded stay
available for dispatch while a company-wide hint waits, and each check
still reads the request again from the database.

**Reason and benefit**
On the hosted test database, `readGitHubResponseRun` ran 6.7 times a
second for no change in state. In a local reproduction, the call rate
drops from about one call for each parked request each second to one
call for each parked request each 30 seconds.

**Breaking changes**
None.

## What Changed

- Add `response-sweep-scope.ts` with the shared scope type of the two
GitHub response sweeps and `githubResponseReconstructionIntervalMs`: 30
seconds for a company-wide scope, 0 for all other scopes.
- Use it as `reconstructionIntervalMs` in the acknowledgement sweep and
the terminal-response sweep. Both files now import the shared scope type
instead of declaring it.
- Add `response-sweep-scope.test.ts`. It registers both real sweeps on a
real database with 6 parked requests (3 for each sweep) and a controlled
clock. It sends company-wide hints and issue hints, and counts
`readGitHubResponseRun` statements on each pass.

## Verification

Cause, from a local reproduction with the real delivery coordinator and
both real sweeps on embedded PostgreSQL 18.1, with 8 parked requests (4
for each sweep), each with a succeeded native run that waits for its
presentation:

| Hints | Hints a second | `readGitHubResponseRun` calls a second |
| --- | --- | --- |
| None | 0 | 0 |
| Company-wide run milestone (`{ companyId, runId }`) | 9.7 | 8.75 |
| Company-wide run milestone | 1.0 | 7.98 |
| Run milestone for an unrelated issue | 9.7 | 0 |

So the call rate is about one call for each parked request each second,
from company-wide hints only. The hosted rate (6.7 a second) is
consistent with about 7 parked GitHub requests in one company.

Red/green regression test:

- `cd server && npx vitest run
src/services/chat-channels/github/response-sweep-scope.test.ts`
- Red (without the two `reconstructionIntervalMs` lines): `expected [ 6,
6, 6, 6, 6 ] to deeply equal [ 6, +0, +0, +0, +0 ]`. Each of 5
company-wide hints rebuilt all 6 parked requests.
- Green: 2 of 2 pass. The first company-wide hint rebuilds once (6
reads). The next 4 hints read nothing. After 30 seconds, the waiting
hint rebuilds (6 reads). Issue hints still read exactly the one request
of that issue at once, and a hint for another company reads nothing. No
acknowledgement is sent for a parked request.

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed
(all GitHub chat-channel integration tests, including the
acknowledgement and terminal-response sweeps).
- `cd server && npx tsc --noEmit`: passed.
- After the rebase onto #21577: both new test files pass (4 of 4), and
the server typecheck passes.
- `pnpm verify` selected most of the server suite, because the sweep
modules are imported widely. I stopped that broad local run. Required CI
supplies that coverage.

## Risks

- A change that only a company-wide hint reports can take up to 30
seconds to rebuild a GitHub response sweep. I checked the callers.
Changes to one request use issue, delivery, conversation or lease hints:
run status changes with a task, native presentation commits
(`status-decision-committer.ts`, `native-run-finalizer.ts`,
`native-chat-review-presentation.ts`), publications, fast responses, and
the publication lease. Those stay immediate. Company-wide hints come
from runs with no task, from a comment merged into an active run (the
wake queue sends only the run ID), and from company authority changes. A
run with no task cannot change the latest run of a GitHub request's
task. For a merged comment, the active run's next status change sends an
issue hint, and staging the request's response sends a delivery hint;
both rebuild at once. Only a mid-run ownership update can wait up to 30
seconds. For an authority change, each check reads the request and its
authority again before it acts.
- The deadline reconciliation snapshot reports these two sweeps as not
reconciled while a company-wide hint waits (up to 30 seconds). This is
the same as the external-object refresh sweep.
- The startup rebuild (`*`) and failure retries do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local PostgreSQL
reproductions).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:05:45 -06:00
DottaandClaude Opus 5.5 340157d1d6 perf(recovery): scope run lifecycle recovery hints to the run's task graph (#21552)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues in the background. Two of
them, `stranded-assignments` and `assigned-work-invariant`, find
assigned tasks that have no live run and restart them.
> - Every run dispatch and every run state change sent a recovery hint
for the whole company. On each hint these queues reload every eligible
open task of the company and check each one again, at most every 30
seconds per queue.
> - With live runs the hint comes all the time, so the queues never
stop. Their cost is (run activity) x (open-task backlog). On a hosted
instance with about 1,400 open tasks and 5 live runs, the server main
thread stayed at 87-92% of a core.
> - A run event can only change the tasks of its own task graph. This
pull request makes run and task producers name that graph, and keeps one
slow full pass as a backstop.
> - The benefit is that a run event costs a few checks, whatever the
size of the backlog, and a missed hint still delays recovery by at most
five minutes.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The issue recovery queues in `server/src/services/recovery/work.ts`
(`stranded-assignments`, `assigned-work-invariant`,
`resolved-dependencies`, `stale-issue-locks`) and the producers that
wake them on run dispatch, run start, run state change, run finalization
and accounting settlement.

**Subsystem affected**

Server background recovery (`server/src/services/recovery/`,
`server/src/services/scoped-deadline-work.ts`, the notification helpers
in `server/src/services/*-notifications.ts`).

**Current behavior**

`notifyRunDispatch`, `notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` sent `notifyIssueRecovery(tx, {
companyId })`. Most run lifecycle paths call one of them, often two or
three in the same transaction (for example `notifyIssueReviewState`).
Some other run paths sent the company hint directly: the wake status
update, the execution-control deadline writes, legacy run
terminalization, scheduled retries, accounting settlement, task checkout
and task document writes. Each company hint makes each task queue reload
all eligible open tasks of the company, 100 per page, and check each
task again. A company hint is reconstructed at most every 30 seconds per
queue, so one run event per 30 seconds is enough to keep a full pass
running all the time. `issue-recovery-deadlines.test.ts` pinned this: a
company hint re-checks every eligible task.

**Proposed behavior**

Design. For each task queue, a run event of run R on task T can change:

| Queue | What the check reads | What one run event can change |
| --- | --- | --- |
| `stranded-assignments` | T's own runs, wakes, handoffs, questions,
recovery actions, goal, assignee eligibility | T (its runs). The parent
of T and the tasks T blocks when the event also writes T's status. |
| `assigned-work-invariant` | the same plus open blockers and open
children | T, the parent of T (open children), the tasks T blocks (open
blockers) |
| `resolved-dependencies` | blocked tasks whose blockers are all done |
the tasks T blocks |
| `stale-issue-locks` | tasks whose checkout or execution run is R | T |

The assignee's agent status (`running`, `idle`, `error`) does not change
whether the agent is invokable
(`packages/shared/src/agent-eligibility.ts`), and a freed concurrency
slot starts the agent's next queued run through the
`queued-run-dispatch` queue, which keeps its company hint. So the
agent's other tasks are not in the set.

- `notifyIssueGraphRecovery(tx, { companyId, runId?, issueId? })` sends
`{ companyId, graphRunId?, graphIssueId? }` to the four task queues. It
runs no query in the producer transaction.
- When a task queue loads that hint, it resolves the graph with one
bounded, indexed query (`resolveIssueGraph`): the task of the run
(`issue_id`, `native_issue_id`, or the saved `issueId`/`taskId`,
validated and cast so the primary key is used), its parent, and the
tasks it blocks. The four queues share the result for 2 seconds. With no
task or more than 100 tasks, the queue turns the hint into the plain
company hint, so task-less runs coalesce under its 30-second gate
instead of each starting its own full pass. The lookup runs after
commit, so it sees the written state. Graph hints do not wait for the
30-second reconstruction gate (they load at most 100 tasks); hints for
the same run coalesce while dirty, and the lookup is shared across the
four queues, so it runs about once per pass instead of once per producer
call. An earlier round ran the lookup in every producer transaction: 34
lookups/s at 1,500 open tasks and 70/s at 5,000.
- The three company-level queues (`orphan-blockers`,
`active-recovery-actions`, `task-watchdogs`) keep the company hint. They
hold one entry per company or per watchdog, not per open task.
- `notifyRunDispatch`, `notifyScheduledRetryDispatch`,
`notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` use it when the scope names a run
or a task. A scope with only a company keeps the company hint.
- Run paths that knew the run or the task but sent only the company now
pass it. This includes three `writeExecutionControl` deadline writes,
legacy run terminalization (`notifyExecutionRunWork`) and native safe
replacement. Those helpers also forward the scope to
`notifyLiveRunRecovery` and `notifyLiveResourceRecovery`, so their hints
to the live-run queues now carry the run id.
- Recovery action changes (`issue-recovery-actions.ts`: create,
supersede, update, exhaust, resolve) name their source task. A recovery
action is a fact of its source task for these queues. The company-level
`active-recovery-actions` queue still gets the company hint. In the
fixture, failing runs create and resolve recovery actions every few
seconds, and that alone kept a full pass running.
- `writeExecutionControl` and `notifyExecutionRunWork` send the
attention retention hint for a task through
`notifyIssueAttentionRetentionWork` (the task, its parents and the tasks
it blocks), a superset of the single-task hint they sent before. A
recovery action of a blocker can change the blocker attention of its
dependents, so the narrower recovery action scope needs this.
- `promoteDeferredWorkIntents` scopes `lock_released` with a task to
that task, and `accounting_settled` to the run when the company has no
active hard-stop budget policy. Only a reached hard stop can refuse work
because of accounting (`evaluateAdmissionInTransaction`), so without
such a policy the settlement of one run cannot admit another task. The
other hold releases keep the company hint.

Backstop. Each scoped hint also sends `{ companyId, backstop: true }` to
the same four queues. Nothing else arms it. The queue runs it as a full
company pass, but not before 5 minutes after the last full pass of that
queue for that company or of the startup pass. It is a scheduler
deadline (`nextWakeAt` reports it), but it holds no idle intent and does
not mark the queue unreconciled. So:

- Startup still reconstructs every company.
- A failed scoped load retries with the existing back-off (30 seconds to
5 minutes). The backstop is pending in any case.
- A graph with no task or more than 100 tasks loads the whole company at
once.
- A task that a scoped hint missed is checked at most
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes) after the event. While runs
continue, each queue runs at most one full pass per company per 5
minutes. With no activity, nothing is scheduled.

Why arming on activity only cannot leave a stranded task undetected when
the instance then goes quiet. A task becomes stranded only through a
write or through a run that stops without a write:

1. The write names the task's graph (run and task producers). The scoped
pass checks the task at once.
2. The write sends a company hint (pause, budget, agent and company
configuration, tree holds, the producers listed below). The company pass
checks it within the 30-second gate, as before.
3. The write is a run event whose graph does not contain the task. That
same event armed the backstop. The backstop deadline stays scheduled
while the instance is quiet and checks the task at most 5 minutes later.
4. A run stops without any write (process lost, controller gone). The
existing deadline paths find it: `stale-issue-locks` waits for the
stored controller lease, and `active-run-silence` and
`orphan-active-leases` act on their own deadlines. Each of them
terminalizes the run, which is a run write and falls under 1 or 3. The
new test `restarts a task whose run vanished without a final event once
its lease lapses` covers this with the production backstop.
5. The process restarts or a quiet instance is woken by a restart. The
startup pass reconstructs every company.
6. A check that found a task waiting (grace, back-off, monitor) returns
its own deadline. That entry keeps its deadline whatever hints arrive.

Five minutes is the existing upper bound of recovery back-off in this
area: the maximum retry delay of a failed scoped-deadline check
(`scoped-deadline-work.ts`), and `DEFERRED_WAKE_PENDING_BOUND_MS`, the
time `assigned-work-sweep.ts` already trusts a deferred wake before it
calls a task stranded.

Producers that keep the company hint, and why. They can change
eligibility of tasks outside one task graph, or I could not show that
their effect stays in the graph. None of them fires once per run event
in the fixture.

- Eligibility for many tasks: company status (`companies.ts`), budgets
(`budgets.ts`), agent lifecycle and configuration (`agent-lifecycle.ts`,
`agent-configuration.ts`), instance settings (`instance-settings.ts`,
`*`), tree holds (`issue-tree-control.ts`), company deletion and import
(`company-deletion.ts`, `company-portability.ts`), and the hold releases
`pause_lifted`, `tree_hold_released`, `scheduling_resumed` and
`reconciler_backstop`. `accounting_settled` also keeps it when the
company has an active hard-stop budget policy.
- Task writes that can move a task between graphs or change eligibility:
task creation and a task update that changes the parent in `issues.ts`
(other task updates and checkout releases now name the task), document
delete and restore paths in `documents.ts`, task watchdogs
(`task-watchdogs.ts`), system and workspace recovery
(`system-recovery.ts`, `task-retry.ts`,
`workspace-recovery-capture-policy.ts`,
`workspace-recovery-stop-proof.ts`).
- Conversation and chat paths: `agent-conversations.ts`, the daily-cap
release in `heartbeat/queue.ts`, the Telegram attachment failure,
`chat-attachment-reuse.ts`, `native-runner-file-handoff.ts`,
`notifyChatRunMilestones` without a task or run (company authority
change), and `notifyNativeStatusReassessment` from `access.ts`.
- `writeExecutionControl` and `notifyExecutionRunWork` callers that pass
only a company.

**Reason and benefit**

Background work must not grow with the backlog of open tasks. The checks
a run event causes now depend on its task graph, not on the company.

## What Changed

- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery`, `resolveIssueGraph`, the scope matcher and
decoder, `ISSUE_RECOVERY_MAX_ISSUE_IDS` (100) and
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes).
- `server/src/services/recovery/work.ts`: the task queues resolve a
graph hint when they load it (shared for 2 seconds), remember the last
full pass per company, and gate the backstop hint on it.
`registerIssueRecoveryWork` takes an optional `backstopMs`.
- `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects`
accepts `issueIds`.
- `server/src/services/scoped-deadline-work.ts`: an optional
`backstop(scope)` that returns the earliest time of a backstop hint. A
pending backstop hint keeps its deadline but does not count as pending
work or as unreconciled.
- Producers: `issue-recovery-actions.ts`,
`execution-control-notifications.ts`, `run-dispatch-notifications.ts`,
`native-status-reassessment-notifications.ts`,
`external-conversation-settlement-notifications.ts`,
`work-intent-outbox.ts`, `run-cost-accounting.ts`,
`heartbeat/run-lifecycle.ts`, `heartbeat/runtime-execution.ts`,
`heartbeat/retries.ts`, `heartbeat.ts`, `legacy-execution-recovery.ts`,
`issues.ts` (checkout), `documents.ts` (task document writes),
`modules/wake-queue/adapters/postgres.ts` (recovery runs for a task),
`heartbeat/queue.ts` (wake enqueue), `recovery/service.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-safe-replacement.ts`,
`modules/run-dispatch/adapters/workspace-busy-wait.ts` (scheduled
retries).
- `bounds subsidiary scans for100 assignment candidates`: a resolved
recovery action now adds one scoped load per task queue (bound 8 to 11),
and the coalescing check sends one company hint before the burst of 100,
because the first full pass after the resolve is now a backstop pass
with its own key.
- `checks only blocked tasks whose blockers are done`: uses the shared
recording harness and runs out the backstop before its quiet check,
because the blocker's status change is now a graph hint (its 300
dependents overflow to a company load).
- Tests. `issue-recovery.test-fixture.ts` and
`live-recovery-work.test.ts` register the queues with `backstopMs: 0`,
so their quiet checks keep the earlier timing. The new tests use the
production value.

## Verification

New tests. I saw each red test fail before the change.

- `issue-recovery-deadlines.test.ts` `re-checks only the tasks a run
lifecycle event can affect, whatever the backlog` (red before): a run of
the agent on a task with a parent and a dependent, 1,000 backlog tasks
of another agent, and a second open task of the same agent. Seven real
producers (`notifyNativeStatusReassessment`, `notifyExecutionRunWork`,
`notifyRunDispatch`, `promoteDeferredWorkIntents` lock release and
accounting settlement, recovery action create and resolve) check exactly
the 3 tasks of the graph. The recovery action steps were red before the
third commit. With 4,000 more backlog tasks the event issues the same
number of statements. Before the change it checked 1,004 tasks.
- `resolves a run's task graph through index lookups, not the backlog`
(red before the second commit, from a Greptile finding): with 2,000
backlog tasks the graph lookup read 2,005 issue rows
(`pg_stat_xact_user_tables`) because it compared `id::text`. It now
validates and casts the saved id, reads fewer than 50 rows, and returns
exactly the run's graph.
- `finds a stranded task outside the affected set within the backstop
bound` (red before): a task inserted without a hint is not checked by
the run event, and is checked at most 5 minutes after the last full
pass. After 20 quiet minutes, a run event checks a new unrelated task
within 5 seconds, because the last full pass is older than 5 minutes.
Then the queues are quiet.
- `keeps the company pass for hold releases that can admit any task`
(guard, passes before and after): accounting settlement with a hard-stop
policy and a lifted pause still check every task.
- `falls back to the company pass when a run affects more tasks than a
hint names` (guard): a run task that blocks 100 more tasks checks the
whole company.
- `limits an ordinary comment to its issue among one thousand open
assignments`: the old final assertion (a company hint checks all 1,000
tasks) stays, and the test now also runs out the backstop and ends
quiet.
- `issue-recovery-assignments.test.ts`, real heartbeat, real producers,
production backstop (guards; they also pass with the old company hint):
a task whose run failed, a task whose run the system cancelled, a run
that vanished without a final event and is found when its controller
lease lapses, and an agent's queued task that starts when its previous
run ends. Each one recovers within 15 seconds, well before the backstop.
- `scoped-deadline-work.test.ts` `keeps a backstop hint's deadline
without holding the worker busy` (red before).

Suites, each file alone, in `server/` (issue recovery, live recovery,
recovery action and attention suites against dedicated local `_test`
databases): `issue-recovery-deadlines` (13/13),
`issue-recovery-assignments` (7/7), `issue-recovery-work-lifecycle`
(14/14), `scoped-deadline-work` (24/24), `live-recovery-work` (51/51),
`assigned-work-invariant` (32/32), `assigned-work-sweep-round-trips`
(2/2), `saved-work-recovery-overdue` (4/4),
`issue-recovery-dependencies` (5/5), `issue-recovery-conversations`
(3/3), `issue-recovery-actions` (112/112), `attention-retention-work`
(55/55). `recovery-deadline-acceptance` skips without its external
database. `pnpm check:module-boundaries` passes.

Local benchmark. The real server ran against PostgreSQL with
`log_statement=all` and instrumented counters (not committed). Fixture:
1 company, 60 agents, open tasks at 150 / 1,500 / 5,000 (52% in review,
39% blocked behind open blockers, 3% in progress, 2% to do), with
process agents that fail at once so recovery re-dispatches runs all the
time. The tables are analyzed after loading. Each window is 180 s,
starting 150 s after the server is ready, so each "after" window
contains one backstop pass. Base is `master` at 54c3616ad9; after is
this branch at dcd170e29f. The last commit (graph hints without the
30-second gate, task updates and checkout releases scoped) is covered by
the tests below but not re-measured. The machine was shared (load
average 9-16, one base window at 45). Run churn differs between runs, so
the table gives counts per run event (runs created in the window).

| Open tasks | Runs created (/s) base → after | Stranded checks per run
base → after | Assigned-work checks per run base → after |
`agent_task_sessions` reads (/s, 4 shapes, stranded check) | Statements
per run event base → after |
| --- | --- | --- | --- | --- | --- |
| 150 | 0.08 → 0.14 | 45.1 → 6.2 | 2.00 → 0.96 | 3.75 → 1.27 | 2,260 →
1,257 |
| 1,500 | 1.09 → 1.00 | 22.0 → 3.9 | 1.24 → 1.53 | 26.5 → 7.2 | 1,313 →
1,060 |
| 5,000 | 1.12 → 1.99 | 20.4 → 1.6 | 4.10 → 0.91 | 29.3 → 8.1 | 1,400 →
829 |

New statements: the task graph lookup (`with seed(id) ...`) at 0.8 / 9.3
/ 18.5 per second, and scoped loads of the task queues (`select "id",
"company_id" from "issues" ...` grew from 0.7 / 6.6 / 7.8 to 2.9 / 20.6
/ 34.4 per second, each a lookup of at most 100 ids). Stale-lock checks
grew from 0 / 50 / 52 to 22 / 190 / 445 per window: each run event now
checks the locks of its own task at once instead of waiting for the next
full pass. In an earlier round the graph lookup ran in every producer
transaction: 34/s at 1,500 and 70/s at 5,000.

With no activity the server issues no statements before and after this
change (quiet checks in `issue-recovery-deadlines`,
`issue-recovery-assignments`, `live-recovery-work`).

## Risks

- A task outside the graph of the run whose state the event changes is
now found by the backstop, at most 5 minutes later, instead of within 30
seconds. One known case: from #21427, `assigned-work-invariant` counts a
held recovery wake as pending while its agent has queued recovery runs.
When the last one starts, the continuation queue delivers the held wake;
the invariant check of that task waits for the backstop.
- A producer that names a task but changes something wider would now
miss the wider effect. Producers that change eligibility for many tasks
(pause, budget, agent and company configuration, tree holds) still send
the company hint.
- The graph is resolved after commit, from the written state. A parent
change is not a run event and keeps its company hint.
- The live-recovery and issue-recovery suites opt out of the backstop
deadline with `backstopMs: 0`.
- The execution-control deadline writes and legacy run terminalization
now pass the run id. `writeExecutionControl` also forwards that scope to
the live-run queues (`active-run-silence`, `native-process-sessions` and
others), so those hints now name the run instead of the company. Those
queues already handle run-scoped hints from native finalization.
- Test-only option: `registerIssueRecoveryWork(..., { backstopMs })`.
The shared recovery fixtures pass `0` so their quiet checks keep the
earlier timing; the new tests use the production value.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
nightly/v2026.1114.0-nightly.0 canary/v2026.1114.0-canary.4
2026-11-14 01:29:22 -06:00
DottaandClaude Opus 5.5 c1d3f90b37 fix(recovery): keep the resolved-dependency probes per task whatever the statistics (#21574)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `resolved-dependencies` recovery queue loads the blocked tasks
whose blockers are all done (#21517). It runs this load on recovery
hints.
> - The load has two correlated subqueries on `issue_relations`.
PostgreSQL turns them into a semi join and an anti join, and lets the
statistics choose how.
> - After a bulk change in a company that the statistics have not seen
(a large import, a restore, a big cleanup), the planner estimates one
row for the company. It then picks nested loops whose inner side scans
every relation of the company and filters by task id. On a local copy
with 5,000 open tasks the query ran for more than 20 seconds (once 19
minutes) until the table was analyzed; after `ANALYZE` it takes 3.7 ms.
> - This pull request keeps both subqueries as per-task subplans, so
each one is an index probe by (company, task id) whatever the statistics
say.
> - The benefit is that the load stays proportional to the blocked tasks
it reads, also right after bulk changes. This is prevention: the hosted
instance does not show this query as slow today.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a local database that had been analyzed with a smaller dataset and
then loaded with 5,000 open tasks in a new company, the
`resolved-dependencies` load query did not finish within 20 seconds; in
a benchmark run it stayed active for 19 minutes after its server
stopped.

**Steps to reproduce**

1. Load a company with about 1,500 tasks and run `ANALYZE issues,
issue_relations`.
2. With autovacuum off for these tables, load another company with 5,000
tasks, 1,900 of them blocked with `blocks` relations.
3. Run the load query of `resolved-dependencies` for the new company.

Plan with stale statistics (abridged):

```
Limit
  ->  Nested Loop Semi Join  (rows=1)
        Join Filter: (issues.id = r.related_issue_id)
        ->  Nested Loop Anti Join  (rows=1)
              Join Filter: (r_1.related_issue_id = issues.id)
              ->  Index Scan using issues_company_id_uq on issues  Index Cond: (company_id = ...)
              ->  Nested Loop
                    ->  Index Scan using issue_relations_company_related_issue_idx on issue_relations r_1
                          Index Cond: (company_id = ...)
                    ->  Index Scan using issues_company_id_uq on issues blocker  Index Cond: (company_id = ...)
        ->  Index Scan using issue_relations_company_related_issue_idx on issue_relations r  Index Cond: (company_id = ...)
```

The index condition holds only `company_id`. The task id is a join
filter, so each blocked task rescans every relation (and every task, for
the blocker) of the company.

**Expected behavior**

Each blocked task probes its own relations through
`issue_relations_company_related_issue_idx` (company_id,
related_issue_id) and its blockers through `issues_company_id_uq`
(company_id, id).

## What Changed

- `server/src/services/recovery/work.ts`: the candidate predicate is a
named export, `resolvedDependencyCandidate`, and both subqueries end
with `offset 0`. PostgreSQL does not pull a subquery with `OFFSET` up
into a join, so it stays a correlated subplan. Its index condition then
includes the outer task id. The predicate and its results are otherwise
unchanged.
- `server/src/__tests__/resolved-dependency-candidates.test.ts`: two
tests.

Indexes that the subplans use (all existing):
`issue_relations_company_related_issue_idx` (company_id,
related_issue_id) for both relation probes, and `issues_company_id_uq`
(company_id, id) for the blocker status.
`issue_relations_company_issue_idx` (company_id, issue_id) is not needed
here.

## Verification

- `selects the same tasks as the earlier predicate`: 40 blocked tasks
with a done blocker, an open blocker, a done and a cancelled blocker, or
no blocker. The new predicate returns the same 10 tasks as the earlier
one.
- `probes the relations of each task, whatever the planner statistics
say`: analyzes a small company, then loads 3,000 blocked tasks in a new
company. The plan (`EXPLAIN (FORMAT JSON)`) has two subplans and no semi
or anti join, and the load returns 100 rows within a 5-second statement
timeout. I saw it fail first with the earlier predicate: the plan had a
semi join and an anti join.
- Local plan on the stale database: EXPLAIN ANALYZE shows `SubPlan 1`
and `SubPlan 2` with `Index Cond: ((company_id = issues.company_id) AND
(related_issue_id = issues.id))`, 1,933 probes each, 4.9 ms.
- `issue-recovery-deadlines.test.ts` and
`issue-recovery-dependencies.test.ts` pass (14 tests). `pnpm
check:module-boundaries` passes.

## Risks

- With fresh statistics the planner could use a merge or hash join for a
large page; the subplan form gives up that choice. Each blocked task now
costs two index probes, which is the same order as the fresh-statistics
plan (3.7 ms before, 4.9 ms after on 1,933 blocked tasks, measured once
each on a loaded machine).
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:28:55 -06:00
DottaandClaude Opus 5.5 6b6c94a4ea perf(chat): read the GitHub response run by issue run indexes (#21577)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel keeps one response comment for each GitHub
request. Two background sweeps (acknowledgements and terminal responses)
and the acknowledgement step read the latest run of the assigned agent
for the request's issue with `readGitHubResponseRun`
> - That function sends two statements. Both match runs with
`coalesce(native_issue_id::text, context_snapshot->>'issueId') = <issue
id>`. No index has that expression, so each statement reads every run of
the agent and decompresses the large `context_snapshot` of each legacy
run
> - On a hosted test database, these two statements were the two most
expensive statements by database time. In a 701 s window, each ran 6.7
times a second, for 203 s and 137 s of database time (about 30 ms and 20
ms for each call). Together they kept about half of one database session
busy
> - This pull request reads the latest run with two index lookups
(native binding, and legacy context without a native binding), the same
way #20671 changed the terminal-response sweep. It decides request
ownership on that one run in the same statement
> - The benefit is that each call reads a few index entries in one
statement instead of the agent's whole run history in two statements.
The results do not change, and no migration is necessary

## Linked Issues or Issue Description

Refs #20671 (the same rewrite for the terminal-response discovery page;
its Risks section names this function as the remaining case).

**What existing behavior does this improve?**
`readGitHubResponseRun` in
`server/src/services/chat-channels/github/response-run-state.ts`. The
GitHub acknowledgement sweep calls it once for each candidate request.
The terminal-response sweep and the acknowledgement step call it through
`githubResponseExecution`.

**Subsystem affected**
GitHub chat channel: request acknowledgement and terminal-response
recovery.

**Current behavior**
The function sends two statements:

1. The latest run of the agent for the issue that woke for the request:
`heartbeat_runs` joined to the inbound `chat_message_links` row of the
delivery on `wakeCommentId` or `wakeCommentIds`, ordered by `created_at
desc`, limit 1.
2. The latest run of the agent for the issue, with the
`responseRunColumns` projection, ordered by `created_at desc, id desc`,
limit 1.

Both statements filter on `coalesce(native_issue_id::text,
context_snapshot->>'issueId') = $issueId`. The planner can only use
`heartbeat_runs_company_agent_started_idx` or
`heartbeat_runs_company_created_at_desc_idx`. It reads all runs of the
agent and evaluates the `coalesce` on each row, which decompresses the
context of each legacy run. On the hosted test database,
`heartbeat_runs` has about 31,000 rows and 1.96 GB, because
`context_snapshot` is large.

**Proposed behavior**
One statement:

- The latest run ID comes from two disjoint branches, each `order by
created_at desc, id desc limit 1`: runs with `native_issue_id =
$issueId` (through
`heartbeat_runs_company_native_issue_contract_id_uq`), and runs with
`native_issue_id is null and context_snapshot->>'issueId' = $issueId`
(through `heartbeat_runs_company_ctx_issue_created_idx`). Together the
branches are exactly the rows where the old `coalesce` equals the issue
ID.
- The statement reads that one run by primary key with the same
`responseRunColumns` projection.
- An `exists` subquery on the inbound `chat_message_links` row of the
delivery decides whether that run woke for the request. It uses the same
`wakeCommentId` / `wakeCommentIds` match as before, but on one run only.
- When the issue ID is null, the function returns no run without a
query. The old statements could not match a null issue ID either.

**Reason and benefit**
These were the two most expensive statements on the hosted test
database. On a local fixture with the hosted shape, the median call goes
from 111.6 ms to 1.03 ms (about 108 times faster), and the buffers for
one call go from about 73,000 to 98.

**Breaking changes**
None. The function signature, the returned run, and the ownership result
do not change.

## What Changed

- `readGitHubResponseRun` reads the latest run of the agent for the
issue through the two index-backed branches, and decides request
ownership on that run in the same statement. Two statements become one.
- Add `response-run-state.test.ts`. It runs the old two-statement
function (kept in the test as the oracle) and the new function on the
same inputs, and it requires equal results. It also requires a plan
bound: one call must read fewer than 150 `heartbeat_runs` rows while the
agent has 1,500 unrelated runs.
- Add `scripts/benchmark-github-response-run-state.ts`. It loads the
baseline module from a given commit, compares the complete output of
both versions for every input, measures both, and prints the plans.

## Verification

Why the old statements were slow, and the thesis: each statement filters
the agent's whole run history with an expression that no index has, and
detoasts the context of every legacy run. If each precedence branch uses
its own issue index and ownership is checked on the one latest run, the
cost no longer depends on the agent's run count, and the results stay
identical.

Red/green regression test:

- `cd server && npx vitest run
src/services/chat-channels/github/response-run-state.test.ts`
- Red (test file on the old `response-run-state.ts`): the plan bound
fails with `expected 3597 to be less than 150`. The equivalence test
passes, because the oracle is the old code.
- Green (this change): 2 of 2 pass.

The equivalence fixture has two companies, two agents each, 60 issues
each, and one or two GitHub requests for each issue. Runs cover these
cases: native binding, legacy context without a native binding, a native
binding to another issue while the legacy context names this issue,
another agent, another company, `wakeCommentId`, `wakeCommentIds` as an
array, `wakeCommentIds` as a JSON string, a link to the outbound message
of the delivery, a link to another company's comment, a link to the
sibling request of the same issue, a request with no inbound comment,
ties on `created_at`, and a null issue ID. It also calls with another
company's endpoint. The test requires more than 20 results in each
outcome (owned, latest run not owned, no run) and both binding kinds.

Local benchmark with the hosted shape:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-github-response-run-state.ts --baseline-ref=3c3e6c2d75
```

Fixture (synthetic): one company, 4 agents with about 7,750 runs each
(31,000 runs), 2,000 issues, 300 GitHub requests. Each run has about 60
KiB of incompressible context, and `heartbeat_runs` is 1,932 MB. Runs
are native or legacy at random, and about half link to the request
comment. Embedded PostgreSQL 18.1 on an Apple laptop. The hosted
database is Neon, so absolute times are local measurements only. The
benchmark compares the complete output for all 600 inputs (262 owned,
329 latest run not owned, 9 no run): equal. It then times 20 inputs with
runs (two agents), with 2 warmups and 7 alternating samples.

| | Statements per call | Median ms per call (range) | Buffers for one
call |
| --- | --- | --- | --- |
| Before | 2 | 111.6 (100.9–240.5) | 18,358 hit + 26,498 read, then
14,997 hit + 13,489 read |
| After | 1 | 1.03 (0.81–1.39) | 80 hit + 18 read |

Plans for one call. Before: statement 1 is a `Bitmap Heap Scan` on
`heartbeat_runs_company_agent_started_idx` that removes 7,674 rows by
filter (75.7 ms). Statement 2 is an `Index Scan` on
`heartbeat_runs_company_created_at_desc_idx` that removes 19,801 rows by
filter (60.2 ms). After: one `Index Scan` on `heartbeat_runs_pkey`. Its
`InitPlan` is a `Merge Append` of two `Limit` index scans on
`heartbeat_runs_company_native_issue_contract_id_uq` and
`heartbeat_runs_company_ctx_issue_created_idx`, and its `SubPlan` is one
`Index Scan` on `chat_message_links_delivery_idx` (0.128 ms).

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed
(all GitHub chat-channel integration tests, including the
acknowledgement and terminal-response sweeps).
- `cd server && npx tsc --noEmit`: passed.
- `pnpm verify` exited 2 because a file under `scripts/` changed, which
requires the broad CI coverage. Required CI covers the rest.

Why the function runs 6.7 times a second. The acknowledgement sweep and
the terminal-response sweep rebuild their candidate set when a hint
arrives, and they call this function once for each candidate on each
rebuild. A run milestone for a run with no issue (for example, every
status change of a run without a task in `setRunStatus`, or a comment
merged into an active run in the wake queue) sends a company-wide hint
to both sweeps. So in a busy company, both sweeps rebuild about once a
second (their minimum pass interval), and each rebuild reads every
parked candidate again, although nothing about those candidates changed.
In a local reproduction with 8 parked candidates (4 for each sweep) and
the real coordinator and workers, 10 company-wide milestone hints a
second gave 8.75 calls a second; 1 hint a second gave 7.98 calls a
second; 10 hints a second for an unrelated issue gave 0 calls; and no
hints gave 0 calls. So the hosted rate is consistent with about 7 parked
GitHub requests in one company and a steady stream of company-wide run
milestones. This pull request makes each call cheap. A separate pull
request will address the repeated rebuilds.

## Risks

- Low risk. The change is one function. The run match is the same set of
runs: `native_issue_id::text = <uuid text>` is equal to `native_issue_id
= <uuid>` for the canonical UUID text that callers pass (the issue IDs
come from UUID columns), and a null `native_issue_id` makes the old
`coalesce` use the legacy context.
- Ownership: the old code compared the latest run that woke for the
request (ordered by `created_at` only) with the latest run (ordered by
`created_at, id`). That is true exactly when the latest run woke for the
request, except when two runs that both woke for the request tie on
`created_at` with the latest run. In that case the old result depended
on the physical row order. The new code always returns the deterministic
answer (the latest run woke for the request).
- The number of calls does not change in this pull request.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 00:59:25 -06:00
Nicky LeachandClaude Opus 5.5 e977c3b922 feat(native-runtime): end the wait for the provider's own turn when a person comments (#21455)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner part of this work is complete in #21316 and the pull
requests below it: when a controller sends `providerStartedTurns` on
`turn.start`, the Runner keeps Claude while a background shell is
running, and gives the cycle of Claude to the controller as a turn that
the provider started.
> - Part 2 gave the consumer a wait with a limit and two inputs: a hook
that ends the wait early, and a start time.
> - This pull request gives both from the server for a native ACPX
Claude run. A new comment from a person ends the wait. The start time
comes from an event that is already saved, so a restart of Core does not
make the wait longer.
> - The benefit is that a person who comments does not wait up to 10
minutes for the agent to see it.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one of four parts on the Core side. They are
stacked on #21316:

1. The driver admits a turn that the provider started.
2. The consumer waits for that turn, for a limited time.
3. The server ends the wait when a person comments, and keeps the clock
across a restart.
4. The transport sends the opt-in. Only this part changes what a
production run does.

This is part 3. Its base branch is `feat/core-provider-turn-consumer`.
Refs #19912 (design).

**What existing behavior does this improve?**

Nothing visible yet. What Core gives to the consumer of a native ACPX
Claude run.

**Subsystem affected**

Server: `server/src/services/native-runtime`.

**Current behavior**

- A comment from a person during a run is queued as a deferred wake. The
next run delivers it. This does not change.
- Core has no wait for a provider-started turn.

**Proposed behavior**

- For a native ACPX Claude run, Core gives `executeNativeSession` the
option `providerTurnWait`.
- One delivery worker does the person-comment check for all runs that
wait. The server registers it at startup with the shared delivery work
coordinator, as it registers the cost accounting worker. The worker
listens to the existing notification for deferred wakes. On each notice
it runs one query for each run that waits: a comment from a person on
this issue, made after the start of the run, that a deferred wake of
this agent names. A comment that the query finds ends the wait of that
run. The run then takes the path of today, and the next wake delivers
the comment as today.
- If a database read fails, that run stays owed a check, and the
coordinator runs the worker again after 5 seconds, as for each other
delivery worker. A read that fails for one run does not stop the check
of the other runs. When no run is owed a check, the worker has no
deadline and does no query.
- The start time of the wait is the time of the first saved
`turn.completed` event of the run that has `awaitingProviderTurn`. With
no such event it is empty.
- Each query has the company id in its condition.

**Reason and benefit**

- The comment window starts at the start of the run, not at the start of
the wait. A comment made during the prompt turn must not wait for the
limit.

## What Changed

- `server/src/services/native-runtime/native-provider-turn-wait.ts`
(new, 109 lines) and 4 lines in `native-session-executor.ts`.
- `server/src/index.ts` (4 lines): the registration of the worker, its
stop at server close, and its place in the list of delivery workers that
must be ready. `delivery-work-notifications.ts` (1 line): the queue key
of the worker. The key needs no database change, because each topic uses
the one shared channel.
- `server/src/vendor/paperclip-runner/index.ts`: one type export.
- Tests: `native-provider-turn-wait.test.ts` (new, 252 lines) and one
case in `native-usage-accounting.test.ts` (25 lines).

## State saved by the previous release

- No new persisted state and no schema change. The start time is read
from `heartbeat_run_events`.
- **A run that the previous release started:** it has no waiting
terminal, so the start time is empty.
- **Core restarts during a wait:** the new process reads the start time,
so the limit is the original one. The consumer then continues the wait
(#21454), and the worker checks the comments again from the start of the
run, so a comment that was made before the restart still ends the wait.
- **Rollback of Core during a wait:** the previous Core stops at the
waiting terminal and finalizes the run as `native_finalization_missing`,
as today.

## Verification

At the head of this branch:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json` and `pnpm run build:typescript` in
`packages/paperclip-runner`; `node
scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in
`server`; `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed.
- `native-provider-turn-wait.test.ts`, 10 tests on embedded Postgres
with the real coordinator: a comment from a person in a deferred wake
ends the wait; a comment from an agent, a comment before the run, and a
comment of a different company do not; the start time comes from the
first waiting terminal, and a run of the previous release has none.
- The 6 new tests: a read that fails one time is tried again after 5
seconds and finds a comment that is already saved, with no new notice; a
notice during a read that fails gives a new check; a read that fails for
one run does not hold back a second run, and each run ends only for its
own comment; a run that starts to wait during a check is checked; a wait
that is aborted is removed, nothing checks it again and no deadline
stays; with no run owed a check there is no deadline and no query in one
hour of test time.
- The test helper now registers the wake intent before it writes the
deferred wake, in the same transaction. This is the order of each
production writer (`modules/wake-queue/adapters/postgres.ts` and
`services/heartbeat/queue.ts`). Production code did not change for this.
- Mutation checks: a read that fails clears the owed check (3 tests
fail); an aborted wait is not removed (the abort test fails).
- Also run, unchanged and green: `delivery-work-coordinator.test.ts`,
`budget-enforcement-worker.test.ts`,
`server-startup-feedback-export.test.ts`, `startup-timeline.test.ts`.
The 7 files together: 1,699 passed, 7 skipped.
- `native-usage-accounting.test.ts`: a prompt turn and the
provider-started turn after it are each counted one time, with the
events as the executor receives them. The cost of the provider-started
turn is the session total in the cumulative channel, and it is not added
two times. The accounting code is not changed.

Not run: a run through the executor with a database; the full suite. CI
supplies the broad coverage.

## Risks

- **One more query at each start of a native ACPX Claude run,** on
`heartbeat_run_events` by company and run.
- **The worker is registered in each server process, and it is idle with
no wait.** A notice with no run that waits does no query. No wait starts
before part 4.
- **After an abort during a retry, the 5 second retry of the coordinator
runs one more time.** It finds nothing owed and does no query. The
coordinator has no call to cancel the retry of one worker.
- **A process that does not register the worker does no comment check.**
The server registers it with no condition at startup, and native runs
execute only in that process. Only a unit test can be in this state.
- **A comment ends the wait but does not interrupt a turn that is
running.** That is the rule of part 2.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:55:11 -08:00
DottaandPaperclip 2102492806 fix(discord): refresh due command registration after task drains (#21573)
## Thinking Path

> - Paperclip runs agents and connects their work to chat services.
> - Discord command registration uses a shared worker and saved
deadlines.
> - A task drain can hold that worker while registration is already due.
> - The coordinator can keep a 30-second fallback after the drain ends.
> - This change refreshes the command owner on drain release or observed
expiry.
> - Normal admission, earliest-time and provider retry checks still
apply.

## Linked Issues or Issue Description

Refs: #17557, #21562. Related: #15532 checks credential reads when
registration is not due. This change handles resumption of due work in
the separate command worker.

**What happened?**

A due command registration can wait for the coordinator's fallback after
a task drain ends. The worker retains the request but has no
drain-release subscription. This delays command work and its return to
an idle state.

**Expected behavior**

Resume due command work when admission returns. Preserve future earliest
times and provider failure backoffs. Leave an empty or stopped worker
idle.

**Steps to reproduce**

1. Register the normal command worker while an idle task drain is
active.
2. Retain a due registration and capture the coordinator's fallback.
3. Release the drain, or observe its expiry, before that fallback.
4. Check that the due endpoint is checked before the fallback. Both
regression cases fail before this change.

**Paperclip version or commit**

Main base `3c15acd3fc1b200ef0e36ac904a03cd5fcba00c0`.

**Deployment mode**

Built from source. The focused tests use the real coordinator and
command owner with controlled registration operations. They do not
contact Discord.

## What Changed

- Subscribe the command owner to normal drain changes and observed
expiry.
- Wake it only when current admission allows work and its existing next
time is due.
- Remove both subscriptions through the normal worker stop callback.
- Add eight controls for release, expiry, another closed gate, future
earliest time, provider backoff, empty state and both stop paths.

## Verification

Local checks used the integrated branch that exposed this contract gap.
This main port contains the same two tested file afterimages. Its
admission, scheduler and fairness modules match those local inputs.
Main's coordinator and DB modules have other changes. Fresh CI on
`aec6ecc65717aa6372bad526e044ec4aa3888bec` now qualifies this main
context.

- Both release and expiry cases fail at the intended missing
endpoint-check assertion before the product change. No setup or timeout
failure is used as causal proof.
- All eight focused cases pass after the change. No case is excluded.
- The complete normal Server typecheck and affected-file program checks
pass on the integrated branch.
- Normal committed Source SDK verification passes on this main branch.
- Exact-head main CI passes, including typecheck, build, the general and
serialized suites, and all eight new controls (142 ms) in server shard
2/12. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/43565579653).
A full local repository build and suite were not run in this checkout.
- Greptile reviewed the exact head at 5/5 with no actionable findings.
The complete edited summary and all paginated review threads were
inspected; no unresolved threads remain.

Run the focused controls with:

```sh
pnpm --filter @paperclipai/server exec vitest run src/services/chat-channels/discord/command-work.test.ts --maxWorkers=1 --no-file-parallelism --retry=0
```

The existing admission-drain contract remains accurate. This fix adds no
configuration or command, so no documentation change is required. Full
transport sleep acceptance and deployed suspension are separate claims.

## Risks

A release event can request an earlier pass for already due work. The
existing `canRun`, earliest-time, batching and per-item backoff checks
remain active. Both worker stop and coordinator stop must remove the
subscriptions; the controls cover each path. No scheduler policy, claim
or provider deadline changes.

## Model Used

OpenAI `gpt-6.1-sol` assisted with implementation and independent
review. The Codex managing agent performed technical review and
verification. The host does not expose its exact runtime model ID or
context-window size. Work used source inspection, code edits and local
test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (not
required: the existing contract remains accurate)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-14 00:46:49 -06:00
DottaandClaude Opus 5.5 3c3e6c2d75 fix(retention): stop rescanning unchanged pinned workspace snapshots (#21571)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote workspaces keep saved versions in the
`execution_workspace_snapshots` ledger, and a retention sweep retires
versions that no reference holds
> - Every execution notification wakes this sweep, and the sweep
examined every pinned version again on each pass
> - Some pins never change. A migration cohort supersession leaves
failed operations with `lastFailure` and no `retiredSave`, and these
hold their snapshots until a separate review releases them
> - On a busy server the sweep ran four statements per pinned row, many
times each second, on a main thread that was already saturated
> - This pull request records the reference that pins a version and
leaves that version out of the candidate query until that reference
releases
> - The benefit is that an execution notification costs one candidate
query, not one transaction per pinned row

## Linked Issues or Issue Description

No public issue exists. Bug report:

**What happened?**
A hosted instance ran four statements at 13.4 calls per second each: the
workspace read, the importing-parent read and the operation read in
`pinned()`, and the locked ledger row read. The sweep in
`server/src/services/remote-workspace-snapshot-retention.ts` caused
them. A pinned superseded row keeps `status = 'superseded'` with
`delete_after = null`, so the candidate query selects it on every pass.
Each pass examines up to 50 rows. A truncated page runs again after one
second. Each run-work notification (`notifyExecutionRunWork`) sets
`rescanRequested`, which starts the scan again. The sequence stopped at
the operation read because a failed operation with `lastFailure` and no
`retiredSave` pinned these rows.
`supersedeExpiredRemoteWorkspaceMigrationCohort` writes such operations,
and they hold their snapshots permanently by design (see
`doc/DATABASE.md`, "Workspace cohort admission proofs").

**Expected behavior**
A pinned version is examined again only when the reference that pins it
can have changed. An idle or busy server does not repeat reads for
unchanged pinned history. A released version is retired on its normal
deadline.

**Steps to reproduce**
1. Create 300 superseded snapshots. Pin each with a failed
`remote_workspace_migration` operation that has `lastFailure` and no
`retiredSave`.
2. Register the retention worker on a delivery coordinator and let the
first pass finish.
3. Send `notifyExecutionRunWork` about 13 times each second for 60
seconds.
4. Count statements with `pg_stat_statements`.

**Paperclip version or commit**
`master` at 6dd0349d52

**Additional context**
The local reproduction showed the same four statements at equal call
counts, as on the hosted instance (see Verification).

## What Changed

- `pinned()` returns the first reference that holds the version (current
pointer, importing parent, operation, unfinished child, run, or recovery
action) and its ID, not a boolean.
- The sweep stores that reference in `metadata.retentionPin` when it
pins a version. It writes again only when the reference changes or a
deadline must be cleared.
- The candidate query leaves out a version with no deadline while its
recorded reference still holds. Each pin condition is one SQL builder.
`pinned()` and the candidate query use the same builder, so the
exclusion can never hold a version that `pinned()` would release.
- When the recorded reference releases, the version comes back, and
`pinned()` checks every reference again. It records the next reference
that holds, or schedules or retires the version.
- The record is removed when the sweep schedules or claims a version.
- The worker coalesces wakes for up to one second
(`REMOTE_WORKSPACE_SNAPSHOT_COALESCE_MS`). Retention deadlines are days,
so this delay has no effect on retirement.
- `doc/DATABASE.md` describes the record.
- No schema migration. The record is in the existing `metadata` JSONB
column.

## Verification

Red/green regression tests in
`server/src/services/remote-workspace-snapshot-retention.test.ts`:
- `leaves %s-pinned history out of later passes, across restarts, until
that pin is released`. This runs once for each of the six pin kinds. It
pins the version and checks that a new service instance (a restart)
examines zero rows. Then it releases the pin and checks that the version
is deleted.
- `re-examines a pinned snapshot when its recorded pin is released but
another still holds it`.
- `revisits a failed save's recorded pin once its abandoned upload is
settled`.
- `does not re-examine unchanged pinned history on execution
notifications`. This uses the coordinator and `notifyExecutionRunWork`.

On the parent commit, the first eight new tests fail (6 pin kinds, the
second-pin case, and the notification case). With the fix, all 31 tests
pass.

Related suites pass locally (212 tests):
`remote-workspaces.integration.test.ts`,
`remote-workspace-snapshot-reuse.test.ts`,
`remote-workspace-cohort-migration.test.ts`,
`snapshot-attempt-upload-abort.test.ts`, and
`remote-workspace-snapshot-retention.test.ts`. `tsc --noEmit -p
server/tsconfig.json` is clean.

Load measurement. Local PostgreSQL 18 with `pg_stat_statements`. 300
pinned rows. About 13 run-work notifications each second for 60 seconds,
after the first pass finished:

| | Before | After |
| --- | --- | --- |
| Pinned rows examined again | 30,250 – 39,000 / min | 0 / min |
| Sweep statements | ~151,000 – 196,000 / min | 88 / min |
| All statements, including the test's own notification transactions |
153,327 – 197,205 / min | 1,647 / min |
| Sweep database execution time | ~1,285 ms / min | ~52 ms / min |
| Candidate query | 0.07 ms, ~10 / min | 0.88 ms, 58 / min |

Before: `begin`, the locked row read, the workspace read, the
importing-parent read and the operation read each ran 30,250 times in 60
seconds. This is the same signature as on the hosted instance. After:
the sweep ran 58 candidate queries and 30 deadline reads. With
per-notification wakes and no coalescing, the sweep ran 776 candidate
queries and 776 deadline reads (1,552 / min).

I did not verify the change on a hosted instance.

## Risks

- A recorded pin is checked against only its own reference. The check
uses the same SQL condition as `pinned()`, so a version cannot stay
excluded after that reference releases. The parametrized test covers
each pin kind.
- A version can be pinned by more than one reference. When the recorded
reference releases, the sweep checks all of them again and records the
next one.
- A producer that rewrites the row's metadata and drops the record
causes one extra examination. A producer that sets a deadline makes the
record inactive, because the exclusion applies only when `delete_after`
is null.
- Versions that have no record yet, which includes all existing pinned
rows, are examined once after deploy. The examination writes the record.
- The candidate query does one primary-key lookup for each pinned row.
It takes about 0.9 ms with 300 pinned rows, and the time grows linearly
with the number of pinned rows.
- Pinned history from failed migration cohorts stays pinned. This change
does not alter that hold.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, local PostgreSQL)

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 00:05:32 -06:00
Nicky LeachandClaude Opus 5.5 b1f608296e feat(runner): keep a run open for a limited time while the Runner waits for the provider's own turn (#21454)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner part of this work is complete in #21316 and the pull
requests below it: when a controller sends `providerStartedTurns` on
`turn.start`, the Runner keeps Claude while a background shell is
running, and gives the cycle of Claude to the controller as a turn that
the provider started.
> - `consumeTurn` reads the events of one turn and returns at the
terminal. It already continues past a terminal for a Codex goal.
> - This pull request makes it continue past a terminal that waits for a
turn of the provider, and it limits that wait.
> - The benefit is that a run continues in the turn that Claude starts,
and a wait always ends.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one of four parts on the Core side. They are
stacked on #21316:

1. The driver admits a turn that the provider started.
2. The consumer waits for that turn, for a limited time.
3. The server ends the wait when a person comments, and keeps the clock
across a restart.
4. The transport sends the opt-in. Only this part changes what a
production run does.

This is part 2. Its base branch is `feat/core-provider-turn-driver`.
Refs #19912 (design).

**What existing behavior does this improve?**

Nothing in production yet. What the consumer of a native session does
after a turn that ended with no result while the Runner waits.

**Subsystem affected**

Paperclip Runner package, the controller side:
`src/native-session-runtime.ts` and a new module
`src/native-provider-turn-wait.ts`.

**Current behavior**

- `consumeTurn` returns at the first terminal. With no result, the run
is finalized as `native_finalization_missing`.

**Proposed behavior**

- At a terminal with `awaitingProviderTurn` and no result, `consumeTurn`
continues to read. It consumes the provider-started turn as the next
turn of the same run. A turn that reports a result finalizes as a normal
run.
- The wait has a limit of 10 minutes from the first terminal that waits.
A later terminal that waits does not start the clock again. The limit is
never later than 5 seconds before the timeout of the run.
- The limit and an early end stop the wait only. They never interrupt a
provider-started turn that is running; that turn stays under the timeout
of the run. If that turn ends and the Runner waits again, the run takes
the path of today at once.
- When the wait ends with no turn active, the consumer returns the last
terminal, and the existing code for a run with no result decides
(`resolveMissingResult`, then `native_finalization_missing`). No new
failure code.
- The wait ends at once when the driver gives the acknowledgement of
part 1. This covers Stop, the limit, and a Runner that says the wait is
over.
- `executeNativeSession` gets one new option, `providerTurnWait`: the
start time of a wait that an earlier controller process saw, and a hook
that ends the wait early.
- When the wait ends a turn, the consumer first waits for the receipts
of the text deltas that are committed in batches (#21460), as each other
return path of `consumeTurn` does. Without this, a delta that arrived
during the wait was not in the event count of the turn, and a delta that
failed to commit was not reported.
- When the interrupt that ends the wait is not available or fails, the
limit forces the end of the wait. If that forced end starts a new turn
(the continuation that a restart owes), the consumer continues to read
that turn. The forced-end signal is used one time only, and the read
that is open at that moment is kept until it gets its event. Before this
fix, the consumer returned the terminal of the old turn and lost the
first event of the new turn, and the run could end as
`native_finalization_missing`.
- If the controller restarts during the wait, the run continues to wait.
When the saved checkpoint still shows the first turn as active, the
recovered session gives the waiting terminal again, and the wait starts
again with the start time that was recorded. When the checkpoint was
saved after the waiting terminal, the consumer finds that terminal in
the journal, starts the wait again, and does not send the
disposition-only recovery prompt. The consumer does this only if two
conditions are true: the recovered session has the waiting state
(`awaitingProviderTurn`, saved by the driver since #21453), and the
terminal in the journal has the marker. If one condition is false, the
run uses the recovery path of today. When a wait that started again ends
with no turn, the existing code for a run with no result decides, as for
each other wait.

**Reason and benefit**

- The 10 minutes are the maximum time of a foreground command in the
pinned Claude Code.
- A limit on the wait, not on the work, means that a long test run in
the turn of Claude is not cut.

## What Changed

- `src/native-provider-turn-wait.ts` (new, 129 lines) and
`src/native-session-runtime.ts` (58 lines added, 4 removed),
`src/index.ts` (1 export).
- `src/native-session-runtime.provider-turn.test.ts`: written again for
the new behavior (557 lines added, 61 removed).

## State saved by the previous release

- This part adds no persisted state. The wait is in memory. Part 3 reads
its start time from an event row that is already saved.

## Verification

At the head of this branch:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json` and `pnpm run build:typescript` in
`packages/paperclip-runner`; `node
scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in
`server`; `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed.
- `src/native-session-runtime.provider-turn.test.ts`: 30 tests. They
cover: a controller restart with the first turn still active in the
checkpoint (the wait continues; a provider-started turn settles its
result; with no turn the limit ends the wait on the recorded clock); a
controller restart with the checkpoint saved after the waiting terminal
(the same two results, no recovery prompt, no event of the journal
appended again); a limit that passed before the restart ends the wait at
once, with and without an interrupt; the recovery prompt of today when
the terminal in the journal has no marker, when the journal has no
terminal, and when the recovered session has no waiting state; a second
restart during the wait; a forced end that starts a new turn settles the
result of the new turn, with each event of that turn appended one time
and in order; an event that reaches the open read after the forced end
is appended exactly one time; a delta that arrives during the wait is
counted when the limit ends the turn (the test fails when the new wait
for receipts is removed); continue and settle the result of the
provider-started turn; a second wait does not start the clock again; the
limit with no turn active; the limit and a comment during a running turn
(not interrupted); a running turn that ends and waits again after the
limit; Stop; a `run.terminal` that the Runner held; the Runner stream
ends during the wait; the limit under the timeout of the run; a start
time from an earlier controller; a plan that a provider-started turn
proposes settles as the plan review wait.
- `src/native-session-runtime.test.ts`, not changed by this PR: 297
passed.
- Mutation checks, each made tests fail: no continue at a waiting
terminal; no limit; no early end; no clock from an earlier controller;
the clock starts again at each wait; a running turn is interrupted at
the limit.
- Mutation checks for the forced end, each made the two new tests fail:
the forced-end signal used on each loop (the run ends as
`native_finalization_missing`); a new read started after the forced end
(the first event of the new turn is lost).
- Mutation checks for the restart, each made named tests fail: the
recovery prompt not held back; the wait not started again in
`consumeTurn`; the wait started again on the session state alone (the
test with no marker in the journal fails).

Not run: the full suite. CI supplies that coverage.

## Risks

- **No production effect.** No terminal has the marker before part 4.
- **`consumeTurn` runs for each native turn of each agent.** It makes
one small object for each turn and reads one more payload field at a
terminal.
- **If the interrupt that ends the wait is not available or fails,** the
limit forces the end: the consumer returns the last terminal, or
continues to read when the forced end starts a new turn. This is a
fallback path. On a forced return, one read of the event stream stays
open until the session closes, as before this PR.
- **A wait that starts again after a restart holds back the recovery
prompt.** This is the one change to the recovery flow. It needs the two
conditions above, and both exist only for a run that asked for
provider-started turns.
- **Known limit.** The driver does not clear its waiting state when the
consumer ends a wait with an interrupt. A restart in the short time
after that can start the wait again; the recorded clock and the comment
check then end it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 05:59:42 +00:00
DottaandClaude Opus 5.5 3c15acd3fc perf(recovery): look up held runs by key so the fresh-continuation deadline stops scanning run history (#21568)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is cut off, the fresh-continuation fallback continues the
task in a new run. The execution-disposition worker schedules its next
pass with `nextFreshContinuationCheckAt` and lists due work with
`heldExecutionsAwaitingFreshContinuation`
> - Both queries share two predicates (`heldExecutionWhere` and
`recentInterruptionSourceSql`). These predicates matched the run as
`source.id::text = <evidence text>` and the task's runs as
`native_issue_id::text = <issue id>::text`. A cast on an indexed column
prevents an index lookup
> - So for each held record, PostgreSQL read the company's whole run
history (two times), and it decoded each run's wide context JSON. On
staging, the deadline query took about 13.5 s of database time in each
27 s worker cycle. It was the largest statement by time
> - This pull request compares the uuid columns directly. It turns the
recorded run id into a uuid only when the text is the canonical
lowercase form, which is exactly when `id::text = text` can be true
> - The benefit is that each held record costs a few index lookups. The
same records are selected, and the worker cycle no longer keeps a
database session busy

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The fresh-continuation scheduler queries: `nextFreshContinuationCheckAt`
(the disposition worker's deadline),
`heldExecutionsAwaitingFreshContinuation` (the backlog listing, also
probed by the schedule), and the "left alone" count. They are in
`server/src/services/fresh-continuation-fallback.ts`.

**Subsystem affected**
Server: recovery and the execution-disposition worker
(`executionControlSchedule().disposition` in
`execution-control-work.ts`).

**Current behavior**
Neon statement statistics on staging (window of 160 s on 2026-11-14):
the `select
min((issue_recovery_actions.evidence#>>'{freshContinuationCheck,nextCheckAt}')::bigint)
…` statement ran 6 times at about 13.5 s each. That is 81 s of database
time in 160 s, about half of one busy session. All other statements in
the window used less than 35 s together. Staging has 31,399 runs (1.96
GB, wide `context_snapshot`), 75,285 recovery records, and 282 records
with a check deadline.

The local plan shows the cause. For each candidate record:
- `source.id::text = coalesce(evidence->>'runId',
evidence->>'sourceRunId')` cannot use the primary key. PostgreSQL
filters the company's failed runs (6,767 rows removed in each loop, and
6,798 again in the outer semi-join).
- In the recent-interruption check, `max(held.finished_at)` matches runs
of the task with `native_issue_id::text = source_issue_id::text or
issue_id::text = … or context_snapshot->>'issueId' = …`. The casts
cannot use the task-link indexes, so PostgreSQL filters the company's
run history and decodes each run's context JSON (29,937 rows removed in
each loop).

**Proposed behavior**
The queries look up the source run by its primary key and the task's
runs by the existing `(company_id, native_issue_id, …)`, `(company_id,
issue_id, …)` and `(company_id, context_snapshot->>'issueId', …)`
indexes. The results do not change.

**Reason and benefit**
On a staging-shaped local fixture, the deadline query goes from 2,330 ms
to 27 ms (87x), and the backlog listing and count go from 3.7-5.7 s to
19-27 ms. Buffer accesses go from 2.15 M to 28 k.

**Breaking changes**
None. No schema change and no migration.

## What Changed

- `server/src/services/fresh-continuation-fallback.ts`:
- Add `sourceRunKey`. It compares the recorded run id with
`heartbeat_runs.id` through the existing `canonicalExecutionUuid` guard
(cast only when the text matches `^[0-9a-f]{8}-…-[0-9a-f]{12}$`).
`uuid::text` always gives that lowercase form, so this matches exactly
the rows `id::text = text` matched. Uppercase, braced, padded, numeric
and invalid ids still match nothing, and no cast error is possible.
- Use it in `heldExecutionWhere`, `interruptionEligibility` and the
listing's order expression.
- `runOfTask` compares `native_issue_id` and `issue_id` with the issue
id as a uuid. Both callers pass a uuid: the `source_issue_id` column, or
the id of a task row read in the same transaction.
- Export `heldExecutionWhere` and `recentInterruptionSourceSql` so the
regression test can compare them with the previous predicates row by
row.
- Add `server/src/services/fresh-continuation-schedule-queries.test.ts`
(regression test, see Verification).
- Add `scripts/benchmark-fresh-continuation-deadline.ts`. It seeds a
staging-shaped fixture, captures the SQL that the baseline revision and
the working tree send, compares complete positional rows, and reports
timings and plans.

## Verification

- Red/green: `fresh-continuation-schedule-queries.test.ts` "finds the
deadline by run-key and task-link lookups, not a scan of run history per
record" counts the `heartbeat_runs` rows that the deadline query's plan
reads (loops x (rows + rows removed)). With the previous predicates it
fails: 549,166 rows read against a bound of 9,442 (two times the run
count). With this change it passes.
- Equivalence: the same file seeds 900 held records with a seeded random
generator. The records cover these cases: `runId`, `sourceRunId`, both
keys, uppercase, braced, padded, numeric, invalid and unknown ids; a
source run in another company; source statuses (failed, timed_out,
interrupted, cancelled, succeeded, running) with and without
`finished_at`; the assignee and other agents; newer runs linked by
native id, legacy issue id, or context only; unrelated runs;
cancelled-at-dispatch runs; live (queued, running, scheduled_retry)
runs; earlier and later creation times; retries of the source; person
Retry wakes; old and recent finish times; restart, person-owned and
sandbox-resume error codes; and issue statuses and assignees. For every
record it compares `heldExecutionWhere()`, `heldExecutionWhere({
includeActive: true })` and `recentInterruptionSourceSql(now)` with
verbatim copies of the previous predicates. It also compares
`nextFreshContinuationCheckAt` and the full ordered
`heldExecutionsAwaitingFreshContinuation` rows with the previous
queries. The test checks that each predicate is true and false for more
than 5 records, so the comparison is meaningful.
- Focused suites pass locally: `fresh-continuation-fallback.test.ts` and
`fresh-continuation-schedule-queries.test.ts` (82 tests). Related
recovery suites pass: `fresh-continuation-retry-delivery`, `task-retry`,
`fresh-continuation-move-forward-offer`,
`execution-recovery-dead-controller`,
`automatic-run-brake-recovery-passes`, `execution-control-work`,
`recovery-wake-herd` (with the new test: 8 files, 252 tests). `node
server/scripts/typecheck-layers.mjs` passes.
- `pnpm verify` exits 2 (the new benchmark script is a global CI input),
so CI supplies the broad coverage.
- Benchmark: local embedded PostgreSQL 18.1, `jit=on`,
`shared_buffers=128MB`, macOS arm64, a shared machine with other work.
Baseline `67652580ee`. Fixture: 5,669 issues, 31,351 runs with 60 KiB
incompressible context JSON (1,962 MB), 75,285 recovery records (282
with a check deadline, a third overdue). One warm-up and 5 alternating
timed pairs. Both versions return identical positional rows for every
query.

| Query | Baseline median (samples, ms) | This PR median (samples, ms) |
Speedup | Shared buffers |
  | --- | --- | --- | --- | --- |
| `nextFreshContinuationCheckAt` | 2,330 (1,962-2,421) | 27 (21-28) |
87x | 2,149,802 -> 28,227 |
| `heldExecutionsAwaitingFreshContinuation` limit 5 | 3,692
(3,296-4,014) | 19 (18-24) | 194x | 2,831,539 -> 23,970 |
| same, limit 1 (schedule probe) | 3,554 (3,210-4,861) | 20 (18-24) |
179x | 2,831,539 -> 23,970 |
| left-alone count | 5,746 (5,090-5,992) | 27 (25-28) | 213x | 5,435,614
-> 40,953 |

With 1 KiB contexts (rows stored inline) the deadline query goes from
4,096 ms to 19 ms. Staging measured 13.5 s for the deadline query. The
local fixture reproduces the same plan shape, but the staging data
distribution and Neon storage are not reproduced exactly, so the staging
gain is not verified yet.

  ```sh
pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-fresh-continuation-deadline.ts \
--baseline-ref=67652580ee --context-kib=60 --samples=5 --warmups=1
--print-plans
  ```

- Plan after the change: the source run is an `Index Scan using
heartbeat_runs_pkey` (0 rows removed), and each task-run probe is a
`BitmapOr` over the native-id, issue-id and context-issue indexes. The
newer-run `not exists` already used that `BitmapOr` before. It does not
change.

## Risks

- Low risk. The SQL change is limited to how two keys are compared. The
regression test compares the old and new predicates for every record of
a varied fixture.
- `runOfTask` now casts its issue id to uuid. A caller that passes a
non-uuid text would get an error instead of no match. The two callers
pass a uuid column and a task row's id.
- Follow-up, not changed here: the disposition schedule calls
`deadline()`, which returns "retry in 15 s" when the earliest check is
already due. An active hold that the disposition pass does not pick up
(for example, one that records its run only as `sourceRunId`, which
`unresolvedExecutionDispositions` does not read) keeps the deadline
overdue, so the worker runs again every 15 s with nothing to do. A local
probe confirmed this. This explains a fixed cadence of 15 s plus the
query time (about 27 s on staging). After this change each such pass
costs milliseconds, not seconds.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, local PostgreSQL benchmarks and
tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
behavior or command change; no doc update needed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.3
2026-11-13 23:37:19 -06:00
DottaandClaude Opus 5.5 3ae7aba643 perf(db): reuse dedicated connections instead of opening one per call (#21567)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server uses `withDedicatedDbConnection` (packages/db) when an
action must hold one database session outside the main pool: advisory
transaction locks, lock keepalive, and `pg_settings` reads for workspace
lifecycle, finalization, source locks, skill caches and status cards
> - Each call built a new postgres.js client, opened a new physical
connection (TCP, TLS and startup through the pooler), built a new
drizzle client with the full schema, and then ended all of it
> - On staging the dedicated application name shows about 21 connection
setups per second, and the main thread is pinned (event-loop utilization
0.965)
> - This pull request keeps a small set of idle single-connection
clients per owner and reuses them
> - The benefit is that a dedicated action costs about 0.2 ms of
main-thread time instead of 4 to 8 ms, with the same exclusive-session
guarantee

## Linked Issues or Issue Description

**What existing behavior does this improve?**

`withDedicatedDbConnection` opens and closes one new database client and
one new physical connection for every call.

**Subsystem affected**

Database client (`packages/db`) and the server services that take
dedicated advisory-lock sessions.

**Current behavior**

Each call builds a new postgres.js client, opens a new connection (TCP,
TLS and startup through the pooler), and builds a new drizzle client
with the full schema. On staging (Neon statement statistics, 326 s
window) `SET application_name='paperclip-workspace-finalization-lock'`
ran 21.4 times per second. Only this factory uses that application name.
In a 30 s staging CPU profile, drizzle `relations.ts` was 2.2%
inclusive, `n:net` 5.8% and postgres.js construction 0.3%. Part of that
is this per-call setup.

**Proposed behavior**

A caller still gets exclusive use of one session for the whole action.
The session and its drizzle client are reused by later actions, so the
per-call cost is the action's own queries, not a full connection setup.

**Reason and benefit**

The server main thread is saturated on staging (event-loop utilization
0.965). Reuse removes about 5 ms of main-thread work per call.

**Breaking changes**

None. The exported function keeps its signature.
`closeDedicatedDbConnections` and `DEDICATED_CONNECTION_IDLE_MAX` are
new exports.

## What Changed

- `packages/db/src/client.ts`: each `createDb` owner now has one set of
dedicated single-connection clients, shared with every private client
derived from that owner (so nested dedicated actions use the same set).
- An action takes an idle client (last in, first out) or opens a new
one. It never waits. Callers hold advisory locks and nest dedicated
actions, so a waiting queue could deadlock.
- On success the client returns to the idle set if fewer than
`DEDICATED_CONNECTION_IDLE_MAX` (4) are idle. Otherwise it is closed.
- When the action throws, the client is closed and never reused. This
covers a failed or cancelled transaction, a broken socket, and an
interrupted lock wait.
- When an action ends the client itself (the source-lock cancel path
calls `$client.end`), the client is never reused.
- Idle clients keep the owner's idle timeout and max lifetime, so
postgres.js still closes idle physical connections after
`DATABASE_IDLE_TIMEOUT_SECONDS`.
- New `closeDedicatedDbConnections(db, timeoutSeconds)` closes the idle
set and stops retention. Ending the owner `$client` does the same. A
later action still works with a connection that closes on release.
- `server/src/index.ts`: shutdown closes the dedicated set next to the
other private pools.
- `doc/DATABASE.md`: documents the idle cap, the "not an admission
limit" rule, and the transaction-scoped session state contract.
- Session state audit: every current caller uses `pg_advisory_xact_lock`
/ `pg_try_advisory_xact_lock` and `set_config(..., true)`. No caller
uses session advisory locks, `SET` without `LOCAL`, `LISTEN`, or
temporary tables. A reset round trip per call is therefore not added.

## Verification

- New `packages/db/src/dedicated-connection-pool.test.ts` (embedded
PostgreSQL, 10 tests).
- Red on `origin/master`: 200 sequential calls used 200 distinct
sessions (`expected 200 to be 1`). All 10 tests failed.
- Green with this change, 3 runs in a row: 200 sequential calls use 1
session; 7 concurrent callers each hold a different session for the
whole transaction, and afterwards at most 4 stay open and later calls
reuse them; a failed transaction's connection is closed; a terminated
socket during the action is not reused; an idle connection terminated by
the server reconnects; a client the action ended is not reused;
transaction-local settings do not leak; shutdown and owner `$client.end`
close every retained connection; nested actions do not block.
- Focused suites pass: packages/db `work-signals`,
`prepared-queries-postgres`, `response-capture-postgres`,
`checkpoint-pool`, `client`, `client-options`,
`client-teardown-registry`, `postgres-connection-recovery`,
`client-disconnect-replay` (132 tests). Server suites that use dedicated
connections: `skill-package-store`, `status-card-evaluation-work`
(updated: it now proves advisory locks are released, since sessions are
reused), `agent-instruction-working-copies`,
`workspace-recovery-retry-copyback`, `native-execution-lease-batch`,
`company-skills-empty-runtime`, `decisions-service`,
`legacy-controller-lease`, `server-startup-feedback-export` (467 tests).
- `pnpm verify --explain` selects 2779 files for a
`packages/db/src/client.ts` change. Per AGENTS.md I ran the focused
checks above and let CI give the broad coverage.
- Benchmark: 200 calls of `withDedicatedDbConnection(db, d =>
d.transaction(tx => tx.execute(select pg_try_advisory_xact_lock(42))))`
against local embedded PostgreSQL 18, 10 warm-up calls, 5 alternating
before/after process pairs per mode. Main-thread time is
`eventLoopUtilization().active`.

| Mode | Variant | Wall ms/call (median, range) | Main-thread ms/call
(median, range) |
| --- | --- | --- | --- |
| TLS | before | 35.55 (32.93–37.52) | 5.18 (4.68–7.95) |
| TLS | after | 0.19 (0.17–0.21) | 0.18 (0.17–0.20) |
| Plain | before | 18.39 (17.91–20.53) | 3.65 (3.58–4.30) |
| Plain | after | 0.17 (0.13–0.18) | 0.13 (0.12–0.14) |

At the staging rate of about 21 calls per second, this removes roughly
100 ms of main-thread work per second (TLS figures; staging CPUs may be
slower).

## Risks

- A retained connection that the server or network drops while idle:
postgres.js reconnects on the next query when it has seen the close
(tested). If the drop and the reuse race, the first query of that action
fails like a main-pool query would. Before this change, a dedicated
action always started on a fresh connection.
- Through a transaction-mode pooler, `SET application_name=...` is a
server-side parameter switch, so its count may not drop as much as the
client-side connection count. Measure the deployed effect with pooler
client connections and main-thread CPU, not only that statement.
- Concurrency above 4 opens extra connections exactly as before; they
close on release. Total retained connections per owner grow by at most 4
idle clients, which still close after the idle timeout.
- An action that sets session-level state would now leak it to the next
action. The contract is documented on `withDedicatedDbConnection` and in
`doc/DATABASE.md`; no current caller does this.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:37:04 -06:00
DottaandClaude Opus 5.5 c36b96ae00 fix(recovery): hold the start of queued automatic runs while a company's automatic-run brake is on (#21561)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The recovery system starts automatic runs: continuations, retries,
and sweep replacements
> - #21443 added a per-company automatic-run brake. It stops recovery
from creating new automatic runs while a company's runs keep failing
before they start
> - The brake did not hold automatic runs that were already queued. The
dispatcher still started them, so a burst of queued recovery runs kept
failing in the same way
> - This pull request adds the brake's start gate to the queued-run
dispatch pass, and wakes a held company at its next try without a
polling loop
> - The benefit is that a failing company stops spending runs on
automatic work, people's work still starts at once, and held work starts
again on its own when the brake releases

## Linked Issues or Issue Description

Refs #21443
Refs #21433
Refs #21427
Refs #21505

**What happened?**
While the automatic-run brake was open for a company, its already-queued
automatic runs (continuations and retries) still started. Each one
failed before it started, for the same cause that opened the brake.

**Expected behavior**
A queued automatic run must stay queued while the brake is open. Only
the brake's single probe, and then the half-open release rate (2, 4, 8
per window), can start one. A person's work (a Retry, or a continuation
of a person's request) must never be held. Held runs must start again on
their own, with nobody acting.

**Steps to reproduce**
1. Make 20 automatic runs in one company fail before they start, so the
brake opens.
2. Queue automatic continuations for that company.
3. Run a dispatch pass. Before this change, the continuations start.

## What Changed

- `server/src/services/heartbeat/queue.ts`: the dispatch pass asks
`reserveAutomaticRunStart` after the tried/backoff skip and the
claim-attempt budget check, and before the budget increment and the full
row read. A held run spends no claim attempt, does no row read, and does
not enter the declined-claim backoff. If the brake grants the start (its
probe or a half-open release start) but the claim then returns null,
throws, or finds the row gone, the pass gives the start back, so another
automatic run can use it in the same pass. The dispatcher's
person-origin answer is passed as is, and it is `undefined` only when
the dispatcher did not check origin (an agent with one queued run). Then
the brake's own cached person check decides.
- `server/src/services/heartbeat/queued-run-line.ts`:
`queuedRunDispatchFacts` also returns `personOriginChecked`, so the
queue knows when the dispatcher's negative answer is real.
- `server/src/services/heartbeat/dispatch-work.ts`: the queued-run
dispatch deadline worker adds companies whose next try is due to its
scopes, and includes the next try time in its next wake. A held company
wakes for its next try without a polling loop. If the brake's bounded
wake list had to drop a company, the worker wakes all companies at the
earliest dropped time.
- `server/src/services/recovery/automatic-run-brake.ts`: when the brake
holds a run, it records the company's next try time (probe time or the
next release window) in a bounded map. A probe or half-open release
grant returns a release that gives the start back with a compare-and-set
on its own run or window. It exports `reserveAutomaticRunStart`,
`takeDueAutomaticRunWakes`, and `nextAutomaticRunWakeAt`. The brake now
uses `isRecoveryRun`, `isPersonOwnedFailureCode`, and
`isRestartClassFailureCode` from `continuation-cause.ts` directly. The
rules are the same as before; the lazy loader keeps only the imports
that would make an import cycle.
- `server/src/__tests__/automatic-run-start-gate.test.ts` (new, 11 tests
through the real heartbeat queue): bounded queries for 250 held runs
over 100 passes, and for 1,500 held runs (past the 1,024-run cache) over
3 passes; a declined probe claim gives the probe back and another
automatic run starts in the same pass, and the probe stays with that run
on a later pass; an unused half-open start goes back to its window; with
276 held companies (past the 256-company wake list) every company is
still woken; a work run behind 250 held runs starts in the same pass; a
person's Retry is never held; the only queued run is not held when it
continues a person's Retry; exactly one probe starts across two server
instances; the release gives 2, 6, 14 cumulative starts over three
windows; held runs are not in the declined-claim backoff after the
pause; the dispatch worker wakes the company at the next try; and an
end-to-end test that a held automatic run stays queued while the brake
is open, then starts and succeeds on its own at the next try.
- Two small test updates in `automatic-run-brake.test.ts` and
`automatic-run-brake-import.test.ts` for the new cache size field and
the moved rule.

- `server/src/__tests__/automatic-run-brake-recovery-passes.test.ts`:
one test for the sweep path from #21505 (continue a blocked task behind
a run refused at dispatch). The sweep only records the continuation (no
wake, no new run), and `deliverReconciledExecutions`, which the brake
already gates, creates the run only after the pause. No second gate is
needed on that path.

## Verification

- `npx vitest run src/__tests__/automatic-run-start-gate.test.ts`: 11
passed.
- Red before for the review fixes: on the first PR head, the 1,500-run
cost test made 3,848 queries (bound 130), the declined-probe test
started 0 runs (expected 1), and the 276-company wake test missed 20
companies.
- Red before: with the `queue.ts` gate line removed, 5 of the 8 tests
fail (bounded pass, single probe, release rate, backoff, end-to-end).
- Red before for the person case: with the dispatcher's `false` passed
through as final, "never holds the only queued run when it continues a
person's Retry" fails (0 started). It passes after the change.
- `automatic-run-brake.test.ts` 19 passed,
`automatic-run-brake-import.test.ts` 1 passed,
`automatic-run-brake-recovery-passes.test.ts` 28 passed,
`failed-continuation-continues-again.test.ts` 32 passed (after merging
master with #21505).
- Nearby queue tests (on the final head): `heartbeat-run-dispatch-work`
15, `queued-run-dispatch-stacking` 6,
`heartbeat-queued-run-claim-isolation` 10,
`assigned-work-stalled-queued-run` 12, all passed.
- `server` `tsc --noEmit` is clean.

## Risks

- Behavior change only while a company's brake is open or half-open.
When the brake is closed, the gate returns from its cache with no query.
- The first time the gate sees a non-person automatic run while the
brake is on, the brake does one person-origin query for that run. The
result is cached per run (bounded to 1,024 runs).
- No migration. No API or UI change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:33:53 -06:00
DottaandClaude Opus 5.5 84cb28f9b7 fix(connections): keep machine sign-in secrets and settings in one locked commit (#21566)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21473 added machine sign-in (MCP OAuth client credentials) for
organization connections
> - Auto-merge squashed #21473 from an earlier head, so its last review
fixes did not reach master
> - Those fixes close a P1 race and two smaller setup gaps that Greptile
found on #21473
> - This pull request applies that last commit to current master, with
its tests
> - The benefit is that a client ID and its secret always stay together,
and key switching and app selection work as intended

## Linked Issues or Issue Description

Refs #21473

**What existing behavior does this improve?**
Machine sign-in setup and rotation, added in #21473.

**Problem**
- A connection test wrote a replacement client secret before checking
whether the connection changed. If the save was then refused, the secret
stayed and could pair with another client ID, so the next token request
failed.
- Switching a client-secret connection to a Paperclip key always created
an RS256 key, even when the sign-in service accepts only ES256.
- Machine sign-in dropped the setup's app ID and created a separate app.

**Proposed improvement**
Write the secret and the settings in one locked transaction after the
change check. Record the accepted signing algorithms and use them. Pass
the app ID through.

## What Changed

- `client-credentials-service.ts`: `saveConnection` locks the row,
checks that the sign-in method and credential references are unchanged
and that the connection is not removed, then writes any new credential
and the settings in the same transaction. Test, secret rotation and key
staging all use it.
- Setup records `tokenEndpointAuthSigningAlgValuesSupported`. Switching
from a secret to a key uses `chooseAssertionAlgorithm` on it.
- UI: `MachineSignInSetup` takes `applicationId`, and the custom-server
flow passes `prefill.applicationId`.
- Tests: two new server tests and one UI expectation.

## Verification

- `server/src/__tests__/tool-access-client-credentials.test.ts`: 25/25
pass. The 2 new tests fail on master's service code ("never leaves a
refused test's secret behind with another client ID", "creates the kind
of key the sign-in service accepts when switching from a secret").
- `MachineSignInSetup.test.tsx` and
`GenericMcpConnectStep.machine.test.tsx`: 8/8. The app ID expectation
fails on master's component.
- Server `tsc --noEmit` clean. UI `tsc` clean except the pre-existing
generated Storybook catalog imports.

## Risks

Low. The change is limited to machine sign-in. A concurrent change now
returns 409 `client_credentials_changed` instead of saving partly. No
migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 05:30:14 +00:00
DottaandClaude Opus 5.5 bbed32450c perf(load-model): model recorded native run rates, the recorded event mix and coalesced delta batches (#21549)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `scripts/benchmark-agent-scaling-load.ts` (#21418) estimates how
many concurrent agents one server process can carry
> - It assumed 20 text deltas/s and 1 other event/s per agent, and every
other event was an `item.completed`
> - Recorded hosted native runs show about 0.2 deltas/s and 0.5 other
events/s, with a mix in which 39% of the other events are server trace
spans that take a cheaper path
> - This pull request makes the recorded rates and mix the model's
default, keeps the old rates as an explicit stress profile, adds a burst
profile, and mirrors the delta batching of #21520
> - The benefit is a model that measures what production does, so the
next optimizations target the real per-agent cost

## Linked Issues or Issue Description

Refs #21418 (load model and plan). Refs #21520 (delta batching that the
model now mirrors).

**What existing behavior does this improve?**
The load model's default rates, its event mix and its delta batching
rule, and the plan document's measurements.

**Subsystem affected**
`scripts/benchmark-agent-scaling-load.ts`, its test
`server/src/__tests__/agent-scaling-load-model.test.ts`, and
`doc/plans/2026-11-13-thousand-concurrent-agents.md`.

**Current behavior**
The model defaults to 20 deltas/s and 1 other event/s per agent. Every
other event is a native `item.completed` with a chat-progress wake and a
checkpoint save. It cuts a delta batch on the timer even while the
previous batch commits.

**Proposed behavior**
- `PROFILE=recorded` (default): 0.2 deltas/s and 0.5 other events/s per
agent. `PROFILE=burst`: 2.5 and 3, a run's busiest minute.
`PROFILE=stress`: the earlier 20 and 1. API requests stay at 0.3/s. Each
rate can still be set on its own.
- `EVENT_MIX=recorded` (default) sends other events in the recorded
proportions. A `run.performance.span` is an ordinary append with no wake
and no checkpoint, as the server's run trace writes it.
`tool.execution.*` and `item.completed` are native appends with a
chat-progress wake, then a checkpoint save. `EVENT_MIX=<type>` sends one
type only.
- Delta batching follows #21520: deltas that arrive while a batch
commits all join one next batch.
- The report adds `profile` and `eventsIssuedByType`.

**Reason and benefit**
At the recorded rates the per-agent cost is the non-delta runner events
and their checkpoint saves, not deltas. The plan and the next
optimizations need that picture.

**Breaking changes**
None for the product. Anyone who runs the model with no options now gets
the recorded profile; `PROFILE=stress` gives the earlier load.

## What Changed

- Profiles, the recorded event mix (a 100-slot cycle per agent, offset
by agent so the fleet sends the mix at every moment), the span append
path, and the coalesced delta batches.
- The model test now checks the profile in the report and that both
server trace spans and runner events with a progress wake are in the
mix, for the recorded profile and for the stress profile with one
transaction per delta.
- The plan has a new section "Load model at recorded rates": the
recorded source (40 recent native runs on one hosted workspace, as
aggregates), the profiles, the mix and its server paths, fresh runs at
500, 1,000 and 2,000 agents plus burst and stress, and what they show.
It records the 3-connection checkpoint pool as a computed per-process
limit on a hosted database, and that SQL building in Drizzle costs about
as much main-thread time as result parsing on the hosted instance. A new
section "Considered and not done" records why `run.performance.span`
rows stay (about 3% of the statements per agent, with readers) and why
checkpoint saves per critical event only are not done now.

## Verification

- `server/src/__tests__/agent-scaling-load-model.test.ts` passes (2
tests). The new assertions fail against the previous script: it reports
no `profile` and no `eventsIssuedByType`.
- `server/src/__tests__/agent-scaling-load-model.test.ts` also checks
each profile's rates with no overrides, and that a rate set on its own
wins.
- Load model runs, recorded in the plan (60 s, main pool of 10, embedded
PostgreSQL on the same machine, each run started below a 1-minute load
average of 15): recorded rates at 1,000 agents (event-loop utilization
0.61) and 2,000 (0.85) finish every operation with no failure; 500
agents all in their busiest minute run at 0.95 with p99 lateness under
0.4 s. About 4.4 statements per agent per second at recorded rates, 35%
of them on the checkpoint pool.
- Earlier runs at load averages of 20–55 had shown one process falling
behind at 500 burst agents and at 2,000 recorded; the quieter runs after
the model fixes (event order, spans off the runner stream) do not, and
the plan says so. The checkpoint pool as a per-process limit is now
stated as a computed limit for a hosted round trip (about 400 agents in
a busy minute at 1 ms), not as measured.
- Greptile findings fixed: a waiting delta batch no longer takes deltas
sent after the next runner event, takes every delta queued when it
starts, and spans no longer join the runner event stream.

## Risks

- Low: a benchmark script, its test and a plan document. No product code
changes.
- The machine that measured the plan's table was shared; the plan marks
the rows that the machine distorted.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:13:07 -06:00
DottaandClaude Opus 5.5 6dd0349d52 fix(ui): name a queued turn's hold once, in the turn header (#21564)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task's chat shows its run as a turn: a header line with the run's
state, then the turn body
> - #21405 and #21447 made a queued turn name why it waits: "Waiting for
Codie · …", "Waiting for PAP-5656 to finish", "Waiting to start"
> - On staging, the turn body repeated that label under the header
("Waiting for PAP-5656 to finish" twice). The tail used by other
adapters showed a fixed "Waiting to start..." even while the pill above
said "Working" or named a hold
> - #21447 removed the repeat only for the agent's waiting line
> - This pull request names every queued hold once, in the turn header
> - The benefit is one clear statement of why a task waits, with no
repeat and no contradicting line

## Linked Issues or Issue Description

Refs #21405, #21447

**What happened?**

On staging (revision 6b631dd47), a routine task waiting on the routine's
open execution showed "Waiting for PAP-5656 to finish · 31s" in the turn
header and "Waiting for PAP-5656 to finish" again in the turn body.
Other server holds (maintenance, agent setup, server restart, previous
run) and "Waiting to start" had the same repeat. For adapters other than
the Paperclip runner, the tail under the live run pill always said
"Waiting to start..." for a queued run, even while the pill said
"Working" (the 30-second startup feedback) or named a hold.

**Expected behavior**

A queued turn names its hold once, in the header. The turn body adds no
second line for it.

**Why does it matter?**

A repeated or contradicting line makes the waiting state harder to read.
DESIGN.md asks for one name per state.

## What Changed

- `TaskChatRunnerTurn`: a queued turn that has a run and a hold label
(agent queue, server hold, or "Waiting to start") no longer renders the
current-activity line under the header. Startup preparation before a run
exists ("Taking longer than usual…") keeps its slot, as DESIGN.md
requires.
- `TaskChatThread`: the live tail for other adapters no longer shows the
fixed "Waiting to start..." for a queued run. The live run pill above
names the wait.
- DESIGN.md: a queued turn's hold is named once, in the turn header.
- Storybook: the `WaitingForRoutine` story renders the full turn and
checks that the hold appears once.

## Verification

- Red/green: `TaskChatRunnerTurn.test.tsx`. The server-hold cases
(maintenance, agent setup, setup failed, server restart, previous run,
and the new routine hold) and the "Waiting to start" case now assert
that the label appears exactly once and that the turn body has no
current-activity line. Without the change, the 6 existing cases fail.
With it, all 52 pass. The preparation-label case still passes (its slot
stays).
- `vitest run` on `src/components/task-chat` and
`TaskChatThread.test.tsx`: 1,436 passed. The UI typechecks.
- The updated stories render in headless Chromium; the routine story
shows the hold once.
- After deploy: check on staging.

## Risks

- Low risk. This is a UI-only change. A queued turn with a hold shows
one line fewer. Running turns, terminal turns and pre-run preparation
are unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, headless browser).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:07:08 -06:00
DottaandClaude Opus 5.5 351b69941c perf(checkpoint): write a routine controller checkpoint save in one statement (#21553)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote runner's controller saves a checkpoint after each runner
event it commits, on a 3-connection pool, before it acknowledges the
event
> - Each save is a transaction of 4 statements (5 where commit
notifications are on): BEGIN, the owner lock with the transaction id,
the write, COMMIT
> - What forces the transaction is the live-run recovery wake that every
save registers. Recovery does not react to a routine save
> - This pull request writes a routine save (new or changed event,
command and cursor records) with one statement that checks the owner and
writes, and keeps today's transaction for every other save
> - The benefit is one statement instead of four for most checkpoint
saves, on the pool that limits one process in a busy minute

## Linked Issues or Issue Description

Refs #21528 (owner lock returns the transaction id). Refs #21549 (load
model at recorded rates; the plan records the checkpoint pool as a
per-process limit).

**What existing behavior does this improve?**
`nativeRunnerCheckpointWriter` in
`server/src/services/native-runtime/native-runner-checkpoint.ts`.

**Subsystem affected**
Controller checkpoint persistence for remote native runs.

**Current behavior**
Every save locks the coordinator row (`native_run_finalizations` `for
update`), checks the owner, calls `notifyLiveRunRecovery` (6 recovery
queues), and upserts the changed parts, in one transaction.

**Proposed behavior**
A routine save is one statement: a CTE locks the coordinator row only
while the lease owner, boot id, generation and an unexpired lease
(`clock_timestamp()`) all match, and the upsert writes only through that
row. It registers no recovery wake. If the owner does not match, it
writes nothing and the save takes today's full path, which runs the
complete owner check (successor rules included) and fails with
`native_runner_checkpoint_owner_lost` when it should.

A save is routine when it has changed parts, no reset, no removed parts,
no header change, no predecessor, and the run is its own authority.
Every other save (the first save of a controller, which is a reset, a
save after a failed save, header or retirement changes, removals,
source-run consumers) is unchanged.

**Reason and benefit**
At recorded rates about a third of one process's statements are
checkpoint saves. In a busy minute the 3-connection pool cannot keep up
at 500 agents (#21549).

**Breaking changes**
None. No schema change.

## What Changed

- `ROUTINE_CHECKPOINT_SAVE_SQL` and the routine branch in
`nativeRunnerCheckpointWriter`.

## Why a routine save needs no recovery wake

I read the six consumers of `notifyLiveRunRecovery`:
- `nativeProcessSessions`, `nativeQuestionCancellations`,
`activeExecutionRecovery`, `settledNativeProcesses`
(`live-recovery-work.ts`): each selects candidates from `heartbeat_runs`
status, runtime mode, `native_run_finalizations` phase, lease and retry
times, the context snapshot and process identity. None reads checkpoint
rows to decide eligibility.
- `strandedAgentStatus` (same file, `agents` rows) and
`activeRunSilence` (`active-run-watchdog/adapters/deadline-work.ts`,
`heartbeat_runs.last_output_at`): neither reads checkpoint rows.
- The checkpoint rows matter to recovery in two places:
`listExpiredNativeRunnerAuthorities` requires that a checkpoint exists,
and the remote bootstrap proof reads its parts. A checkpoint first
appears with a reset save, which keeps its wake. A routine save only
adds event and command parts, which can only take a run further from the
bootstrap proof; recovery reads the current rows when its own deadline
(lease expiry) fires.
- The intent of the wake also fenced a recovery pass against a save in
flight. The routine save is one atomic statement and holds the
coordinator row lock while it writes, so a recovery claim that takes
that row waits for it, and a reader sees the save whole or not at all.

## Verification

- Red/green on embedded PostgreSQL
(`native-runner-checkpoint-save.test.ts`); the new tests failed against
`origin/master` first:
- A routine save is one statement (`with owner as ... for update ...
insert into native_runner_checkpoints`) and registers no recovery wake
(before: 4 statements and a wake).
- A reset save keeps BEGIN, the owner lock with the transaction id, and
COMMIT, and still registers its recovery wake.
- A routine save that waits while another connection takes over the
lease (new owner and generation) falls back to the full path and fails
with `native_runner_checkpoint_owner_lost`; the checkpoint is unchanged.
- The same race where the new owner also resets the checkpoint (deletes
every part and writes its own header): the save fails the same way and
the checkpoint holds only the new owner's header.
- A lease that expires while the routine save waits for the coordinator
row lock (the holder does not change the row) writes nothing: the
statement judges the lease after the lock, as
`lockNativeRunnerCheckpointOwner` does. My first version checked the
lease in the locking CTE itself, and PostgreSQL does not re-check a
condition after a lock wait on an unchanged row; the existing
`native-runner-restart-recovery.integration.test.ts` test "fences
checkpoint delivery when ownership expires during an unchanged row-lock
wait" failed on it, and so does the new test. Both pass now.
- The existing test "refuses a save after the owner's lease expired and
writes nothing" passes through the new fallback.
- Mutation checks: with the owner conditions removed from the statement,
both race tests and the expired-lease test fail; with the lease check
moved back into the locking CTE, the lock-wait expiry test fails.
- Other focused suites, one file at a time:
`native-runner-restart-recovery.integration` (231 passed, 14 skipped),
`native-runner-restart-checkpoint-cost.integration` (1; it greps for
`insert into "native_runner_checkpoints"`, so the statement quotes the
table name), `native-recovered-owner-followup` (97),
`native-controller-recovery.integration` (30),
`native-runner-checkpoint-authority.integration` (1),
`native-restart-recovery-isolation` (1), the port suite with conformance
(105), `heartbeat-native-restart-suspension` (115), `live-recovery-work`
(51 tests pass; on this busy machine its teardown hook timed out, as it
has before on master). `pnpm --filter @paperclipai/server run typecheck`
passes.
- Statements per routine save: 4 → 1 (5 → 1 where commit notifications
are on). I did not rerun the load model for this change: the machine is
too busy for a fair timing comparison today, and the statement count is
exact.

- CPU per operation (a local script, not committed: 600 routine saves of
one event part and the cursor for one run on embedded PostgreSQL, one
connection, three rounds, `origin/master` code against this branch):
main-thread CPU per save 0.38–0.56 ms before and 0.10–0.12 ms after
(encryption included in both); wall time per save 0.45–0.73 ms before
and 0.22–0.30 ms after.

## Risks

- A routine save no longer wakes the six recovery queues. Their own
deadlines and the 30 s reconstruction still run; see the reading above
for why no decision depends on a routine save.
- The owner check of the routine statement accepts only the exact active
owner. Any other state goes to the unchanged full path, so a successor
or a finished result keeps today's rules.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:58:06 -06:00
DottaandClaude Opus 5.5 67652580ee perf(tool-access): index enabled policies so each decision checks only candidates (#21563)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool access policies decide whether an agent can call each tool, and
an MCP tools/list response decides access once for each listed tool
> - Each decision matched the raw selector of every enabled policy two
times, so one listing cost tools x policies selector checks on the
server's main thread
> - On staging, a company with 363 enabled policies kept the main thread
at 95-98% of a core: policy matching was 22% of CPU time, with single 80
ms runs inside the selector match
> - This pull request compiles the enabled policies once for each
decision cache and indexes them by the selector key that rejects most
(catalog entry, then tool name)
> - The benefit is that each decision checks only the policies that can
match its tool, with the same decisions, so a listing no longer blocks
the event loop

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Tool access decisions for MCP tool listings
(`toolAccessPolicyService.decide`, called by `decideToolsForListing` in
the tool gateway).

**Subsystem affected**
Server: tool access policy evaluation
(`server/src/services/tool-access-policy.ts`).

**Current behavior**
Each decision loops over all enabled policies two times. The first loop
matches the selector and schema-checks the conditions of every
conditioned policy. The second loop evaluates conditions and matches the
selector of every policy. The selector match reads and allocates every
selector key in a fixed order, so a catalog entry mismatch is found only
at the end. A listing of 800 tools with 365 policies does 305,600
selector evaluations. On staging this was 22% of main-thread time
(event-loop utilization 0.965, delay p99 144 ms, max 710 ms).

**Proposed behavior**
The decision cache holds the enabled policies compiled once: parsed
selectors, an index by catalog entry and tool name, and a memoized
conditions schema check. Each decision evaluates only the candidate
policies, in the original priority and creation order.

**Reason and benefit**
Listing cost becomes proportional to the policies that can match each
tool. In the benchmark, one listing goes from about 164 ms to 12 ms of
main-thread time, and the longest event-loop block goes from 84 ms to 4
ms.

**Breaking changes**
None. The policy data model, API, and listing contract do not change.

## What Changed

- Add `server/src/services/tool-access/policy-selector-index.ts`. It
compiles a policy selector into a matcher with the same semantics as the
raw `selectorMatches`, and it indexes policies by `catalogEntryId`, then
by `toolName` (matched against the tool name and the upstream tool name)
for policies without a catalog entry selector. Policies with neither key
are candidates for every call. Candidates keep their original order.
- `decideInternal` caches the compiled index under the same
`enabled-policies:${companyId}` key. The new `matchEnabledToolPolicies`
keeps the two-step semantics: the first selected policy with unsupported
conditions denies the call before any match applies, then the selected
policies with matching conditions go to the unchanged decision loop
(block, rate limit, trust rule, require approval, allow, explanations).
- Decisions without a cache, and decisions that consume a rate limit
(which read fresh rows), compile the rows they read in the same way.
- The conditions schema check runs once per policy and cache, not once
per decision.
- Conditions are now evaluated only for policies whose selector matches.
Before, a malformed legacy condition on an unrelated policy could throw
in every decision. A selected policy with such conditions still gets the
unsupported-conditions deny first.
- Add `server/scripts/benchmark-tool-policy-matching.ts` and a section
in `doc/mcp-discovery-performance.md`.

## Verification

- Red/green regression:
`server/src/__tests__/tool-access-policy-index.test.ts` "reads policy
selectors once per decision batch, however many tools it decides". It
decides 40 tools with one cache against a staging-shaped set of 365
policies and counts reads of the raw policy selectors. On
`origin/master` it fails: 308,110 reads for 40 tools against 7,711 for
the first tool. With this change the count stays at the first-tool
count. The test also checks that cached and uncached decisions are
equal.
- Equivalence: the same file compares the compiled selector with the raw
`selectorMatches` for 404 selectors x 300 contexts (including empty,
whitespace, non-string, string-plural, duplicate and contradictory
values). It compares the indexed policy selection with the old unindexed
two-step selection (copied into the test) for 5 random sets of 400
policies x 400 contexts, and for the staging-shaped set over 800 tools.
The sets include catalog-entry-only, catalog-entry+connection,
multi-key, tool-name, unindexed, conditioned, unsupported-condition,
block, rate_limit, trust_rule, allow and legacy policy types with mixed
priorities.
- Index edge cases: an earlier tool-name policy (matched through the
upstream name), an unindexed policy and an empty catalog-entry selector
apply before a catalog-entry block; an unsupported-condition policy in
another bucket still denies. A DB-backed test checks the same through
`decide`, with and without a cache.
- Focused suites pass locally: `tool-access-policy-index`,
`tool-access-policy-service`, `tool-gateway-listing-memory`,
`connector-tool-policy`, `tool-gateway`, `tool-gateway-service` (6
files, 406 tests). `node server/scripts/typecheck-layers.mjs` passes.
- `pnpm verify` exits 2 (the new benchmark script is a broad input), so
CI supplies the broad coverage.
- Benchmark, local embedded PostgreSQL, macOS arm64, Node 24. One
listing of 800 tools, 365 enabled policies (staging shape plus the 2
fixture policies), 16 concurrent decisions with one cache as in the
gateway. Three runs of 9 alternating rounds after 2 warm-up rounds. Run
3 uses the event-loop delay histogram that also covers the end of the
listing. Both versions return identical decisions.

| Per listing | `origin/master` median (min-max) | This PR median
(min-max) |
  | --- | --- | --- |
  | Elapsed, run 1 | 157.5 ms (146.1-192.8) | 11.4 ms (10.3-15.3) |
  | Elapsed, run 2 | 165.3 ms (161.7-185.4) | 11.9 ms (10.1-14.0) |
  | Process CPU, run 1 | 160.5 ms (149.7-188.9) | 13.7 ms (11.1-21.0) |
  | Process CPU, run 2 | 162.8 ms (157.2-192.7) | 12.3 ms (9.6-20.5) |
  | Elapsed, run 3 | 163.9 ms (151.6-185.3) | 12.4 ms (10.4-18.5) |
  | Process CPU, run 3 | 165.4 ms (155.7-214.9) | 14.5 ms (10.6-20.2) |
| Longest event-loop block, run 3 | 84.1 ms (76.9-95.1) | 3.7 ms
(3.0-5.0) |
  | Selector evaluations | 305,600 | 1,164 |
  | Raw selector property reads | 6,155,830 | 9,490 (compile only) |

  ```sh
cd server && pnpm exec tsx scripts/benchmark-tool-policy-matching.ts
--baseline-ref origin/master --tools 800 --rounds 9
  ```

- Not verified: the hosted staging profile after deploy.

## Risks

- Low risk. The decision loop is unchanged; only the set of policies
that reach it is narrowed, and every left-out policy has a selector that
rejects the call. The equivalence tests compare against the old
selection.
- The index lives only in the per-listing decision cache (or for one
uncached decision). There is no cross-request cache, so policy edits
apply on the next listing as before.
- The cached index memoizes the conditions schema check on shared
entries. The check is pure, so concurrent decisions in one listing get
the same result.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test and benchmark runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:51:43 -06:00
Nicky LeachandClaude Fable 5.2 82ce8e64ba feat(runner): let the session driver admit a turn that the provider started (#21453)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner part of this work is complete in #21316 and the pull
requests below it: when a controller sends `providerStartedTurns` on
`turn.start`, the Runner keeps Claude while a background shell is
running, and gives the cycle of Claude to the controller as a turn that
the provider started.
> - Core reads Runner events through the Codex app-server driver, also
for ACPX agents. The driver refuses a turn that the controller did not
start, except a goal turn of Codex.
> - This pull request lets the driver accept a turn that the provider
started, only directly after a turn that the Runner marked as waiting.
> - The benefit is that the next parts can use that turn. Nothing
reaches this code until the last part sends the opt-in.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one of four parts on the Core side. They are
stacked on #21316:

1. The driver admits a turn that the provider started.
2. The consumer waits for that turn, for a limited time.
3. The server ends the wait when a person comments, and keeps the clock
across a restart.
4. The transport sends the opt-in. Only this part changes what a
production run does.

This is part 1. Its base branch is
`feat/adopt-claude-provider-started-turn`. Refs #19912 (design).

**What existing behavior does this improve?**

Nothing in production yet. What the driver does with a turn that the
provider started.

**Subsystem affected**

Paperclip Runner package, the controller side: `src/drivers/codex` (the
driver that Core uses to read a native session).

**Current behavior**

- A `turn/started` that the controller did not ask for fails the session
with `turn_binding_mismatch`, except for an active Codex goal.
- An interrupt with no active turn fails with "already terminal".

**Proposed behavior**

- A turn terminal that the Runner marked with `awaitingProviderTurn` and
that has no result puts the driver in a waiting state. The driver event
`turn.completed` carries the marker.
- In that state, a `turn/started` with `origin: "provider"` is admitted
the same way as a goal turn, and the driver event carries `origin:
"provider"`. In each other state it still fails closed.
- In that state with no active turn, an interrupt gives an
acknowledgement item with the status `provider_turn_wait_ended`. It
sends nothing to the Runner. The driver also gives that acknowledgement
when it gets the internal `paperclip/runTerminal` notification with a
private marker; a notification without the marker does nothing.
- An interrupt for a provider-started turn that is running is sent to
the Runner only when it names that turn.
- The driver saves the waiting state in the persisted session
(`awaitingProviderTurn`, written only during a wait). After a controller
restart, the restored session still admits the provider-started turn and
still acknowledges an interrupt during the wait. The driver restores the
state only when the session has no active turn, no stored result and at
least one finished turn, so a new run cannot take the wait of the run
before it.
- When a restart finds that the saved active turn is finished, the turn
read of the transport and `reconcile()` of the driver carry the same
marker as the live `turn/completed` notification.
`runnerd-codex-transport.ts` copies `awaitingProviderTurn: true` from
the saved `turn.completed` event to the turn object of a read.
- The driver ends the waiting state when the Runner reports the run
terminal.

**Reason and benefit**

The consumer in the next part needs these three signals: a marked
terminal, an admitted turn, and a clear end of the wait.

## What Changed

- `codex-session-terminal.ts`, `codex-session-notifications.ts`,
`codex-session-state.ts`, `codex-harness-session.ts`,
`codex-driver-values.ts`: 37 lines added, 4 removed.
- `codex-app-server-driver.provider-turn.test.ts` (new, 232 lines).
- `contracts/harness-driver.ts`, `contracts/native-session-backend.ts`,
`backends/harness-driver-backend.ts`, `codex-app-server-driver-impl.ts`:
the optional field in the two persisted shapes and its copy in each
direction.
- `live/runnerd-codex-transport.ts`: the marker on a recovered turn
terminal. The loop that builds the earlier terminals of a `thread/read`
is now the function `runnerdPriorTurnTerminals`, with the same logic, so
that a test can call it.

## State saved by the previous release

- One new optional field in the persisted driver session:
`awaitingProviderTurn`. A snapshot outside a wait has the shape of the
previous release.
- **State from the previous release, read by this release:** the field
is absent, so the driver does not wait. This is the behavior of today.
- **State from this release, read by the previous release:** the
previous reader copies the fields that it knows and ignores this one. No
reader rejects it, and the integrity digest covers the stored object as
written.

## Verification

At the head of this branch:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json` and `pnpm run build:typescript` in
`packages/paperclip-runner`; `node
scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in
`server`; `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed.
- 6 driver tests: a provider-started turn after a marked terminal is
admitted and can report a result; a provider-started turn with no marked
terminal fails closed; an interrupt during the wait is acknowledged; a
running provider-started turn is not interrupted by the end of the wait;
the marked notification ends the wait and a notification with no marker
does not; a normal terminal is not changed.
- Restart tests: a session restored from a snapshot with the waiting
state admits the provider-started turn and settles its result; a
snapshot outside a wait has no new field; a restored session with no
waiting state refuses a provider-started turn as today; an interrupt
during a restored wait is acknowledged. A saved active turn that the
read reports finished with the marker gives a `turn.completed` with
`awaitingProviderTurn: true`; with no marker it is an ordinary terminal.
- `harness-driver-backend.test.ts`: the field goes through a serialized
checkpoint only during a wait. `runnerd-codex-transport.test.ts` (324
passed): the marker is on a recovered terminal only for a
`turn.completed` with a boolean `true`.
- `src/native-session-runtime.test.ts`: 297 passed, with the real
`paperclip-runnerd`.
- Mutation checks for the restart work, each made a named test fail: no
restore of the state; no copy in the backend; no copy in the transport;
no marker argument in `reconcile()`; no end of the wait at the run
terminal.
- Mutation checks: the acknowledgement branch removed, the admission
removed, and the marker check removed each made a test fail.

Not run: the full suite. CI supplies that coverage.

## Risks

- **No production effect.** The Runner sends the marker only after the
opt-in of part 4.
- **The admission rule is shared with Codex goal turns.** The existing
goal tests pass with no change.
- **Known limit after a controller restart.** If the Runner already runs
a provider-started turn and the saved active turn is the turn before it,
recovery fails closed, as today for each unexpected active turn.
- **A controller turn that starts during a wait** clears the waiting
state when its `turn/started` arrives, not when it is sent. A snapshot
in that short time can hold an active turn and the field together; the
restore rule drops the field.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 20:45:49 -08:00
Dotta ca7d42ba71 fix(heartbeat): refresh queued dispatch when task drains end (#21562)
## Thinking Path

> - Paperclip runs agent work for a company.
> - A task drain holds new run starts while maintenance or idle handoff
is in progress.
> - A queued dispatcher can keep an unreconciled request and a 30-second
retry while the drain is active.
> - Ending the drain can resume domain work without refreshing that
dispatcher's saved deadline.
> - This change refreshes the pending queued owner when a drain ends or
its expiry is observed.
> - Normal admission checks and real retry deadlines still apply.

## Linked Issues or Issue Description

Related: #20640. That change refreshes deferred receipt workers. This
change refreshes the separate queued-run dispatcher.

**What happened?**

A queued-work notification received during a drain could leave a
30-second fallback in the dispatcher after the drain ended. The owner
stayed unreconciled after useful work had settled. This delayed its next
pass and the instance's idle state.

**Expected behavior**

Refresh an unreconciled queued owner when normal dispatch becomes
allowed. Keep other drain holds, admission rules and scheduled retry
deadlines.

**Steps to reproduce**

1. Let the normal dispatcher complete its first pass.
2. Start an idle task drain and commit a company-scoped queued-work
notification.
3. Stop the drain, or observe its expiry, before the saved fallback is
due.
4. Check that the scoped queued pass runs before that fallback. The two
regression cases fail on the unchanged owner and pass with the fix.

**Paperclip version or commit**

The main-specific patch is based on
`f7b1651799d8c0c660a76266878c420a12f0beca`. The bug was found in
assembled sleep acceptance. The affected queued-dispatch, drain and
notification modules match main.

## What Changed

- Listen for normal drain changes and observed expiry in the queued-run
owner.
- Wake that owner only when dispatch is allowed and its reconciliation
is pending.
- Forward the deadline worker's optional stop callback after its stop
fence.
- Remove both subscriptions when the owner or coordinator stops.
- Add release, expiry, remaining-gate, shutdown, retry-deadline and idle
controls.

## Verification

Local checks used the integrated feature branch that exposed the issue.
The main-specific port preserves main's existing `canRun` logic and
excludes the private branch's extra cancellation hook.

- Both new release and expiry cases failed at the intended scoped-resume
assertion before the product change.
- Six dispatch cases pass with the normal package import.
- Fourteen existing deadline controls and nineteen queue-order controls
pass. Their inputs did not change during the test-import correction.
- The complete normal Server typecheck and its three affected-file
program checks pass.
- The first typecheck caught a test import outside the Server root. The
import now uses the existing public package wildcard. The corrected
six-case suite and normal typecheck pass.
- Normal Source SDK input verification passes.
- Exact-head CI at `fd623d3ba9345e5ec8f252c55932ff797cd57e8f` completed:
54 passing checks, four expected skips and one neutral result; Snyk
passed. Greptile reviewed this exact head at 5/5 with no actionable
findings. All review threads were read; none remain open. A full local
repository build and suite were not run in this checkout.

The existing admission-drain contract in `doc/execution-semantics.md`
already requires prompt admission after drain release. This fix restores
that contract. No configuration or command documentation changes are
needed.

## Risks

Drain events can request an earlier pass for a pending owner. The normal
permission, reconciliation, coalescing and earliest-time checks remain
active. The scheduled retry owner has no new listener. Shutdown must
remove both listeners; the coordinator-stop control covers that path.

This PR does not claim deployed suspension or complete transport sleep
acceptance.

## Model Used

OpenAI `gpt-6.1-sol` assisted with implementation and independent
review. The Codex managing agent performed technical review and
verification. The host does not expose its exact runtime model ID or
context-window size. Tools used include source inspection, code edits
and local test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (not
required: the existing contract remains accurate)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-13 22:45:26 -06:00
DottaandClaude Opus 5.5 9824277f65 perf(run-events): append events whose only wake is chat progress in one statement (#21559)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Most runner events that are not text deltas
(`tool.execution.started/progressed/completed`, `item.completed`) wake
the chat progress projection
> - That wake is registered inside the append's transaction, so each
such event takes 4 statements (BEGIN, the run lock read, the append,
COMMIT), where an event without a wake takes 1 (#21511)
> - The chat progress projection rebuilds itself from the stored events;
the wake only tells it when to look
> - This pull request writes an event whose only wake is chat progress
with the one-statement append, and sends the wake after that statement
commits
> - The benefit is 1 statement instead of 4 for most non-delta runner
events, with response, reassessment, retention and recovery wakes
unchanged

## Linked Issues or Issue Description

Refs #21511 (one-statement append). Refs #21528 (lock read returns the
transaction id). Refs #21549 (load model at recorded rates).

**What existing behavior does this improve?**
`appendHeartbeatRunEvent` for native source events whose only wake is
chat progress.

**Subsystem affected**
Server run event ingest (`heartbeat-run-events.ts`), the chat progress
notification helper, and two additive exports in
`packages/db/src/work-signals.ts`.

**Current behavior**
A progress event opens a transaction, reads the run lock with
`runtime_mode`, the bound task and the transaction id, probes for a
replay, registers the chat-progress wake as a tracked intent, appends,
and commits.

**Proposed behavior**
When chat progress is the event's only wake, the append takes the
one-statement path of #21511. That statement also returns the run's
`runtime_mode` and bound task id from the row it locks, and, where
commit notifications are on, calls `pg_notify` inside the same statement
for runs that are native and bound to a task, so other server processes
get the wake at commit. After the statement returns, this process
publishes the wake as settled to its own listeners. A replay, a gap, a
changed run or any other fallback takes the existing transaction, which
registers the wake as before.

**Reason and benefit**
At recorded rates these events are about 61% of the non-delta events
(#21549), so this cuts about 3 statements per such event.

**Breaking changes**
None. No schema change.

## What Changed

- `appendHeartbeatRunEvent`: `progressOnly` (progress set, every other
wake clear) also qualifies for the one-statement path.
- `APPEND_NEW_NATIVE_SOURCE_EVENTS_WITH_PROGRESS_SQL`: the one-statement
append, plus `runtime_mode` and the bound task id (`boundRunIssueIdSql`,
rendered once) from the locked row, plus a conditional `pg_notify`.
- `chat-native-progress-notifications.ts`:
`chatNativeProgressCommitNotification` and
`publishChatNativeProgressSettled`.
- `work-signals.ts` (additive): `databaseWorkCommitNotification(owner,
topic)` returns the cross-process payload when notifications are on;
`publishDatabaseWorkSettled(owner, topic, hint)` publishes a settled
wake to this process's listeners.

## Verification

- Red/green on embedded PostgreSQL. Each new test failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "appends an event whose
only wake is chat progress in one statement and wakes progress after it
commits": 1 statement (before: a transaction), the wake is published
once as settled with no intent, and a replay publishes nothing.
- "keeps the transaction for the last events of a turn and a run, and
one statement for progress-only events":
`tool.execution.started/progressed/completed` and a non-final
`item.completed` take one statement; a final `item.completed`,
`run.result.accepted`, `turn.completed`, `turn.failed`, `turn.cancelled`
and `turn.interrupted` keep the transaction.
- "sends the progress wake to other server processes with the one
statement's commit": with commit notifications on, another connection
that listens on the work channel receives exactly one chat-progress
notification, and a replay sends none.
- `chat-native-progress-work.test.ts` "wakes the projection from a
one-statement progress event after the statement commits": the real chat
progress worker publishes the progress for an event appended in one
statement.
- Mutation check: without the settled publish after the statement, that
test fails.
- Lost-wake repair (condition b): an event row stored without any wake
models a lost hint.
- "repairs a lost progress wake with the run's next progress event":
nothing is published for the stored event; the run's next progress event
publishes the progress.
- "repairs a lost progress wake with the startup reconcile": a worker
that starts after the stored event publishes it.
- The existing response-wake test now uses a final reply and still shows
the intent before the write, and the race test of #21528 uses a final
reply so it still covers the transaction path.
- Other focused suites, one file at a time: the port suite with
conformance (105), `native-observer-replay` (1),
`paperclip-semantic-conformance` (1), `native-interaction-bridge` (7),
`heartbeat-run-event-sequencing` (5), `native-event-boundary-golden`
(1), `heartbeat-native-restart-suspension` (115), `live-recovery-work`
(51), `work-signals` (25), atomic-append (31),
`chat-native-progress-work` (38). `pnpm --filter @paperclipai/server run
typecheck` and `pnpm check:module-boundaries` pass.

- CPU per operation (a local script, not committed: 800
`tool.execution.progressed` appends to one native task run on embedded
PostgreSQL, one connection, three rounds, `origin/master` code against
this branch): main-thread CPU per event 0.28–0.50 ms before and 0.10 ms
after in the warm rounds; wall time per event 0.84–1.04 ms before and
0.52–0.53 ms after.
- Inside a caller's transaction: new test "keeps a progress-only event
inside a caller's transaction on the locked path, and settles its wake
only when that transaction ends". `publishDatabaseWorkSettled` publishes
at once only outside a transaction; inside one it waits for the
outermost commit and drops the wake on rollback (new
`work-signals.test.ts` test, which failed with the unguarded publish).
The one-statement path is only taken when the owner is not a transaction
(`!is(db, PgTransaction)`, #21511), so the two guards agree.

## Risks

- **Lost wake.** If the statement's outcome is unknown (the connection
drops after the server ran it), this process sends no wake for that
event. In a chat channel (Slack, Telegram) the progress line for that
task can then stay one step behind (for example it does not move from
"working" to the latest tool step) until the run's next runner event, a
run milestone, or a server restart. A turn normally ends with an event
that keeps the transaction
(`turn.completed/failed/cancelled/interrupted`), and the run's
completion notifies chat run milestones, which include chat progress
(`notifyChatRunMilestones` in `run-completion.ts` and
`native-run-finalizer.ts`). So progress cannot stay stale after a run
ends; in the worst case it lags one step for the rest of a turn. The
projection rebuilds from stored events, so nothing is lost or
duplicated.
- The chat-progress wake no longer publishes an intent before the write.
Intents let the coordinator fence a pass against an in-flight write; the
settled wake after the commit makes the projection run again after the
row is visible.
- `packages/db/src/work-signals.ts` gets two additive exports and no
change to existing behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:37:35 -06:00
DottaandClaude Opus 5.5 8c0c7c6abb perf(ui): load the @task mention pool after the task page settles; add a cold-load perf spec (#21557)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People often open a task by its URL, for example from a link in a
message or after a reload
> - #20592 made a hard load of a task URL read the chat's first frame
from one `chat-bootstrap` request
> - The task page still sends a large read as it mounts: 100 full
recently updated tasks for the `@task` mention picker. On staging this
read returned 395 KB and took 3.9 s of server time
> - The hosted database is CPU-bound, so reads that do not affect the
first frame must not compete with the reads that do
> - Also, no test measured a cold direct load under realistic latency or
guarded the one-bootstrap request shape
> - This pull request starts the mention pool after the task page
settles. It also adds a cold-load perf spec with a delay on each API
request and an always-on request-shape test
> - The benefit is less concurrent server work during a cold task open,
and a regression guard for the one-round-trip first frame

## Linked Issues or Issue Description

Refs #20592 (hard-load chat bootstrap that this pull request measures
and guards). Refs #20355 (the `chat-bootstrap` endpoints).

**What existing behavior does this improve?**
The task page loads the `@task` mention pool when it mounts.

**Subsystem affected**
Board UI task page (`ui/src/pages/IssueDetail.tsx`) and the issue-detail
perf harness (`tests/perf/issue-detail`).

**Current behavior**
`GET /api/companies/:id/issues?limit=100&sortField=updated&sortDir=desc`
starts with the chat's first-frame reads. It starts before the first
chat frame in 14 of 14 local cold loads. The picker needs it only after
the person types `@`.

**Proposed behavior**
The read starts after the page has painted and its own reads have
finished. On a page that keeps reading, it starts after at most 5 s.

**Reason and benefit**
The read never affects the first frame. Deferring it removes a large
concurrent read from the busiest part of a cold open.

**Breaking changes**
None. If the person types `@task` in the first moments after a cold
load, the task suggestions can appear about half a second later.

## What Changed

- `ui/src/hooks/usePageSettled.ts`: new hook. It returns `true` once
`whenPageSettles` runs (the page has painted, the tab is visible, no
query is fetching, and the browser is idle), or after
`PAGE_SETTLED_MAX_WAIT_MS` (5 s). The cap keeps a long-running read from
keeping the picker empty.
- `ui/src/pages/IssueDetail.tsx`: the mention-pool query is enabled only
when `usePageSettled()` is true.
- `ui/src/hooks/usePageSettled.test.tsx`: tests that the hook waits for
in-flight reads and that the cap applies.
- `ui/src/pages/IssueDetail.test.tsx`: mocks the hook to `true`, so
existing tests keep their behavior.
- `tests/perf/issue-detail/cold-bootstrap.perf.spec.ts`: new spec. It
delays every API response (300 ms by default) and opens the task URL in
a new browser context, with the JavaScript in the HTTP cache and no app
storage. The finish event is the first readable frame: the history is
revealed, the latest saved message is visible and the composer is
editable.
- The request-shape test always runs. It requires one `chat-bootstrap`
request and no per-endpoint first-frame read before the frame. Reads
that start after the live socket is created are excluded, because they
are its catch-up refresh.
- The benchmark runs only when `PAPERCLIP_ISSUE_PERF_COLD_RUNS` is set.
It writes the median, the spread, the request count before the frame,
the last requests before the frame, the JavaScript bytes transferred and
a request timeline.
- The delay uses CDP `Fetch` interception instead of Playwright routing,
because routing disables the HTTP cache.
- `tests/perf/issue-detail/README.md`: documents the spec.
- My earlier draft also added a deadline for a stalled bootstrap. Master
already has one (`CHAT_BOOTSTRAP_SHARE_DEADLINE_MS`, 5 s), so this pull
request does not include it.

## Verification

Measurement setup:
- Production UI builds of master `4a78124a3e` and of this branch, served
by one local server (`PAPERCLIP_UI_DEV_MIDDLEWARE=false`, embedded
PostgreSQL).
- A delay of 300 ms on each API request.
- Cold direct task URL with 24 saved comments and a plan.
- Master and the branch alternated: 2 rounds of 7 runs each.
- The host was shared with other agent sessions.

| Build | Runs | First readable frame, median | p25–p75 | Min–max | API
requests before frame (median) | Mention pool started before frame |
|---|---:|---:|---:|---:|---:|---:|
| master | 14 | 1,422 ms | 1,336–1,525 | 1,284–7,398 | 50 | 14 / 14 |
| this branch | 14 | 1,390 ms | 1,338–1,457 | 1,272–2,207 | 49 | 0 / 14
|

- The spec adds the API delay through CDP request interception, so the
HTTP cache stays on. The measured loads downloaded 0 KiB of JavaScript
in every run. The spec records this as `jsTransferBytes`. Each build has
one outlier (7,398 ms on master, 2,207 ms on this branch), from host
contention.
- With this branch, the mention pool starts after the frame in every run
(350–400 ms after it in an earlier set).
- The first-frame timing is neutral locally. The local database answers
the pool read in milliseconds, so it does not compete with other reads.
The intended benefit is lower concurrent server load on a busy database.
This pull request does not show it as a local latency gain.
- On master and on this branch, the hard load adopts the bootstrap and
the frame is no longer held by task reads. The frame now comes right
after the shell reads return: `auth/get-session` and
`instance/settings/experimental`, then `companies` and
`workspace-preferences/me`. Profiling marks confirm this: the chat
thread mounts after `workspace-preferences/me` returns, and all
readiness gates are ready within 30 ms.
- The request-shape test passed 5 of 5 on master and 5 of 5 on this
branch.
- `layout-stability.spec.ts`, `plan-startup.spec.ts` and
`cold-bootstrap.perf.spec.ts` against this branch's build: 18 passed.
- `ui` Vitest for `usePageSettled.test.tsx` and `IssueDetail.test.tsx`:
278 passed. The `ui` typecheck passed. `pnpm check:token-gates` passed.
- `pnpm verify` exits 2 because `tests/perf/issue-detail/README.md` is a
global CI input. Required CI supplies the broad coverage.

To reproduce the measurement:

```sh
PAPERCLIP_ISSUE_PERF_BASE_URL=http://127.0.0.1:3287 PAPERCLIP_ISSUE_PERF_COLD_RUNS=7 PAPERCLIP_ISSUE_PERF_LABEL=branch \
  pnpm exec playwright test --config tests/perf/issue-detail/playwright.config.ts cold-bootstrap -g benchmark
```

## Risks

- Low risk. The only product change is when one read starts.
- If the page keeps a query in flight, the pool waits up to 5 s. During
that time, `@task` suggestions list only agents, projects and members.
- The request-shape test treats reads that start after the live socket
is created as catch-up refreshes. If the socket connects before the task
page mounts, such a test would not see a first-frame read that started
later. The test still requires exactly one bootstrap request before the
frame.
- The perf spec is not in the default CI browser suites. It runs when
the issue-detail harness runs.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file editing, Playwright browser runs). It wrote
the change, the tests and the measurements.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.2
2026-11-13 22:30:03 -06:00
DottaandClaude Opus 5.5 54c3616ad9 fix(recovery): ignore the non-assignee comment wake marker in the failed-preparation proof (#21550)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a Runner run fails in setup before any provider work, a
person's Retry must start a new run
> - Admission proves "no provider work" for a failed preparation by
checking that the run record holds only known, harmless fields
> - The run profile allowlist did not include
`aiConnectionNonAssigneeCommentWake`, a boolean that dispatch writes
when it admits a run from a non-assignee comment
> - So a run with that marker never passed the proof. Its Retry waited
on `previous_execution_unverified` forever, even after saved-work
recovery finished and the workspace was handed on unchanged
> - This pull request accepts the marker (boolean only) in the proof,
and in the related workspace-wait parent proof
> - The benefit is that these tasks get a working Retry, and a Retry
that was saved earlier starts by itself once the proof passes

## Linked Issues or Issue Description

Refs #21481, #21497.

**What happened?**
On a hosted staging instance, a Runner run failed in setup ("RPC call
environmentExecute timed out") while it prepared. Saved-work recovery
finished ("Saved work recovered."). The workspace was handed on
unchanged for this run. A person pressed Retry two times. Both Retries
waited on `previous_execution_unverified`, and no run started.

**Expected behavior**
A run that failed in setup before any provider work
(`executionRecovery.providerWorkStarted: false`, no process, no
execution events, stopped sandbox) gets a working Retry. A saved Retry
starts by itself when its gate clears.

**Steps to reproduce**
1. Let dispatch admit a Runner run from a non-assignee comment, so the
run profile has `aiConnectionNonAssigneeCommentWake: true`.
2. Make the run fail in setup during preparation.
3. Press Retry. See that the request waits on
`previous_execution_unverified`.

**Paperclip version or commit**
Observed on staging at `e82c4f0a31`; still present on `master` at
`6b631dd477`.

## What Changed

- `server/src/services/cancelled-native-startup.ts`: the
failed-preparation proof accepts `aiConnectionNonAssigneeCommentWake`
when its value is a boolean. The workspace-wait parent proof ignores the
same key.
- `server/src/services/legacy-preparation-retry-hold.test.ts`:
staging-shape tests for a recovered setup failure (failed restore, a
workspace that still names its location, the marker, all together), and
a Retry that attaches to an earlier refused request and then starts.

## Verification

- `cd server && npx vitest run
src/services/legacy-preparation-retry-hold.test.ts`: 12 passed. Without
the fix, the `comment_wake_marker`, `all` and `attached_after_refusal`
cases fail (red), and the other cases pass.
- Focused test file only (the machine was loaded). CI supplies
typecheck, build and the full suite.

## Risks

- Low risk. The marker only records why dispatch admitted the run. Every
other proof clause (no process, no execution events, no native result,
stopped leases, settled preparation evidence, workspace repair) is
unchanged. A non-boolean value still fails the proof.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:02:55 -06:00
DottaandClaude Opus 5.5 6b9e45787f fix(recovery): continue a task whose system retry was refused at claim behind a saved-work hold (#21505)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops before it finishes, automatic recovery continues
the task, through the fresh-continuation fallback, in a new run that
starts from the saved work
> - A follow-up run (for example one started by a new comment or by
finished subtasks) can fail before any provider work and leave a
saved-work hold. The system then queues its own retry, and the dispatch
gate cancels that retry at claim with
`execution_reconciliation_required` because the hold is still in place
> - The fallback takes over the hold when that happens, but it continued
only "recent interruptions". That test required the failed run to be a
recovery run itself or to fail with a restart code. A first follow-up
run that failed with `adapter_failed` passed neither test, even though
the system had already decided to retry it. So the fallback answered "a
person must press Retry". The backlog pass used the same test, and the
assigned-work sweep never listed the task. The task stayed blocked with
no run, no pending path and no Retry button
> - This pull request treats a system retry refused at dispatch as proof
that the system continues the task. It also stops cancelled-at-dispatch
attempts from counting against the hourly budget, lets the sweep
continue a blocked task whose latest run was refused at dispatch, and
keeps recovery notices from asking for a Retry that the task does not
offer
> - The benefit is that such a task continues by itself in exactly one
fresh run from the saved work, whether the saved work was recovered
before or after the refusal, and a person's own failed Retry keeps
today's behaviour

## Linked Issues or Issue Description

Refs #21394, #21348, #21363, #21478

**What happened?**
A task's follow-up run, a legacy run that was not a recovery run, failed
before any provider work with `adapter_failed`. Its files could not be
restored, so it left a saved-work hold
(`legacy_execution_requires_reconciliation`, `workspaceRestoreFailure:
restore_failed`). The system queued a retry of that run. The dispatch
gate cancelled the retry at claim with
`execution_reconciliation_required`, and the "older run hold" path
handed the hold to the fresh-continuation fallback
(`run-dispatch/adapters/postgres.ts` `settleOlderRunHold`). The fallback
found the refused run and asked `interruptionEligibility` whether the
interruption was recent. That predicate required
`continuesAutomaticallySql(source)` (a recovery run, or a restart,
deploy or bootstrap code), so it returned `person_owned` /
`retry_needed`. The stranded reconciler then blocked the task with
"Press Retry", but the saved-work hold reports that Retry is not
available. This happened twice in a local restart-chaos wave. In one
case the saved work was recovered two minutes before the claim. In the
other it was recovered thirty seconds after the claim.

**Expected behavior**
A task never rests blocked with no live run, no pending path and no
Retry. While the saved-work recovery runs, that recovery is the pending
path. When it finishes, one continuation starts from the saved work
through the fallback (isolation, the shared hourly cap). A run that was
cancelled at dispatch before it started does not use up the automatic
budget. A person's own failed Retry is still left for that person.

**Steps to reproduce**
1. Fail a follow-up run (wake reason `issue_children_completed`) before
provider work with `adapter_failed` and `workspaceRestoreFailure:
restore_failed`.
2. Queue the system's retry of that run (`retryOfRunId`, wake reason
`issue_continuation_needed`) and claim it. The dispatch gate cancels it
with `execution_reconciliation_required`.
3. Before this change, no continuation is recorded, the stranded
reconciler blocks the task, and nothing ever continues it.

**Paperclip version or commit**
master at 2756be2f32

## What Changed

- `fresh-continuation-fallback.ts`, `interruptionEligibility`: the
interruption counts as one the system continues when the failed run's
cause is system-owned and a later run that the dispatch gate refused
before it started is a system recovery run (`recoveryRunSql`, which
excludes a person's Retry). A failed run that was itself a person's
Retry (`task_retry` or `explicitUserContinuation`) never qualifies this
way. The 24-hour recency bound is unchanged. The claim-gate hand-off,
the backlog pass and its deadline query all use this one predicate.
- `recovery/continuation-ledger.ts`,
`recovery/task-automatic-attempts.ts`: one shared rule,
`startedOrPendingAutomaticRetry`. An automatic retry that the dispatch
gate cancelled before it started (`cancelled`, no `startedAt`,
`execution_reconciliation_required`) does not count toward the hourly
cap in either ledger.
- `recovery/assigned-work-sweep.ts`:
- When the latest run was refused at dispatch, the sweep continues from
the run it was refused for through the fallback. Before, it gave up.
- A blocked task with an active hold and a recent refused-at-dispatch
latest run is now listed as a suspect. The check hands it to the
fallback before it answers `left_to_hold_owner`. A pending recovery
becomes `waiting_on_recovery` with its recheck time.
- Holds that wait for a person's configuration or confirmation are still
excluded.
- Review follow-up (`recovery/task-automatic-attempts.ts`,
`recovery/assigned-work-sweep.ts`):
- The task ledger counts a sweep's fresh-continuation record (`trigger:
assigned_work`) itself. It no longer depends on a companion activity
entry, and an activity entry marked `continuedFromSavedWork` is not
counted a second time. Every sweep continuation counts once, whatever
path recorded it.
- When the sweep claims a continuation, it marks its claim entry
`continuedFromSavedWork` inside the transaction that records the fresh
continuation. A restart between the two writes can no longer leave one
attempt counted twice.
- The sweep's continuation behind a refused run spends the per-minute
repair budget (`ASSIGNED_WORK_REPAIRS_PER_MINUTE`). The excess is
deferred with a `retryAt`.
- `recovery/stranded-notice.ts`, `recovery/service.ts`: when the task's
execution blocker offers no Retry, the escalation notice and the
recovery action's next action leave out every Retry instruction. They
say that Paperclip is holding the task until its saved work is settled.
- No change to the dispatch gate (`heartbeat/queue.ts`, the restore-hold
predicate in `run-dispatch/adapters/postgres.ts`, or
`execution-blocker.ts`). The recorded continuation settles the hold, so
the delivered run is not refused again.

## Verification

Red before the change, green after (embedded Postgres, real services:
`terminalizeLegacyExecution`, the dispatch adapter's
`cancelStaleQueuedRun`, `reconcileStrandedAssignedIssues`,
`deliverReconciledExecutions`, `continueHeldExecutions`,
`recoveryService().assignedWork`):
- `server/src/services/fresh-continuation-fallback.test.ts`
- "continues a follow-up … when the saved work was recovered before the
system retry was refused at claim": red (`continuationDelivery:
"invalidated"`, no continuation). Green: the continuation is recorded at
the claim, and the stranded reconciler does not block the task. The
delivered run is queued and passes the claim gate (`not_stale`).
- "… recovered after the system retry was refused at claim": red (no
pending check). Green: the claim records `exact_recovery_pending` /
`saved_work_recovery_running`. When the recovery completes, the backlog
pass continues the task, and the delivered run passes the claim gate.
- "continues a follow-up … with its files in place, after its system
retry was refused at claim": red, then green. This is the control
without a restore failure.
- "still leaves a person's failed Retry of a follow-up for that person
…": stays `person_owned` / `retry_needed`, before and after.
- "counts only automatic retries that started, or still may, against the
task's hourly budget": new.
- `server/src/__tests__/failed-continuation-continues-again.test.ts`,
"the sweep continues a blocked task whose system retry was refused at
claim behind a recovered saved-work hold": red (the task was not a
suspect). Green: `repaired`, a pending continuation in the hold, the
task back in `todo`, and no wake enqueued. The next check reports a
pending intent.
- Same file, "a continuation the sweep records behind a refused run
counts once toward the task's hourly cap": red (5 counted after 5
attempts plus the new continuation), green (6, and the transient-retry
cap check refuses the next attempt).
- Same file, "continuations behind refused runs spend the sweep's
per-minute repair budget": 20 tasks. Red (20 continued in one pass),
green (10 continue, 10 are deferred with a `retryAt`, and the 10
continue in the next pass).
- Same file, "a sweep continuation interrupted right after it was
recorded still counts once": the continuation commits, then the
post-commit activity write fails (a simulated server stop). Red: 6
counted for 5 real attempts. Green: 5 counted, and the next automatic
retry is still allowed.
- `server/src/services/recovery/stranded-notice.test.ts`, "never tells a
person to press Retry when the task offers no Retry": new.

Local runs on this branch, one file at a time:
- `fresh-continuation-fallback.test.ts`: 77 passed. The suite's
`afterAll` teardown hit the 30 s hook timeout on a heavily loaded
machine.
- `failed-continuation-continues-again.test.ts`: 32 passed
- `assigned-work-invariant.test.ts`: 31 passed
- `stranded-notice.test.ts`: 33 passed
- CI on the first head (ce4431410b): every required check passed
- `server` `tsc --noEmit`: clean
- `pnpm verify --explain` selects most of the repository (2666 files),
so required CI supplies the broad coverage.

## Risks

- More tasks continue automatically. Each continuation goes through the
existing fallback. The 24-hour recency bound, the per-task hourly cap
(6), isolation of the next run, and every person-owned gate
(person-owned failure codes, a person's Stop, review, a person-owned
hold, a person's failed Retry) are unchanged.
- The hourly cap now ignores automatic retries that never started. A
task whose retries are refused at dispatch again and again is still
bounded: the refusal hands the hold to the fallback, and the fallback
records one continuation per source run.
- Known separate gap, not changed here: after an interrupted
preparation's holds are resolved (`settleInterruptedNativeBootstrap`),
`scheduleBoundedRetry` can report `exhausted` or `blocked`, and nothing
follows that up. The assigned-work sweep may now pick such a task up
through the refused-at-dispatch path, but this pull request does not
prove that.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs)

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:01:39 -06:00
Nicky LeachandClaude Fable 5.2 f7b1651799 feat(acpx-sidecar): adopt Claude's own cycle as a provider-started turn when the Runner arms it (#21316)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - The parts before this one give the adapter an ordered stream of
those cycles (#21421), declare the contract (#21422), and make
`paperclip-runnerd` wait for such a turn (#21423).
> - This pull request is the last part in the Runner. The sidecar reads
the stream, and gives a cycle to `paperclip-runnerd` as a
provider-started turn only when `paperclip-runnerd` armed it.
> - The benefit is that a later change in Core can let a run continue
when background work finishes. Until Core sends the opt-in, a production
run does the same as today.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request was first opened with the full change (49 files). It
is now split. The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. This pull request: the sidecar adopts a cycle.

Its base branch is `feat/runnerd-await-provider-turn` (#21423). The code
at the head of this branch is the same as before the split; only the
base changed. Review comments made before the split are about code that
is now in the parts above. Refs #19912 (design).

**What existing behavior does this improve?**

What the ACPX sidecar does with a cycle that Claude starts by itself.

**Subsystem affected**

Paperclip Runner: `src/cli/acpx-runtime-sidecar.ts`. Claude only.

**Current behavior**

- The sidecar does not read the cycle stream of the adapter (#21421), so
each cycle is declined and the background state becomes unknown.
- The sidecar does not know the command `session.provider_turn`, so
`paperclip-runnerd` (#21423) never waits.

**Proposed behavior**

- The command `session.provider_turn` with `armed: true` arms the
sidecar for one turn. The sidecar accepts it only for the terminal that
reported a running background shell, with no active turn and a provider
that is alive. One flag holds the arming. It is cleared at each turn
start and turn end, on suspend and close, and when the provider is lost.
- One loop, `consumeProviderCycles`, reads the stream. For each cycle:
- if the sidecar is armed and no turn is active, the sidecar adopts the
cycle: it emits `runtime.turn_started` with `{ "origin": "provider" }`,
then the events, the usage and the terminal of the cycle, as for a
normal turn;
- if the cycle cannot be decided yet (the prompt turn is still settling,
or `paperclip-runnerd` did not arm yet), the sidecar holds it for a
maximum of 5 seconds, and then adopts or drops it;
- in each other case the sidecar drops the cycle. The stream ends, and
the background state becomes unknown, so `paperclip-runnerd` restarts
the provider at the end of the turn as today.
- A Paperclip tool call or a permission request of a held cycle waits
for the decision of its own cycle. It is never bound to the prompt turn
or to a different cycle.
- If an adopted cycle proposes a plan (`ExitPlanMode`), the sidecar
publishes the same `plan.proposed` event as for a controller turn, and
the turn ends with the same terminal: completed, with no
`backgroundTasks`, so `paperclip-runnerd` does not wait again. A
controller turn is not changed.
- If the provider exits while the sidecar is armed with no turn active,
the sidecar sends a `runtime.diagnostic` with the code
`ACPX_PROVIDER_PROCESS_LOST`, and `paperclip-runnerd` ends the wait.
- `turn.cancel` for a provider-started turn cancels the cycle and closes
the provider.
- With no arming, nothing is shown. No controller sends the opt-in yet,
so `paperclip-runnerd` never arms.

**Reason and benefit**

- Only `paperclip-runnerd` decides that a cycle becomes a turn. The
sidecar cannot start a turn by itself.
- One reader of one ordered stream means that two cycles cannot overlap.

## What Changed

- `src/cli/acpx-runtime-sidecar.ts` (127 lines added): the command, the
arming flag, `consumeProviderCycles`, `adoptProviderCycle`,
`pumpProviderTurn`, and the wait of `waitForTool` for a cycle that is
not decided.
- `src/cli/acpx-runtime-sidecar.provider-turn.test.ts` (new, 320 lines)
and two fake hosts in `src/test-support` (227 lines). One host follows a
script. The second runs the real cycle stream and the real background
task tracker.
- `src/cli/acpx-runtime-sidecar.test.ts` (2 lines): one more parameter
for a test harness.

No schema, no Rust and no persisted state changes in this part.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `pnpm run typecheck:typescript`, `npx tsc --noEmit -p
tsconfig.test.json` and `pnpm run build:typescript`: no errors. The
server check `node scripts/verify-runner-vendor-dependencies.mjs`, the
cache guard and the contract tests: passed.
- `npx vitest run src/cli src/drivers/acpx
src/live/runnerd-acpx-provider-turn-wait.test.ts
src/drivers/runner-tool-bridge.test.ts
src/native-session-runtime.provider-turn.test.ts`: `128 passed (128)`
files, `2147 passed`, 9 skipped.
- `cargo test -p paperclip-runner-core --lib` and the `acpx_*` suites:
all passed. No Rust changed in this part.
- The sidecar tests of this part: a cycle that starts before the arming
command, or while the prompt turn settles, is held and then adopted with
its tool call; a cycle is never shown when `paperclip-runnerd` did not
arm, disarmed, armed for a different terminal, armed for a quiet
terminal, or armed after the provider exited; two cycles in sequence
with `{ live: 2 }`, then `{ live: 1 }`, then an unknown state; cancel; a
lost provider; Codex gets no change.
- Mutation checks, each made the named tests fail: the sidecar arms with
no matching terminal; no hold; the old hold condition that drops a cycle
while the prompt turn settles; no wait of a tool call for the decision.
- Live run with a real Claude account, not in CI, with the real ACP
runtime and the real Anthropic API (model `claude-sonnet-5-5`), at the
adapter level: one background shell gave `{ live: 1 }`, one cycle with
its own usage, then an unknown state. Two shells of 8 s and 28 s gave `{
live: 2 }`, `{ live: 1 }`, then unknown. All Paperclip tool calls were
admitted, and none was refused.

Not run: a live run through the real sidecar process,
`paperclip-runnerd` and Core (that path needs an API key, not a local
sign-in); Linux; `pnpm verify --full`. CI supplies the broad coverage.

## Risks

- **The feature is off.** No controller sends the opt-in, so the sidecar
is never armed. The new code that runs today is the loop that reads the
stream: with no arming it drops each cycle, and the background state
becomes unknown as before.
- **A follow-up for a command that finishes while Claude reports on an
earlier one is lost.** The Runner waits again only while a background
command is still running. The run ends as it does today. The reason is
in #21421.
- **A cycle that starts in the last second of a prompt turn** can have
its first messages in the prompt turn and the rest in the
provider-started turn. Tool calls stay safe: the admission rule of
#21278 refuses a call that the active turn did not announce.
- **The 5 second hold** is not tested under load. A cycle that is held
longer is dropped, and the run takes the path of today.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 19:50:22 -08:00
DottaandClaude Opus 5.5 87d363a5c0 perf(ui): reveal a returning Mine visit from the saved cache (#20594)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Mine inbox is the board's main list of work for one person; the
app saves its query cache in `localStorage` so a returning visit can
paint from it
> - On a hard load the saved cache gave no benefit: Mine restored its
rows but still held them until the first approvals, join-request and
dismissal reads answered, because those items sort in among task rows by
time and a late one would push rows down (#20407)
> - Those three reads were not saved, so every returning visit waited on
the network for items it showed a minute before
> - This pull request saves what Mine showed from those reads, as one
record per user and company, and lets it answer the wait on a returning
visit while every read refreshes in the background
> - The benefit is that a returning visit shows its rows with no API
response at all; the first visit keeps its bounded wait unchanged

## Linked Issues or Issue Description

**What happened?**
A returning hard load of `/:companyPrefix/issues?view=mine` with the
saved cache present (`paperclip:query-cache:v1:<user>:<company>`) showed
rows no sooner than with the cache cleared. On the hosted staging
instance, rows appeared about 40 ms after `GET
/companies/:id/join-requests?status=pending_approval` returned in 7 of
10 runs.

**Steps to reproduce**
1. Open Mine once so the app saves its query cache.
2. Delay API responses (for example 300 ms) and reload.
3. Rows appear only after the approvals and join-request reads answer.

**Expected behavior**
Rows show from the saved cache at once. The reads refresh in the
background.

Related: #20407 (kept approvals and join requests in the reveal gate),
#20408 and #20419 (Mine snapshot and its perf guard), #20591 (removes
the second wave of reads; merged), #20592 (task chat preload; merged).

## What Changed

- `ui/src/lib/inbox-mine-reveal-sources.ts` (new): the record `["inbox",
companyId, "mine-reveal-sources", userId]` holds the approvals visible
in Mine, the pending join requests, and the dismissals of those items
and of Mine's failed runs. A record over 256 KB is replaced by an empty
`incomplete` record that never answers the wait. `readMineRevealSources`
accepts a record only for the same user and company.
- `ui/src/lib/persisted-query-cache.ts`: the saved cache keeps that
record per user and company, only when its data names the same user and
company as its key.
- `ui/src/pages/Inbox.tsx`:
- Mine writes the record once approvals, join requests and dismissals
all have server answers.
- On a returning visit with a restored Mine snapshot, the saved
approvals, join requests and dismissals stand in while their reads are
pending, and after a transient failure. A refusal (401/403/404) or a
server answer replaces them. The first-attempt wait is skipped only when
saved values stand in; the first visit is unchanged.
- Selection fix: the selected row's key is recorded only when the
selection moves. Before, every list change recorded the row now at the
old index before the re-anchor ran, so a row inserted above the
selection (a late approval, or any live insert) moved the selection to
another row.
- `ui/src/hooks/useInboxBadge.ts`: `useInboxDismissals` takes an
optional saved `fallback` and returns `hasServerData`. Other callers are
unchanged.
- Tests:
- `Inbox.test.tsx`: a returning visit reveals rows and the saved
approval while the reads are pending, then keeps selection on the same
row when a newer approval arrives above it; a transient failure keeps
the saved approval and a 403 drops it; a record stamped for another user
is ignored and the page waits as a first visit does.
- `inbox-mine-reveal-sources.test.ts`: projection, size limit,
user/company checks, persistence scope.
- `persisted-query-cache.test.ts`: the record round-trips for the same
user, restores as invalidated, and another user's record is not saved.
- `useInboxDismissals.test.tsx`: fallback while pending, after a
transient failure, dropped after a refusal, and no fallback.
- `tests/perf/collections/mine-returning-visit.spec.ts` +
`returning-visit-fixture.ts`: production-build browser checks and the
timing case; README updated.

## Verification

- `cd ui && pnpm exec vitest run src/pages src/lib src/hooks
src/components/coordinator`: 453 files, 5574 tests pass.
- Browser checks on a disposable local instance (production UI build,
`local_trusted`, embedded PostgreSQL, Chromium headless, 1440×900):
`PAPERCLIP_COLLECTION_PERF_BASE_URL=http://127.0.0.1:<port> pnpm exec
playwright test --config tests/perf/collections/playwright.config.ts
mine-returning-visit.spec.ts`
- With every read of the company held, task rows and saved approvals
show and no company read has answered. Fails on `master`, passes here.
- An approval dismissed after the cache was saved stands in, then leaves
when the reads answer. Fails on `master` (no saved approvals), passes
here.
- Another company's Mine never shows the saved company's rows. Passes on
both.
- Timing (5 fresh contexts per arm, alternating saved/cleared, every API
response +300 ms, same instance and fixture, host load average about
20). "Responses before rows" counts API responses that finished before
the first row:

| Build | Saved cache: median (range) | Responses before rows | Cleared
cache: median (range) |
  | --- | --- | --- | --- |
| `master` (`99edcd28a7`) | 1028 ms (906–3011) | 16–19 | 1502 ms
(1423–1592) |
  | This PR | 625 ms (589–672) | 0 | 1632 ms (1359–1692) |
  | This PR + #20591 | 625 ms (604–671) | 0 | 1495 ms (1398–1585) |

Saved cache: difference of medians 403 ms. The remaining 625 ms is
JavaScript download and route-chunk loading in a fresh browser context;
no API response precedes the rows. Cleared cache: no change is claimed;
the spread overlaps.
- Existing collection specs on this build: `parent-context.spec.ts`
passes; `startup-regressions.spec.ts` has 4 failures (partial
failure/context error and two 429 cases) that fail the same way on the
`master` build. `mine-workspaces.spec.ts` was not run (needs a database
URL for its fixture).
- Review follow-up (9da56b5515), each with a red/green test that fails
on the previous head:
- Once a saved record exists, each source's answer replaces its part at
once and a refusal (401/403/404) empties it, without waiting for the
other reads, so a reload never shows refused join requests or approvals.
- The selected row's key is also recorded when the recorded row has left
the list, so a new item arriving above the fallback row never becomes
the selection or the archive target.
- An optimistic dismissal and its rollback start from the saved
dismissals that stand in, not from an empty list.
After the rebase onto `origin/master`: `cd ui && pnpm exec vitest run
src/pages src/lib src/hooks src/context src/components/coordinator`
passes (a first run under host load average ~35 had 4 timeouts and two
files missing new dependencies; after `agent-preflight` the affected
files pass, 223 tests).
- `pnpm check:token-gates`: clean. `pnpm verify` selected broader
coverage (exit 2, perf fixture is a global CI input); required CI
supplies it.
- Not run: the hosted staging page, the full local Vitest suite.

## Risks

- Saved approvals can be stale until the read answers (or after a
transient failure). A stale approval shows briefly and its actions go to
the server, which answers with the current state. A 401/403/404 drops
the saved values.
- The record holds approval payloads and join requests for the signed-in
user's company in `localStorage`. It uses the saved cache's existing
projection (secret-like fields removed), the same per-user revocation on
sign-out, and a 256 KB limit.
- #20591 is merged, so this branch (rebased on it) gets one read of each
shell request. Without it, a returning visit still reconnects the live
socket when the session is verified; because rows now reveal sooner,
that reconnect refetches more reads (here: session, health, company list
and agents twice). With both, every shell read runs once.
- Relation to #20592 (task chat preload): both edit
`ui/src/pages/Inbox.tsx` in different places and merge cleanly in either
order (checked with `git merge-tree`). This PR does not preload
anything. #20592 starts its chat preload when `allLoaded` turns true;
with this PR that happens sooner on a returning visit, while the reads
still refresh in the background, so its preload overlaps that refresh.
#20592 is merged and this branch is rebased on it with no conflicts.
- The selection change affects every Inbox tab: the key is recorded when
the selection moves, not on every list change. Archive and removal still
clamp as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, Playwright). Standard context window,
reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 21:49:17 -06:00
Devin FoleyandPaperclip b7922570e1 fix(runtime): verify GitHub bridge health before agent startup (#21537)
Verify the inactive GitHub credential endpoint from the execution target
before publishing launcher settings. If HTTP/2 fails during startup,
stop it and verify one file-bridge attempt. Stop an unverified final
bridge before returning isolated anonymous launchers.

Regression tests cover early transport loss, delayed loss reporting,
credential isolation, and failed file-bridge cleanup. The master update
keeps the current MCP approval fixture.

Verification: 96 focused tests and server typecheck pass after conflict
resolution. Full CI passes on 843cd9945e. Greptile is 5/5 with no
unresolved review threads.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 19:41:17 -08:00
Nicky LeachandClaude Opus 5.5 4a78124a3e feat(runner): wait for a provider-started Claude turn when the controller asks (#21423)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - `paperclip-runnerd` ends a run when a turn ends. For a Claude turn
that ended with no result while a background shell is running, this
stops the shell and the run fails.
> - #21278 binds each tool call to its turn, and #21293 reports how many
background shells are running at the end of a turn. Part 4 declared the
contract.
> - This pull request makes `paperclip-runnerd` wait for a turn that
Claude starts by itself, when the controller asks for it, and admit that
turn into the same run.
> - The benefit is that the Runner has one place that decides to wait
and one place that ends a wait.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one part of a change that was first opened as one
large pull request (#21316). The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. #21316: the sidecar adopts a cycle.

This is part 5. Its base branch is `feat/prp-provider-turn-contract`.
After each part, a production run does the same as today, because no
controller sends the opt-in. Refs #19912 (design).

**What existing behavior does this improve?**

What `paperclip-runnerd` does after a Claude turn ends with no result
while a background shell is running. Nothing changes until a controller
sends the opt-in.

**Subsystem affected**

Paperclip Runner: the ACPX provider backend in `runner-core` (Rust).
Claude only.

**Current behavior**

- After each completed turn the Runner emits `run.terminal`. A turn with
a running background shell gets the provider restart, and the shell
stops.

**Proposed behavior**

Nothing changes unless the controller sends `providerStartedTurns: true`
in the payload of `turn.start`. No controller sends it yet.

With that field:

- The Runner keeps the provider and waits when all of these are true:
the agent is Claude; the turn completed; the run has no accepted
`paperclip_finish` or `paperclip_block` result; the terminal event
reported `backgroundTasks` with `live` above zero; each condition of
#21293 for a kept process is true; the sidecar accepted
`session.provider_turn` with `armed: true`.
- `turn.completed` then has `awaitingProviderTurn: true`, and no
`run.terminal` follows. The Runner holds that `run.terminal`.
- When the sidecar reports a turn that the provider started, the Runner
admits it only during a wait and publishes `turn.started` with `origin:
"provider"`. In each other case that report fails the session closed.
- When a wait ends with no turn active, the Runner publishes the held
`run.terminal`. This applies when the provider process exits, the
sidecar exits, the controller cancels or stops the run, the session is
closed or suspended, and after a Runner restart.
- The held `run.terminal` and the removal of the wait are saved in one
write, and the controller gets the event from the next poll. A crash
after a cancel or a stop cannot lose it, and it is never sent two times.
- A controller `turn.start` during a wait replaces the provider first,
as before, and the held event is dropped.
- A `run.attach` from a different run during a wait ends the wait when
the attach succeeds. The Runner then discards the held `run.terminal`,
because that event is of the old run and must not end the new run. This
is the same rule that a Runner restart applies. The wait is removed only
in the state that the attach saves for the new run. If the attach fails,
for example when the old session does not stop, the old run keeps the
wait and gets its held `run.terminal` one time, from the next poll or
from a cancel or a stop. A `run.attach` from the same run publishes the
held event to that run.
- The Runner waits at most 5 seconds for the sidecar to answer a request
to arm or to disarm a provider-started turn. This is the limit that turn
cancellation already uses. Before, the limit was the 30 seconds of an
ordinary request, and a cancel, a stop or a shutdown had to wait behind
it. No answer is handled as a lost sidecar: the session fails closed,
and the Runner still publishes the held `run.terminal` one time.
- The sidecar of this branch does not know `session.provider_turn`, so
the arming request fails and the Runner does not wait. The behavior
comes with part 6.

**Reason and benefit**

- The wait is one entry in the saved session state. One rule ends it at
each save, so each new way to leave the open session ends the wait with
no new code.
- A wait must not outlive the provider or the sidecar, and a cancel must
stop it.

## What Changed

- `acpx_provider_turn_wait.rs` (new, 230 lines with its unit tests):
`provider_turn_wait { turn_id, held_run_terminal }` in the saved state;
the rule that ends a wait at save; `settle_turn_events`, which decides
at one place to arm, to create the wait and to put
`awaitingProviderTurn` on `turn.completed`.
- `acpx_provider_turn.rs` (new, 119 lines): the offer of a turn at its
terminal, the arming request, and `admit_provider_turn`.
- `acpx_provider_session.rs`, `acpx_provider_backend.rs`,
`acpx_provider_state.rs`, `provider_events.rs`,
`acpx_park_preparation.rs`: the opt-in from `turn.start`; the record
that the run has a result; the provider-started turn in the state and in
the PRP events; cancel and stop with no turn; no turn-end restart and no
park during a wait.
- `acpx_sidecar_transport.rs` (11 lines): during a wait, a poll with no
event also checks that the sidecar process is alive. A child process can
keep the output stream of a dead sidecar open.
- `bin/fake-acpx-sidecar.rs` (232 lines): provider-started turns and
failures for tests.
- Tests (1,164 lines): `tests/acpx_quiet_turn_continuation.rs`,
`tests/support/acpx_quiet_turn_restart.rs`,
`src/live/runnerd-acpx-provider-turn-wait.test.ts` (the real daemon),
`src/native-session-runtime.provider-turn.test.ts` (the controller of
today).
- `runner/DURABLE_TRANSPORT.md`: the rules.

## State saved by the previous release

- One new optional field in the saved ACPX session state:
`providerTurnWait`. It is written only during a wait. In each other
state the saved file has the shape of the previous release.
- **State from the previous release, read by this release:** the field
is absent, so there is no wait. A unit test covers this.
- **State from this release, read by the previous release (a
rollback):** outside a wait, the file loads as before. During a wait,
the previous Runner refuses the file with its "ACPX provider state is
malformed" error, because its loader refuses each unknown field. Only a
run that waits at the moment of a rollback is in this case, and no run
waits before a controller sends the opt-in.
- **Runner restart during a wait:** the wait is restored. The provider
is a new process, so the wait is over, and the controller receives the
held `run.terminal` one time. A test with the real daemon covers this.
- **Controller of today or of the previous release:** `turn.start` has
no opt-in. A test runs a turn with a running shell and a turn with an
unknown background state, and compares the events, the commands after
the turn, the restart, and the park state before and after the restart.
They are the same.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run
build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs
test/protocol-contract.test.mjs`, the server check `node
scripts/verify-runner-vendor-dependencies.mjs`, `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt
--check` and `cargo build -p paperclip-runner-core --bins` (no new
warning): all passed.
- `cargo test -p paperclip-runner-core --lib`: 644 passed, 4 ignored.
All 17 `acpx_*` suites passed.
- `src/live/runnerd-acpx-provider-turn-wait.test.ts`, 5 cases with the
real `paperclip-runnerd` and the fake sidecar through PRP: a wait ends
with exactly one `run.terminal` when the sidecar exits while a child
process keeps its output open, when the provider is lost, when the
controller cancels, and after a Runner restart; with no opt-in,
`run.terminal` follows the turn and the sidecar is never armed.
- Rust tests: wait and admit; no wait with a result; no wait with no
opt-in; an active goal neither marks nor arms a wait; a sidecar that
refuses to arm; a provider-started turn that the Runner did not wait for
fails the session closed; cancel and stop during a wait; provider and
sidecar loss; restart; the saved shape in both directions.
- `src/native-session-runtime.provider-turn.test.ts`: the real
`consumeTurn` of today gets the event sequence with no opt-in and
finalizes it as `native_finalization_missing`, as today.
- Attach during a wait, in `tests/support/acpx_quiet_turn_restart.rs`: a
different run attaches after the snapshot reports `warmAttachReady:
true`; the new run gets no `run.terminal`, the saved state has no wait,
and the old session is closed. A same-run attach gives exactly one
`run.terminal`. An attach that fails when the old sidecar is dead leaves
the wait with the old run, and the old run then gets exactly one
`run.terminal` from a poll, `run.cancel`, `turn.interrupt` or
`turn.stop`. Mutation checks: with no fix, the old `run.terminal` is in
the pending events of the new run; with the wait ended before the attach
is complete, the old run gets no `run.terminal` after a failed attach.
- No answer from the sidecar, in
`tests/support/acpx_quiet_turn_restart.rs`: the fake sidecar does not
answer the disarm request; `run.cancel` and `turn.interrupt` each return
`already_settled` in less than 10 seconds, exactly one `run.terminal`
reaches the run, `session.close` completes in less than 2 seconds with
no second `run.terminal`, and the provider process is gone. With no
answer to the arm request, the turn ends with its `run.terminal` and
with no wait. With the limit removed, each test fails at 30 seconds.
- Mutation checks, each made the named tests fail: the rule that ends a
wait at save; no `run.terminal` after a restart; arming with no opt-in;
the marker set with no wait; the sidecar process check removed.

Not run: a rollback with the binary of the previous release; a run with
the opt-in and a sidecar that answers the arming request with an error
(the code reads each error as not armed); Linux; the full suite. CI
supplies the broad coverage.

## Risks

- **The feature is off.** No controller sends the opt-in.
- **A rollback of the Runner during a wait** makes the previous Runner
refuse that saved session. See "State saved by the previous release".
- **The wait has no time limit in the Runner.** The controller must
close, suspend or cancel. Core will own that limit.
- **Park is refused during a wait.** The controller must end the wait
before it parks.
- **A new run can attach during a wait.** The attach is not blocked,
because the Runner has no time limit on the wait and a new run can send
only `run.attach`. After an attach that succeeds, the old run gets no
`run.terminal` from the Runner; the controller must not depend on that
event. The request that disarms a sidecar that the attach keeps has no
test, because no test can keep a session across an attach during a wait.
- **After a sidecar that does not answer, `runner.suspend` returns an
error at once** ("provider lifetime cleanup is not yet proven"), and
`session.close` completes. This is the behavior of today for each lost
sidecar; this PR does not change it.
- **The shared code that changed** runs for each Claude turn: the record
of a run result, and the settle step at each turn end.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1114.0-canary.1
2026-11-13 19:21:10 -08:00
DottaandClaude Fable 5.2 689f3b190e fix(recovery): pause automatic recovery runs while a company's runs fail before starting (#21443)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Automatic recovery continues interrupted and failed tasks without a
person, with a per-task limit of 6 automatic attempts per hour
> - There is no company-level limit. When the sandbox provider refuses
to create sandboxes, every task fails the same way, and recovery keeps
creating new runs for each task
> - On a hosted instance, one hour of a provider outage made 510
automatic runs, 587 failures and 0 successes across 289 tasks. No task
had more than 4 runs, so the per-task limit did not engage. Each start
asked a refusing provider for a sandbox and used server memory, which
made the outage longer
> - Recovery must still continue every task by itself. The answer is to
wait while the infrastructure fails, not to give the task to a person
> - This pull request adds a company-scoped pause for automatic work: it
opens when recent automatic runs fail before the agent starts, the
recovery passes stop creating new automatic runs while it is open, and
it closes by itself when runs start normally again
> - The benefit is that an infrastructure outage no longer multiplies
itself through recovery, and recovery resumes with no person acting

## Linked Issues or Issue Description

**What happened?**
During a sandbox provider outage, automatic recovery created hundreds of
new runs per hour across hundreds of tasks. All of them failed before
the agent did any work. The per-task limit did not engage because no
single task reached it.

**Expected behavior**
When most recent automatic runs of a company fail before they start,
recovery waits instead of creating more runs. It tries again at a small
rate. It resumes at full rate when runs start normally again. A person's
work is never held.

**Steps to reproduce**
1. Make the sandbox provider refuse lease creation (HTTP 503) for a
company with many assigned tasks.
2. Let recovery continue the failed tasks.
3. Count automatic runs created per hour: it grows with the number of
tasks, and every run fails at sandbox creation.

Refs #21394 (continuation classification this change will reuse when it
lands), #21433 (start-pass rewrite that the start hook will build on).

## What Changed

- New table `automatic_run_brakes` (one row per company, cascade delete
with the company; migration `0480`). Migration `0481` builds
`heartbeat_runs_company_finished_idx (company_id, finished_at desc)
where finished_at is not null` concurrently, so the signal reads the
runs that finished in the window directly. It stores the state (closed,
open, half_open), the reason with the window numbers, and the
compare-and-set fields that two servers share. A company with no row is
closed.
- New module `server/src/services/recovery/automatic-run-brake.ts`:
- Signal: on the database clock, the automatic runs of the company that
finished in the last 10 minutes (and since the last state change),
newest 500 by finish time, plus up to 200 running runs. A run counts as
a failure only when it ended in its own start phase: no provider work
(`executionRecovery.providerWorkStarted = false` or the native
`nativePreDispatchFailure` record), a legacy run still preparing, or a
transient setup code. A run interrupted by a server stop, restart or
lease loss is neutral, so a release swap does not open the pause. A run
that succeeded, or ended for a person-owned or task reason, counts as
healthy.
- Opens at 12 or more counted runs with 80% or more failures, or on 5
sandbox refusals (`environment_provider_unavailable`) in a row.
- Each outcome counts once, at its own time: a finished run at its
finish time, a running run that got past setup at its start time. Read
time never orders evidence.
- Closes in two steps: open moves to half_open after at least 2 minutes
when the newest 2 counted runs are healthy or the newest one succeeded;
half_open releases at a doubling rate per minute (2, 4, 8, 16, 32, 64)
and then closes. It reopens from half_open if the signal trips again. It
closes by itself after 15 minutes with no automatic activity.
- Reads are one indexed query per company, cached for 5 seconds in a
bounded in-process map (256 companies). Each state change writes one
structured log line
(`automatic_run_brake.opened|half_open|closed|reopened`) with the
numbers.
- It also contains the start gate `holdAutomaticRunStart` (one probe at
a time, at most one new probe per 60 s, release budget per window by
compare-and-set, no I/O while the cached "held until" time has not
passed, person origin taken as an argument with a cached fallback). This
pull request does not call it from the start pass; see Risks.
- Recovery passes defer creating runs while the pause is open. The gate
is only where a run is created:
- `deliverReconciledExecutions`: after the agent-room check and right
before the wake, the delivery asks for a creation slot; when the pause
holds it, the continuation stays pending. The held backlog
(`continueHeldExecutions`) is not gated: recording a pending
continuation creates no run and costs nothing, and its delivery passes
this gate once.
- Assigned-work sweep `act()`: after the process rate limit and before
it claims a continuation, it asks for a creation slot; when held, it
returns `deferred_rate_limit` with the next try time, so nothing is
written to the per-task attempt ledger. New assignments with no earlier
run are not deferred.
- A reserved slot is given back (compare-and-set on its own timestamp)
when no run is created: the wake is refused or throws, the task is
capped, the claim changed, or the task went back to review. The
give-back is conditional in SQL on no run for that task having been
created since the reservation, so a wake that commits its run and then
throws keeps the slot.
- The sweep's overdue saved-work recovery path returns before the slot
is asked for: it starts a recovery attempt, not a run, and is never
held.
- Person work is never deferred. The gate is called only at the delivery
(`agentHasRoom`) and the sweep (`act()`), and each call passes the task
and the run it continues. While the pause is open, the gate does not
hold the record when any of these is true; it holds only when none is.
With no task or no source run it does not hold (fails open).

| Shape | How it is recognised |
| --- | --- |
| A Retry or "move forward" waiting for the task
(`deferred_issue_execution`, `queued` or `claimed`, no run yet) | an
unstarted wake request from a user for the task |
| A Retry folded (`coalesced`) into the continuation, or a Retry while
the delivery is pending | a wake request from a user for the task,
requested after the source run finished (any status) |
| A person's message or a new assignment by a person after the last run
| the same: a wake request from a user for the task after the source run
finished |
| A held task selected because a person pressed Retry (the refused Retry
run) | a run for the task created after the source run finished whose
wake is a person reason (`task_retry`, chat message), or that has
`taskRetryRequestId` or `explicitUserContinuation` |
| A continuation of a person's Retry (any depth) | the source run's
chain reaches a person's run (`listPersonOriginSourceRunIds`, the
dispatch rule) |
| A Retry during pending delivery that becomes the continuation itself
(`deliverContinuationAsRun`) | not gated: that run is created by the
person's own wake, never by these passes |
| The held backlog's saved-input path | not gated: recording or
releasing a held continuation creates no run here |
- To prevent a deadlock, a pass may still create one automatic run per
minute per company when no automatic run of the company started in the
last 2 minutes, however many runs wait (compare-and-set, shared by
servers).
- `GET /api/companies/:id/assigned-work-invariant` now returns
`automaticWork` with the state, the time it paused, the window numbers
and a plain sentence, for example "Automatic work is paused for this
company since T because 47 of 50 recent automatic runs failed before
starting."
- `listPersonOriginSourceRunIds` in `heartbeat/queued-run-line.ts` is
now exported, unchanged, so the gate uses the same person-origin rule as
dispatch.
- The outcome rules (person-owned codes, restart codes, transient setup
codes, native pre-dispatch record) come from their owning modules
through one memoised dynamic import. A static import of
`heartbeat/retries.ts` made a module cycle at server start.

### What the pause consumes, and how waste is bounded

| Token | Taken | Given back or bounded |
| --- | --- | --- |
| Creation slot (one per minute per company while open) | right before
the run is created, after every other check | released only when no run
for the task was committed after the reservation |
| Probe slot (start gate) | when the start gate lets one automatic run
through | another run may take it when the reserved run has not started
within 10 seconds (claim declined, agent paused or full) or was
cancelled, deleted or finished |
| Gradual-release start (start gate) | one per run the start gate lets
through | a declined claim spends one start of that window; the window
ends within 60 seconds and the next one doubles |
| Probe success count | once per probe run, by compare-and-set on its id
| a probe run counts once; cancelled or deleted probes count as neither
success nor failure |

### When the pause opens, stays open and closes

| State | Leaves when | Bound |
| --- | --- | --- |
| closed | opens when, in the last 10 minutes since the last change, 12
or more counted automatic runs finished and 80% or more failed before
starting, or the newest 5 counted runs were sandbox refusals | none
needed |
| open | moves to half_open when it has been open 2 minutes or more and
the newest 2 counted runs are healthy, or the newest one succeeded |
evidence arrives at least once a minute: when no automatic run started
in the last 2 minutes, a pass may create one per minute |
| open | closes when no automatic run started or finished for 15 minutes
(queued runs do not count) | 15 minutes after the last activity |
| open | stays open only while new automatic runs keep failing before
starting | each new try is spaced by the existing passes and by one
creation per minute when nothing starts |
| half_open | closes after 6 release minutes (2, 4, 8, 16, 32, 64) | 6
minutes |
| half_open | reopens when the signal trips again, counted only since
half_open began | none needed |

Restart-class endings (server stop, restart, lease loss, process loss,
interrupted native session) are neutral in every state, so a release
swap does not open the pause.

## Verification

- `npx vitest run src/__tests__/automatic-run-brake.test.ts` (14 tests,
pass): classification; opens on 20 failures before start; 50 queued
automatic runs give exactly one probe and no failure or attempt; a
person's Retry, a new assignment, a run a person joined and the
continuation of a person's Retry are not held; mixed and person-owned
outcomes stay closed; old data is ignored; 40 working runs interrupted
by a restart stay closed, then the same continuations failing at sandbox
acquisition open it; 5 refusals in a row open it; failed probe waits for
the interval, two probes past setup move to half_open, the release gives
exactly 2, 4, 8, 16, 32, 64 starts per window and then closes; reopen
from half_open; probe slot frees after cancel, delete or no start; quiet
close after 15 minutes; two instances on one database agree and start
one probe; one creation per minute while open across servers; 250 held
runs over 100 passes use at most 5 queries (at most 255 with the
person-origin fallback, one per run, once); caches stay at 256 companies
and 1024 runs.
- `npx vitest run
src/__tests__/automatic-run-brake-recovery-passes.test.ts` (pass; one
test per person shape above at the delivery and the sweep, with the
pause open and no creation slot, plus a control that still holds a
continuation no person asked for): includes "no pause record: every pass
behaves as before", "the waiting run cannot start: a pass continues a
task on another agent, then the pause closes", "one held task spends one
creation slot", and "a person's Retry or message is never paused at the
delivery, the sweep or the held backlog".: sweep, delivery and held
backlog create nothing while open, write no attempts, and continue by
themselves after recovery; the sweep still continues one task a minute
when nothing automatic waits. With the three deferral lines removed, all
4 fail (red), and they pass with them (green).
- The pause reads as closed and logs once when its run rules cannot load
(test in `automatic-run-brake.test.ts`).
- `npx vitest run src/__tests__/automatic-run-brake-import.test.ts`: the
module loads after the heartbeat service, in the server's import order.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify`: see the comment below for the result. CI supplies the
full suite.

## Risks

- This pull request does not hold starts. Already queued runs and
`transient_failure_retry` runs still start while the pause is open; they
act as the natural tries. The start gate goes in as a follow-up on top
of #21433, at a point before that pass counts a claim attempt and reads
the full run row, so a held run costs no attempt and no read.
- The sweep reuses the `deferred_rate_limit` outcome for a deferral, to
keep its outcome type unchanged while #21394 edits that file.
- The automatic-run predicate is built from master exports
(`RECOVERY_WAKE_REASONS`, retry markers). It will switch to
`isAutomaticContinuationRun` when #21394 lands.
- Migration `0480` adds one small table.
- Migration `0481` (`0481_heartbeat_runs_company_finished_index.sql`)
runs `CREATE INDEX CONCURRENTLY IF NOT EXISTS` as a top-level statement:
it is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` in
`packages/db/src/client.ts`, like 0470 and 0476, and the runner drops
and rebuilds an invalid index after an interrupted build. The index is
declared in the Drizzle schema, so `pnpm db:generate` reports no
changes, and `pnpm --filter @paperclipai/db check:migrations` passes. On
a large `heartbeat_runs` table the build is online (no write lock) but
takes time proportional to the table. After it exists, it is one more
index on a hot table: each run that finishes writes one more index
entry.
- No user-visible text changes outside the operator report.

## Model Used

- Claude Fable 5.2 (Anthropic), via Claude Code with tool use (shell,
file edits, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 21:20:09 -06:00
DottaandClaude Fable 5.2 6826800425 feat(ci): report the required checks on merge-queue commits (#21545)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each merge to `master` publishes the standard image, and the time
from merge to published image limits how fast a fix reaches users
> - #21096 lets `docker.yml` push an image that the merge queue already
built, but the merge queue is not on
> - The `master` ruleset requires `ci / verify` and `ci / e2e`. Only
`pr.yml` reports them, and only for pull requests
> - A merge queue waits for the same two names on its merge-group
commit. No workflow reports them there, so every queued merge would
stall
> - This pull request adds a workflow that reports both names on
merge-group commits
> - The benefit is that the owner can turn the merge queue on with one
ruleset change

## Linked Issues or Issue Description

Refs #21096

**What existing behavior does this improve?**

The merge-queue image build from #21096. It cannot run until the merge
queue is on, and the merge queue cannot be turned on safely.

**Current behavior**

`ci / verify` and `ci / e2e` come from `pr.yml`, which runs on
`pull_request` only. With a merge queue, GitHub waits for both checks on
the merge-group commit. They never arrive. Each queued pull request
would time out and leave the queue.

**Proposed behavior**

`merge-queue-checks.yml` runs on `merge_group` and reports `ci / verify`
and `ci / e2e`. It runs no tests. It does not check out the queued
commit. It has no token permissions and no secrets.

**Reason and benefit**

A pull request enters the queue only after both checks pass on the pull
request. A queued merge therefore has the same protection as a direct
merge today. The exact-source gate still tests each `master` commit
before publication. Real tests on the combined tree can replace these
jobs later under the same names.

## What Changed

- Added `.github/workflows/merge-queue-checks.yml`. It reports `ci /
verify` and `ci / e2e` on merge-group commits.
- Added `scripts/__tests__/merge-queue-checks-workflow.test.mjs`. It
fails if the names differ from the names that `pr.yml` reports, or if
the workflow gains a checkout, an action, a secret, a token permission
or a self-hosted runner.
- `pr-trusted.yml` now runs that test and
`scripts/merge-group-image.test.mjs` in a step that skips test files an
older checkout does not have. The second test came with #21096 but no CI
step ran it.
- `scripts/merge-group-image.mjs find` now waits up to 15 minutes for a
merge-queue image build that is still queued or running. The queue can
land a commit before its image build ends. Without the wait, the
`master` push always builds from source.
- `doc/RELEASING.md` explains the workflow and lists the ruleset
settings that turn the queue on and off.

## Verification

- `node --test scripts/__tests__/merge-queue-checks-workflow.test.mjs`:
2 pass.
- `node --test scripts/merge-group-image.test.mjs`: 10 pass, with new
cases for unfinished builds.
- `node --test scripts/__tests__/release-verify-workflow.test.mjs`:
pass.
- `actionlint` on both changed workflows: no findings.
- Not verified: no `merge_group` event can fire before the queue is on.
The first queued merge is the live test. `pr.yml` calls
`pr-trusted.yml@master`, so the new CI step runs only after this merges.

Owner steps after merge:

1. In the `master` ruleset, add **Require merge queue**: Squash, build
concurrency 5, group size 1 to 5, status check timeout 60 minutes.
2. After the first queued merge, confirm that the `master` commit SHA
equals the SHA of its merge-group runs.

## Risks

- Low risk while the queue is off. The workflow has no trigger other
than `merge_group`.
- With the queue on, the queue does not test the combined tree. Two pull
requests that pass alone can still break `master` together. This is the
same as today.
- A merge-queue image build that hangs delays the source build by up to
15 minutes. A build that fails does not.
- If GitHub does not accept a job named `ci / verify` for the required
context, queued merges time out. Rollback: remove the merge queue rule.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 21:18:13 -06:00
DottaandClaude Opus 5.5 f6c5986b61 feat(connections): machine sign-in (MCP OAuth client credentials) for organization connections (#21473)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through connections to remote MCP servers
> - Today an organization connection must sign in as a person (browser
sign-in) or use a static key
> - Service accounts and CI-style integrations have no person who can
sign in
> - The MCP OAuth Client Credentials extension (2026-07-28) defines
machine-to-machine sign-in for this case
> - This pull request adds "Machine sign-in" for organization
connections, with a Paperclip-held signing key (recommended) or a client
secret
> - The benefit is that organizations can connect automation-owned MCP
servers safely, with credentials only in the vault and bound to one
sign-in service

## Linked Issues or Issue Description

**Subsystem affected**
Connections: remote MCP OAuth (server `tool-access/oauth`, tool gateway
token refresh) and the custom MCP server setup flow in the UI.

**Problem or motivation**
Organization-owned automation (service accounts, CI integrations) cannot
connect an MCP server that requires OAuth, because no person can
complete a browser sign-in. The MCP OAuth Client Credentials extension
(`io.modelcontextprotocol/oauth-client-credentials`) solves this with
`grant_type=client_credentials` and RFC 7523 signed client assertions.

**Proposed solution**
Add machine sign-in for organization connections only. Paperclip creates
a keypair per connection, keeps the private key in the company vault,
and shows the public key (PEM and JWK) for the admin to register. A
client secret is also supported. The sign-in service (issuer and token
endpoint) is pinned at setup. Tokens use the existing organization grant
refresh path, so caching, concurrent dedupe and 401 re-request reuse the
existing lease and single flight.

**Alternatives considered**
Reuse the deployment-wide `PAPERCLIP_TOOL_OAUTH_*` client env path that
already sent `client_credentials`. Rejected: it is not per connection,
it keeps secrets in env, and it has no issuer pin. Route machine sign-in
through `connectGalleryApp`. Rejected: its rollback deletes the draft on
a failed test, which would lose a key the admin already registered.

**Roadmap alignment**
Phase 1 of the enterprise MCP plan (no enterprise single sign-on
needed). OIDC login and enterprise-managed authorization (ID-JAG) are
later phases.

## What Changed

- `oauth/client-credentials.ts`: keypair generation (RS256, or ES256
when the service requires it), RFC 7638 key IDs, and RFC 7523 assertions
(`iss` = `sub` = client ID, `aud` = token endpoint, 60 s `exp`, unique
`jti`) with Node `crypto` only.
- `oauth/client.ts`: the token exchange accepts a signed assertion and
then sends only `client_assertion_type` and `client_assertion` (no
`client_id` or secret), as the extension specifies.
- `oauth/client-credentials-service.ts`: discovery of the sign-in
service (protected-resource metadata, then RFC 8414/OIDC metadata with
issuer check, or a token URL entered directly), issuer pinning,
vault-only key and secret handling, draft creation, test-and-activate,
and no-downtime rotation (test the new secret first; stage a new key and
switch only after the provider accepts it).
- `oauth/grant-refresh.ts` and `oauth/connection-refresh.ts`: machine
sign-in requests a new access token with the connection's own
credentials instead of needing a refresh token. It never falls back to
deployment-wide OAuth clients.
- `catalog/discovery.ts`: a rejected token on a machine sign-in
connection never re-runs discovery or rewrites the pinned endpoints.
- `mcp-http.ts` and the tool gateway: declare
`io.modelcontextprotocol/oauth-client-credentials` in client
capabilities only for connections that use it.
- Routes (with OpenAPI and activity logging): `POST
/companies/{companyId}/tools/machine-sign-in`, `GET|POST
/tool-connections/{id}/machine-sign-in[/test|/rotate|/rotate/complete]`.
Removal activity now records `machineSignIn: true`. The runner API
treats these routes as sensitive.
- Shared validators and types for machine sign-in.
- UI: "Machine sign-in" under Advanced authentication in the custom MCP
server flow (organization connections only), `MachineSignInSetup`
(choose Paperclip key or client secret, register the public key, test),
and Storybook stories for each setup state.
- Docs: `doc/connections/MCP-CLIENT-CREDENTIALS.md`, linked from
`doc/connections/README.md` and `GENERIC-REMOTE-MCP.md`.

## Verification

- `server/src/__tests__/tool-access-client-credentials.test.ts`: 16/16
pass. It uses a fake sign-in service that verifies real signatures
against registered keys, and a fake MCP server. It covers RS256 and
ES256 assertion claims and signatures, assertion-only client
authentication, the secret flow (`client_secret_basic`), discovery and
direct token URL, issuer pinning (refuses a different sign-in service
and sends it nothing), the pinned endpoint surviving a hostile
challenge, the token cache (0 token requests while live), concurrent
dedupe (5 callers on 2 service instances produce 1 token request), 401
re-request, key and secret rotation, a refused personal connection,
secret redaction in responses, rows and activity, and vault cleanup on
removal.
- Red/green: with the grant-refresh machine sign-in branch and the
discovery pin guard disabled, 5 runtime tests fail (pin, cache/dedupe,
401 re-request, both rotations). Restored: 16/16 pass. The other tests
cover new routes and services that do not exist on `master`.
- `ui/src/features/connections/MachineSignInSetup.test.tsx`: 3/3.
`ConnectionSetupFlow.architecture` and `.oauth` tests: 19/19.
- `openapi-routes.test.ts`: 51/51. `runner-api.test.ts`: 2153/2153.
`mcp-http.test.ts`: 9/9. Existing `tool-access-service.test.ts` refresh
and legacy client-credentials tests: 54/54 (filtered).
- Server package `tsc --noEmit`: clean. UI `tsc`: clean except the
pre-existing generated Storybook catalog imports.
- Not run locally (machine under heavy load): full suites, repo-wide
typecheck, browser check of the live app. CI covers breadth.

## Risks

- No migration. New config lives in `config.oauth` (`grantType:
"client_credentials"` plus `clientAuth`). Older env-configured
client-credentials connections have no `clientAuth` and keep their
original path.
- The capability declaration adds one extension key only for machine
sign-in connections. Other requests are unchanged.
- Rotation has no settings-page control yet; it is available through the
API. This is a follow-up.
- Low risk to agent boot and time to first token: a live token is read
exactly like a browser sign-in token.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:10:32 +00:00
DottaandClaude Opus 5.5 6b631dd477 fix(recovery): check held retained orphans only for candidate leases (#21542)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The ownership reconciliation pass lists in-progress records of each
kind every 30 seconds, and settles the records whose owner is gone
> - One kind, the retained orphan sandbox, lists leases with a large SQL
filter that has correlated subqueries on `heartbeat_runs`, `issues`,
`issue_recovery_actions` and `environment_leases`
> - The planner ran those subqueries for every released, failed and
expired lease in history, before the cheap orphan predicates. The cost
grew with lease history, not with the number of orphans
> - On a busy hosted database one page of this listing ran for 17 to 138
seconds as one statement. It held an AccessShareLock on `heartbeat_runs`
for all of that time, so an online `ALTER TABLE heartbeat_runs ADD
COLUMN` with a 5 second lock timeout failed on every try
> - This pull request runs a cheap candidate filter first and evaluates
the held checks only for candidate rows
> - The benefit is that the listing cost now follows the number of
candidates. Schema changes on `heartbeat_runs` no longer wait behind it,
and the pass reads far fewer pages

## Linked Issues or Issue Description

No public issue exists. This describes the bug in the format of the bug
report template.

**What happened?**
A hosted instance could not apply a pending migration that adds a column
to `heartbeat_runs`. The deployer sets `lock_timeout = 5s` and tries
three times. All three tries timed out. The database showed one session
that ran `select "id", "company_id", "heartbeat_run_id", "issue_id",
"metadata", "updated_at" from "environment_leases" ...` in state
`active` for up to 138 seconds. Its transaction start and query start
differed by less than 0.1 seconds, so it was one statement. It held an
AccessShareLock on `heartbeat_runs`. That instance had about 18,600
lease rows: 13,240 expired, 4,695 released, 610 failed, 69 retained, 17
active and 9 pending cleanup. Each failed try also stalled all
statements on `heartbeat_runs` for up to 5 seconds.

**Expected behavior**
The ownership listing must finish in a short time that does not grow
with lease history. It must not keep a lock on `heartbeat_runs` long
enough to block an online schema change.

**Steps to reproduce**
1. Use a database with many terminal environment leases (expired,
released or failed) and few or no retained orphan sandboxes.
2. Run the `environment_retained_orphan_sandbox` listing
(`OWNABLE_KINDS`, `listInProgress`), or wait for the ownership
reconciliation pass.
3. Run `EXPLAIN (ANALYZE, BUFFERS)` on the listing. The `heartbeat_runs`
primary key subplans run once for each terminal lease, not only for
candidates.
4. While the listing runs, run `ALTER TABLE heartbeat_runs ADD COLUMN
probe jsonb` with `lock_timeout = '5s'`. The ALTER waits until the
listing ends.

**Paperclip version or commit**
`master` at 0a96f972e1.

**Database mode**
Hosted PostgreSQL. Reproduced on embedded PostgreSQL 18.

**Additional context**
The 138 second session was not a transaction held open across remote
work. The call path is `registerOwnershipReconciliationWork` →
`runOwnershipReconciliationPass(db)` → `kind.listInProgress(db, 200,
offset)` on the pool, with no transaction. Only the `leaseKind` and
`operationKind` listings send this exact unqualified column list. Of
these, only this kind's filter reads `heartbeat_runs`.

## What Changed

- `releasableRetainedOrphanCondition` and `nextRetainedOrphanDueAt`
first apply `retainedOrphanCandidateCondition`. This is a filter with no
subqueries and no NULL results: cleanup status, run, provider, lease
status, lease policy, and the metadata keys of the three retained orphan
shapes. Expired, active and pending-cleanup leases fail it at once.
- The held checks (always held and softly held) now run inside `CASE
WHEN <retained orphan condition> THEN ... ELSE false END`. PostgreSQL
evaluates a CASE in order, so the correlated subqueries run only for
candidates. The planner can no longer split the `NOT (a OR b OR ...)`
into separate top-level subplans and run them before the candidate
predicates.
- The two `heartbeat_runs` checks of the always-held part are now one
scalar subquery with a primary key lookup. It reads
`runner_profile_json->'nativeWorkspaceSync'` once through a fenced
lateral, instead of about 12 `#>>` path reads. A scalar subquery cannot
become a hashed subplan, so the planner cannot replace it with a full
scan of `heartbeat_runs`.
- The listed rows, the order and the paging do not change.
`softlyHeldSql` is now exported so that the test can rebuild the
previous condition.
- The paging stays OFFSET-based. The `listInProgress(db, limit, offset)`
contract is shared by all ownership kinds and by the I2 rule, which
compares consecutive passes. Keyset paging is a change to that shared
contract, so this pull request does not make it.

## Verification

- `cd server && npx vitest run
src/__tests__/environment-retained-orphan-listing-cost.test.ts
src/__tests__/environment-retained-orphan-sandbox.test.ts
src/__tests__/abandoned-workspace-files-hold.test.ts`: 3 files, 55 tests
pass.
- New test `environment-retained-orphan-listing-cost.test.ts`:
- Equivalence: four seeded random fixtures of 600 leases each, across
two companies. The fixtures cover every lease status and policy, runs of
each status and runtime mode, valid and invalid native workspace
descriptors, every always-held metadata key, sibling leases on the same
sandbox, open and resolved recovery actions, open runs on the issue, and
past, future and absent expiry. Every page (limit 7 and 200) equals the
previous SQL, which is kept only in the test. `nextRetainedOrphanDueAt`
also equals the previous query.
- Red/green: over 2,000 history leases with the hosted status mix (71%
expired, 25% released, 4% failed), the test counts the `heartbeat_runs`
rows that the listing reads, from `EXPLAIN (ANALYZE, FORMAT JSON)`. On
`master` it reads 1,760 rows for 1 candidate, so the test fails. With
this change the count is within the bound of 10 per candidate.
- Incident regression: while listings run in a loop over 5,000 history
leases, `ALTER TABLE heartbeat_runs ADD COLUMN ...` and `DROP COLUMN`
with `lock_timeout = '5s'` complete in less than 5 seconds.
- Local measurement (embedded PostgreSQL 18, Apple M-series, warm
cache), not in CI:
- 18,640 leases with the hosted status mix and a 19 KB runner profile
per run. `master`: 422 ms, 296,771 buffers (12,357 read). This change:
26 ms, 36,004 buffers. The `heartbeat_runs` subplans run 13 times (once
for each candidate), not once for each lease.
- 300,000 history leases. `master`: one listing page takes 2,822 ms, and
the ALTER under load waits 2,448 ms. This change: 212 ms, and the ALTER
takes 96 ms.
- `pnpm check:module-boundaries` passes.

## Risks

- Low risk. The listing and the due-time query return the same rows, and
the equivalence test compares them. The settle is unchanged and still
re-checks every hold under the row lock.
- The candidate filter must stay a superset of
`retainedOrphanSandboxCondition`. If a later change adds a retained
orphan shape and does not update the candidate filter, the listing skips
that shape. The equivalence test catches this only for the shapes in the
fixture.
- The listing still reads each lease row once with cheap predicates. A
partial index on the candidate filter could make it scale only with
candidates. This pull request does not add a migration.
- The ownership pass still lists all kinds every 30 seconds when its
fingerprint changes. This pull request does not change that cadence.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edit, embedded PostgreSQL runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:48:21 -06:00
DottaandClaude Opus 5.5 1dee321844 perf(run-events): read the wake's transaction id and the replay probe in the lock read (#21528)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every native event that announces work to another part of the server
(chat progress, a final response, an evidence outcome, recovery)
registers a wake in its transaction before it writes the event
> - Such an event takes six statements: BEGIN, the run lock read, a
replay probe, a read of the transaction id for the wake, the append, and
COMMIT. A controller checkpoint save takes five: BEGIN, the owner lock,
the transaction id, the write, COMMIT
> - In the agent scaling load model these two paths are now the largest
per-agent cost after #21511 and #21520, and other events fall behind
first at 600 agents
> - This pull request returns the transaction id from the lock read that
each path already sends, and moves the replay probe into the run lock
read when it is safe
> - The benefit is four statements for such an event and four for a
checkpoint save, with the same locks, the same wake order and the same
replay rule

## Linked Issues or Issue Description

Refs #21511 and #21520 (the delta batch path). Refs #21418 (agent
scaling plan and load model).

**What existing behavior does this improve?**
The statement count of native events that register a wake
(`appendNativeSourceEvent`) and of controller checkpoint saves
(`nativeRunnerCheckpointWriter`).

**Subsystem affected**
`packages/db/src/work-signals.ts` (one new export),
`server/src/services/heartbeat-run-events.ts`,
`server/src/services/native-runtime/native-runner-checkpoint.ts`.

**Current behavior**
`signalDatabaseWork` sends `select pg_current_xact_id()` for the first
wake of each transaction. A native event with a wake reads the run lock,
then probes for its replay in a separate statement, because a statement
that waits for the lock keeps its earlier snapshot.

**Proposed behavior**
- The run lock read and the checkpoint owner lock read also return
`pg_current_xact_id()::text`. Both already lock a row, so the
transaction has an id at that point. `primeDatabaseWorkTransactionId`
gives it to the transaction's wake state, and the wake sends no
statement for it.
- For an event that may register a progress, response, reassessment or
retention wake, the run lock read also reads `next_event_seq` from its
own snapshot and checks for a replay in that snapshot. Every event
append advances `next_event_seq` in its transaction. So when the locked
row has the same `next_event_seq` as the snapshot, no append committed
in between, and the probe result is current. Otherwise, or when the
probe finds a replay, the separate probe runs as before.

**Reason and benefit**
Fewer round trips while a pool connection is held in a transaction. In
the load model these paths hold the main pool and the three-connection
checkpoint pool longest per event.

**Breaking changes**
None. No schema change.

## What Changed

- `primeDatabaseWorkTransactionId(connection, xid)` (`work-signals.ts`,
exported from `@paperclipai/db`): inside a transaction whose id is not
yet known, it records the id. Outside a transaction it does nothing. An
uncertain commit still resolves with `pg_xact_status` on the same id.
- `appendNativeSourceEvent`: the lock read returns the transaction id
when the event has any wake. For a probing wake it is one statement: a
CTE that reads `next_event_seq` from the snapshot, a CTE that locks the
run row (`for no key update`, unchanged), and `exists (...)` for the
replay. The wake is still registered after the replay check and before
the append.
- `lockNativeRunnerCheckpointOwner` returns the transaction id with its
owner lock and primes the wake state. Its return value and every caller
are unchanged.

## Verification

- Red/green on embedded PostgreSQL; each new assertion failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "registers a native
event's wakes before its write and only when the event is new": four
statements (before: six), the wake's intent and settlement are
published, and a replay publishes no wake.
- New: "finds a replay that another writer committed while the lock read
waited, and registers no wake for it". Another connection holds the run
row (the test waits until PostgreSQL reports the append waiting for that
lock), stores the same event and moves `next_event_seq`; the waiting
append returns `duplicate`, runs the separate probe, and publishes no
wake. With the `next_event_seq` comparison removed, this test fails.
- New `native-runner-checkpoint-save.test.ts`: a save is BEGIN, the
owner lock with the transaction id, the write, COMMIT (before: five
statements), and its recovery wake is published; a save after the lease
expired writes nothing and fails with
`native_runner_checkpoint_owner_lost`.
- `work-signals.test.ts`: a primed transaction sends no id statement for
its wakes; a primed transaction whose connection is killed after its
write stays pending and then resolves through `reconcileDatabaseWork`,
with no row stored.
- Added after merging `origin/master` (with #21511, #21520 and #21436)
into this branch:
- "gives the wake the transaction id of the transaction that took the
run lock": inside a caller's transaction, an event with a progress wake
sends no separate `pg_current_xact_id()` statement; the transaction's
connection is then killed, and `reconcileDatabaseWork` checks
`pg_xact_status` with exactly the id that `pg_current_xact_id()` returns
on that same transaction. Priming a wrong id makes it fail.
- "primes nothing when the lock read fails, so the transaction is not
left uncertain": a lock read that fails on `lock_timeout` leaves no
pending work; after the same failure inside a savepoint, the next signal
on that transaction still sends its own `pg_current_xact_id()`. Priming
before the lock read makes it fail.
- Both call sites pass only the value returned by the statement they
just ran on the same `tx`: `appendNativeSourceEvent` (`tx.execute(...)`
then `primeDatabaseWorkTransactionId(tx, run.xid)`) and
`lockNativeRunnerCheckpointOwner` (`tx.with(locked).select(...)` then
`primeDatabaseWorkTransactionId(tx, xid)`).
- Retest on the combined tree, one file at a time: atomic-append 28
(including the equivalence test and both race tests), the port suite
with conformance 105, checkpoint-save 2, `live-recovery-work` 51,
`work-signals` 25, runner `native-session-runtime` 297; server typecheck
passes. No code change was needed for the combination.
- Other focused suites pass, one file at a time: the port suite with its
conformance suite (100), `chat-native-progress-work` (35),
`native-observer-replay` (1), `heartbeat-run-event-sequencing` (5),
`native-event-boundary-golden` (1), `native-recovered-owner-followup`
(97), `native-restart-recovery-isolation` (1),
`heartbeat-native-restart-suspension` (115), `live-recovery-work` (51),
`work-signals` (25).
- `pnpm --filter @paperclipai/server run typecheck` and `pnpm
check:module-boundaries` pass.
- Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in
#21418). Stacked: this pull request merged locally with #21511, and the
model's local `DELTA_COALESCE=1` mode that mirrors #21520 (the model
applies the port's batching rules itself; that mode is not in any pull
request). Default pool of 10 connections; embedded PostgreSQL on the
same, shared machine; runs one after the other with a pause between
them. No run had a failed operation, a failed checkpoint save or a lost
lease. "Before" rows are `origin/master` from the same session.

| Agents | Code | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 /
12,000 | 278 / 3,895 |
| 200 | stacked | 60 s | 7.9 | 0.31 | 2,725 | 102 / 253 | 12,000 /
12,000 | 1 / 70 |
| 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 /
24,000 | 1,904 / 60,154 |
| 400 | stacked | 60 s | 8.6 | 0.60 | 5,609 | 117 / 955 | 24,000 /
24,000 | 2 / 1,274 |
| 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 /
36,000 | 10,822 / 52,171 |
| 600 | #21511 + #21520 only | 120 s | 11.4 | 0.72 | 6,730 | 1,147 /
6,335 | 61,158 / 72,000 | 11,678 / 21,212 |
| 600 | stacked | 120 s | 9.6 | 0.93 | 10,290 | 656 / 4,233 | 71,895 /
72,000 | 1,136 / 4,980 |
| 600 | stacked (repeat) | 120 s | 9.8 | 0.90 | 9,579 | 648 / 4,509 |
71,958 / 72,000 | 1,319 / 5,712 |
| 800 | stacked | 60 s | 9.7 | 0.98 | 10,971 | 920 / 3,305 | 46,244 /
48,000 | 1,974 / 3,835 |

With the three changes, one process finishes 99.9% of the offered work
at 600 agents, but its event-loop utilization is 0.90–0.93, above the
0.75 target, and p99 lateness is about 5 s. It saturates between 600 and
800 agents.
- Statements: an event with a wake 6 → 4; a checkpoint save 5 → 4 (6 → 5
where commit-time notifications are on).

## Risks

- The transaction id is read in a statement that already assigns one (a
row lock), so no transaction gets an id it did not have before.
- The probe in the lock read is used only when `next_event_seq` did not
change between the snapshot and the lock. Every writer of
`heartbeat_run_events` moves `next_event_seq` in the same transaction
(native and ordinary appends, the run coordinator store, the retry
queue). A future writer that inserts events without it would make this
probe miss a concurrent replay; the append statement then still finds
the replay and stores nothing, but the wake would already be registered
(a spurious wake, not a lost or duplicate event).
- This pull request and #21511 both edit `heartbeat-run-events.ts`, in
different functions.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:48:17 -06:00
Devin FoleyandPaperclip dc018f9bc0 Keep agent-owned connections healthy during catalog refresh (#21539)
Use dedicated agent grants for catalog refresh, GitHub health checks, and reconnect status. Preserve shared connection health when a different agent lacks authorization.

Add PostgreSQL and HTTP regressions for credential selection, stale health recovery, caller isolation, and actual provider rejection. Correct the MCP demo approval fixture to submit a valid saved-card option through normal answer validation.

Validation: 629 focused tests, server typecheck, all CI checks, and Greptile 5/5 with no unresolved review comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 18:45:20 -08:00
DottaandClaude Opus 5.5 833689a3fc fix(queue): say a long-queued turn is waiting to start, and name the routine execution that holds it (#21447)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21405 made a task that waits for its agent's concurrency limit show
"Waiting for <agent> · N slots max · P of Q in line · Configure"
> - On hosted staging, after that deploy, a routine task stayed queued
for 8 minutes while its agent had 19 free slots. Its chat said "Working
for 7m · Thinking", and the server label was a plain "Queued"
> - The real cause was the routine's one-execution-at-a-time rule: the
previous execution of the same routine was still open with its run. That
is correct scheduling, but the UI did not say it
> - A queued turn also kept the startup feedback ("Working") for as long
as it stayed queued, with no limit
> - This pull request names the routine hold and limits the startup
feedback to 30 seconds
> - The benefit is that a queued task always says that it waits, and why
when the server knows

## Linked Issues or Issue Description

Refs #21405

**What happened?**

On staging, one routine task's continuation run was queued at 17:39:03Z
and claimed at 17:47:10Z. The agent ran 1 of 20 runs. The chat showed
"Working for 7m · Thinking". The run's projection label was "Queued"
with no cause.

**Why did it happen?**

1. The routine allows one open execution with a run at a time (unique
index `issues_open_routine_execution_uq`). The previous execution of the
same routine was still open and held its run until 17:47:10.172Z. The
waiting run was claimed 345 ms after that task closed. The claim was
refused with "Another execution for this routine is already in progress"
until then. This is the routine's intended policy, not a scheduling bug.
2. The execution projection did not name that hold, so the server label
was "Queued".
3. The task chat keeps the startup feedback ("Working" and "Thinking")
for a queued turn with no named hold (a DESIGN.md rule). That rule had
no time limit.

**Expected behavior**

A routine task that waits for the routine's open execution says "Waiting
for <task> to finish". A queued turn that has no named hold says
"Waiting to start" after a short grace period.

## What Changed

- Server: `executionProjectionsForRuns` reads, in one query per batch,
whether another open task of the same routine holds the routine's
execution. For a never-started queued run that waits on it, the
projection sets the cause `routine_execution_in_progress`, the label
"Waiting for <task identifier> to finish", and a next action. This check
comes before the agent-limit check, because a free slot would not start
the run.
- UI: a new hook, `useWaitingToStart`, keeps the startup feedback for 30
seconds. After that, a queued turn with no named hold shows "Waiting to
start": an hourglass, no spinner, no shimmer. The task chat's runner
header, its current-activity tail and the live run pill use it. The hook
uses one timer at the threshold and no per-second ticks.
- DESIGN.md: the startup-feedback rule now has the 30-second limit and
the routine hold.
- Storybook: `WaitingToStart` and `WaitingForRoutine` stories, with play
checks.

**Why 30 seconds:** DESIGN.md keeps "Working" from message submission
through admission, so that a normal start does not flicker. On staging,
runs that were not held started 3 to 8 seconds after they were queued
(from the company's run list between 17:30 and 17:48Z). 30 seconds is
several times that, and shorter than any real wait the user should be
told about.

## Verification

- Red/green server test (embedded Postgres),
`execution-projection-agent-queue.test.ts`: a routine task waits while
another open task of the same routine holds a run. The test checks the
label "Waiting for Q-3093 to finish" and the cause. It checks that the
holder is not marked as waiting on itself, and that the label returns to
plain "Queued" when the holder closes. It fails without the server
change.
- Red/green UI tests: `TaskChatRunnerTurn.test.tsx` (startup feedback
until the threshold, then "Waiting to start" in the header and the tail;
a running turn is never called waiting) and
`TaskChatLiveRunPill.test.tsx` (the label changes, with no spinner or
shimmer). Both fail without the UI change.
- `vitest run` on the four `execution-projection-*` server suites (36
tests) and all `task-chat` and `TaskChatThread` UI suites (1,414 tests).
UI and server typecheck.
- The new stories render in headless Chromium.
- On staging, before this change, the #21405 waiting line was confirmed
with a temporary 1-slot agent: "Waiting for Queue Check (temp) · 1 slot
max · 2 of 2 in line · Configure". The temporary tasks and agent were
cancelled and terminated.
- After deploy: confirm the new labels on staging.

## Risks

- Low risk. There is no schema change.
- One extra query per projection batch that includes a never-started
queued run on a routine task. It is a join on the company's
routine-execution tasks through the existing `issues_company_origin_idx`
index.
- A queued turn that is admitted after 30 seconds, for a reason the
server does not name, now says "Waiting to start" instead of "Working".
This is the intended change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, Chrome for staging checks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:22:07 -06:00
DottaandClaude Opus 5.5 a82e235828 fix(heartbeat): keep recovery behind people's work, guarantee a start to tasks people wait on, and admit recovery a few runs at a time (#21427)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts an agent's queued runs when a slot is
free. #21234, #21277, #21349 and #21391 order them: person work first, a
recovery share, aging, and a share for the longest-waiting run
> - On hosted staging, one agent with 20 slots had 295 queued runs. 139
of them were system recovery wakes that entered the queue within one
hour. Runs that people asked for waited 7 hours
> - Two rules let this happen. Recovery took every fourth start even
when a person's run waited, and an aged recovery run moved into the
agent-work tier ahead of todo tasks. The continuation producer also
admitted any continuation held past 5 minutes, however deep the queue
was
> - This pull request keeps recovery behind people's work, gives a
person's long-waiting run a bounded start, re-checks a recovery run
before it starts, and admits recovery to an agent's queue a few runs at
a time
> - The benefit is that a task a person asked for starts within a
bounded number of dispatches, and a recovery burst cannot flood an
agent's queue

## Linked Issues or Issue Description

Refs #21234, #21277, #21349, #21391, #21405, #21433, #21502

**What happened?**

On hosted staging (2026-11-13), one agent had `maxConcurrentRuns` 20 and
all 20 slots held hour-long runs. 295 runs were queued. 139 had wake
reason `issue_recovery_action_restored` (source `execution.reconciled`),
enqueued between about 10:46Z and 11:47Z. Two tasks that people created
were stalled by a restart at 07:25Z. Their next runs were queued at
07:29Z and were still queued 7 hours later. Both were system wakes: one
`issue_monitor_due`, one `issue_continuation_needed` from the
assigned-work sweep. Recent starts had waited from 1 to 290 minutes.

**Why did it happen?**

1. The recovery share gave every fourth start to system recovery, also
while a person's run waited.
2. A recovery run that waited one hour aged into the agent-work tier.
Recovery tasks are usually in progress, so they sorted ahead of todo
tasks in that tier.
3. The oldest-run share served the oldest aged run of any class. Older
agent work and older recovery came before a person's run.
4. Only a wake that a person requested counted as person work. A system
wake on a task that a person created and waits on (a monitor check, a
continuation after a restart) competed as ordinary work or recovery.
5. `deliverReconciledExecutions` admitted a continuation held past
`RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 minutes) whatever the queue
depth. At one per agent per 15-second pass, one agent got about 139
continuations in an hour.

**Expected behavior**

A run that a person created or asked for never waits behind system
recovery. Waiting never fails or cancels it. A long wait moves it
forward, so it starts within a bounded number of dispatches. Recovery
enters the queue a few runs at a time. A recovery run that is no longer
needed is dropped quietly.

**Steps to reproduce**

1. Give an agent 20 slots, all busy.
2. Queue 60 agent assignments, then two system wakes on tasks people
created (a monitor check and a continuation), then 30 later person runs
on in-progress tasks, then a burst of 139 recovery wakes.
3. Free one slot at a time while people and agents keep adding work.
4. See that the monitor check starts only at dispatch 244.

## What Changed

- `server/src/services/heartbeat/queued-run-dispatch-order.ts`:
- System recovery has its own tiers below agent work (fresh, then
backlog). Aging moves old backlog only into the fresh recovery tier,
never ahead of agent work.
- The recovery share applies only when no ready person run waits. It
comes out of agent work, never out of a person's turn. Within the share,
the longest-waiting aged recovery run goes first.
- The oldest-run share serves the longest-waiting aged run that a person
asked for or whose task a person created (`issues.created_by_user_id`),
first. This includes a system continuation of such a task. Then it
serves the longest-waiting aged agent-work run. It never serves other
recovery.
- New dispatch fact `personTask`, read with the queued tasks' status and
priority (same query).
- `queuedRunDispatchLine` (the O(n log n) place-in-line order from
#21405) follows the same rules. `queued-run-line.test.ts` still proves
it equals the dispatcher's order over 400 random queues, now with
`personTask` in them.
- `server/src/services/heartbeat/queued-run-line.ts`: the start-history
counters use `dispatchHistoryStart` and `dispatchCountersFromStarts`, so
dispatch, the place-in-line view and the tests share one rule.
- New pure helpers `oldestShareCandidate`, `dispatchHistoryStart` and
`dispatchCountersFromStarts`. The queue and the tests use the same
counter rule (which includes the #21391 rule that a person's recovery
restart does not use up the recovery share).
- `server/src/services/heartbeat/queue.ts`:
- At claim, a system recovery run (`issue_recovery_action_restored`,
`issue_continuation_needed`, not requested by a person) is re-checked.
If the task is reassigned, or another run continues it (started after it
was queued, running, a queued non-recovery run, or a newer queued
recovery run), the run is cancelled before it starts with
`recovery_no_longer_needed` and its wake is skipped. This is the same
quiet path as the existing closed-task check: no failure, no provider
work. A person's undelivered message on the run still keeps it.
- `server/src/services/recovery-queue-admission.ts` (new):
`QUEUED_RECOVERY_RUNS_PER_AGENT` = 3 and the queue-depth check.
- `server/src/services/execution-recovery-resolution.ts`: a continuation
is admitted only while its agent has fewer than 3 queued recovery runs,
fresh or backlog. The 5-minute hold bound still orders held
continuations, but no longer overrides the cap. The rest stays pending
in its recovery action. The existing continuation pass (every 15 s while
any continuation is pending) admits the next one after a queued recovery
run starts or is dropped. No new polling.
- `server/src/services/recovery/legacy-workspace-rehome.ts`: the
pending-wake delivery for rehomed and released-writer tasks uses the
same cap. A held wake waits 60 s for its next turn and does not count as
a refusal.
- The oldest-run share counts as served only by a start that people wait
on while its candidate is one, so older agent work never uses up a
person's turn (`servesOldestShare`, used by the dispatcher, the
place-in-line order and the start history, which now records
`personWaiting`).
- `server/src/services/recovery/assigned-work-sweep.ts`: a rehome or
released-writer wake held by the cap (`HELD_RECOVERY_WAKE_KEYS`) counts
as the task's pending path while its assignee has queued recovery runs.
- `server/src/services/recovery/assigned-work-sweep.ts` (with #21502's
stalled-queued-run replacement): a start of work people wait on does not
count as passing over a queued run that people do not wait on. Waiting
behind people's work is intended, so the sweep does not cancel such a
run and replace it with a recovery continuation. Test:
`assigned-work-stalled-queued-run.test.ts`, "leaves an automatic run
that waits behind work on tasks people created". It fails without the
rule.
- `server/src/services/recovery/assigned-work-sweep.ts`: a pending
continuation still counts as the task's wake path past
`PENDING_CONTINUATION_BOUND_MS` while its agent has queued recovery
runs, so the sweep does not treat a held task as stranded.

**Bounds**

- A run a person asked for, or any run on a task a person created: once
it has waited past `QUEUED_RUN_AGING_MS` (1 hour), it starts within
`OLDEST_DISPATCH_SHARE` × (such aged runs queued before it + 1)
dispatches. Before that, runs a person requested already go first. There
is no time limit, and waiting never cancels it.
- Recovery run, proposed bounded behavior: at most 3 queued per agent.
When no person run waits, an aged recovery run starts within
`RECOVERY_DISPATCH_SHARE` × (aged recovery runs before it + 1)
dispatches. At start it is re-validated against the current task, and it
is dropped quietly if stale. While people keep the agent busy, recovery
waits. It is held durably in its recovery action, not lost and not
failed.

## Verification

Regression test that reproduces today's shape
(`server/src/services/heartbeat/queued-run-dispatch-order.test.ts`,
"deep per-agent queue: a recovery burst behind 20 busy slots"). It uses
20 busy slots, one dispatch per freed slot, 60 older agent assignments,
the two 07:29 system wakes on tasks people created (monitor check and
continuation), 30 later person runs, a 139-run recovery burst, and new
person and assignment arrivals on every step. The same model, run
against master's dispatcher and this one over 600 dispatches:

| | Before (master) | After |
| --- | --- | --- |
| Dispatch at which the 07:29 monitor check starts | 244 (≈12 h at ~20
starts/h) | 4 |
| Dispatch at which the 07:29 continuation starts | 5 | 8 |
| Recovery starts while a person-requested run waited | 89 | 0 |

With person-requested runs in the 07:29 slots instead (the first version
of the test), master starts them at dispatch 244 and 248, and this
change at 4 and 8.

- Red on master: the new dispatch tests fail (`expected -1 to be greater
than or equal to 0`, and recovery started ahead of agent assignments).
Green with this change.
- Admission cap, embedded Postgres (`recovery-wake-herd.test.ts`,
"queues at most a few recovery runs per agent..."): with the cap
removed, 12 of 12 continuations enter one agent's queue in 12 passes.
With the cap, 3 enter, the rest stay pending, and all 12 are delivered
as queued runs leave. The queue never holds more than 3 recovery runs.
- Older agent work (`queued-run-dispatch-order.test.ts`): with 60 older,
higher-priority assignments and no person arrivals, an aged
`issue_monitor_due` run on a person-created task starts within
`OLDEST_DISPATCH_SHARE` in dispatch and in the place-in-line order. With
the old reset rule it does not start in 8 dispatches.
- Held restored-task wake (`legacy-workspace-rehome.test.ts`): with the
agent's recovery queue full, the wake stays pending with no failure
counted, the sweep reads the task as `pending_intent`, and the wake is
delivered after a queued recovery run leaves.
- Start-time re-check, embedded Postgres, real heartbeat queue
(`heartbeat-stale-queue-invalidation.test.ts`): a recovery run whose
task was reassigned, or that a newer run already continued, is cancelled
with `recovery_no_longer_needed`, its wake is `skipped`, and the adapter
is never called. A recovery run that the task still needs starts.
- Updated tests that encoded the old rules: recovery's share and aging
now come out of agent work, not out of a person's turn. Sweep liveness
while held is covered.
- `pnpm --filter @paperclipai/server exec tsc --noEmit`: passes.

## Risks

- Behavior change: a system run on a task a person created now takes the
oldest-run share ahead of older agent work. On an agent whose queue is
mostly such tasks, agent-made subtasks get that share less often. They
still start in normal order.
- Behavior change: while people keep an agent fully busy, system
recovery for that agent waits. This is intended. Recovery is held
durably and starts once no person run waits.
- Behavior change: a queued system recovery run can be cancelled at
start as no longer needed. It is cancelled before any provider work,
with the same quiet outcome as a continuation whose task closed.
- A continuation held behind the cap stays pending longer than 7
minutes. The sweep now treats it as live while its agent has queued
recovery runs. If such queued runs can never start (for example
dependency-blocked), the next continuations wait behind them.
- One extra indexed read per recovery claim, on the existing
`(company_id, context_snapshot->>'issueId', created_at)` index. No
schema change. Mixed versions are safe: an old server keeps its old
order until it is replaced.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:21:35 -06:00
DottaandClaude Opus 5.5 a726dce4f0 feat(mcp): show live progress for long connected-app tool calls in the task chat (#21449)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call tools from connected apps through the Paperclip MCP
gateway
> - Some tool calls take a long time: exports, builds, large searches
> - The MCP progress utility lets an app report how far a call is, but
Paperclip ignored it
> - So the task chat showed only a spinner until the call finished, with
no sign of life
> - This pull request reads the app's progress and shows it on the
tool's row while the call runs
> - The benefit is that people can see that a long call is working, how
far it is, and how long it has run

## Linked Issues or Issue Description

**Subsystem affected**
Connected apps (MCP gateway) and the task chat.

**Problem or motivation**
A long tool call from a connected MCP app shows only a spinner in the
task chat. Paperclip does not send a `progressToken`, and it drops
`notifications/progress` from the app.

**Proposed solution**
Send a `progressToken` on each `tools/call` that an agent run makes.
Read the app's progress notifications from the call's response stream.
Show a progress bar (or an indeterminate sweep), the latest message, and
elapsed time on the running tool row. Keep the latest state so that a
reload mid-call shows it.

**Alternatives considered**
A new transport for progress. The existing company live-events socket
already carries per-run events with run read checks, so this change uses
it.

**Additional context**
Spec:
https://modelcontextprotocol.io/specification/2026-07-28/basic/utilities/progress

## What Changed

- `mcp-http.ts`: `mcpHttpRequestInit` can add `_meta.progressToken`.
`readMcpHttpResponse` gives each notification on the response stream to
an observer before the result. The observer cannot fail or delay the
call. New `mcpProgressNotificationOf` matches a notification to its
token.
- `tool-gateway.ts`: remote HTTP and local stdio `tools/call` from an
agent run send a per-call token and forward progress to a reporter.
Calls without a run send no token.
- New `server/src/services/tool-call-progress.ts`: checks each update
(finite, increasing, plain single-line text of at most 200 characters).
It publishes a `heartbeat.run.tool_progress` live event at most every
250 ms (latest wins) and saves the latest update on the invocation at
most every 1 s. Nothing is awaited by the call. A final `done` event
goes out when the call ends.
- DB: nullable `tool_invocations.progress` jsonb column (migration
`0479`, `ADD COLUMN` only). A save never writes onto a finished call.
- Live delivery: the new event follows its run like transcript output. A
client gets it only for runs it follows, after the run read check, in
the run's company only.
- New route `GET /api/issues/:issueId/tool-call-progress` (task read
check) returns the latest progress of the task's running calls, for a
reload mid-call. Documented in `docs/api/issues.md`.
- UI: `ToolCallProgressProvider` (one per chat) keeps the latest update
per call. A running tool row subscribes and shows a bar or sweep, the
message, the percent and elapsed time (design tokens, reduced motion
honored). Finished rows drop it and never fetch or re-render for
progress. `LiveUpdatesProvider` ignores the event for query
invalidation.
- `packages/kv-demo-mcp-server`: new `slow_with_progress` demo tool
(`steps`, `stepMs`, `withTotal`) in its own file.
- Storybook: `Tasks/Tool progress` stories (running with total, running
without total, done).
- Docs: new `doc/connections/MCP-PROGRESS.md`;
`MCP-PROTOCOL-VERSIONS.md` no longer lists progress as unsupported.

## Verification

- Red/green: on the original source, 5 of the new server tests fail
(token in `_meta`, SSE with interleaved progress and result,
notification matching, run-interest delivery, gateway end-to-end). With
this change they pass.
- `server/src/__tests__/mcp-http.test.ts` (13),
`tool-call-progress.test.ts` (9: throttling, latest wins, bounded text,
out-of-order drop, finish, failures never throw, company/run/tool
scoping), `tool-call-progress-store.test.ts` (2, embedded Postgres:
company and task scoping, no write onto a finished call),
`realtime/live-event-delivery.test.ts` (run interest),
`tool-gateway.test.ts` end-to-end against the real SDK kv demo server
over Streamable HTTP: 6 provider updates over about 1.2 s give 3 to 7
throttled events, all for the right company, run, task and tool, the
last one `done`; no event reaches another company; the invocation keeps
the progress; the result is unchanged.
- UI: `tool-call-progress.test.tsx` (10: name matching, row states,
plain-text rendering, drop on complete, restore after reload, no fetch
for finished rows). All 67 task-chat test files pass.
- `packages/kv-demo-mcp-server`: 13 tests, including progress over HTTP
with and without a total.
- Storybook: checked the three stories in a browser. The bar fills to
40%, the sweep animates, the elapsed time ticks, and the done row shows
no progress.
- `pnpm verify`: see the CI checks for the final revision.

## Risks

- Low. The migration adds one nullable column.
- An SSE response is now bounded per message, not per stream, so handled
notifications (such as progress) do not use up the result's size limit.
Each message is still bounded.
- Progress code runs only while a run's tool call is in flight. It adds
a few bytes to the request and throttled, fire-and-forget work. Agent
boot and time to first token do not change.
- A row finds its call by run and tool name. If one run calls the same
tool twice at the same time, both rows show the newer call's progress.
- Progress from a call that became an MCP Task is not shown.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:14:22 -06:00
DottaandClaude Fable 5.2 5d0b27acca fix(run-events): watch both late appends for rejection as soon as they start (#21536)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server stores each run event through `appendHeartbeatRunEvent`
and `appendHeartbeatRunEventBatch`
> - Pull request #21511 added a test that deletes a run while two
appends wait for its row lock
> - Both appends must fail, but the test attached the rejection check of
the second append only after it awaited the first
> - When the second append failed first, its rejection had no handler,
and Vitest failed the whole shard with one unhandled error
> - This pull request attaches both rejection checks when the appends
start
> - The benefit is that server test shards on master and on unrelated
pull requests pass again

## Linked Issues or Issue Description

Refs #21511

**What happened?**

Server test shards failed on master and on unrelated pull requests.
Every test passed. Vitest reported one unhandled error: `Error:
heartbeat_run_event_binding_mismatch`, from
`server/src/__tests__/heartbeat-run-event-atomic-append.test.ts`.

**Expected behavior**

The test awaits both expected failures. No rejection stays without a
handler.

**Steps to reproduce**

Run the general server test shard that contains
`heartbeat-run-event-atomic-append.test.ts` on master at `cefef8f11d`.
Examples: master "verify / General tests (server (1/10))", pull request
#21443 "ci / General tests (server (9/12))", pull request #21505 "ci /
General tests (server (1/12))".

## What Changed

- In the test "waits for Stop on the run row and appends after it, and
fails without rows after a run deletion", the two `expect(...).rejects`
checks are now created immediately after the two appends start. The test
awaits them after it releases the deletion.
- No product code changed. The product code rejects correctly; the test
left one rejection without a handler for a short time.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-run-event-atomic-append.test.ts`: 25 of 25 tests
pass, three runs in a row, no unhandled error.
- The failure before this change is the master shard named above. It
depends on which append rejects first, so I did not get a local
reproduction on demand.

## Risks

Low risk. The change is in one test. The assertions are the same.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 20:07:31 -06:00
DottaandClaude Opus 5.5 81d06cfc97 feat(mcp-tasks): add a demo MCP server that uses Tasks, test against it, and keep audit records out of agent errors (#21464)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call tools in connected apps through the tool gateway, which
speaks MCP to each app
> - MCP Tasks (#21263) lets an app answer a long tool call with a
durable task. It shipped behind the experimental setting
`enableMcpTasks`, off by default
> - No public app returns tasks without credentials, so nobody had seen
the feature work against a real server, and it was hard to understand
> - This pull request adds a small demo app that uses Tasks, tests the
gateway against it over HTTP, and records a live end-to-end run
> - The live run found one bug: every task stop sent the call's audit
record, with a server stack trace, to the agent. This pull request fixes
it
> - The benefit is a repeatable way to try and test Tasks,
plain-language docs, and evidence for the decision to turn the setting
on

## Linked Issues or Issue Description

Refs #21389
Refs #21263

## What Changed

- Add an MCP Tasks demo server to `packages/kv-demo-mcp-server` (new bin
`paperclip-tasks-demo-mcp-server`; run `node
packages/kv-demo-mcp-server/dist/tasks/main.js` after a build). It is
stateless MCP 2026-07-28 with `server/discover`, and it declares
`io.modelcontextprotocol/tasks`. It has six tools: `slow_report`,
`needs_approval` (form question), `will_fail`, `cancellable`, `expires`,
and `quick_echo`. It saves task state to a file, so it can restart
mid-task. It needs no credentials. It has a status page at `/` and JSON
at `/api/state`.
- I put it in the existing demo package, so there is no new workspace
package and no `pnpm-lock.yaml` change.
- Add `server/src/__tests__/mcp-tasks-demo-server.test.ts`. It runs the
real gateway against the real demo server over HTTP, with one test per
tool plus a restart.
- Fix: `server/src/routes/tool-gateway.ts` no longer sends
`details.execution` to callers. That record has request IDs and the
server-side error stack. The audit still keeps it. Add
`server/src/__tests__/tool-gateway-agent-errors.test.ts` (red before the
fix, green after).
- Docs: `doc/connections/MCP-TASKS.md` now has "In plain words", "What a
person sees", "Try it yourself", the live-run results, and a
recommendation. The package README documents the demo server.
- `enableMcpTasks` stays off by default.

## Verification

- `packages/kv-demo-mcp-server`: `npx vitest run` gives 25/25 pass (13
new demo-server tests). `tsc -p tsconfig.test.json` is clean.
- `server`: `mcp-tasks-demo-server.test.ts` gives 8/8 pass.
`tool-gateway-agent-errors.test.ts` gives 2/2 pass, and it failed 2/2
before the fix.
- Existing suites that cover the changed route and Tasks:
`mcp-tasks.test.ts` and `tool-gateway-discovery-http.test.ts` (results
in a comment below).
- `pnpm verify` exits 2, because the doc is a global CI input. Required
CI gives the broad coverage.
- Live end-to-end run on a local `test-drive` instance with
`enableMcpTasks` on. The agent was Codex `gpt-6-luna` at low effort. The
demo was connected through "Connect your own MCP server":

| Behaviour | Result | `tools/call` count at the app |
| --- | --- | --- |
| Polling (`slow_report`, 45 s) | Delivered after 22 polls | 1 |
| Question (`needs_approval`) | The card was answered in the UI.
`tasks/update` was sent, and the task finished | 1 |
| Failure (`will_fail`) | `provider_task_failed` with the app's message
| 1 |
| Cancel (`cancellable`) | Stop on the run sent `tasks/cancel` | 1 |
| Expiry (`expires`, 15 s) | `provider_task_expired`, no hang | 1 |
| Restart (`slow_report`, 120 s) | The server was down for 2 min 23 s.
The result was delivered once, and the agent was woken once | 1 |

After the run, all 6 `tool_mcp_tasks` rows were settled and none had a
next poll.

## Risks

- Low risk. The demo server is new code in a demo package, and the bin
is additive.
- The route change removes `execution` from tool-gateway error
responses. No UI code or test reads that field from these responses. The
audit and tool-call events still record it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:47:58 -06:00