Commit Graph
4209 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 e7f31705da perf(release): stage the verified host release daemon instead of compiling it in publish_canary (#21555)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every master push runs a canary, and only one canary runs at a time,
so its duration sets how often merged work ships
> - `publish_canary` takes about 30 min. 8 min of it is `pnpm build`,
and 4 min 40 s of that is an uncached `cargo build --release` of
`paperclip-runnerd`
> - The release then replaces that binary with the exact-source daemons
that `release_runner_binaries` already built and verified for all four
targets
> - This pull request stages the verified daemon for the publisher's own
target during a release, instead of compiling it
> - The benefit is about 4.5 min less per canary, with the same verified
bytes published

## Linked Issues or Issue Description

**What happened?**
In canary run 43487884481 (2026-11-13), the `Publish canary` step took
28.6 min. Its `pnpm build` ran 19:42-19:50. The
`packages/paperclip-runner build` took 5.3 min, and its `cargo build`
reported `Finished release profile ... in 4m 40s`. Every other package
waited for it. `release.sh` then runs `assemble-daemons`, which
overwrites `dist/bin` with the four exact-source release daemons.

**Expected behavior**
A release does not compile a binary that it discards. The Runner build
stamp stays valid, so the server build still reuses the Runner build.

Refs #21554 (the other canary speed-up)

## What Changed

- `packages/paperclip-runner/scripts/build-runner-binary.mjs` (new):
when `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR` is set, it checks the
release daemons with the existing `releaseDaemonManifest` (source
revision equal to `HEAD`, digest, architecture, build metadata) and
stages the host target's daemon at `dist/bin` with the existing
`stageRunnerBinary`. Otherwise it runs `pnpm run build:binary` as
before.
- The Runner `build` chain calls this script in place of `pnpm run
build:binary`. `build:binary` itself and the image build are unchanged.
- `stage-release-runner-binaries.mjs` exports `metadataContract`.
- Tests: `stage-runner-binary.test.mjs` (staging per target, wrong
source, bad digest, missing target, non-release fallback);
`runner-image-identity.test.ts` step list.
- `doc/RELEASING.md` describes the behavior.

## Verification

- `node --test
packages/paperclip-runner/scripts/stage-runner-binary.test.mjs`: 9 of 9
pass.
- `server`: `vitest run src/__tests__/runner-image-identity.test.ts`: 42
of 42 pass.
- With the real daemon artifacts of canary run 43487884481 (receipts
pointed at this commit): `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR=...
pnpm run build` in `packages/paperclip-runner` finished in 15 s with no
`cargo` line. The build stamp was recorded, and
`server/scripts/prepare-runner-vendor.mjs` then reported
`action:reuse,reason:inputs_unchanged`. The staged daemon runs.
- After merge: the next canary's `packages/paperclip-runner build` log
shows "Staged the verified exact-source linux-x64 release daemon" and no
`Compiling` lines.

## Risks

- Low. A release still fails closed: a daemon with the wrong source,
digest, architecture or metadata stops the build, and `assemble-daemons`
checks all four again afterwards.
- Local, Docker and CI builds without the variable are unchanged.
- On macOS, staging re-signs the binary, as it does for a compiled one.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:40:21 -06:00
DottaandPaperclip c17ef9249d fix(comments): keep saved authorization cards from blocking replies (#21744)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People use task comments to ask for help with pending requests.
> - Saving a comment also checks whether it replaces any pending
requests.
> - A stored request that fails today's payload schema can abort that
comment write.
> - The connection gateway could write such a request without validating
its generated target.
> - This change validates new authorization cards, reads older custom
identities, and keeps malformed requests from blocking comments.
> - Comments remain available while authorization and review gates
retain their authority.

## Linked Issues or Issue Description

**What happened?**

A connection authorization card can contain a generated custom target
key longer than 120 characters. The gateway writes the card directly.
Later, the interaction reader rejects it. Both the task interaction list
and comment submission then return a validation error. Other malformed
saved requests can also abort a comment write.

**Expected behavior**

People can ask questions on a task while authorization is pending. New
cards meet the shared schema. Older generated identities remain
readable. A malformed request never becomes approval because a person
posted a comment.

**Steps to reproduce**

1. Use a personal connection with a long UID and no grant for the
current user.
2. Invoke a tool. The gateway saves an authorization request with the
long UID in its target key.
3. Open the task or post a comment. Reading the saved payload fails the
key length limit.

Related PR searches found no existing repair for this defect. This is a
focused bug fix, with no roadmap feature overlap.

## What Changed

- Validate generated connection authorization payloads before either
insert or update. Use a stable SHA-256 target identity from the existing
connection/user idempotency key. Bound its display label.
- Read older confirmation custom keys and labels without the new-input
length limits. Preserve their exact identities. Keep target types, URLs,
document revisions, and resolver checks strict. New requests still use
the strict shared schema.
- During current and historical comment supersession, keep an invalid
saved request pending, report its ID and schema locations, and continue
with valid siblings. Database and other unexpected errors still
propagate.
- Keep this behavior in the gateway writer and interaction owner. The
two existing source files exceed 1,000 lines; the additions extend their
existing write/hydration boundaries rather than introduce another
ownership path.

## Verification

- Reproduced the deployed failure through the browser: comment POST and
interaction GET both returned HTTP 400 for `target.key` exceeding 120
characters. The person's draft was restored.
- Ran the new regressions before the repair: four expected failures and
one short-identity control pass. Failures came from real payload parsing
and real PostgreSQL comment writes, not fixture setup.
- After repair, all 257 tests passed in the complete affected suites:
`issue-thread-interactions-service.test.ts`, `tool-gateway.test.ts`, and
`issue-thread-interactions.test.ts`.
- Safety cases cover fresh and reused authorization cards, repeated
missing-grant calls, refusing another user's credentials, two persisted
follow-ups, strict new-input limits, wrong-company resolution, unsafe
URLs, invalid document revisions, malformed question and confirmation
payloads, and valid sibling supersession.
- Local browser acceptance used an isolated, credential-free source
instance on macOS arm64 with embedded PostgreSQL and Chrome. A card
shaped like an older saved long-key authorization request remained
visible. Two comments saved through Send, including a second after
reload. The original comment persisted and the authorization request
remained pending. No model execution was configured.
- Browser acceptance was repeated on final commit
`58b57dfe104b2ab421c7f24c89db4c29fcfda8ec` after rebase. The server
health and ownership check confirmed that exact serving revision. Both
comments persisted and the authorization stayed pending. Both owned
instances were then stopped. All 257 affected tests passed again after
rebase. The final fixture typing correction passed its six affected
PostgreSQL cases. Greptile reviewed final head
`58b57dfe104b2ab421c7f24c89db4c29fcfda8ec` at 5/5, with zero review
threads or open findings. Required CI is green on that head: aggregate
`ci / verify` and `ci / e2e` both passed. The repository CI run is
https://github.com/paperclipai/paperclip/actions/runs/43710675824 . All
55 completed checks passed; four inapplicable checks were skipped. The
non-required Superagent Security Scan remains in progress at 35%, with
no annotations or reported findings. This external scan is still
unverified; it was not retried or marked as passed.
- `git diff --check` and the module boundary check pass. All seven local
server typecheck layers pass on the final fixture revision. An earlier
run found fixture typing errors; those were corrected and rerun
successfully. Full repo tests, release qualification, and deployment
were not run.

## Risks

- Stored custom keys and labels may exceed current input limits. Only
those length limits change for saved confirmation payloads; unsafe links
and invalid document targets still fail validation.
- A malformed request stays pending and still needs repair. It does not
receive consent, disappear, or block a comment. Its read or resolution
can still return a validation error.
- This PR does not deploy the fix or recover a remote task. The original
environment remains affected until rollout.

## Model Used

OpenAI GPT-6-based Codex, with code editing, shell execution, browser
tools, and PostgreSQL-backed tests. The host does not expose a more
specific model version or context size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
(behavior and limits are recorded in this PR)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-15 06:10:35 -06:00
DottaandClaude Opus 5.5 689d62661d fix(server): wake a delivery task unscoped when it cannot apply settled hints (#21686)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The delivery work coordinator wakes background tasks when a database
work topic settles, and passes the settled scope hints to the task's
`onSettled`
> - The coordinator calls `onSettled(hints)` before `wake()`
> - If `onSettled` throws on a hint that it does not recognise, the task
is not woken. Only a generic listener warning is logged
> - Hints now cross server processes (#21681), so a hint shape from
another server version can arrive during a deploy overlap
> - This pull request makes the coordinator log a warning, apply the
settlement unscoped once, and always wake the task
> - The benefit is that a hint that a task cannot read never makes the
task miss committed work

## Linked Issues or Issue Description

Follow-up to #21681 (merged). This branch is based on master.

**What existing behavior does this improve?**
How the delivery work coordinator handles an `onSettled` callback that
throws.

**Subsystem affected**
`server/src/services/delivery-work-coordinator.ts`.

**Current behavior**
`subscribeDatabaseWork` calls the coordinator listener, which calls
`task.onSettled?.(hints)` and then `wake()`. If `onSettled` throws,
`wake()` does not run. The work-signal publisher catches the error and
logs "Database work listener failed". The task stays asleep until a
different event or deadline wakes it.

**Proposed behavior**
If `onSettled(hints)` throws, the coordinator logs a warning and calls
`onSettled(undefined)` once, which is the unscoped settlement. If that
also throws, it logs a second warning. In all cases it then wakes the
task.

**Reason and benefit**
A settled event means committed work can be ready. A task must never
miss it because a hint could not be applied.

**Breaking changes**
None.

## What Changed

- `delivery-work-coordinator.ts`: a settled event goes through
`applySettled(hints)`. It tries `onSettled(hints)`. If that throws and
hints were given, it tries `onSettled(undefined)` one time. Each failure
logs `Delivery settlement could not be applied for <queue>` with
`process.emitWarning`. Then `wake()` runs.
- Unscoped means `onSettled(undefined)`. Each registered task already
defines what that means. For example, scoped deadline queues, dispatch
work, muse maintenance and the revocation index mark `*` dirty.
Counter-style tasks set their request flag. `chat-action-work` does
nothing. The fallback is bounded: it calls `onSettled` at most twice per
event, it does not recurse, and it does not schedule another settlement.
So it cannot loop.
- `work-signals.test.ts`: a new test pins one rule of #21681 for the
local path and the remote path. An unhinted `signalDatabaseWork` in the
same transaction as a hinted one gives a settlement that is scoped to
the hinted set. The owner of `work-signals.ts` confirmed that this is
deliberate. The repository does not allow code comments in pull
requests, so this sentence and the test document it.

## Verification

- `cd server && npx vitest run
src/__tests__/delivery-work-coordinator.test.ts`: 31 pass. New tests:
- An `onSettled` that throws on an unknown hint shape: the task gets
`[hint]` and then the unscoped settlement, it is woken once, and one
warning is logged. `onError` is not called.
- An `onSettled` that always throws: the task gets 2 calls and is woken
once, with 2 warnings. A later unscoped event gives 1 call, 1 wake and 1
warning. No more runs happen after 60 seconds, so there is no loop.
- A normal `onSettled`: it gets the hints once, the task is woken once,
and no warning is logged.
- Red/green: with the coordinator from master, the 2 throwing tests
fail. `onSettled` is called only with the hints, and there is no
unscoped settlement and no wake. With this branch, all tests pass.
- `cd packages/db && npx vitest run src/work-signals.test.ts -t
alongside`: passes on real PostgreSQL. The local listener and the
receiving process both get `["company-1"]`.

## Risks

- Low risk. The change applies only when `onSettled` throws. Before,
that path skipped the wake. Now it wakes with an unscoped settlement.
For scoped deadline queues an unscoped settlement is a `*` reload. That
is the existing safe fallback, and the reconstruction interval of the
queue still throttles it.
- A task whose `onSettled` throws on every event now runs on every
settled event and logs a warning each time. Before, it did not run. The
warning makes such a defect visible.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:02:54 -06:00
DottaandClaude Opus 5.5 83541e89ad perf(recovery): re-check only changed tasks between hourly full backstop passes (#21729)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues on one Node main thread.
`stranded-assignments` and `assigned-work-invariant` find assigned tasks
that have no live run and restart them.
> - #21552 scoped run hints to the run's task graph and kept a company
"backstop" pass to catch a missed hint. Any graph hint arms the
backstop, and the backstop runs at most every 5 minutes after the last
full pass.
> - The backstop reloads and re-checks every open assigned task of the
company. With any trickle of activity it runs every 5 minutes. On the
hosted staging instance that is about 850 tasks every 5 minutes, about
2.8 checks per second at idle, close to the measured 3.2 per second.
That is work proportional to history, not to activity.
> - This pull request makes the 5-minute backstop re-check only tasks
whose own rows changed since the previous look, with one set-based
query. A full company pass runs at most once an hour, still armed only
by activity.
> - The benefit is no re-checks between hourly passes when nothing
changed: idle statements drop from 59 per second to 5.5 per second in a
staging-shaped fixture. A task whose own state changed without a hint is
still recovered within 5 minutes.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The missed-hint backstop of the `stranded-assignments` and
`assigned-work-invariant` recovery queues
(`server/src/services/recovery/work.ts`).

**Subsystem affected**

Server: issue recovery background work.

**Current behavior**

Each graph hint arms a `{companyId, backstop: true}` scope. When it
opens (5 minutes after the last full pass), the queue loads every open
assigned task of the company and checks each one again: about 22
statements per task for the stranded check. With a hint at least every 5
minutes, this repeats every 5 minutes, whatever changed.

**Proposed behavior**

Between full passes, the backstop loads only the tasks whose own rows
changed since the previous look (minus a 5-minute overlap for late
commits): the task row, its runs, its recovery actions, its wake
requests, its thread interactions, its comments, or its assignee agent.
A full company pass runs when the last one is an hour old. A real change
still sends its own hint and gets an immediate targeted check.

**Reason and benefit**

On staging at idle, the stranded check runs about 3.2 times per second
with zero runs. A fixture with staging's population shape attributes all
of it to the backstop.

## What Changed

- `server/src/services/recovery/work.ts`:
- `fullPasses()` keeps the time of the last full pass and of the last
changed-tasks pass per company. The backstop gate is 5 minutes after the
later of the two.
- A backstop scan decides on its first page: a full load when the last
full pass is at least `ISSUE_RECOVERY_FULL_BACKSTOP_MS` (1 hour) old,
otherwise a changed-tasks load since the previous look minus
`ISSUE_RECOVERY_BACKSTOP_MS`. All pages of one scan use the same
decision.
- Each scan records its start time on its first page. A completed scan
advances the saved full or changed time only to that start, so a change
behind the cursor of a slow scan falls in the next window.
- The stranded queue adds `changedTaskCondition(since)` to its load
query. The assigned-work queue passes `changedSince` to `listSuspects`.
Both return `keepUnseen: true` for a changed-tasks load.
- A new `fullBackstopMs` option. When a caller sets only `backstopMs`
(tests), every backstop stays a full pass, as before.
- With `PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments`, each `check`
event now names why the task was loaded: `deadline`, `reload` (unscoped
reconstruction), `backstop`, `company` (company-wide hint) or `hint`
(task or graph hint).
- `server/src/services/recovery/changed-tasks.ts` (new): the set-based
"changed since" condition. Each part uses an existing index: `issues
(company_id, updated_at)`, runs by `(company_id, context issueId)`,
wakes by `(company_id, payload issueId)`, interactions and comments by
`(company_id, issue_id)`.
- `server/src/services/scoped-deadline-work.ts`: a load result may set
`keepUnseen`. Then a completed scan does not drop the pending entries it
did not load, so their saved deadlines stay.
- `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects`
accepts `changedSince`.
- `server/src/services/issue-recovery-notifications.ts`:
`ISSUE_RECOVERY_FULL_BACKSTOP_MS = 60 * 60_000`.
- Tests:
- `server/src/__tests__/issue-recovery-idle-backstop.test.ts` (new): the
staging-shaped fixture, see Verification.
- The staging-shaped population helper and the fixture's statement meter
come from #21724, now merged into this branch.
- `server/src/__tests__/issue-recovery-work-debug-attribution.test.ts`:
the check event now names its load reason (`hint` or, when the backstop
loaded the task last, `backstop`) instead of `load`.
- `server/src/__tests__/issue-recovery-deadlines.test.ts`: the backlog
helper writes its rows with the test's fake clock, as production writes
use the server clock. Without it, rows stamped with the database clock
look older than the fake-clock window.

## Verification

- Fixture (`issue-recovery-idle-backstop.test.ts`, embedded PostgreSQL,
the real recovery, dispatch and retry-cancellation workers, fake clock,
statement meter, `PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:1`):
one company, 50 agents, 780 in-review tasks with succeeded native runs
and no execution state, 40 in-progress and 30 to-do tasks of a paused
agent, 12 tasks with an active agent-owned recovery action, 3 tasks of a
terminated agent. After boot, one unrelated graph hint per minute stands
in for idle activity.
- Wake reasons on master over 30 idle minutes with that trickle: 5,100
stranded checks, all `backstop`; 0 from hints, 0 from company hints, 0
from deadlines. That is 2.83 checks per second and 106,632 statements
(59.2 per second). With no hint at all after boot, master still made one
backstop pass (850 checks) in 30 minutes, armed by boot activity.
- This branch, same fixture:

  | | master | this branch |
  | --- | --- | --- |
  | Stranded checks, first 30 idle minutes | 5,100 | 0 |
  | Statements, first 30 idle minutes | 106,632 | 1,044 |
| Stranded checks, one idle hour (one full pass) | about 10,200 | 850 |
  | Statements per second over that hour | 59.2 | 5.5 |

- Missed hint: a task inserted with no hint is checked and gets a run
within 5 minutes plus one pass (`ISSUE_RECOVERY_BACKSTOP_MS + 2 min`
asserted).
- A change outside the task's own rows (its agent resumed by a raw
update that sets no `updated_at`, with no hint) is not seen by the
5-minute pass. It is checked and gets a run after more than 10 minutes
and within 62 minutes, through the hourly full pass.
- Prompt pickup with the backstop closed: a finished run
(`notifyNativeStatusReassessment`), a user comment, a hold release
(`pause_lifted`, which a budget raise and an agent resume send) and an
agent resume each get a check of the affected task within 60 seconds.
The company-wide events re-check the whole company at 25 tasks per pass,
as before.
- Two regression tests in `issue-recovery-deadlines.test.ts`, red on
379dda9e26 and green on 802a76c153:
- "keeps a saved deadline of a task that a changed-tasks backstop pass
does not load": a task with a 16-minute saved deadline is checked at
that deadline after two changed-tasks passes that did not load it.
- "starts the next changed-tasks window from the start of a slow
backstop scan": each check advances the clock by 20 s, so the scan spans
more than 10 minutes. A task changed behind the cursor early in the scan
is read by the next changed-tasks pass.
- #21552 backstop tests in `issue-recovery-deadlines.test.ts` pass,
including "finds a stranded task outside the affected set within the
backstop bound", which now runs with the production defaults.
- Regression, embedded PostgreSQL: issue-recovery-idle-backstop,
issue-recovery-deadlines, scoped-deadline-work,
assigned-work-recheck-idle, live-recovery-work (98 tests; "resolves a
run's task graph through index lookups" failed once under 3 parallel
workers and passes alone; it does not touch this change);
live-recovery-work, assigned-work-recheck-idle, assigned-work-invariant,
recovery-deadline-acceptance, issue-recovery-recorded-decision-idle,
continuation-delivery-unwakeable-agent-idle.
- `tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- **The missed-hint bound changes for one class of change. The owner
must see this.** A change that touches the task's own rows (task, runs,
recovery actions, wakes, interactions, comments, or the assignee agent's
row) and sends no hint is still recovered within 5 minutes plus one
pass. A change elsewhere that sends no hint (company or budget rows,
tree holds, an execution-state participant agent, or a write that does
not set `updated_at`) is now recovered within 60 minutes plus one pass,
where it was 5 minutes. Every such writer that I found sends a company
or graph hint, which still triggers an immediate check.
- The changed-tasks window trusts `updated_at` stamps. A transaction
that commits more than 5 minutes after its stamp can be missed until the
next full pass.
- The fixture cannot run the full set of workers that `index.ts`
registers. It runs the recovery, dispatch and retry-cancellation
workers, and stands in for other activity with one graph hint per
minute.
- Not changed here: a company-wide hint (`{companyId}` with no task)
still reloads and re-checks every task, at most every 30 seconds. About
20 writers publish one, for example `documents.ts`, `task-watchdogs.ts`,
`issue-tree-control.ts`, `issues.ts`, `heartbeat/queue.ts` and
`run-dispatch-notifications.ts` without an issue. In the fixture these
came only from boot escalations, not at idle.
- At boot, a graph hint that arrives before the first full
reconstruction completes still opens one full backstop pass.
- master (with #21724) is merged into this branch. The changed-tasks
backstop and #21724's `beforeChecks` pass with its shared review-wait
read work together: the focused tests of both pull requests pass on the
merged tree (107 tests). On the merged tree the hour that holds the full
pass reads 7.7 MB instead of 15.0 MB, because of #21724's lean reads.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 05:50:59 -06:00
DottaandClaude Opus 5.5 07d5a6ee17 perf(db): carry scope hints in cross-process work notifications (#21681)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Server processes share durable queues and wake each other with a
commit-bound PostgreSQL NOTIFY
> - The NOTIFY payload carries only `{version, origin, topic}`, so the
receiver gets an unscoped "settled" event
> - In `scoped-deadline-work.ts` an unscoped settle becomes `*`, which
reloads every candidate of that queue on the whole instance
> - On a large instance one cross-process notification can re-check
about 850 open tasks, at about 22 statements each
> - This blocks more than one server process per stack, and it costs
work during every deploy overlap
> - This pull request puts the topic's hints in the NOTIFY payload, so
the receiver reloads only the hinted scopes
> - The benefit is that a scoped commit in one process costs the same
scoped reload in every other process

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Cross-process work notifications between server processes on one
database.

**Subsystem affected**
`packages/db` work signals (`work-signals.ts`) and their consumers, such
as `server/src/services/scoped-deadline-work.ts`.

**Current behavior**
`receiveDatabaseWorkNotification` publishes "settled" with no hints. A
scoped deadline queue then reloads all of its candidates, throttled only
by its reconstruction interval.

**Proposed behavior**
The NOTIFY payload carries the deduplicated hints of its topic. The
receiving process publishes "settled" with the same hints that a local
commit publishes. Overflow and malformed input fall back to the old
unscoped behavior.

**Reason and benefit**
A scoped commit no longer causes an instance-wide reload in other
processes. This is necessary before a stack runs more than one server
process.

**Breaking changes**
None. Old and new processes interoperate in both directions (see Risks).

## What Changed

- `packages/db/src/work-signals.ts`: the per-level pending notifications
are now a map of topic to `{ hints, unscoped }`. Savepoint merge and
rollback keep the same rules as before.
- The hints follow the local "settled" semantics: the union of the hints
from `signalDatabaseWork` and `publishDatabaseWorkSettled`. An unhinted
`publishDatabaseWorkSettled`, `notifyDatabaseWorkOnCommit`, or a topic
with no hints sends the unscoped payload.
- A hinted payload is `{version: 1, origin, topic, hints: [...]}`. The
field is additive and the version stays 1.
- Each payload is at most 1,024 bytes
(`DATABASE_WORK_NOTIFICATION_MAX_BYTES`), because current receivers drop
any payload longer than 1,024 characters before they parse it. Many
hints split into several payloads. The payloads of one topic are at most
7,000 bytes in total (`DATABASE_WORK_TOPIC_NOTIFICATIONS_MAX_BYTES`). If
a topic exceeds this cap, or one hint alone does not fit, the topic
sends one unscoped payload.
- All payloads still go in the one batched `select pg_notify(...) from
jsonb_array_elements_text(...)` statement at commit. There are no added
statements or awaits on the commit path.
- `receiveDatabaseWorkNotification` accepts payloads up to 8,000
characters (the PostgreSQL limit). It keeps the drop of own-origin
payloads and of payloads without a usable origin or topic. A missing,
malformed, or empty hint list, or a version other than 1, gives an
unscoped wake. It never throws.
- The `remote_notification` debug event now logs the received hints
(`hints`: a JSON array, or `null` when unscoped).
- Tests: new payload, compatibility and fallback tests in
`packages/db/src/work-signals.test.ts`, and a new two-process test on
real PostgreSQL in
`server/src/__tests__/scoped-deadline-work-notifications.test.ts`.

## Verification

- `cd server && npx vitest run
src/__tests__/scoped-deadline-work-notifications.test.ts`: two pools on
one embedded PostgreSQL database. Process A commits work. Process B runs
a real delivery coordinator with `registerScopedDeadlineWork` over 800
candidate rows in 20 companies.
- Master (red): a commit hinted with one company makes B load `*`: 1
load, 801 rows.
- This branch (green): B loads only `company-7`: 1 load, 41 rows. A
commit with two hints loads only those two companies (80 rows). An
unscoped commit still loads all 801 rows.
- `cd packages/db && npx vitest run src/work-signals.test.ts` (32 tests,
real PostgreSQL). New tests:
- deduplicated hints per topic, including a released savepoint and a
rolled-back savepoint
- split into payloads of at most 1,024 bytes, at most 7,000 bytes in
total, and the union of hints is complete
  - overflow and one oversized hint fall back to one unscoped payload
- malformed hint lists and unknown versions wake unscoped; unusable
payloads do not wake; own-origin payloads are dropped; nothing throws
- an old-format payload `{version, origin, topic}` gives an unscoped
wake
- every new payload passes a copy of the receiver from master, which
reads it as an unscoped wake for the correct topic
  - the debug event logs the received hints
- With master's `work-signals.ts`, 5 of these tests fail (red). With
this branch, all pass.
- Also run: `scoped-deadline-work.test.ts`,
`delivery-work-coordinator.test.ts`, `deployment-notifications.test.ts`,
`run-start-handover.test.ts`, `heartbeat-run-start-handover.test.ts`,
`heartbeat-run-event-atomic-append.test.ts` (111 tests pass),
`work-signal-debug.test.ts` (12 pass), and `tsc --noEmit` for
`packages/db`.

## Risks

- Mixed deploys, new sender to old receiver: every payload is version 1
and at most 1,024 bytes. The old receiver ignores the `hints` field and
wakes unscoped, as today. A commit with many hints sends several
payloads, so an old receiver can get up to about 7 unscoped wakes for
one commit. The coordinator merges them into one `*` reload.
- The sender bounds bytes, and old receivers compare `.length` in UTF-16
code units. A string never has more UTF-16 code units than UTF-8 bytes,
so a payload of at most 1,024 bytes always passes the old check.
- PostgreSQL collapses identical NOTIFY payloads in one transaction. The
chunks of one topic always carry different hints, so PostgreSQL drops
none of them.
- Mixed deploys, old sender to new receiver: the old payload has no
`hints`, so the new receiver wakes unscoped, as today.
- I did not use one large payload of about 7 KB: current receivers drop
any payload over 1,024 characters before they parse it, so they would
miss the wake during a deploy overlap.
- Consumers that read hints (for example `chat-action-work`,
`approval-queue-seeding-work`) now get the same hints from another
process that they already get from a local commit. Before, they got no
hints from another process.
- The receiver accepts hint strings from the database channel. Consumers
decode them in the same way as local hints. `scoped-deadline-work`
already turns a decode failure into a `*` retry.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 05:38:06 -06:00
DottaandClaude Opus 5.5 819f61f1fd fix(evals): repair four cohort-15 chat cases and prefix board links in completion feedback (#21682)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat-feature eval catalog runs real chat journeys. It grades
saved state, rendered links and run ledgers.
> - Chat-eval cohort 15 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `36101ebcb9`)
failed four cases where the agent did the right thing or where our own
server text misled it.
> - One failure was a harness race introduced by #21667. Two failures
came from graders that read free-form text with keyword regexes. One
failure came from an unprefixed approval link in the server's completion
feedback, plus a grader that required a literal prefix.
> - This pull request fixes the harness step, replaces the free-form
graders with saved-state and clicked-link checks, and makes the server
emit company-prefixed board links.
> - The benefit is that these cases measure the feature again. The
retained correct outcomes pass, and the relevant broken outcomes still
fail.

## Linked Issues or Issue Description

Refs #21667

**What happened?**
In chat-eval cohort 15, four cases failed:
- `projects.share-project`: the harness followed the agent's ID-based
project link. The app then rewrote the URL to the project slug and
redrew the page. The owner step timed out because Riley's option in the
people picker was "detached from the DOM". #21667 introduced this path.
- `skills.author-skill`: the saved skill was a correct, richer
checklist. Per-sentence keyword regexes rejected it. For example,
"ignore the mouse" tripped the forbidden word "ignore".
- `inbox-search.show-decisions`: the reply linked the right approval and
tasks and said the question was "addressed to a different teammate". A
regex required the word "Riley".
- `agents.hire-needs-approval`: both replies linked `/approvals/<id>`
without the company prefix. The agent copied that link from
`server/src/services/native-runtime/native-completion-feedback.ts`. The
grader required a literal `/<prefix>/approvals/<id>`, although the board
redirects the unprefixed route.

**Expected behavior**
Correct agent behaviour passes. Grading uses saved state, setup literals
and rendered or clicked links, not free-form wording. The server gives
agents canonical company-prefixed board links.

**Steps to reproduce**
1. Use the retained cohort-15 evidence for the four cases.
2. Run the old graders on the retained skill markdown and decisions
reply. Both fail.
3. For share-project, open a project configuration page by ID while the
app rewrites the route after a delay, then use the access picker before
the rewrite completes.
4. For hire-needs-approval, finish a task with a pending linked
approval. The feedback links `/approvals/<id>`.

**Paperclip version or commit**
master `36101ebcb9` (cohort 15); fixed on top of current master.

## What Changed

- `projects.share-project` (`internal-privacy.ts`,
`runtime-support.ts`): after the harness follows the agent's link, it
waits until the URL settles on the project's canonical route key. This
reuses the #21667 approach for agent profiles: `settleAgentProfileRoute`
now calls a shared `settleRouteKey`, and `projectRouteKey` follows the
app's `projectRouteRef`. The owner's picker selection
(`addProjectAccessMember`) now retries as one unit. If the dialog or
picker closes, or an option detaches, it reopens them, searches again
and selects again. It continues only when the picker shows the person
and Add is enabled.
- `skills.author-skill` (`integrator-skills.ts`): removed the
keyword-regex checklist grader, the waiver regexes, and the reply
regexes for "global activation" and "attached to me" claims. The
saved-state checks stay: one agent-created writable skill, its current
revision authored by the responding agent, the same bytes on the skill
and the revision, the Skills/Studio inspection with reload, and
assignment scope from saved assignments and audit rows. The checklist
check now requires the setup literals `375`, `WCAG` and `AA` as whole
tokens in the saved SKILL.md and in the rendered editor. Whether the
skill still asks for keyboard-only navigation and visible labels, and
whether it waives a check, is left to human review. Oracle version
`skill-authoring.v10`.
- `inbox-search.show-decisions` (`internal-decision-search.ts`): removed
all reply regexes, including the `Riley` requirement and the
wording-based separation checks. The grade now uses saved state (pending
decisions unchanged, answered history unchanged, each person's own
attention queue and resolver) and rendered links to the exact approval,
the question's task and the history task. Oracle version
`decision-search.v3`.
- Product (`native-completion-feedback.ts`): the completion feedback now
links `/<prefix>/approvals/<id>` and `/<prefix>/issues/<identifier>`.
This fixes the approval link and the four sibling task links in the same
file.
- `agents.hire-needs-approval` (`runtime-hiring.ts`): the harness now
clicks each reply's rendered link to its approval and records where the
UI lands. The grade accepts a prefixed link, or an unprefixed link that
the board redirect lands on `/<prefix>/approvals/<id>` with the proposed
hire visible. Without such a link, the harness still opens the approval
directly so later phases keep their evidence, and the grade fails.
Oracle version `runtime-hiring.v4`.
- No eval YAML changed, so the TestOps manifest does not change.
`chat-catalog-descriptors.json` records the three oracle version bumps.

## Verification

- Red/green tests (each new test fails on the old code and passes on the
new code):
- `internal-privacy-share-route.test.ts` (real Chromium): the project
page rewrites its route after 0, 400 and 1500 ms, and one case closes
the picker while its first results render. With the old flow, the 400 ms
rewrite and the closed picker both time out on the option click (the
cohort-15 failure). With the fix, Riley is added on the settled route in
all cases.
- `integrator-skills.test.ts`: the retained cohort-15 SKILL.md passes.
The old grader rejects it. A stale skill body, another author, and a
lost 375 width still fail.
- `internal-decision-search.test.ts`: the retained cohort-15 reply and a
reworded reply pass. The old grader rejects the retained reply. A
missing or wrong approval, question-task or history-task link fails.
- `native-completion-feedback.test.ts`: a pending linked approval
produces `[the pending approval](/<prefix>/approvals/<id>)`, and a
pending review stage produces a prefixed task link. The old code
produces `/approvals/<id>`.
- `runtime-hiring.test.ts`: unprefixed links that the redirect lands on
the right approval pass. The old grader rejects them. A link that did
not land, landed on another approval, was not clicked, or is another
href still fails.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog`: all passed (one negative control in `internal.test.ts`
changed from a reply wording to "approved while listing").
- `pnpm test:e2e:runner:typecheck` and `server` `tsc --noEmit`: passed.
- `server` vitest for `native-completion-feedback`,
`native-current-wake-completion-feedback` and
`heartbeat/run-completion`: 201 tests passed.
- Narrow paid run of the four full case ids on this branch: run
43657390762 (results are added in a comment).

## Risks

- Low risk. The product change only adds the company prefix to links in
agent feedback text. It adds one indexed company read, and only on the
branches that return a link.
- Less is graded automatically, by design. `skills.author-skill` no
longer checks that the skill keeps the keyboard-only and visible-label
checks or that it does not waive a check. `inbox-search.show-decisions`
no longer checks how the reply names resolvers or separates the person's
decision from others' and from history. `skills.author-skill` no longer
checks whether the reply describes the assignment scope truthfully. A
person reviews this meaning. Saved state and links still catch the
plausible broken outcomes listed above.
- `agents.hire-needs-approval` now needs a reply link that really opens
the approval. A reply without a link to its approval fails as before.

## Model Used

- Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use
and code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 05:34:56 -06:00
DottaandClaude Opus 5.5 ca23330c69 fix(recovery): continue a task on its own when its saved files are permanently lost (#21541)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops early, Paperclip recovers its saved files and then
continues the task automatically
> - Sometimes the sandbox is gone. Then saved-work recovery ends with
`source_unavailable` (use another strategy)
> - In that state the task showed Retry. Retry worked, because it
continues in a fresh workspace. But nothing continued the task
automatically
> - A task must never need a person's Retry for a system failure. The
target is 0%
> - This pull request lets the system take the same fresh-workspace step
that Retry takes, when the saved files are permanently lost
> - The benefit is that these tasks continue on their own, and the sweep
reports them correctly

## Linked Issues or Issue Description

Refs #21478

**What happened?**
A legacy run on a remote sandbox failed in setup. Its unsaved files were
held for saved-work recovery. The sandbox was gone, so the recovery
ended `source_unavailable` / `use_alternative_strategy`. The task then
showed Retry. For about 2 hours, the assigned-work sweep logged
`retry_offered` with outcome `not_started` every 15 minutes. A person
had to press Retry.

**Root cause**
The fresh-continuation fallback returned `person_owned` /
`retry_needed`. It does this when a later run of the task was withheld
at dispatch (here, a scheduled retry that the hold blocked) and the
source run does not continue automatically. The source was a person's
earlier Retry that failed with `setup_failed`, so it was not an
automatic recovery run and not a restart-class failure. Without the
withheld run, the same task already continued automatically.
Restore-hold ownership (`isOwnedReconciliationHold`) and the legacy
refusal in `inspectFallback` were not the blocker.

**Expected behavior**
When saved-work recovery proves that the files are permanently gone, the
system continues the task in a fresh workspace on its own. This is the
same path that Retry takes.

## What Changed

- `fresh-continuation-fallback.ts`: the eligibility rule
(`interruptionEligibility`, which the fallback and the backlog pass
share) also accepts a system-owned source run when a hold of that run
has saved-work recovery `exhausted` with a permanent unverifiable-source
failure (`UNVERIFIABLE_SOURCE_CODES` ×
`PERMANENT_SOURCE_RETRY_CLASSES`).
- `recovery/assigned-work-sweep.ts`: admission can settle the hold and
coalesce the sweep's wake into a pending fresh continuation. The sweep
now reports this as `repaired` / `queued` (`continuedFromSavedWork`).
Before, it reported `enqueue_refused` / `not_started`.
- `legacy-execution-recovery.ts`: the stranded sweep calls
`terminalizeLegacyExecution` for the same failed run on every pass. Its
restore-hold branch rewrote the hold every time, with a new
`automaticRecovery.recordedAt`, a new `updatedAt`, and
`continuationDelivery: "invalidated"`. The branch now returns early when
the hold already records the same no-replay decision for this run, or
when a fresh continuation took the hold over. Finding: a continuation
that was already recorded was not clobbered, because the existing
`executionReconciliation.runId` guard returned early. But a hold with a
`coalesced` delivery state would have been reset, and the churn kept
bumping `updatedAt`.
- New regression test `legacy-source-unavailable-continuation.test.ts`.
The existing `assigned-work-invariant` test now expects the corrected
sweep outcome.

## Verification

- Red/green: `legacy-source-unavailable-continuation.test.ts` uses
`failedRemotePreparation()` with a `destroyed` termination receipt,
`recoverRetainedWorkspace`, a withheld scheduled retry, and a source
that was a person's Retry. Then it runs
`assignedWorkSweep(...).checkIssue` with the real
`heartbeatService(db).wakeup`.
- Without the fix: outcome `enqueue_refused`, and the fallback returns
`person_owned` / `retry_needed`.
- With the fix: outcome `repaired`. A stranded pass keeps
`continuationDelivery: "pending"`. `deliverReconciledExecutions` starts
the successor run. The task no longer shows Retry. There is no
`not_started` entry.
- Red/green: the "revisits" test fails without the
`terminalizeLegacyExecution` guard, because the hold evidence changes.
It passes with the guard.
- Locally passing: `fresh-continuation-fallback`,
`fresh-continuation-move-forward-offer`,
`fresh-continuation-retry-delivery`, `legacy-execution-recovery`, both
`assigned-work-invariant` suites, `assigned-work-stalled-queued-run`,
`assigned-work-sweep-round-trips`, `saved-work-recovery-overdue`, and
the new test file (10 files, 225 tests). `tsc --noEmit` passes for
`server`.
- Not run locally: the full suite. CI covers it.

## Risks

- Behavior shift: a task whose source failure is system-owned now
continues automatically after its saved files are proven permanently
gone, even when an earlier wake was withheld. The new run does not
replay the old run. It starts in a fresh workspace, with the same notes
and quarantine that Retry uses. Person-owned failure codes still wait
for a person.
- The new hold guard can skip a rewrite. It skips only when the hold is
already resolved with the same no-replay decision and the same
restore-recovery data. A changed decision is still written; the test
covers this.
- No schema changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, local Vitest against embedded Postgres).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 05:22:00 -06:00
DottaandClaude Opus 5.5 68f641ba0b fix(recovery): continue a native run cut off by a sandbox client timeout (#21736)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime runs an agent's provider turn in a sandbox (for
example Daytona) and records each failure on the run and its native
coordinator
> - A Daytona SDK request can hit its client-side deadline during a
provider turn. The run then fails with `native_session_interrupted` in
an ambiguous state, because the turn started and its outcome is unknown
> - That failure blocks the task under a board-owned watchdog hold. No
recovery owner selected the hold, because its cause is not an
execution-reconciliation cause. The task never ran again, and only a
person's Retry could continue it
> - Recovery from an infrastructure fault must work automatically,
without a person
> - This pull request lets the execution-disposition pass hand that hold
to the existing fresh-continuation fallback
> - The benefit is that the task continues once, in an isolated fresh
run, and the cut-off turn is never replayed

## Linked Issues or Issue Description

**What happened?**
A native run on a Daytona execution environment (native runtime, an
OpenRouter model) started its provider turn. A Daytona request then
failed with `HTTP request timed out after 300000ms waiting for a
response. This is a client-side deadline
(DaytonaConfig.requestTimeoutMs, or the per-call operation/execution
timeout); any operation already started on the server may still be
running.` The run failed with `native_session_interrupted`. The task had
no further run: no automatic retry and no continuation.

**Expected behavior**
The task continues automatically, exactly once, in a fresh run from its
saved work. The cut-off provider turn is not replayed.

**Steps to reproduce**
1. Start a native run for an assigned task. Let its provider turn start
(a `turn.started` event is saved).
2. Make a sandbox request in that turn fail with the Daytona client-side
timeout error.
3. The executor saves the coordinator as `terminal_failure` with
`recoveryMode: ambiguous_state`, blocks the task, and writes an active
board hold with cause `native_session_interrupted`.
4. Run the execution-disposition worker
(`settleUnrecoverableExecutions`). Before this change, nothing happens
and the task stays blocked.

**Paperclip version or commit**
`master` at 9ebc276b42 (observed on a build from 5acf27bc3).

**Deployment mode**
Hosted cloud with a Daytona execution environment.

Causal path:
- The Daytona error is not a named class, so
`nativeSessionFailureSourceCode` returns the generic
`native_session_interrupted`.
- `nativeProviderRecoveryEvidence` returns `ambiguous_state`: provider
events exist, but there is no checkpoint with a provider identity. The
executor then makes the failure terminal on the first attempt (no
same-run resume) and writes a board hold whose cause is
`native_session_interrupted`.
- `unresolvedExecutionDispositions` and `heldExecutionWhere` select only
holds with `EXECUTION_RECONCILIATION_CAUSES` (or a narrow
`resume_native_run` shape). The safe replacement lane accepts
`native_session_interrupted` only for the historical
listener-unavailable case. The assigned-work sweep leaves a task blocked
by an active hold to the hold's owner. So no owner continued the task.

## What Changed

- `server/src/services/execution-recovery-resolution.ts`: the
disposition pass also selects an active watchdog hold with a
restart-class cause (`RESTART_CLASS_FAILURE_CODES`) for a terminal
native run with no retry pending, unless the safe replacement lane owns
that run. It hands the run to `continueFreshInTransaction` and records
the fallback's result on the hold, so a run that does not continue is
checked again only when due.
- `server/src/services/fresh-continuation-fallback.ts`: the fallback
treats an active watchdog hold with a restart-class cause as its own.
When it continues the task, it settles that hold and unblocks the task,
as it does for reconciliation holds.
- New red/green test
`server/src/services/native-runtime/native-client-timeout-continuation.integration.test.ts`.
It drives the real native executor with a provider turn that starts and
then fails with the Daytona client-timeout error.

The cut-off turn is not replayed. The fresh continuation isolates the
old run as it does for every continuation: the old run loses its write
authority, its remote workspace epoch is fenced (or the continuation
gets a new workspace), and the coordinator records `replacementDenied:
fresh_continuation`. The fallback keeps its existing checks:
person-owned causes, exact recovery still under way, and the per-task
hourly cap.

## Verification

- Red on `master`: `npx vitest run
src/services/native-runtime/native-client-timeout-continuation.integration.test.ts`
fails with no continuation recorded.
- Green with this change. The test asserts that the run fails with
`native_session_interrupted` and the Daytona error text in an ambiguous
state, then that two disposition passes record exactly one fresh
continuation, the task goes from `blocked` to `todo`, no active hold
remains, the old run is marked `replacementDenied`, and two
continuation-delivery passes wake the assignee exactly once.
- Focused suites (all tests that use the disposition pass, the
fresh-continuation fallback, or the execution-control schedule, 28
files): 27 files pass. `chat-channels.integration.test.ts` has 9
failures that are unrelated to this change (see the PR comment for the
`master` baseline).
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify --explain` selects 2751 of 3578 files (a near-full run).
I did not run it locally. CI supplies that coverage.

## Risks

- Behavior change: a terminal native interruption hold that before
stayed with the board now continues automatically in a fresh run. This
is the intended product behavior. The fallback still refuses
person-owned causes and still respects the per-task cap.
- Holds with restart-class causes are now settled by the fallback when
it continues a run. Before, they could stay active next to the
continuation record.
- No migration. No API or UI contract change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits) and code execution for tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:15:41 +00:00
DottaandClaude Opus 5.5 01c1e708e9 fix(runner): stop a turn that keeps resending a refused tool call and continue fresh (#21734)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs call Paperclip tools, and
`RepeatedToolFailureGuard` refuses an exact repeat of a call that
already failed validation twice
> - The refusals are not counted and never end the turn, so a model
whose host drops an argument can resend the same refused call until the
run's wall clock
> - In a recorded restart-chaos wave, an opencode/DeepSeek agent sent
the same `PATCH /api/issues/{id}` through `call_api` 431 times in 40
minutes without `body`; 427 were refused. The run held an agent slot and
delayed other tasks
> - The system knows the next step (stop the turn and continue on a
fresh session), so the harness must own it, with no Retry banner
> - This pull request counts refusals, stops the provider turn after a
bounded number with the classified cause `agent_tool_call_loop`, and
queues an automatic fresh-session continuation that tells the model to
use a dedicated tool
> - It also records the declared argument names that each opencode tool
call delivered, so we can tell whether the bridge or the model drops
`body`
> - The benefit is that a looping agent frees its slot in seconds
instead of 40 minutes, and the task continues by itself

## Linked Issues or Issue Description

**What happened?**
An opencode agent on DeepSeek created two subtasks and then tried to set
their descriptions with `call_api` `PATCH /api/issues/{id}`. Every call
arrived with `operationId`, `pathParams`, `query` and `responseText`,
but without `body`. The guard refused each repeat. The model said "I
need to include body" more than 20 times and kept sending the same call:
431 calls, 427 refused, over 40 minutes. The run stayed active and its
agent slot delayed other queued work.

**Expected behavior**
After a small, bounded number of refusals, Paperclip stops the provider
turn and continues the task automatically on a fresh session, with a
hint to use a dedicated Paperclip tool. The board shows no Retry banner
because the system owns this recovery.

**Steps to reproduce**
1. Run a native agent whose model resends one tool call with a missing
required argument.
2. The guard refuses the third and later identical calls.
3. On master, the refusals never end the turn.

**Paperclip version or commit**
`master` at 6400698a8e.

## What Changed

- `repeated-tool-failure-guard.ts`: counts refusals per exact call and
per run. At 6 refusals of one call, or 20 refusals in the run, it throws
`ToolCallLoopError` (`code: "agent_tool_call_loop"`, tool, counts and
argument names, never values). The message tells the model the turn
stops and gives a short hint.
- `paperclip-runner-tool-authority.ts`:
`scheduleToolCallLoopContinuation` writes one idempotent native
status-wake outbox row (key `agent_tool_call_loop:<runId>`) with
`forceFreshSession: true` and the hint as the wake instruction. The
outbox dispatcher admits it with `allowRunCoalescing: false`, so it
never merges into a deferred comment wake or the active run. An assignee
run uses `issue_continuation_needed`. A completion-review run uses
`native_completion_review` with its review assignment, which the
dispatcher revalidates, so the fresh reviewer session keeps the review.
- `native-session-executor.ts`: the session authority epoch queues that
continuation when the guard raises the loop error. The governed-wait
lookup finds the wake on the next tool event and ends the provider turn
as `yielded` with a `same_agent` continuation, the same way the
connection-tools refresh ends a turn. If the wake cannot be queued, the
refusal still reaches the model and the error is logged.
- `status-decision-committer.ts`: a yielded continuation with that key
binds to the queued wake instead of creating a second one.
- `status-arbiter.ts` and `native-run-finalizer.ts`: a pending review
run that ended through this continuation skips the
`native_review_unresolved` resume recovery, because the queued review
wake owns the next reviewer run.
- `mcp-bridge.ts` and `opencode-proxy-finish-diagnostics.ts`: the
opencode tool-delivery diagnostic records `argumentNames` (only names
declared in the tool's input schema, sorted, at most 1,024 characters of
JSON), `argumentNamesTruncated` and `undeclaredArgumentCount`. Values
and undeclared names are never recorded. The forwarder's closed key
allowlist accepts the new fields and validates them, so a record stays
under its 2,048-character limit.

### Threshold choice

- **6 refusals of one exact call.** The guard already runs a call twice
and then refuses with an explicit correction. Six more refusals (eight
identical attempts in total) give a model that reads errors many chances
to change its arguments. In the recorded run the model never changed
them in 427 refusals, so six refusals end that loop within seconds.
- **20 refusals in the run.** A model can alternate between a few
refused variants and keep each one under 6. Refusals happen only for
exact repeats of a certain validation failure, so a healthy run almost
never reaches 20.

### Previous-release story

The guard counters live in memory for one run's tool authority, as
before. A deploy or restart during a run starts the counters again at
zero on the new process; that only delays the stop. The continuation is
an ordinary wake row. If a server on the previous release finalizes a
run that a new server parked, it does not know the key and queues its
own `same_agent` continuation too; the issue execution lock lets one run
and the other coalesces.

## Verification

- Red then green, `repeated-tool-failure-guard.test.ts`: "after N
refusals the guard raises a turn-stopping loop error" and "stops the
turn once refusals across different calls reach the run limit". Both
fail on master.
- Red then green, `native-session-executor.test.ts`: "a turn whose model
repeats a refused call N times ends with agent_tool_call_loop and
schedules a continuation". With the epoch handling disabled it fails (13
calls instead of 8, nothing parked). Also "keeps the refused call's
error when the loop continuation cannot be queued".
- `paperclip-runner-tool-authority.test.ts`: "queues one fresh-session
continuation outbox for the assignee / a completion reviewer stopped by
a tool-call loop" (real Postgres; idempotent).
- `native-tool-call-loop-continuation.test.ts`: with a comment wake
already deferred behind the looping run, the continuation keeps its own
keyed deferred wake, the comment wake is unchanged, and after release a
run starts with `forceFreshSession` and the loop cause.
- `status-arbiter.test.ts`: a pending reviewer stopped by a loop
releases its checkout without a resume recovery.
- `native-status-arbiter-corpus.test.ts`: the existing wake-binding test
now also covers the `agent_tool_call_loop` key.
- `mcp-bridge-delivery.test.ts` and
`opencode-proxy-finish-diagnostics.test.ts`: argument names are
recorded, values and undeclared names are not, invalid records are
dropped, and a boundary test with 32 names of 64 characters stays
forwardable with `argumentNamesTruncated: true`.
- Server: the six touched test files pass (1755 tests). Runner:
`src/drivers/opencode` and `src/cli` pass (569 tests). `tsc --noEmit`
passes for `server` and `packages/paperclip-runner`.
- `pnpm verify` exits 2 (runner package changes need broad coverage), so
required CI supplies the broad run.

## Risks

- A run that hits the limit ends its turn early. The limits are far
above what a model that reads its errors needs, and the task continues
automatically on a fresh session.
- The fresh session loses the in-turn conversation. The task, its
comments, documents and workspace stay, and the continuation names the
cause and the hint.
- The diagnostic now carries declared argument names. These are schema
names, not values, and the forwarder validates the shape before it
forwards a record.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21360

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 01:52:50 -06:00
DottaandClaude Opus 5.5 c7efca1eaf perf(recovery): read an agent's own reporting chain and each pass's review waits once (#21724)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs its background recovery queues on one Node main
thread. Each database statement and each byte returned costs main-thread
time. The target is 1,000 concurrent agents per server.
> - On the idle hosted staging instance, the stranded-task check is the
top remaining cost. Each check read the full agent row (with its
configuration blobs) and then every agent of the company, only to decide
whether this one agent can run. Per check it also opened a
repeatable-read budget transaction for the same company and agent.
> - The org-chain evaluation only looks at the agent's own managers.
Reading the whole company is work proportional to the size of the
company, for an answer that depends on a few rows.
> - The same check also read the external review wait of each in-review
task one task at a time.
> - This pull request reads only the agent's reporting chain (one
recursive query up `reports_to`), gives the stranded check a lean agent
read, and reads the review waits of a pass's tasks in one call.
> - The benefit is half the bytes per check (17.2 KB to 8.6 KB), 87
percent fewer rows, and no per-task review-wait chain for tasks that
wait, with the same answers.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The per-task reads of the stranded-task recovery check
(`reconcileStrandedAssignedIssues`) and of every caller of
`evaluateAgentInvokabilityFromDb` and `agentService.getById`.

**Subsystem affected**

Server: issue recovery and agent invokability.

**Current behavior**

`evaluateAgentInvokabilityFromDb` reads `id, company_id, name,
reports_to, status` for every agent of the company on each call.
`getById` does the same for `orgChainHealth`. The stranded check calls
it once per task, after a full-row agent read with the configuration
blobs. On staging at idle these are the `agents where company_id = $1`
read at 4.7/s and the full `agents where id = $1` read at 4.2/s. The
check also calls `listExternalReviewWaits` for one task at a time (the
`chat_message_links` / `chat_github_reviews` reads at about 3/s on
staging).

**Proposed behavior**

Read only the agent's managers, with the same cycle, missing-manager,
cross-company and terminated rules. Read nothing when the agent has no
manager or when its own status already decides the answer. Read the
stranded check's agent with only the fields it uses. Read the review
waits of all tasks of a pass in one call, and use that answer only to
skip a task that waits.

**Reason and benefit**

At idle, the average result size on staging is about 17 KB per
statement. The stranded check is about 22 statements and 17 KB per task,
and it runs about 3 times per second there.

## What Changed

- `server/src/lib/agent-reporting-chain.ts` (new):
`readAgentReportingChain` reads the agent's managers with one recursive
CTE up `reports_to`, limited to the agent's company. `UNION` stops the
walk at a cycle. A root agent costs no statement.
- `server/src/services/agent-invokability.ts`:
`evaluateAgentInvokabilityFromDb` evaluates against the chain, not the
company list. It reads nothing when the agent's own status is not
invokable, because the status then decides the answer and the chain is
not part of it. Every caller benefits with no change: wake enqueue,
queued-run claim, retries, run dispatch, wake-queue promotion,
continuation delivery, task watchdogs, thread interactions, native
completion feedback, native session resumption, native review
participants, agent conversations and recovery.
- `server/src/lib/agent-records.ts`: `getById` computes `orgChainHealth`
from the chain, not from the company list.
- `server/src/services/recovery/service.ts`, stranded check:
- The agent read selects only `id, company_id, name, reports_to, status,
lifecycle_state, lifecycle_holds`, not the full row.
- `hasPersistedDurableWaitPath` takes the pass. When the task is in the
pass, the first in-review task reads the external review waits of every
task of the pass with one `listExternalReviewWaits` call. A task in that
answer waits and is skipped. A task not in that answer is read again
alone, exactly as before, so a missing answer never lets a check go
further than a fresh read would.
- A company-wide call without a pass gets a pass of its in-review
candidates.
- `server/src/services/scoped-deadline-work.ts`: an optional
`beforeChecks(entries)` hook gets the entries of the batch before their
checks run.
- `server/src/services/recovery/pass-reads.ts` (new) and
`server/src/services/recovery/work.ts`: the task queues build a pass
from the batch in `beforeChecks` and give it to each check of that
batch.
- Tests:
- `server/src/__tests__/agent-reporting-chain.test.ts` (new): property
test, see Verification.
- `server/src/__tests__/issue-recovery-pass-reads.test.ts` (new):
statements and bytes per check, per pass and per full worker drain.
- `server/src/__tests__/issue-recovery-review-wait-pass.test.ts` (new):
review-wait answers and reads, see Verification.
- `server/src/__tests__/monthly-spend-service.test.ts`: the
select-sequence stub no longer serves a company agent list to `getById`.
- `server/src/__tests__/helpers/recovery-population.ts` (new): seeds a
company with an org chart, review tasks with succeeded native runs, and
optional paused, terminated and recovery-action tasks.
- `server/src/__tests__/issue-recovery.test-fixture.ts`: an optional
statement meter, installed before the first connection.
- `server/src/__tests__/agent-record-read-projections.test.ts`:
`getById` now makes one typed `agents` select; the chain is a raw query.

### Why the shared review-wait read cannot start or escalate work that a
fresh read would not

- A pass is one coordinator batch: at most 25 checks, one after the
other. Its read is never older than the start of the batch. Nothing is
kept across batches.
- Only the positive answer is used: "this task waits on an external
review or reply", which leads to a skip. A skip only delays the task to
its next check, and the end of a wait sends its own hint.
- A negative answer is not used. The task is read again alone, so every
decision that goes further (an enqueue or an escalation) uses a fresh
read, as today.
- A first version also kept the agent's invokability and the budget
answer for the pass. Review found that a kept permissive answer can
reach an escalation, which does not re-check those facts the way an
enqueue does. That reuse is removed: those reads stay per task. No
authorization, pause, approval or budget check is removed or reordered.

## Verification

- Equality proof (`agent-reporting-chain.test.ts`, embedded PostgreSQL):
- 60 generated org charts, 2 companies each, up to 45 agents. Shapes:
deep chains (depth 44), trees, full cycles and mixed graphs.
- Managers are missing (inserted with foreign keys off), in another
company, terminated, paused or pending approval at random depths. Agent
statuses include an unknown value.
- For each agent, the test compares the old list-based result with the
new result. `evaluateAgentInvokabilityFromDb` deep-equals
`evaluateAgentInvokability(agent, companyRows)`.
`getAgentWorkEligibility` on the chain deep-equals it on the company
list (this is the `getById` path). The test also asserts that every
block reason and every chain-health reason occurred.
- The list-based evaluation has no depth cap (`reporting_chain_too_deep`
is declared but never returned), so the chain read has none either. A
cycle ends the walk in both.
- Mutation check: when the recursive step does not walk up, both tests
fail.
- Review-wait equality (`issue-recovery-review-wait-pass.test.ts`,
embedded PostgreSQL): two tasks with a direct chat reply wait, one task
with an inherited GitHub review wait, one task with none, and one task
with an inherited GitHub review wait in another company. One call for
all of them gives the same per-task answer as one call per task, for
both companies.
- Review-wait reads per pass (same file): the pass makes 2 bound-run
message reads (one shared, one for the task whose candidate is not a
wait) where per-task reads make 3, and 89 statements instead of 91. The
check results are identical. On master the pass is ignored, so the test
fails (3 is not less than 3).
- Statements and bytes (`issue-recovery-pass-reads.test.ts`, 50 agents,
200 review tasks with succeeded native runs and no review waits,
embedded PostgreSQL, statement meter on the client sockets):

  | | master | this branch |
  | --- | --- | --- |
  | One check: statements | 22 | 22 |
  | One check: bytes received | 17,170 | 8,617 |
  | One check: rows | 57 | 8 |
  | 25 checks: bytes | 429,331 | 215,505 |
| Real worker, full company drain (200 checks): statements | 4,414 |
4,422 |
  | Real worker, full company drain: bytes | 3,455,957 | 1,749,701 |
  | Real worker, full company drain: rows | 11,600 | 1,800 |
  | Company agent-list reads per drain | 200 | 0 |

Here no task waits, so each batch pays one extra review-wait statement
(8 per drain). Where tasks wait, the shared read replaces the per-task
review, publication and authorization reads.
- Regression after the review change, embedded PostgreSQL (664 tests):
scoped-deadline-work, heartbeat-process-recovery,
issue-recovery-deadlines, assigned-work-invariant,
recovery-deadline-acceptance, heartbeat-issue-liveness-escalation,
slack-conversation-lifecycle. Also issue-list-review-wait-authorization,
external-review-wait, monthly-spend-service.
- Regression of the first version, embedded PostgreSQL (757 tests):
issue-assignee-invokability-routes, agent-record-read-projections,
issue-graph-liveness-equivalence, heartbeat-paused-assignee-new-task,
heartbeat-process-recovery, heartbeat-issue-liveness-escalation,
assigned-work-invariant, recovery-deadline-acceptance,
issue-recovery-actions, heartbeat-refused-assignment-wake,
failed-continuation-continues-again, agent-invokability,
agent-cross-tenant-authz-routes, heartbeat-stop-queued-task,
issue-tree-pause-resume-run.
- `tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- A test double that gives `evaluateAgentInvokabilityFromDb` an object
with only `select` now needs `execute`, and a select-sequence stub of
`getById` gets one select fewer. CI found one (`monthly-spend-service`),
fixed here.
- A batch with no waiting task pays one extra cheap review-wait
statement.
- This branch and #21729 both touch
`server/src/services/recovery/work.ts` in nearby lines. The second one
to merge may need a small rebase.
- `server/src/services/recovery/service.ts` was already past 1,000
lines. This change adds about 25 lines to it, beside the check it
changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 01:34:08 -06:00
DottaandClaude Opus 5.5 41a3520a2f fix(issues): count a blocker covered when its wake already created a queued run (#21731)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A blocked parent task shows "covered" when its blockers have work
that will start by itself, and "needs attention" when a person must act
> - The blocker-attention read counts a blocker as active only through
its current execution run, or through a waiting wake that has not
created a run yet
> - A child whose wake already created a queued run that has not started
matches neither, so the parent shows "needs attention" and lands in the
blocked inbox
> - That work starts by itself when an agent slot frees, so the
attention is false and asks a person to act for no reason
> - This pull request also counts a blocker as active when its waiting
wake points at a queued or running run
> - The benefit is that the board and the blocked inbox stop showing
false attention while queued child work waits for a slot

## Linked Issues or Issue Description

**What happened?**
A parent task was blocked by two child tasks. Each child had a queued
run that a wake had created, and the runs waited for agent slots. The
parent showed `needs_attention` / `attention_required` and appeared in
the blocked inbox. The children started about 25 minutes later and the
parent then finished without a person.

**Expected behavior**
The parent shows `covered` with reason `active_child` while a child's
queued run waits to start.

**Steps to reproduce**
1. Create a blocked parent task with one child blocker that is assigned
to an agent.
2. Insert a queued wake for the child whose `runId` points at a queued
`heartbeat_runs` row, and leave the child's `executionRunId` empty.
3. List blocked tasks. The parent shows `needs_attention`.

**Paperclip version or commit**
`master` at 6400698a8e.

## What Changed

- `server/src/services/issues.ts`: the blocker-attention wake query
left-joins `heartbeat_runs` on the wake's `runId`. A queued or deferred
wake counts as an active path when its `runId` is empty or when its run
is `queued` or `running`.
- `server/src/__tests__/issue-blocker-attention.test.ts`: two tests. One
covers a child whose wake created a queued run that has not started. One
makes sure that a wake whose run already finished does not cover the
blocker.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/issue-blocker-attention.test.ts`: 27 passed.
- Red on master: "classifies a blocked parent as covered when its
child's wake already created a queued run that has not started" fails
with `needs_attention` / `attention_required`. Green with this change.
- Query plan (embedded Postgres, `enable_seqscan = off`): an index scan
on `agent_wakeup_requests_company_payload_issue_idx` (company and
payload issue id), then a nested-loop left join through
`heartbeat_runs_pkey`. The query count does not change, so there is no
N+1.
- `pnpm verify`.

## Risks

- Low risk. The change adds coverage only for a wake whose run is still
`queued` or `running`. A wake whose run finished, failed or was
cancelled stays uncovered, as before.
- The join is a primary-key lookup per wake row, inside the same chunked
query, so the board read cost does not change in a meaningful way.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 01:23:38 -06:00
DottaandClaude Opus 5.5 61c315ebab perf(chat): keep GitHub review check deadlines on company-wide rebuilds (#21725)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel shows a check run on each pull request it
reviews. The review-check sweep (`review-check-work.ts`) keeps one
`github_review_check` action per check. It republishes the check when
its status changes (queued, running, completed) and polls the pull
request head on a saved deadline.
> - Each time the sweep loads its checks, it computes the current check
status of every `received` check (`githubReviewCheckState`: 3–5
statements) to see whether the status changed.
> - Run lifecycle changes anywhere in a company send a company-wide
hint. So the sweep rebuilds every company's checks every 30 s and
recomputes the status of every waiting check, even when nothing about
those checks changed.
> - This pull request keeps the saved poll deadline on a company-wide
rebuild and reads the status only on targeted hints, at startup, and in
`check()`.
> - The benefit is that an idle company with many waiting checks costs
one page read per rebuild instead of 3–5 statements per check.

## Linked Issues or Issue Description

Refs #21705, #21711 (the same class of idle reload in the GitHub
response sweeps).

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On paperclip.staging at 89de6cbe3e (2026-11-15, 04:47–04:51Z, near idle
at about 0.4 HTTP requests/s), the statements of
`githubReviewCheckState` each ran at about 3/s:

- `select … from chat_github_reviews where endpoint_id … and
repository_id … and pull_number … and head_sha … order by created_at
desc limit 1` (all columns: id, company_id, endpoint_id, issue_id,
run_id, repository_id, …) at 3.04/s;
- `select chat_message_links.comment_id from chat_deliveries join
chat_message_links …` at 3.04/s;
- `select id, status from heartbeat_runs where company_id = $1 and
agent_id = $2 and context_snapshot->>'wakeCommentId' in (…)` at 3.04/s.

Staging has 84 GitHub review checks waiting on runs that have not
started. 84 checks reloaded every 30 s is 2.8 status computations per
second. That matches the observed rate.

**Root cause**

`registerGitHubReviewCheckWork` loads with
`githubReviewCheckReconstructionIntervalMs`, which allows one
company-wide rebuild every 30 s. Run lifecycle changes elsewhere in the
company keep sending `{ companyId }` hints, so the rebuild runs every 30
s. `entry()` calls `githubReviewCheckState` for every `received` row on
every load, only to see whether the status differs from the published
one. A waiting check's status does not change until its review, its run,
its delivery or its connection changes.

**Expected behavior**

A company-wide rebuild reads the waiting checks without recomputing each
status. A change to a check's run or pull request still publishes
promptly.

**Steps to reproduce**

1. Create 30 `received` GitHub review checks whose published status is
`queued`, with a poll deadline 1 hour ahead.
2. Register the review-check sweep. Send a company-wide hint every 30 s.
3. On master, each rebuild runs one page read plus 2–5 statements per
check.

**Paperclip version or commit**

master at 5acf27bc3b.

## What Changed

- `server/src/services/chat-channels/github/review-check-work.ts`:
- A company-wide load (company scope with no endpoint, repository,
number or task) does not call `githubReviewCheckState`. It keeps each
check's saved deadline: the head-poll `retryAt`, or the lease expiry.
Dispatch reads the current status anyway when that deadline comes.
- Startup (no scope) and targeted loads (endpoint, repository, pull
request or task) still read every status, as before. So do `check()` and
dispatch.
- Run lifecycle changes send `{ companyId, issueId }` for the run's
task, and publication lease releases send the pull request scope. Both
still publish a status change at once.
- New `mergeEntry`: an entry from a rebuild that did not read the status
keeps an earlier due time that a status read found. A company-wide
rebuild never postpones a pending publication. `check()` reads the
status again before it dispatches.
- `server/src/services/chat-channels/github/review-check-work.test.ts`
(new, real PostgreSQL):
- `does not recompute the status of every waiting check on each company
rebuild`
- `still publishes a check promptly when its review run starts` (a
task-scoped hint after a review run starts makes only that check due)

- `server/src/__tests__/chat-channels.integration.test.ts`:
- `publishes a local change at once and still recovers a missed head
after backing off` used a company-wide hint to stand for the review run
starting. It now sends the task-scoped hint that a run milestone sends,
and still expects the `in_progress` publication at once.
- `rebuilds GitHub review checks for company-wide hints at most once per
interval` is now `reads no GitHub review check status for company-wide
hints and still reads it for the check's task`. Company-wide hints read
0 statuses. Hints for the check's task still read 1, and hints for
another task or company read 0.

No GitHub check content changes. The published status, title and
conclusion are computed as before.

## Verification

- `pnpm vitest run
src/services/chat-channels/github/review-check-work.test.ts` (server),
30 waiting checks, 4 company rebuilds:
  - master: 244 statements, with 120 check status computations (red).
- this branch: 4 statements (one page read each), with 0 status
computations (green). A task-scoped hint still makes the started check
due at once.
- `pnpm vitest run src/services/chat-channels/github/`: 8 files, 62
passed.
- Related suites (`chat-channels.integration.test.ts`,
`app-private-hostname-gate.test.ts`, `idle-completed-history.test.ts`,
`chat-github-review-repair.test.ts`) filtered to review-check, GitHub,
history and hostname tests: 702 passed before the integration-test
updates below; after updating the two review-check tests, the 15
review-check tests pass.
- `pnpm typecheck` (server): passed.

## Risks

- A status change that sends only a company-wide hint is now published
at the check's next poll deadline, not at the next company rebuild. The
poll backs off to at most 5 minutes. Run lifecycle changes for a review
run carry its task, and connection or configuration changes are also
checked by dispatch, so this affects only hints that name no task or
pull request.
- Startup still reads every status, so recovery after a restart is
unchanged.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL tests) and
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 23:56:53 -06:00
DottaandClaude Opus 5.5 5acf27bc3b perf(server): read the instance role and membership once per GET request in authorization (#21716)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every board request goes through authorization, and the server runs
all requests on one Node main thread
> - The cost on that thread is per database statement
> - One `GET /api/issues/:id` makes several authorization decisions (the
issue, its project, mentioned projects, related task edges), and each
decision reads `instance_user_roles` and `company_memberships` again
> - On a task with a blocker and a project mention, that is 4 role reads
and 4 membership reads in the authorization service for one request, all
with the same answer
> - This pull request shares those two reads among the decisions of one
GET or HEAD request
> - The benefit is fewer statements on every polled board read, with no
authorization cache across requests

## Linked Issues or Issue Description

**What existing behavior does this improve?**
`authorizationService.isInstanceAdmin` and `getActiveMembership` read
the database for each decision in a request.

**Subsystem affected**
Server: authorization service and app middleware.

**Current behavior**
`GET /api/issues/:id` for a plain task sends 3 `instance_user_roles`
reads and 3 `company_memberships` reads (1 each in the session
middleware, 2 each in authorization). A richer task sends 5 and 5.

**Proposed behavior**
A new middleware, `authorizationReadScope`, opens a per-request memo for
GET and HEAD requests. For a session actor, it seeds the instance admin
answer that the session middleware read at the start of the same
request. Inside the scope, `isInstanceAdmin`, `getActiveMembership`,
`issueReadSqlCondition` and `projectReadSqlCondition` read each key
once. The plain task now sends 1 role read (the session middleware) and
2 membership reads (1 in the session middleware, 1 in authorization).
The issue list and summaries routes also drop their extra role reads.
Other methods, background work and other callers keep the old reads. The
memo ends with the request, so a revoked membership takes effect on the
very next request.

**Reason and benefit**
Fewer statements per polled read, with the same decisions.

**Breaking changes**
None.

## Memo lifetime rule

- The memo exists only for GET and HEAD requests.
- It wraps `res.writeHead`, `res.flushHeaders` and `res.write` once and
never reassigns them, so hooks that later middleware installs on the
response stay in place.
- It closes at the first of: `res.writeHead`, `res.flushHeaders`,
`res.write`, `finish` or `close`. After it closes, every read goes to
the database. A stream, a long poll, a download or a callback that runs
after the response therefore never uses the memo.
- Each entry is used for at most 2 seconds
(`AUTHORIZATION_READ_MAX_AGE_MS`), then read again. Correctness does not
depend on a list of routes.
- A rejected read is removed from the memo, so the next decision reads
again.
- Nothing is kept across requests.

Long-lived and streaming routes that I checked:
- `GET /plugins/:pluginId/bridge/stream/:channel` (SSE): authorizes once
at connect with `assertBoardOrgAccess` and `assertCompanyAccess` on
`req.actor`. It does not re-authorize per event. The memo closes at
`writeHead`.
- Streaming downloads (`object.stream.pipe(res)` in assets, access and
issue attachments): they authorize before the stream starts. The memo
closes at the first write.
- `POST /board/chat/stream` and `POST
/companies/:companyId/skill-sources/discover` (NDJSON): POST, so no
memo.
- Live events, plugin API, runner and terminal WebSockets: they use
`server.on("upgrade")`, so they do not pass through Express middleware.

## What Changed

- New `server/src/services/authorization-read-scope.ts`:
`authorizationReadScope` (an AsyncLocalStorage scope for GET and HEAD)
and `readOncePerRequest`.
- `authorizationService.isInstanceAdmin` and `getActiveMembership` use
`readOncePerRequest`. The existing batch snapshot memo still comes
first.
- `issueReadSqlCondition` and `projectReadSqlCondition` use the same
`admin:<userId>` key for their instance admin check.
- For `source: "session"` board actors, the scope seeds `admin:<userId>`
with `req.actor.isInstanceAdmin`, which the session middleware read from
`instance_user_roles` in the same request.
- `createApp` mounts `authorizationReadScope` after the actor and Cloud
control middleware.
- `authorizationReadScope` closes the memo when the response starts or
ends, ignores entries older than 2 seconds, and drops rejected reads.
- New exported `readInstanceAdmin(db, userId)`: the one predicate for
`isInstanceAdmin`, `issueReadSqlCondition` and
`projectReadSqlCondition`.
- New test `authorization-request-reads.test.ts`: one role read and one
membership read in authorization per issue read, and a removed
membership gives 404 on the next request.

## Verification

- Red: without the scope, the issue read sends 3 `instance_user_roles`
statements (`expected ... to have a length of 2 but got 3`).
- Green: with the scope, it sends 1 role read and 2 membership reads.
The issue list plus the summaries request send 1 role read each.
- Revocation: delete the membership row, and the next GET returns 404.
HEAD returns 200, then 404 after the revoke.
- Stream: a GET that flushes headers, then re-reads the membership after
it is deleted mid-stream, sees the deletion.
- Background callback: a callback scheduled in a GET that runs after the
response and after a revoke reads again and sees the deletion.
- Age bound: an entry older than 2 seconds is read again. A rejected
read is not kept.
- Seed predicate: for users with an `instance_admin` row, another role
row, and no row, the session middleware value equals
`readInstanceAdmin`.
- Other actors: an agent key, a board key and a cloud tenant actor get
no seeded value.
- Response hooks: a later middleware that wraps `res.writeHead` and
`res.write` keeps working for JSON and streamed responses. This test
fails with a version that restores the original methods.
- Red: with the previous implementation, the stream, callback, age and
rejection tests fail.
- `pnpm exec vitest run` on `access-routes-hidden-floor`,
`access-routes-permissions-upgrade`, `access-service`,
`agent-auth-middleware`, `authorization-privacy-query`,
`authorization-request-reads`, `authorization-run-context`,
`authorization-service`: 8 files, 177 tests pass. Also
`inbox-mine-candidates`, `issue-read-filter-batching`,
`attention-summary-counts`, `issues-unread-inbox`,
`live-runs-batched-reads`, `privacy-production-review`,
`activity-routes`, `authorization-privacy-query`,
`muse-native-visibility`: 224 tests pass (one
`privacy-production-review` test failed once under machine load and
passed on two reruns).
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- This changes where authorization gets two facts inside one request, so
it needs a human review before merge.
- A GET route that changes a membership or an instance role and then
authorizes again before its response starts, within 2 seconds, would see
the earlier value. GET routes do not change memberships or roles. POST,
PATCH, PUT and DELETE requests do not use the memo.
- A board actor built by the session middleware still carries its own
connect-time `memberships` and `isInstanceAdmin` for the life of a
stream. That is the existing behavior, and this change does not affect
it.
- Nothing is cached across requests.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 22:41:47 -06:00
Nicky LeachandClaude Opus 5.5 a78be8b601 feat(native): send launch preparation and run digests to plugin task prepare (#21717)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Environment plugins can run a native Runner as a plugin task, and a
warm session keeps that Runner between turns
> - On each warm turn Core sends the plugin a `prepare` operation with
the full launch requirements for the new run
> - The requirements' `requirementsDigest` covers the run identity, and
the run identity changes on every turn
> - So a plugin cannot tell that only the run moved, and it must redo
its whole preparation every warm turn
> - This pull request gives the plugin two digests: one that stays the
same while its preparation stays valid, and one that names the run
> - The benefit is that a plugin can transfer only the run identity when
nothing it reconciles changed

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The `prepare` operation that Core sends to an environment plugin for a
plugin Runner task, on cold starts and on warm attaches.

**Subsystem affected**
Native runtime, plugin Runner tasks, the Runner launch contract, and the
plugin SDK environment task contract.

**Current behavior**
A warm attach sends the full requirements. The only digest available,
`requirementsDigest`, includes `runId`, `turnId` and `itemId`, so it
changes on every turn even when the launch is otherwise the same.

**Proposed behavior**
Core computes two digests for every `prepare`:
- `preparationDigest`: the launch requirements an environment reconciles
when it prepares a Runner:
  - the runner, lease and session identity;
  - provider, model and settings;
  - the prompt and instruction text;
- the runtime context's instruction and skill bundles, working copy and
connection instructions.

It leaves out the `runId`, `turnId` and `itemId`, and the runtime
context's MCP assignment and binding (with `aggregateDigest`, which
hashes the assignment). MCP reaches the Runner through Core's tool
authority on every run attachment, never through files the environment
publishes. Core already continues a warm session across an MCP-only
change (`nativeRuntimeContextsMatch` with tool refresh). The binding is
also named per run (`native-mcp:<runId>`). So the digest stays the same
across warm turns of one Runner, lease and session, including when the
agent's MCP access changes.
- `runDigest`: `preparationDigest` plus everything it leaves out:
`runId`, `turnId`, `itemId`, and the MCP assignment and binding. It is
unique to one run's launch.

A plugin that advertises `launchDigestVersion: 1` in its task
description gets both fields on `prepare`.

**Reason and benefit**
A plugin can compare `preparationDigest` with its saved preparation and
transfer only the run identity when they match, instead of a full
preparation.

**Breaking changes**
None. Plugins without `launchDigestVersion` get no new fields.

## What Changed

- `packages/paperclip-runner/src/live/runner-provider-launch.ts`: add
`runnerLaunchPreparationDigest` and `runnerLaunchRunDigest`. The
preparation digest drops the MCP assignment and binding; the run digest
takes the full requirements and includes them. Both use the same
canonical JSON as `runnerLaunchRequirementsDigest`, with their own
schema tags so the three digests never collide. Export them from the
package and the server's vendored runner shim.
- `packages/plugins/sdk/src/environment-tasks.ts`: add optional
`preparationDigest` and `runDigest` to the `prepare` operation, and
optional `launchDigestVersion: 1` to the task description. The
capability gates both digests.
- `server/src/services/native-runtime/plugin-runner-task.ts` and
`native-session-executor.ts`: send both digests on `prepare` when the
description advertises `launchDigestVersion: 1`.
- `server/src/services/environment-task-runtime.ts`: recompute both
digests from the requirements, and refuse a `prepare` whose digests are
missing in part or do not match.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the SDK
public source pin.
- Tests:
- the preparation digest is stable across runs and changes when the
runner, lease, session, instructions, model, approval policy or settings
change;
- with a runtime context, an MCP-only change (the run's binding, no
binding, a changed access set) keeps the preparation digest and changes
the run digest;
- a change to the instruction bundle, instruction entry, a skill bundle,
the skill set, the working copy or the connection instructions changes
the preparation digest;
  - the run digest changes per run;
  - Core sends the digests only when advertised;
- the runtime refuses forged or partial digests before it calls the
plugin.

## Verification

- `npx vitest run src/live/runner-launch-preparation.test.ts` in
`packages/paperclip-runner`: 26 passed.
- `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts
src/__tests__/environment-task-runtime.test.ts` in `server`: 160 passed.
- `npx vitest run source-sdk-public-inputs.test.mjs
source-sdk-evidence.test.ts source-sdk-public-type.test.ts` in
`tests/runner-e2e`: 59 passed.
- `pnpm typecheck` in `server` and in `packages/paperclip-runner`:
clean.
- Not run locally: the full suite. CI covers it.

## Risks

Low risk. The digests are new optional fields behind a plugin capability
version. Core computes them and checks them again before it calls the
plugin, so a plugin can rely on them. They prove only that the launch
requirements are unchanged. A plugin that skips work on a match must
still check its own state, such as files it published into the sandbox.
If the runtime context carries values that change per run, the
preparation digest changes too, so a plugin does its full preparation.
That is safe, but the plugin saves nothing. The MCP binding is the one
such value Core sets per run today, and MCP is excluded entirely. A
plugin that does publish MCP configuration into its sandbox must not
rely on this digest for it. Values outside the requirements are not
covered at all: the account identity and credentials, the GitHub broker,
project IDs, and the plugin's own runtime revision. A plugin must keep
checking those itself. A match lets a plugin skip publishing its launch
configuration, not computing the descriptor: `prepare` must still return
a launch whose `requirementsDigest` matches the current requirements,
which carry the current MCP assignment and binding.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:07:54 -08:00
DottaandClaude Opus 5.5 89de6cbe3e perf(server): list skills without a folder transaction when bundled folders exist (#21715)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI polls `GET /api/companies/:companyId/skills`, and the
server runs all requests on one Node main thread
> - The cost on that thread is per database statement, so a polled read
must stay small
> - Each skills list read reconciles the bundled skill folders, and that
opened two transactions with a company advisory lock, even when all
folders already existed
> - A write-path guard must not run a lock and a transaction on every
poll
> - This pull request checks the current folders with a plain read first
and takes the lock only when a folder must change
> - The benefit is a skills list read with no transaction and no
advisory lock, and 13 fewer statements (58 to 45)

## Linked Issues or Issue Description

**What existing behavior does this improve?**
`folderService.ensureBundledCategory` and `pruneEmptyBundledCategories`
always open a transaction and take `pg_advisory_xact_lock` for the
company folders.

**Subsystem affected**
Server: skill folders service, used by the company skills list.

**Current behavior**
Every `GET /api/companies/:companyId/skills` sends 58 statements,
including 2 transactions (BEGIN, lock, reads, COMMIT), even when nothing
changes.

**Proposed behavior**
`ensureBundledCategory` reads the company skill folders once. When the
bundled root exists and the category folder exists with the expected
name, it returns that folder. `pruneEmptyBundledCategories` computes the
empty categories with plain reads and returns when there is nothing to
delete. In all other cases, both functions take the lock and run the old
code, which reads the folders again under the lock. The same request
sends 45 statements with no transaction.

**Reason and benefit**
The polled read stops taking a company lock, and it sends fewer
statements.

**Breaking changes**
None.

## What Changed

- `ensureBundledCategory`: a plain read fast path when the category
folder and the bundled root exist and the name matches. Create and
rename still run under the company folder lock.
- `pruneEmptyBundledCategories`: a plain read pre-check. The delete
still runs under the lock, and it computes the empty set again under the
lock.
- New test `company-skills-list-statements.test.ts`: a warm skills list
read has no transaction and no advisory lock and at most 45 statements;
two concurrent first ensures create exactly one folder; two concurrent
first list requests create no duplicate folder; a renamed category is
repaired and an empty category is pruned through the locked path.

## Verification

- Red: before the change, the warm read had 2 transactions (`expected 2
to be +0`) and 58 statements.
- Green: after the change, the warm read has 0 transactions, no
`pg_advisory` statement, and 45 statements. The response body is equal
to the first read.
- Concurrency: two `ensureBundledCategory` calls at once return the same
folder, and there is exactly one row for the system key.
- `pnpm exec vitest run` on `company-skills-list-statements`,
`company-skills-service`, `company-skills-routes`,
`company-skills-runtime-concurrency`, `company-skills-catalog-service`,
`company-skills` and the folder tests: 215 tests pass. Two
`company-skills.test.ts` tests in "project workspace skill discovery"
fail on my machine with and without this change. They test file system
discovery and do not use the folders service.
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Low risk. The fast path returns only when the locked path would return
the same folder without a write. Every write still happens under the
lock, after a fresh read.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:42:10 -06:00
DottaandClaude Opus 5.5 30c4d2f4a1 perf(server): read the issue detail with fewer database statements (#21713)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI loads the task page through `GET /api/issues/:id`, and
the server runs all requests on one Node main thread
> - A CPU profile of the server shows that the cost is per database
statement: SQL building, parameter encoding, result parsing and one
socket write each
> - The issue detail read sent 73 statements for a task with two
ancestors, one blocker and a project mention
> - Several of those statements read the issue row again, only to get
values that the route already loaded
> - This pull request passes the loaded issue to those reads, walks the
ancestor chain in one query, and skips reads whose result the response
does not use
> - The benefit is 9 fewer statements on each task page load (73 to 64),
plus one fewer statement for each extra ancestor level, with a
byte-identical response

## Linked Issues or Issue Description

**What existing behavior does this improve?**
`GET /api/issues/:id` reads the same issue row several times in one
request, and walks the ancestor chain with one statement for each level.

**Subsystem affected**
Server: issue routes and the issue, issue reference and project
services.

**Current behavior**
The route loads the issue, then `getRelationSummaries`,
`getCurrentScheduledRetry`, `findMentionedProjectIds`,
`listIssueReferenceSummary` and `getAncestors` each read the issue row
again. `getAncestors` sends one statement for each ancestor. The inbox
archive fields read two activity aggregates even when the issue has no
archive row. The project embed reads the workspace provider catalog, but
the compact embed does not return it. The test fixture request sends 73
statements.

**Proposed behavior**
The route passes the values it already has. The ancestor walk is one
recursive query. The activity aggregates run only when an archive row
exists. The project embed skips the provider catalog. The same fixture
request sends 64 statements, and the JSON response is the same.

**Reason and benefit**
Fewer statements per request means less CPU on the main thread for every
open task page.

**Breaking changes**
None. Other callers of the changed service functions keep the old
behavior, because the new arguments are optional.

## What Changed

- `issueService.getRelationSummaries` and `getCurrentScheduledRetry`
accept an optional known `companyId` and then skip the issue read.
- `issueService.findMentionedProjectIds` accepts the loaded issue
(`companyId`, `title`, `description`).
- `issueService.getAncestors` accepts the known `parentId` and walks the
chain in one recursive query. It keeps the order (nearest first), the
limit of 50, and the stop at a cycle or at a task that is its own
parent.
- `issueService.getActiveInboxArchiveFields` reads the archive row first
and reads the activity aggregates only when an archive row exists.
- `issueReferenceService.listIssueReferenceSummary` accepts an optional
known `companyId`.
- `issueThreadInteractionService.listForIssue` accepts the loaded issue
(`status`, `companyId`, `executionWorkspaceId`). `GET
/api/issues/:id/interactions`, which the board polls, passes it (11 to
10 statements).
- `projectService.getById` accepts `{ workspaceProvider: false }`. The
issue detail route uses it, because `compactIssueProject` never returns
`workspaceProvider`.
- Test helpers: `createStatementCensus` (statements per request, grouped
by statement shape with the server callers), and a board route fixture
with the real session actor middleware. `meterDatabaseStatements` can
record a call stack for each statement.
- New tests: `issue-detail-statements.test.ts` (statement budget,
byte-identical normalized JSON against a fixture file, access rules,
membership removal on the next request, ancestor walk with a cycle) plus
an interactions poll test, and `board-route-statement-census.test.ts`
(census of the polled and page-load board routes).

## Verification

- Red: before the change, the fixture request sent 73 statements, and
the budget test failed (`expected 73 to be less than or equal to 64`).
- Green: after the change, the request sends 64 statements. The
normalized JSON is byte-identical to the fixture file that I wrote
before the change.
- Access: another company's user gets 404, a missing issue gets 404, an
anonymous request gets 404, a hidden issue stays readable for a member,
and a removed membership gets 404 on the very next request.
- `pnpm exec vitest run` on `issue-detail-statements`,
`board-route-statement-census`, `issues-goal-context-routes`,
`issues-service`, `issue-references-service`, `inbox-archive-routes`,
`heartbeat-task-context`, `project-workspace-provider.integration`,
`workspace-provider-failure`, `issue-telemetry-routes`,
`issue-thread-interaction-routes`, `issue-document-restore-routes`,
`heartbeat-referenced-projects`, `project-environments`: 14 files, 414
tests pass.
- `pnpm --filter @paperclipai/server typecheck` passes.
- Interactions: the polled route sends 10 statements instead of 11, with
a response equal to the service result without the known issue.
- Census: `PAPERCLIP_STATEMENT_CENSUS=/tmp/census.txt pnpm exec vitest
run src/__tests__/board-route-statement-census.test.ts` writes
statements per request and the top statement shapes for each board
route.

## Risks

- Low risk. The new arguments are optional, so other callers keep the
old reads.
- When the route passes a known `companyId`, a concurrent delete of the
issue no longer gives a 404 from these sub-reads. The route already
loaded and authorized the issue in the same request, so the response is
the same in practice.
- The JSON fixture file fails when someone changes the issue detail
response. Regenerate it with `PAPERCLIP_WRITE_ISSUE_DETAIL_GOLDEN=1` and
review the diff.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:42:06 -06:00
DottaandClaude Opus 5.5 f4f4902707 perf(chat): read the run state of a GitHub response sweep page in one batch (#21711)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel has two sweeps for request response
comments: the acknowledgement sweep (`acknowledgement-work.ts`) and the
terminal-response sweep (`terminal-response-work.ts`). Each reads pages
of up to 100 requests.
> - For each loaded request, both sweeps call `readGitHubResponseRun`
one row at a time. That is one latest-run statement, plus the run
identity context read and the message link read of
`chatRunSourceMessageId`. A page of 100 requests costs about 300
statements.
> - On paperclip.staging, those two provenance reads ran at about 13/s
each. #21705 stops the idle backlog from being reloaded. This PR makes
the reads that remain cost a constant number of statements per page.
> - The per-row read also had a failure mode. A 403 from one run's
provenance check aborted the acknowledgement sweep's whole page, so no
request of that company was acknowledged until the 403 went away.
> - This pull request adds a batched `readGitHubResponseRuns` and uses
it once per page in both sweeps.
> - The benefit is a constant number of statements per page, and a
provenance failure that affects only its own request.

## Linked Issues or Issue Description

Refs #21705 (the same sweeps; that PR stops reloading requests that
cannot progress).

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On paperclip.staging (2026-11-15, about 01:45–02:25Z),
`provenanceReads.context` (`select … from run_identity_contexts where id
= $1 and run_id = $2 and company_id = $3 and status = $4 limit $5`) ran
at 13.37/s, and `provenanceReads.link` (`select
chat_conversations.issue_id, chat_endpoints.provider from
chat_message_links join … where chat_message_links.company_id = $3 and
comment_id = $4 …`) ran at 13.20/s. Both GitHub response sweeps reach
them through `readGitHubResponseRun` → `chatRunSourceMessageId(db, run,
{ provider: "github" })`, once for each loaded row. `load()` costs one
page read plus three statements per row. `check()` read the run state
twice for each due request (before and after the acknowledgement).

In the acknowledgement sweep, a 403 from `chatRunSourceMessageId` (for
example, a run whose identity context is no longer accepted) threw out
of `entry()` and aborted the whole page. The terminal sweep already kept
such a row due, so the acknowledgement owner could revoke it.

**Expected behavior**

A page of requests costs a constant number of statements. A failure for
one run affects only the requests of that run.

**Steps to reproduce**

1. Create 120 GitHub response requests on 30 tasks. Give each task an
ended run with an accepted identity context.
2. Register the terminal-response sweep. On master, the first pass costs
one page read plus three statements per row.
3. Mark the requests `received`, revoke one run's identity context, and
register the acknowledgement sweep. On master, the load fails with a
403, and none of the 150 requests is acknowledged.

**Paperclip version or commit**

master at ea8c734cb9 (includes #21705).

## What Changed

- `server/src/services/chat-channels/github/response-run-state.ts`:
- The latest-run subquery and the "woke for this request" check are
built once, from literals (single read) or from the columns of a request
row (batch read). The single read keeps the same query shape.
- New `readGitHubResponseRuns(db, inputs)`. It reads the latest run of
every request of a page in one statement: `unnest(…) with ordinality`
and a lateral latest-run subquery, then a join to `heartbeat_runs` by id
and company. It then calls the existing `chatRunSourceMessageIds(db,
runs, { allowLegacyWakeComment: true, provider: "github" })` once for
the page's distinct runs. That helper reuses `provenanceReads`, so the
scope, status filters and joins are the same as the single check.
Results map back by position. A run whose provenance check is forbidden
gives `{ forbidden }` for its requests only.
- `server/src/services/chat-channels/github/response-execution.ts`:
`githubResponseExecution` takes an optional pre-read state. It throws a
`{ forbidden }` state as the same 403 that the single read throws.
- `server/src/services/chat-channels/github/terminal-response-work.ts`:
`load()` reads each page's run state with one `readGitHubResponseRuns`
call. `check()` reads the run state once: after the acknowledgement it
reads the row only, and keeps a request that is still owed for the next
check, which reads its run state again first.
- `server/src/services/chat-channels/github/acknowledgement-work.ts`:
the same batching. A forbidden provenance check keeps that request due,
so the acknowledgement owner revokes it (`authorization_changed`), and
the rest of the page continues. After an acknowledgement, a still-owed
request keeps the deadline from its own row (retry time, fast-response
expiry or lease).
- Tests:
- `terminal-response-work.test.ts`: `reads a page of waiting responses
in a constant number of statements` and `acknowledges every owed request
of a page when one run's requester provenance changed`.
- `response-run-state.test.ts`: `reads a page of requests in one
latest-run statement with the state each single read returns` compares
the batch with the single read for every seeded binding and wake link.
- `response-sweep-scope.test.ts`: `rebuilds parked requests once per
interval for company-wide hints` counted run-state *statements* as a
proxy for requests reloaded. With batching, one statement reads a whole
page, so the count dropped from 6 to 2 and the test failed in CI. The
behavior it guards must stay: a company-wide rebuild reloads every
eligible request once per interval, and repeated hints inside the
interval reload none. The test now counts the requests that each
run-state read names (the length of a batched read's request arrays, or
1 for a single read). The expectations stay 6 / `[6, 0, 0, 0, 0]` / 6,
so the test still proves the same rebuild behavior.

No GitHub comment behavior changes.

## Verification

- `pnpm vitest run
src/services/chat-channels/github/terminal-response-work.test.ts`
(server), staging-shaped fixtures (120 waiting requests on 30 tasks;
codes null / `acknowledgement_unconfirmed` /
`acknowledgement_superseded`):
- before (#21705 only): the first pass costs 364 statements, with 120
run identity reads. With one revoked context, the acknowledgement sweep
acknowledges 0 of 150 owed requests. The 2 new tests fail (red).
- after: the first pass costs 10 statements, with 2 run identity reads
(per page: page, latest runs, context, link, wait keys). All 150 owed
requests are acknowledged, including the 5 whose provenance is
forbidden. 8 of 8 tests pass (green). All 7 files under
`chat-channels/github/`: 60 passed. `idle-completed-history.test.ts`: 75
passed.
- On master before #21705, the first pass was 362 statements, and every
company rebuild repeated it.
- `pnpm vitest run
src/services/chat-channels/github/response-run-state.test.ts`: 3 passed.
- `pnpm vitest run src/__tests__/chat-channels.integration.test.ts -t
"terminal|acknowledg|GitHub response|response comment|GitHub review"`:
192 passed.
- `pnpm typecheck` (server): passed.
- `pnpm verify` on #21705 selected 2,738 of 3,565 files (close to the
full suite). I relied on the focused runs and required CI, per
AGENTS.md.

## Risks

- The batched latest-run statement must return the same run as the
single read. The equivalence test covers native and legacy bindings,
ties, other agents and other companies.
- `check()` no longer reads the run state again right after an
acknowledgement. A request that is still owed keeps the deadline from
its own row and is read in full at its next check. The acknowledgement
still revalidates everything before it writes.
- A database error still rejects the whole page, as before. Only the 403
provenance decision is isolated per request.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL tests) and
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:15:08 -06:00
Nicky LeachandClaude Opus 5.5 8cc26221af perf(native): accept a plugin's in-preparation project-auth check on warm attach (#21710)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Environment plugins can run a native Runner as a plugin task, and a
warm session keeps that Runner between turns
> - On a warm turn Core attaches the kept Runner to the new run: the
plugin prepares the new run, then Core checks managed project
authentication
> - A warm attach has no submit step, so Core sends a separate
`check_project_auth` operation after `prepare`
> - That separate operation is one more plugin dispatch and Launcher
round trip on every warm turn, about 0.12–0.18 s, and it is serial
> - This pull request lets the plugin do the check inside `prepare` and
say so, so Core does not send the second operation
> - The benefit is a faster warm turn with the same proof and the same
failure behavior

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The warm attach of a plugin Runner task (`plugin-runner-task.ts`
`prepareLaunch`). When `launched` is true, Core always calls
`checkProjectAuth` after `prepare`.

**Subsystem affected**
Native runtime, plugin Runner tasks, and the plugin SDK environment task
contract.

**Current behavior**
Cold starts fold the check into submit (`projectAuthCheckAtSubmit`).
Warm attaches send `prepare`, then a separate `check_project_auth`.
Measured on a plugin sandbox, the separate operation costs 0.12–0.18 s
on each warm turn.

**Proposed behavior**
The `prepared` result has a new optional flag, `projectAuthChecked`. A
plugin sets it only when it ran the managed project-authentication check
inside this preparation, for this launch's configuration, provider and
projects, and the check allowed them. Core then records the same lease
proof that an allowed `check_project_auth` records, and skips the
separate check.

**Reason and benefit**
One fewer serial round trip on every warm turn. The plugin already runs
the check for this task; this removes a dispatch, not a check.

**Breaking changes**
None. The flag is optional. A plugin that does not set it, or that saw a
refusal, gets the current behavior. Plugins detect the flag with the SDK
schema before they send it.

## What Changed

- `packages/plugins/sdk/src/environment-tasks.ts`: add optional
`projectAuthChecked: true` to the `prepared` result, with a doc comment
that defines it.
- `server/src/services/environment-task-runtime.ts`: when a `prepared`
result has `projectAuthChecked`, record
`runnerTaskProjectAuthValidation` for the launch's configuration,
provider and projects, without `atSubmit`. This is the proof that
`submit` and `read_file` require.
- `server/src/services/native-runtime/plugin-runner-task.ts`: do not
send `check_project_auth` when the prepared result has
`projectAuthChecked`.
- Tests: a warm attach with the flag makes only `prepare`; without it,
`prepare` then `check_project_auth`. The runtime test covers all four
combinations of `projectAuthCheckAtSubmit` and `projectAuthChecked`.

## Verification

- `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts
src/__tests__/environment-task-runtime.test.ts` in `server/`: 157
passed.
- Not run locally: full typecheck and full suite. CI covers them.

## Risks

Low risk. Core trusts the flag the same way it trusts a plugin's
`check_project_auth` answer and `projectAuthCheckAtSubmit` today. A
refusal still goes through Core's own check, so the conflict error is
unchanged. The plugin side ships separately and sends the flag only when
its SDK accepts it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, tracing queries).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 03:13:48 +00:00
DottaandClaude Opus 5.5 b5ba548112 feat(server): add an instance-admin server metrics endpoint for the scale ramp (#21708)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The scaling plan
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`) has a finish line
that a real-agent ramp harness checks (#21695)
> - Four finish-line criteria need numbers from inside each server
process: main-thread CPU, run event lateness, agent API time before the
handler, and missed lease renewals
> - The server had no way to give them, so every harness run was at best
"incomplete"
> - This pull request adds one instance-admin endpoint that returns
those numbers per process over a short sliding window, off by default,
and connects the harness to it
> - The benefit is that a ramp run can measure every server criterion it
can see, with near-zero cost when on and no cost when off

## Linked Issues or Issue Description

Refs #21695

**Subsystem affected**

Server (instance routes, run event append, native lease renewal),
database client (statement counter), and the scale-ramp harness in
`scripts/scale-ramp/`.

**Problem or motivation**

The scale-ramp harness reads `PAPERCLIP_SCALE_METRICS_URL`, but the
server has no endpoint for it. The criteria `main_thread_cpu`,
`run_event_lateness_p99`, `agent_api_pre_handler_p99` and
`lease_renewals_missed_zero` are therefore always "not measured", and no
run can pass.

**Proposed solution**

Add `GET /api/instance/server-metrics`. It returns a JSON snapshot of
one server process over the last 50 to 60 s. Only an instance admin can
read it. The env flag `PAPERCLIP_SERVER_METRICS` (read once at start,
default off) turns on collection and the route. Off, the path answers
404 and no collector starts.

**Alternatives considered**

- The postgres.js `debug` option for the statement counter: rejected,
because it captures a stack (`new Error()`) for every query.
- Process CPU from `process.cpuUsage()` as `mainThreadCpu`: rejected,
because it includes GC and libuv threads. It is returned as `processCpu`
instead, and `mainThreadCpu` reads the main thread from `/proc` (Linux
only, null elsewhere).
- Default on: the measured cost is negligible (numbers below), but the
event-loop monitor wakes the process every 20 ms when idle, and this is
a new instance-wide surface. It stays off; the runbook says how to turn
it on for a ramp run.

## What Changed

- `server/src/services/server-metrics.ts`: a per-process collector. It
uses `performance.eventLoopUtilization()`, `monitorEventLoopDelay` (20
ms), `process.cpuUsage()`, the main thread's
`/proc/self/task/<pid>/stat` on Linux, and six 10 s buckets of native
`perf_hooks` histograms. One unref'd timer rotates the buckets. Nothing
on the hot path awaits, allocates per call or does I/O.
- `server/src/routes/instance-server-metrics.ts`: `GET
/api/instance/server-metrics`, guarded by the existing
`assertInstanceAdmin` (the check that `/instance/database-backups` and
other instance routes use). It returns numbers, a random per-process
`processId`, `uptimeSec`, `buildRevision` and `windowMs`. It returns no
SQL, names, IDs or hostnames.
- `server/src/app.ts`: when the flag is on, two middlewares time agent
API requests from arrival to the API router, the route is mounted, and
shutdown stops the collector. When the flag is off, nothing is mounted.
- The snapshot also carries `dedicatedConnections` from
`dedicatedDbConnectionStats()` (#21693): `limit`, `inUse`, `waiting` at
request time and `peakInUse`, `queued`, `overLimit`, `nestedOverLimit`
since start. The harness ignores it today; it shows dedicated-connection
queueing in a restart burst.
- `packages/db/src/statement-count.ts` and
`lazy-transaction-queries.ts`: a Drizzle logger that increments one
counter per statement replaces Drizzle's no-op logger. No edit to
`client.ts`.
- `paperclip-control-plane-port.ts`: one call records runner `emittedAt`
to committed row for each committed (not duplicate) runner event.
- `native-execution-lease.ts`: one call counts a lease whose renewal
deadline passed.
- Harness: missed lease renewals are summed per `processId` (several
processes, or a restart, no longer cancel out). A metrics URL on the
target origin gets the board token in the `Authorization` header;
`PAPERCLIP_SCALE_METRICS_TOKEN` still overrides it, and another origin
gets no token.
- Contract: `scripts/scale-ramp/server-metrics.fixture.json` is the
serializer's own output. A server test asserts the serializer produces
exactly this file, and a harness test parses it and checks every field.
- `rules/company-scope-exceptions.mdc`: adds the snapshot as an
instance-wide exception with its reason. The runbook in the plan doc
documents the path, the flag, the credential and each field.

| Field | Provided | Source |
| --- | --- | --- |
| `mainThreadCpu` | yes (Linux; null elsewhere) |
`/proc/self/task/<pid>/stat` utime+stime |
| `eventLoopUtilization` | yes | `performance.eventLoopUtilization()` |
| `eventLoopDelayP99Ms` | yes | `monitorEventLoopDelay`, highest 10 s
p99 in the window |
| `statementsPerSecond` | yes | Drizzle logger counter (not BEGIN/COMMIT
or raw driver calls) |
| `poolWaitP99Ms` | absent | postgres.js has no cheap hook for queue
time |
| `runEventLatenessP99Ms` | yes (null with no events) | runner event
append |
| `agentApiPreHandlerP99Ms` | yes (null with no agent requests) |
arrival and router-entry middlewares |
| `leaseRenewalsMissed` | yes, counter since process start | native
lease deadline timer |

## Verification

- `cd server && npx vitest run
src/__tests__/instance-server-metrics-routes.test.ts
src/__tests__/server-metrics.test.ts`: through `createApp` on the
embedded test database. 200 with the exact field set for an instance
admin; 403 for a company owner who is not an instance admin, for an
agent key, and unauthenticated; 404 and no collector when the flag is
off; values move after agent requests, statements, a late run event and
a missed renewal; the body has no IDs, keys or SQL.
- `cd server && npx vitest run
src/services/native-runtime/native-execution-lease.test.ts
src/services/native-runtime/paperclip-control-plane-port.test.ts
src/__tests__/api-server-timing.test.ts`: the lease and run-event hooks
(red/green checked for the lease hook by removing the call).
- `cd packages/db && npx vitest run src/lazy-transaction-queries.test.ts
src/client-options.test.ts`: the counter counts statements inside
transactions and savepoints.
- `node --test scripts/scale-ramp/scale-ramp.test.ts` and `tsc -p
scripts/scale-ramp/tsconfig.json`: 38 pass, including the fixture
contract and the board-token header.
- `node server/scripts/typecheck-layers.mjs` and `tsc --noEmit` in
`packages/db`: pass.

Overhead, measured on a shared, loaded laptop (load average 8 to 20),
median with min and max:

| Measure | Off | On |
| --- | --- | --- |
| Logger call per statement (5M calls x 7) | 0.28 ns (0.24 to 0.97) |
3.36 ns (3.32 to 6.16) |
| CPU per `select 1` statement, embedded Postgres (3,000 x 10) | 28.7 us
(20.3 to 42.4) | 32.3 us (20.7 to 45.5) |
| CPU per HTTP request, client and server in one process (2,000 x 6) |
67.0 us (40.4 to 107.4) | 64.2 us (40.9 to 83.3) |
| Idle process CPU, 5 s x 3 | 0.01% of a core (0.00 to 2.21) | 0.08%
(0.05 to 0.09) |
| Record one run event (Date.parse + histogram) | - | 148 ns (135 to
188) |
| One snapshot | - | 55 us |

The statement and request differences are inside the run-to-run spread.
The direct cost is about 3 ns per statement, against about 30 us of CPU
per statement.

## Risks

- Low risk. Off by default; off, only the statement counter runs, and it
costs one increment more than Drizzle's no-op logger.
- `mainThreadCpu` assumes Linux USER_HZ of 100 and is null on other
platforms. The harness then uses `eventLoopUtilization`.
- Run event lateness compares the runner's clock with the server's
clock, so clock skew adds to it. Negative values count as zero.
- With several processes, each request reaches one process. The harness
keeps the highest gauge and sums counter growth per process, but a
process it never samples is not seen.
- The statement counter does not count BEGIN/COMMIT or raw postgres.js
calls.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:13:45 -06:00
DottaandClaude Opus 5.5 45125f7748 test(server): stop the accounting identity child from timing out on loaded runners (#21712)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server CI runs the general test suite in 12 shards on shared
runners
> - One accounting test fails at random on unrelated pull requests and
passes on re-run
> - Each failure costs a full re-run of the shard and the `verify` job,
and it teaches people to re-run instead of read
> - The test spawns two helper processes that load about 1,400
TypeScript modules under a 5 second kill timer
> - This pull request makes the helper processes plain `node` processes,
so they do not load the server module graph
> - The benefit is that the test no longer depends on runner load, and
it still uses real exited processes

## Linked Issues or Issue Description

**What happened?**
`server/src/services/run-cost-accounting.test.ts > run cost accounting
work > normal accounting Retry reconstructs an uncommitted legacy
receipt after its real controller and child exited` failed in "General
tests (server (5/12))" three times on 2026-11-15 (UTC), each with the
retry also failing:
- PR #21687, head `2172a5b26a`, job 132643921517
- PR #21687, head `6a1d2fb0ef`, job 132650244103
- PR #21678, head `fc0e96f23c`, job 132663296537

Every failure has the same assertion:
`AssertionError:
{"exitCode":null,"signal":"SIGTERM","timedOut":true,...,"stderrTail":"accounting-identity:program-entered\naccounting-identity:module-imported\n"}:
expected null to be +0`
In five of the six failed attempts, the child stopped after
`module-imported` and before `birth-read`. In one attempt, the child
printed its result but the 5 second timer fired before it exited.

**Expected behavior**
The test passes on a loaded runner. The helper processes are not under
test. Only the accounting Retry decision is under test.

**Steps to reproduce**
1. Slow each module load in the child by 8 ms with an `--import` hook in
`NODE_OPTIONS`. The test passes `NODE_OPTIONS` to the child. This
simulates a loaded runner.
2. Run `npx vitest run src/services/run-cost-accounting.test.ts -t
"normal accounting Retry"` in `server/`.
3. On `master` the test fails with the same `timedOut: true` assertion
as CI.

**Root cause**
The child ran `node --import tsx/esm -e "await
import('controller-process-identity.ts'); await
currentProcessStartedAtMs(); ..."`. That import pulls in
`@paperclipai/shared` and `@paperclipai/db` source through tsx, which is
737 modules. On Linux, `readProcessStartedAt` also dynamically imports
the full runner package shim
(`server/src/vendor/paperclip-runner/index.ts`) to call the 30-line
`/proc` reader. That adds 652 more modules. tsx transpiles all of them
in a cold process under a 5 second `timeoutSec`. On a shared CI runner
this crosses 5 seconds.

**Production impact**
None. In the built server, the runner package is already loaded at boot,
and `currentProcessStartedAtMs()` on Linux reads `/proc` once per
process and caches the result. The cost exists only in a fresh tsx
process that imports source.

## What Changed

- The accounting identity helper child is now `node -e` with no tsx
loader and no module imports. It installs a `SIGUSR2` handler, prints
`ready`, and waits.
- When the test sees `ready`, it reads the child start time with the
product function `observedProcessStartedAt(pid)` and then sends
`SIGUSR2`. The child exits with code 0.
- The test builds the recorded `ControllerProcess` (random boot id,
child pid, `currentProcessSpaceHash()`, observed start time). These are
the same values the child computed for itself before. The child process
is still real, has its own process group, and has exited before the
Retry checks run.
- No retries, skips or timeout changes. The 5 second guard now covers a
plain `node` start, which takes about 40 ms.

## Verification

- Red on `master` with the slow-load hook: `Tests 1 failed`, `timedOut:
true`, the same assertion as CI.
- Green with this change and the same hook: `Tests 1 passed`.
- Full file without the hook: `npx vitest run
src/services/run-cost-accounting.test.ts` passes 34/34.
- Helper loop, 30 runs per variant, macOS, 18 cores with 36 busy-loop
processes (load average about 56):
- old child (macOS path): 29/30 exited in 5 s, p50 2,020 ms, p90 2,234
ms, max 5,172 ms
- old child plus the runner import (as on Linux CI): 29/30, p50 3,663
ms, p90 4,411 ms, max 5,023 ms
  - new child: 30/30, p50 38 ms, p90 43 ms, max 46 ms
- The full test under the same stress: 6/6 before and 6/6 after on
macOS. macOS uses the `ps` path and does not load the runner package, so
this loop does not show the Linux failure. The helper loop and the
slow-load repro do show it.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Other tests with the same pattern:
`native-controller-source-digest.test.ts` also spawns a tsx child, but
with a 60 second timeout. It is not the same cause, so this pull request
does not change it.

## Risks

- Low risk. Only one test changes. No product code changes.
- The test no longer checks that a separate process computes its own
identity stamp. That was not the purpose of this test. Other tests cover
`currentControllerProcessStamp()`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:10:11 -06:00
DottaandClaude Opus 5.5 8adf0a463a perf(db): build the agent request authentication statements once per client (#21707)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - One Node process controls every run of a workspace. At 1,000
concurrent agents, main-thread CPU for each database statement is the
scarce resource.
> - A CPU profile of the hosted server shows the cost is per statement:
building SQL text in Drizzle costs about as much as parsing the result
in postgres.js.
> - Every authenticated agent API request (about 0.3 per agent per
second, plus tool calls) runs three fixed statements in
`actorMiddleware`: the agent row, the run authentication projection, and
the responsible user's company memberships. The server built the SQL
text of all three again for every request.
> - A local micro-benchmark shows that building the text is 15 to 40 µs
of CPU per statement for these shapes, and that a reused prepared query
costs the same as a raw driver call. Drizzle's `tracing.ts` wrapper is a
no-op (0.1 µs).
> - This pull request prepares the three statements once per database
client with placeholders, and executes them with the request's values.
The driver receives the same SQL text and parameters as before.
> - The benefit is about 40% less main-thread CPU for each authenticated
agent request in a local benchmark.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The CPU cost of agent JWT and agent key authentication on every agent
API request.

**Subsystem affected**
Server request authentication (`server/src/middleware/auth.ts`,
`server/src/middleware/agent-run-authentication.ts`).

**Current behavior**
For every agent request, `actorMiddleware` builds three Drizzle queries
from scratch: `agents` by id, the `heartbeat_runs` authentication
projection (with `jsonb_build_object` and two `run_identity_contexts`
subqueries), and the `company_memberships` read with an `exists` on
`auth_users`. Building and preparing them costs 14, 38 and 40 µs of CPU
(median of 7 passes of 5,000 builds).

**Proposed behavior**
Build each statement one time for each database client, with
`sql.placeholder` values. Execute the prepared query with the request's
values. The SQL text, the parameters, and the returned rows stay the
same.

**Reason and benefit**
At 1,000 agents, agent API traffic is several hundred authenticated
requests per second on one main thread. Measured CPU per authenticated
request (local, embedded PostgreSQL, `process.cpuUsage()`, 2,000
requests per pass):

| Run (interleaved) | Before: median (min–max) µs | After: median
(min–max) µs | Load average |
| --- | --- | --- | --- |
| 1 | 436 (363–599) | 200 (183–224) | 10.8 / 10.1 |
| 2 | 489 (430–535) | 312 (212–392) | 11.6 / 13.4 |
| 3 | 461 (442–562) | 281 (247–409) | 13.1 / 12.9 |
| separate | 383 (322–810), 301 (268–476) | 169 (156–439), 193 (174–215)
| 9.5 / 9.0 |

The machine is shared (18 cores), so the spread is wide. Every pair
shows the after run lower. The drop is larger than the build cost alone,
because the request also allocates less (fewer builder objects to
collect).

**Breaking changes**
None. The statements, their parameters and the result shapes do not
change.

**Additional context**
Per-statement cost attribution from the new
`packages/db/scripts/benchmark-statement-cpu.ts` (CPU µs per statement,
median of 5 passes of 2,000, two runs, concurrency 1, load average
20–26):

| Statement | Drizzle builder | Build only (`toSQL`) | Drizzle
`.prepare()` reused | `unsafe(text, params)` | `unsafe(..., {prepare:
true})` |
| --- | --- | --- | --- | --- | --- |
| `issues` parent by id | 105 / 122 | 17 / 18 | 59 / 75 | 57 / 63 | 30 /
33 |
| `issues` full row by id | 277 / 326 | 136 / 153 | 96 / 81 | 97 / 77 |
48 / 33 |
| `company_memberships` by principal | 128 / 103 | 35 / 33 | 71 / 66 |
80 / 63 | 37 / 32 |
| join with an in-list of 10 | 169 / 173 | 57 / 57 | 83 / 88 | 109 / 80
| 42 / 40 |
| update by id | 110 / 106 | 25 / 26 | 75 / 80 | 76 / 73 | 39 / 40 |
| insert `activity_log` | 116 / 125 | 34 / 35 | 65 / 57 | 64 / 55 | 41 /
30 |

Drizzle's no-op tracer (`startActiveSpan`) measured 0.1 µs per call.

## What Changed

- `server/src/services/prepared-statements.ts`: add
`preparedStatement(prepare)`, the single-statement form of the existing
`preparedStatements` helper (one prepared query for each `Db`, held in a
`WeakMap`).
- `server/src/middleware/agent-run-authentication.ts`: split the query
into `agentRunAuthenticationQuery(db, binding)`;
`loadAgentRunAuthentication` executes the query prepared once with
placeholders.
- `server/src/middleware/auth.ts`: add `agentAuthenticationQuery` and
`responsibleUserMembershipsQuery`; the JWT path, the agent key path and
`loadResponsibleUserMemberships` execute their prepared forms.
- `server/src/__tests__/agent-auth-prepared-statements.test.ts` (new):
proves that each prepared statement sends the same SQL text and
parameters and returns strictly equal rows, and that repeated agent
requests do not build the statements again.
- `server/src/__tests__/agent-auth-middleware.test.ts`: the test
double's select chain now supports `.prepare().execute()`.
- `server/src/__tests__/prepared-statements.test.ts`: covers
`preparedStatement`.
- `packages/db/scripts/benchmark-statement-cpu.ts` (new): CPU per
statement for the builder, build only, a reused prepared query, `unsafe`
with and without `prepare: true`, and the no-op tracer. Run with `cd
packages/db && ITERATIONS=2000 PASSES=5 node --import tsx
scripts/benchmark-statement-cpu.ts` (`CONCURRENCY`, `ONLY`, `VARIANTS`
are optional).
- `scripts/benchmark-agent-request-auth-cpu.ts` (new): CPU per
authenticated agent JWT request through `actorMiddleware`. Run with
`pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-agent-request-auth-cpu.ts`.

Both benchmarks start their own embedded PostgreSQL in a temporary
directory.

## Verification

- Red/green: the "builds the request authentication statements once per
database client" test failed on `origin/master` (`select` called 12
times in 3 requests) and passes with this change.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/agent-auth-prepared-statements.test.ts
src/__tests__/prepared-statements.test.ts
src/__tests__/agent-auth-middleware.test.ts
src/__tests__/agent-run-authentication.test.ts
src/__tests__/agent-jwt-request-authentication.test.ts
src/__tests__/auth-session-route.test.ts
src/middleware/cloud-tenant-actor.test.ts
src/__tests__/local-agent-process-boundary.test.ts
src/__tests__/liveness.test.ts src/__tests__/inbox-mine-routes.test.ts
src/__tests__/board-bootstrap-routes.test.ts
src/__tests__/agent-auth-identity-transfer.test.ts
src/__tests__/prepared-statements-driver.test.ts
src/services/fresh-continuation-fallback.test.ts`: all pass.
- CPU before and after: the tables above.

## Risks

- Low risk. The prepared query uses Drizzle's own `PgPreparedQuery`, the
same execute path that the builder uses. The test pins the SQL text, the
parameters and the rows against the builder.
- Drizzle's `.prepare(name)` on postgres.js does not send a statement
name to the database (see #21201). With `DATABASE_PREPARED_STATEMENTS`
unset, the driver sends the same unnamed statement as before, so
connection poolers see no change.
- A test double for `Db` must now support `.prepare().execute()` on
these selects. One test double changed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test and benchmark runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:43:39 -06:00
DottaandClaude Opus 5.5 ea8c734cb9 fix(chat): park GitHub terminal responses that wait for a newer run or task change (#21705)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel posts one App-owned response comment per
accepted request. The terminal-response sweep
(`terminal-response-work.ts`) finds processed, non-final responses whose
task's latest run has ended, and asks the acknowledgement owner to post
the final text.
> - On paperclip.staging, 1,557 processed `github_response_comment` rows
are not final. None changed in the last 5 minutes, and the oldest dates
from 2026-10-28. The sweep reads all of them again on each company-wide
rebuild (every 30 s per company), with one page read plus three
statements per row. Two of those statements run at about 13/s each in
`pg_stat_statements`.
> - The sweep cannot make progress on these rows. Their task's latest
run ended without owning the request (a later request started it), or
the pause is already shown. Nothing that the decision reads changes
until a new run starts, the task status changes, or the row changes.
> - This pull request records that wait on the row and skips it on broad
rebuilds until one of those inputs changes.
> - The benefit is that an idle backlog costs one page read per rebuild
instead of about three statements per row, and no request loses a path
to settle.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On paperclip.staging (2026-11-15, about 01:45–02:25Z), two single-row
statements ran at nearly the same rate:

- `select id, run_id, company_id, status, responsible_user_id,
message_id, parent_context_id from run_identity_contexts where id = $1
and run_id = $2 and company_id = $3 and status = $4 limit $5` at
13.37/s.
- `select chat_conversations.issue_id, chat_endpoints.provider from
chat_message_links join chat_conversations … join chat_endpoints … where
chat_message_links.company_id = $3 and comment_id = $4 …` at 13.20/s.

They are `provenanceReads.context` and `provenanceReads.link` in
`run-origin.ts`. The terminal-response sweep reaches them through
`readGitHubResponseRun` for each row it loads.

The backlog of `github_response_comment` actions on staging:

- `processed`, `delivery_id` set, `result.final` not `true`: 1,557 rows
(the terminal sweep's set). None updated in the last 5 minutes. The
oldest `updated_at` is 2026-10-28.
- `result.code` of those rows: null 1,387, `acknowledgement_unconfirmed`
150, `acknowledgement_superseded` 20.
- The acknowledgement set (`received`, or `failed` and retryable) is
empty.
- Other rows: processed and final 6,645, failed and not retryable 265,
cancelled 116.

**Root cause**

`entry()` returns `null` ("nothing to post now") when
`githubResponseExecution` has no body. That happens when the task's
latest run has ended but does not own the request and the task is not
blocked. It also happens when the task is blocked and the row already
shows this pause (`pausedVersion` matches). The page query still selects
these rows, because the latest run has ended (or the task is blocked)
and the row is not final. Nothing records the decision, so each
company-wide rebuild reads every row again: `readGitHubResponseRun` (one
statement) plus `chatRunSourceMessageId` (the context and link reads)
for each row. The codes do not matter. `acknowledgement_unconfirmed` is
a retry code that a later successful acknowledgement keeps, and
`acknowledgement_superseded` marks a request whose shared PR comment has
a newer version. All three groups wait for the same thing: a newer run
that owns the request, or a task status change.

**Expected behavior**

A response that cannot progress costs nothing until the run, the task or
the response changes. A response that can progress settles as before.

**Steps to reproduce**

1. Create a GitHub endpoint and a task with several processed, non-final
response comments (codes null, `acknowledgement_unconfirmed`,
`acknowledgement_superseded`).
2. Give the task a newer ended run whose identity context names a later
request, so the run does not own the older requests.
3. Register the terminal-response sweep and send a company-wide hint
every 30 s.
4. On master, each rebuild reads every row again with about three
statements per row and writes nothing.

**Paperclip version or commit**

master at 958eda6f16.

## What Changed

- `server/src/services/chat-channels/github/terminal-response-work.ts`:
- The page query reads the task's latest run through one lateral
subquery. The subquery returns the run status (for the existing filter)
and a run version (id, status, `updated_at`). The run-index union and
its filters are unchanged.
- A wait key joins the run version, the task status and the row's
`updated_at`.
- When `entry()` decides that a row has nothing to post, the sweep
records `result.terminalWaitKey` in one batched update per page. The
update writes only if the row has not changed since the read, and it
leaves `updated_at` alone, because the key includes it.
- A broad load (every company, or one company with no narrower scope)
skips rows whose stored key equals the current key. Targeted loads
(task, delivery, conversation, endpoint, repository or number) and
`check()` still read every matching row.
- Only waits that do not depend on the owning run's reply are recorded:
the latest run does not own the request, or the pause is already shown.
A request that waits for its owning run's reply publication or native
presentation is never parked, because a publication commit changes none
of the key's inputs and a restart can lose its lease-release hint. Those
requests stay eligible for every broad load, as on master.
- `check()` records the key too when its revalidation finds nothing to
post.
-
`server/src/services/chat-channels/github/terminal-response-work.test.ts`
(new, real PostgreSQL), with staging-shaped fixtures: 30 tasks with 4
waiting responses each (codes null 89%, `acknowledgement_unconfirmed`
10%, `acknowledgement_superseded` 1%), and a newer ended run per task
that answered a later request:
- `stops re-reading processed, non-final responses that cannot progress
on every company rebuild`
  - `still settles a waiting response once a run that owns it ends`
  - `still pauses waiting responses once their task is blocked`
- `still settles a response after a restart lost the hint of the reply
publication it waited for` (an owning run's formal review is
`processing`; it commits with no hint; after a coordinator restart the
broad load acknowledges the request)
- `server/src/__tests__/chat-channels.integration.test.ts`: `finds owed
terminal GitHub work beyond a full page of unrelated request history`
now expects the 101 retained requests to carry a wait key in addition to
their code.

No GitHub comment is edited, posted or deleted by this change.
User-visible behavior is the same: a waiting request keeps its current
comment until a run that owns it ends or its task is blocked, as before.

## Verification

- `pnpm vitest run
src/services/chat-channels/github/terminal-response-work.test.ts`
(server):
- master: 2 of 3 fail (red). The first pass costs 362 statements for 120
rows (2 page reads + 3 per row). Four company rebuilds cost 1,448
statements, with 480 run identity reads, and write nothing.
- this branch: 4 of 4 pass (green). The first pass costs 364 statements
(the same reads plus 2 batched wait-key writes). Four company rebuilds
cost 4 statements (one page read each) and 0 run identity reads. An
owning run and a blocked task still settle their requests.
- The restart test fails on the first revision of this PR (no
acknowledgement) and passes now.
- `pnpm vitest run
src/services/chat-channels/github/terminal-response-work.test.ts
src/__tests__/chat-channels.integration.test.ts -t
"terminal|acknowledg|GitHub response|…"`: 193 passed.
- `pnpm typecheck` (server): passed.
- `pnpm verify` selected 2,738 of 3,565 test files ("retained
conservatively"), close to the full suite. I stopped it and relied on
the focused runs above plus required CI, per AGENTS.md. Before I stopped
it, two unrelated files failed locally:
`agent-commentary-skills.test.ts` (a local skills file is missing,
ENOENT) and one test in `agent-lifecycle/lifecycle.test.ts` (it passes
in isolation).

## Risks

- The wait key must change whenever a skipped row could progress. Its
inputs are the latest run (id, status, `updated_at`), the task status
and the row version. Run writers set `updated_at`, and the publication
lease sends a targeted hint that bypasses the key. A missed input delays
a final comment until the run, the task or the row next changes. It does
not lose the request.
- The first rebuild after deploy still reads the whole backlog once, and
writes one key update per page. The next PR batches those reads.
- `result` gains one internal key, `terminalWaitKey`. Writers that copy
`result` forward also set `updated_at`, which invalidates a copied key.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL tests) and
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Related: #21674, #21675, #21676 (same class: a read loop that makes no
progress).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:40:39 -06:00
DottaandClaude Opus 5.5 82b243ab07 perf(server): bound dedicated database connections with a fair queue (#21693)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server holds long advisory-lock transactions on dedicated
database connections (`withDedicatedDbConnection`): workspace source and
lifecycle locks, the skill cache lock, agent-directory operations,
status cards and others
> - `withDedicatedDbConnection` kept at most 4 idle connections, but it
had no limit on connections in use. A run start can hold two at once,
because the source lock nests the lifecycle lock
> - After a restart or deploy, a burst of run starts can open hundreds
of connections. That can exceed the database's `max_connections` at the
moment recovery needs the database most. The documented per-process
total (`DATABASE_POOL_MAX` + 2) was also wrong
> - This pull request bounds the dedicated connections in use with a
fair queue, keeps nested and in-transaction callers deadlock-free, and
states the real connection budget
> - The benefit is that a restart burst of 200 or 1,000 run starts
completes within the limit (at most 48 in use with the default limit of
24), where before most of the starts failed with "too many clients"

## Linked Issues or Issue Description

**What existing behavior does this improve?**
`withDedicatedDbConnection`, which gives long advisory-lock transactions
a connection outside the main pool.

**Subsystem affected**
Database client (`packages/db/src/client.ts`) and the connection-sizing
documentation.

**Current behavior**
Nothing bounds the dedicated connections in use; only 4 idle ones are
kept. A run start can hold two at once (the source lock nests the
lifecycle lock). A burst of run starts after a restart can open hundreds
of connections and exceed `max_connections`. The documented per-process
total (`DATABASE_POOL_MAX` + 2) is wrong.

**Proposed behavior**
A per-process limit on dedicated connections in use
(`DATABASE_DEDICATED_CONNECTION_MAX`, default 24) with a first-in,
first-out queue. A caller that already holds a dedicated connection, or
runs inside an open transaction, never waits, because its wait could
deadlock with the holder that blocks it. After
`DATABASE_DEDICATED_CONNECTION_WAIT_MS` (default 30 s), a queued caller
gets a connection over the limit, so a lock cycle that PostgreSQL cannot
see ends in a delay, not a hang. The documentation states the real
per-process budget.

**Reason and benefit**
At 50 agents, staging showed about 9 new connections per second from
this function. At 1,000 agents, a restart burst can take the database
down when recovery needs it most. With the default limit of 24, a burst
of 200 or 1,000 starts completes with at most 48 dedicated connections
in use.

**Additional context**
Refs the 1,000-concurrent-agents plan
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`). Reusing the
outer session for a nested lock was considered. The locks are
transaction-level advisory locks (`pg_advisory_xact_lock`), so a nested
lock in the outer transaction would be held until the outer transaction
ends, which changes lock lifetimes and order.

## What Changed

- `packages/db/src/client.ts`: `withDedicatedDbConnection` admits
callers against a per-owner limit (`DATABASE_DEDICATED_CONNECTION_MAX`,
option `dedicatedConnectionMax`, default 24) with a first-in, first-out
queue. The slot is released after the connection is returned or closed.
- Nested callers (an active dedicated hold of the same owner in the
async context) and callers inside an open transaction of a client from
this process are admitted without a wait. The transaction marker is an
`AsyncLocalStorage` scope that ends when the transaction callback ends.
- A queued caller is admitted over the limit after
`DATABASE_DEDICATED_CONNECTION_WAIT_MS` (option
`dedicatedConnectionWaitMs`, default 30000).
- `dedicatedDbConnectionStats(db)` reports the limit, in-use, peak,
waiting, queued, over-limit and nested-over-limit counts.
- `PRIVATE_POOL_MAX_CONNECTIONS` names the private pool sizes that were
literals, and `databaseConnectionBudget()` returns the per-process worst
case: main pool + 9 private + 2 listeners + 2 x limit + 4 idle dedicated
= 73 with the defaults.
- `server/src/services/skill-package-store.ts`: an isolated package read
takes its package I/O permit before its dedicated connection, and takes
a second dedicated connection only for the cache trim after a publish.
Before, each of up to 32 queued readers held a dedicated connection
while it waited for one of 2 permits. `skillPackageStorageIoDepth()`
reports the queue for the store's test.
- `withDedicatedDbConnection(db, action, { admitWithoutWaiting: true })`
admits a caller at once. The skill package store passes it for the
dedicated connections it takes while it holds a package permit, so a
permit holder never waits for a slot that a cache-lock owner holds while
that owner waits for the permit. The overshoot is bounded by the permits
(2 storage, 2 response).
- `doc/DATABASE.md` and `docs/deploy/database.md`: document the
admission rules, the two new variables and the real connection budget,
which replaces "`DATABASE_POOL_MAX` + 2".

## Verification

- New `packages/db/src/dedicated-connection-admission.test.ts` (embedded
PostgreSQL, `max_connections` 100). A restart burst with the default
limit: each start holds a source advisory lock on a dedicated connection
and nests a lifecycle lock on a second one, 40 ms in all.
- Before (same burst, scratch test on master): 200 starts gave 151
failures ("sorry, too many clients already") and 99 dedicated sessions
open. 1,000 starts gave 823 failures and 99 sessions.
- After: 200 starts gave 0 failures, 45 sessions, peak in use 48, queue
wait p50 541 ms / p99 815 ms. 1,000 starts gave 0 failures, 40 sessions,
peak in use 48, wait p50 1,767 ms / p99 3,422 ms. The wait is the queue
of this synthetic burst of 40 ms holds. Before, the same callers waited
about as long for connection setup, and most then failed.
- Steady state: 20 concurrent holders under the limit are never queued
(`queued: 0`).
- Nested acquisition: with a limit of 2, two holders each take a nested
connection while a third caller is queued. All finish (no deadlock); the
queued caller runs after them.
- In-transaction caller: with a limit of 1, a pooled transaction holds a
row lock and then asks for a dedicated connection, while the one
dedicated holder waits for that row lock. The caller is admitted at once
and both finish. A hard limit would deadlock here.
- Resource holder: with a limit of 1, the slot holder waits for a
resource that a second caller holds. The second caller, with
`admitWithoutWaiting`, is admitted at once and both finish.
- Wait bound: with a limit of 1 and a 100 ms bound, a queued caller is
admitted over the limit after the bound.
- Budget: the sum of the actual `max` of every private pool equals
`databaseConnectionBudget().privatePools`, and the default total is 73.
- `skill-package-store.test.ts` ("does not load queued legacy bodies for
files reads"): 32 readers queue for package I/O while only the 2 active
readers hold dedicated connections, and the 33rd reader still gets
"busy" at once. With the old order and a limit of 24, CI showed only 22
readers reaching that queue. `skill-package-store`,
`skill-version-cache`, `company-skills-empty-runtime`,
`runtime-skill-preservation`, `company-skills-runtime-concurrency` and
`company-skills-service` pass locally. Two `company-skills.test.ts`
workspace-discovery tests fail locally on master too (same result with
this change), so they are not caused by it.
- `packages/db`: client, lazy-transaction, prepared-query,
response-capture, work-signal and dedicated-pool suites, 134 passed.
- `server`: skill-version-cache, status-card-evaluation-work,
git-workspace-expiration, execution-workspace-runtime-control-conflict,
agent-instruction-working-copies, company-skills-empty-runtime and
decisions-service suites: 413 passed. In one combined run under machine
load, git-workspace-expiration failed at file level with all its tests
passing; alone it passed 96 of 96.
- `npx tsc --noEmit` in `packages/db`: passes.

## Risks

- A caller that holds an application-level lock (for example a
filesystem lock) and then waits in the queue can block a dedicated
holder that needs that lock. The 30 s wait bound ends such a cycle with
a delay. The over-limit counter shows it.
- Nested and in-transaction callers do not wait, so the in-use count can
reach twice the limit with one level of nesting. The documented budget
counts this.
- In a large restart burst, run starts now queue for a slot instead of
opening new connections. Steady state (about 10 to 20 dedicated
connections at 1,000 runs) stays under the default limit of 24, so a
run's first turn does not wait there.
- The checkpoint pool (3 connections) can limit throughput at 1,000 runs
on a hosted database. This pull request does not change it; it needs an
owner decision on the hosted connection budget.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:33:14 -06:00
DottaandClaude Opus 5.5 9c230ace94 fix(heartbeat): settle deferred chat wakes that can never resume instead of retrying every pass (#21678)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat deferred-receipt worker resumes wakes that waited
behind a running task execution
> - A chat message that arrives while its task runs is saved as a
deferred chat wake
> - When the task finishes (done or cancelled) first, the saved wake can
never resume, but the sweep retried it forever and touched `updated_at`
on every pass
> - An idle instance did one write and one failed resume about 4 times a
second, and the log line did not say why
> - This pull request gives each such wake one recorded outcome, and
makes every other failure wait on a stored, spaced, bounded next attempt
> - The benefit is an idle database stays idle, and a person sees the
existing "not started" result instead of silence

## Linked Issues or Issue Description

**What happened?**
The stranded deferred-queue sweep in
`server/src/services/heartbeat/queue.ts` selects
`deferred_issue_execution` chat-inbound wakes. On every pass it runs
`update agent_wakeup_requests set updated_at = now()` for each wake.
Then `resumeDeferredChatWake` fails because the task is done or
cancelled, so the chat authority denies the work start
(`task_assignment`). Nothing records the failure, so the next pass
repeats it.

**Steps to reproduce**
1. Save a chat message on a task while a run is active, so the message
becomes a deferred chat wake.
2. Let the run finish and mark the task done.
3. Leave the instance idle. The log shows "saved chat source could not
resume after recovery" again and again for the same request ids.

**Expected behavior**
A wake that can never resume gets one terminal outcome with a stored
reason. A wake that can resume later waits on a stored time. An idle
instance runs no statements for these wakes.

**Paperclip version or commit**
master at 36101ebcb9 and later.

**Relevant logs or output**
On the hosted instance with no active runs: 1,405 failed resumes in 6
minutes (3.9/s) across 363 request ids, plus one write per attempt. With
the test harness, 12 stale wakes over 30 idle minutes caused 1,344
statements, 84 wake writes and 84 failure log lines in one process.

## What Changed

- `heartbeat/queue.ts`: the stranded-queue sweep no longer touches
`updated_at` on chat-inbound wakes. Non-chat saved wakes keep the touch.
It moves `updated_at`, which spaces the worker's 30-second deadline for
wakes with an execution wait, and a board chat continuation
(`ai-personal-connection-recovery` e2e) depends on that spacing. The
chat branch now records the outcome of each resume. The log line now has
`reason`, `outcome` and `attempts`. Unscoped sweeps skip a wake whose
stored next attempt time is in the future.
- `durable-chat-wakeup.ts`: add `recordDeferredChatWakeDecline`,
`clearDeferredChatWakeRetry` and `deferredChatWakeDeclineForError`.
- Settle: set the wake to `skipped` with the reason in `error`, set
`finishedAt`, mark the chat delivery with the existing "Input saved. Its
work request was skipped…" text, and notify the inbound wake notice
worker. That worker shows the existing "This follow-up was not started"
notice where a "queued" notice is visible. Wakes older than 24 hours
settle silently (see Risks).
- Retry: store `payload.chatResume = { attempts, reason, nextAttemptAt
}` and the reason in `error`. Delays are 1 min, 10 min, 1 h, 6 h and 24
h. After five attempts the wake settles as `retry_exhausted:<reason>`.
- Permanent authority reasons (`source_shape`, `delivery_binding`,
`conversation_binding`, `destination_binding`, `source_comment`) settle.
Other reasons, such as principal access, runtime fence, privacy or
infrastructure errors, retry.
- `chat-channels/inbound/wakeups.ts` (the deferred chat wake owner): for
the recovery sweep only, return a decline instead of a silent return or
an authority error:
- The task is done or cancelled and the wake is older than
`CLOSED_TASK_CHAT_WAKE_REOPEN_WINDOW_MS` (1 hour): settle with
`task_closed` and start no run.
- The task is done or cancelled and the wake is recent: reopen the task
to `todo` in the same transaction as the authorization check, then
resume through the normal chat admission (`queueIssueAssignmentWakeup`).
This is what ingress does today for a message on a finished task:
`chat-channels.ts` reopens the bound task and admits the message. An
authorization failure rolls back the reopen.
- Several recent saved messages from the same conversation on the
finished task start one follow-up run. The wake that reached the owner
is authorized first, and any decline is recorded on that wake. Each
other recent message is authorized in a savepoint, and a message that
fails keeps its own receipt. The newest authorized message is admitted,
and the run's `wakeCommentIds` holds the whole batch. Inside the
admission transaction, the durable request's `authorize` callback locks
the other receipts. If another run took one of them, it refuses the
whole batch (`chat_inbound_wakeup_batch_changed`, stored retry).
Otherwise `onAdmitted` marks them `coalesced` with the run id in the
same transaction. A restart or a refused admission leaves them deferred.
- The sweep acts only on `inbound_wakeup` chat sources. Task-retry chat
receipts that share the `chat-inbound:` key keep their own owner.
- A refused start ends in the state that admission already gives a
refused chat message. A paused agent saves `executionWait` and resumes
when the pause lifts. A budget hard stop or a non-invokable agent
produces a skipped receipt or a saved wait. Nothing retries on a timer.
- The task is missing or reassigned: settle. The task is in backlog, or
the chat source is still in ingress: retry.
  - Other callers (task retry, cleanup) keep their old behavior.
- `issue-assignment-wakeup.ts`: add the optional `wakeCommentIds` input.
It goes into the run context so that the follow-up run carries every
folded message.
- `heartbeat/deferred-receipt-work.ts`: the deferred worker schedules a
wake at its stored `nextAttemptAt`. A notification that names the exact
wake (for example, the run it waited on finished) still resumes it at
once.
- Tests: add `deferred-chat-wake-sweep.test.ts`. Update three scenarios
in `chat-channels.integration.test.ts` whose source is permanently gone
and which now settle.

## Verification

- New `server/src/__tests__/deferred-chat-wake-sweep.test.ts` (embedded
PostgreSQL, real deferred-receipt worker and heartbeat queue, fake
clock, statement counter):
- 12 stale wakes on done or cancelled tasks, 30 idle minutes. Before
(master): 1,344 statements, 84 wake updates, 84 failure log lines in the
idle window. After: 0 statements, 0 updates, 0 log lines. Each wake is
`skipped` with `error = task_closed` after one attempt, and no run
starts.
- A recent wake (5 minutes old) on a done task reopens the task and
starts one run.
- Two recent wakes on one done task start exactly one run. That run's
`wakeCommentIds` holds both comments, and both wakes record its run id.
- A recent wake whose start is refused (paused agent) keeps the saved
pause wait, makes 0 statements in 30 idle minutes, and starts one run
when the pause lifts.
- A chat wake whose task row no longer exists settles once as
`task_missing`.
- When another run takes a batched message before admission, the batch
is refused with no run. The next attempt answers the remaining message
alone.
- A 3-day-old and a 3-hour-old wake on finished tasks both settle. Only
the 3-hour-old one gets a delivery text and one "not started" notice. A
second notice pass and a fresh notice worker (a restart) add nothing,
and the old one is never a notice candidate.
- When a newer saved message on the same finished task lost its source
comment, the older valid message still gets its run, and the newer one
is never folded into it.
- A retryable failure (paused endpoint) makes 3 attempts in 30 idle
minutes (at 0, 1 and 11 minutes), with the reason in `error`. Then the
endpoint is unpaused and a notification names the wake, and it starts a
run at once.
- A wake deferred behind a running run makes 0 statements in 10 idle
minutes. It resumes at once when that run finishes.
- Red/green: on master the first three cases fail, and on this branch
all four pass.
- Focused suites pass locally: `queue.test.ts`,
`deferred-execution-receipt-work.test.ts`,
`heartbeat-held-comment-wake.test.ts`, `human-directed-work.test.ts`,
`owner-chat-instruction-authority.test.ts`,
`explicit-native-continuation.test.ts`, and the 20 "reconsiders the same
GitHub reply after stopped native workspace cleanup" cases in
`chat-channels.integration.test.ts`.
- `pnpm verify --explain` selected 2,728 files, so I ran only focused
suites on this shared machine. CI supplies the broad coverage.

## Risks

- On first deploy, the existing stuck wakes drain on their own. On the
hosted instance, the 359 chat-inbound wakes on done or cancelled tasks
are all older than the 1-hour window, so they settle once as
`skipped`/`task_closed` and start 0 runs. All but 2 of the 364 are older
than a day, and none is younger than an hour, so the deploy starts 0
follow-up runs. The 5 on blocked or todo tasks get one resume attempt
each. They then resume, wait on a persisted execution wait, or follow
the stored retry schedule and settle within about 31 hours.
- Product decision: the owner chose "close silently" for old saved
messages. A wake older than `SILENT_CHAT_WAKE_SETTLEMENT_AFTER_MS`
settles with its reason in `error` and `finishedAt`, and has no visible
chat effect. Its delivery text is not written, no notice is signaled,
and the receipt gets `payload.silentSettlement = true`. Both the notice
candidate query (`wakeup-notices.ts`) and `resolveInboundWakeReceipt`
skip such wakes, so a later pass or a restart never edits an old
"queued" notice. The value of 24 hours from the wake's `created_at` was
picked by the reviewer, not the owner, so it is easy to change.
- A wake between the two bounds (older than 1 hour, newer than 24 hours)
keeps the existing visible result. Its "Your follow-up is queued."
notice changes once to "This follow-up was not started."
- Visible edits on deploy: on the hosted instance, only wakes younger
than 24 hours can get one. There were 2 at measurement time, and likely
0 by deploy. The other wakes close silently.
- Product decision: the owner chose "yes, if recent" for a message that
arrived just as its task finished. The freshness bound is
`CLOSED_TASK_CHAT_WAKE_REOPEN_WINDOW_MS` in `durable-chat-wakeup.ts`.
Its value of 1 hour from the wake's `created_at` was picked by the
reviewer, not the owner, so it is easy to change. Set it to 0 to settle
every closed-task wake without a run.
- Non-chat saved wakes keep the per-pass touch. The other 75 deferred
wakes on the hosted instance are not chat-inbound, so this PR does not
change their timing.
- No change to authorization, budget, pause or approval checks. The
reopen runs in the same transaction as the full chat authorization, and
the authorization still decides. No wait is added before a run's first
turn.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:31:40 -06:00
DottaandClaude Opus 5.5 11b09b7234 refactor: remove doc/execution-semantics.md and its references (#21699)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Contributors and agents read the docs under `doc/` for background on
run execution
> - `doc/execution-semantics.md` grew to 6,310 lines of per-PR notes
> - It was hard to read and it drifted from the code
> - The code and tests are the source of truth for execution behavior
> - This pull request removes the file and every reference to it
> - The benefit is one less stale document that readers can mistake for
a requirement

## Linked Issues or Issue Description

**Issue type**

Outdated or unclear documentation.

**Where is the issue?**

`doc/execution-semantics.md`, and the docs, rules, skills, maps, code
comments and log text that link to it.

**What's wrong?**

The file is 6,310 lines of notes added one PR at a time. Much of it
repeats or contradicts the code. Readers and agents treat it as a
contract, but the code and tests decide the behavior.

**Suggested fix**

Delete the file. Remove each link to it. Keep the documents that link to
it. Point at code where a pointer is still useful. Do not move the
content to another doc.

Refs #10664 (an open docs PR that edits this file; it becomes obsolete).

## What Changed

- Delete `doc/execution-semantics.md`.
- Remove links and pointers in `doc/PRODUCT.md`,
`doc/LOW-TRUST-PRESETS.md`, `doc/TASK-WATCHDOG.md`,
`doc/composer-stop.md`, `doc/native-workspace-finalization-recovery.md`,
`doc/run-log-events.md`, `doc/runner-internals.md` and
`doc/workspace-repositories.md`.
- `doc/TASK-WATCHDOG.md` now points at
`server/src/modules/active-run-watchdog/` for the silent active-run
watchdog.
- Remove the link in `rules/upgrade-safe-run-state.mdc` and
`maps/feature-map/recovery.md`.
- Update the skills `diagnose-why-work-stopped`,
`fix-failed-paperclip-run` and `terminal-bench-loop`. The
`fix-failed-paperclip-run` skill now points at
`server/src/services/heartbeat/` and `server/src/services/recovery/`.
- Remove the file from the source anchors in `.devin/wiki.json`.
- Remove the doc name from four code comments in `server/` and `ui/`.
- The failed-attachment log message no longer ends with a link to the
removed doc. Two tests now check the message text instead of the link.
- Leave two dated plan files under `doc/plans/` unchanged. They are
history.

## Verification

- `git grep -i execution-semantics` finds no hits outside `doc/plans/`.
- Every relative Markdown link in the changed files resolves.
- `node scripts/check-maps.mjs` passes.
- `node --test .github/scripts/tests/*.test.mjs` passes (808 tests).
- `pnpm verify` was not run locally. The worktree has no `node_modules`.
CI runs the changed Vitest files.

## Risks

- Low risk. Most changes are docs and comments.
- One runtime log line is shorter. It no longer has a link. No logic
changes.
- Open PRs that edit the deleted file, such as #10664, will conflict.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
shell access.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:29:25 -06:00
DottaandClaude Opus 5.5 adc3356ad3 fix(native): retry a coordinator claim that loses a deadlock instead of failing the run (#21703)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native (Paperclip Runner) run claims its
`native_run_finalizations` coordinator before it starts any provider
work. The claim is one transaction. It locks the coordinator row, then
the run row.
> - A writer that holds the run row and then waits for the coordinator
row closes a lock cycle. PostgreSQL then aborts the claim with a
deadlock (`40P01`), because the claim waited first.
> - The aborted claim rolls back whole and starts no provider work. But
the raw database error failed the run before its first turn.
> - The failure path then preserved the prepared workspace. That stamped
the unclaimed coordinator `native_continuation_requires_reconciliation`,
and `failRun` used it as the run's error code. The task waited for
reconciliation.
> - A transient database fault must never end a run like this. The
target rate of runs that need Retry is 0%.
> - This pull request retries the claim transaction on transient lock
contention with the shared bounded helper.
> - The benefit is that a fresh run that loses a lock race at its claim
continues to its provider turn by itself.

## Linked Issues or Issue Description

**What happened?**
On a hosted QA stack, a fresh native assignment run (not a retry) failed
after about 50 s. The run status was `failed`. The error code was
`native_continuation_requires_reconciliation`. The error text was
`Database query failed (40P01)`. The stderr was empty. The task needed
reconciliation.

**Steps to reproduce**
1. Prepare a native run with an unclaimed coordinator (`observed`,
attempt 0), as runtime selection leaves it.
2. In another transaction, lock the run row.
3. Start the native session. Its claim locks the coordinator and waits
for the run row.
4. In the other transaction, lock the coordinator row.
5. PostgreSQL aborts the claim with `40P01`, and
`executePaperclipNativeSession` rejects with that error.

**Expected behavior**
The claim retries after the competing transaction commits. The run
reaches its provider turn.

**Paperclip version or commit**
`master` at 958eda6f16.

## What Changed

- `native-session-executor.ts`: the coordinator claim transaction runs
inside `retryTransientLockContention` (`40P01`, `40001`, `55P03`; 5
attempts, 100 ms × attempt backoff). Each attempt re-reads and re-proves
the coordinator, the run, cancellation and lineage, so a retry admits
nothing that the first attempt would refuse.
- A startup context that a rolled-back attempt opened is finished, and
the startup owner's previous context is restored, before the next
attempt.
- New PostgreSQL test
`native-coordinator-claim-deadlock.integration.test.ts`: `retries a
claim that lost a deadlock and reaches the provider`.

## Verification

- RED on `master` (958eda6f16): the new test fails.
`executePaperclipNativeSession` rejects with a `40P01` cause, and the
provider is not called.
- GREEN with this change: `pnpm --filter @paperclipai/server exec vitest
run
src/services/native-runtime/native-coordinator-claim-deadlock.integration.test.ts`
passes. The provider is called once and the coordinator is at attempt 1.
- `tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` for the affected files: see the PR comments.

## Risks

- Low risk. The normal path has no contention and does not retry.
- Under contention, the claim waits at most about 1 s more (backoff
100–400 ms) before provider work. This is before `turn.start`, so time
to first token grows only in the contended case.
- The lock order (coordinator, then run) is unchanged. It matches
admission and the runner checkpoint owner lock. The writer that took the
run row first on the QA stack is not identified, because no server logs
or SQL were available. This change makes the claim survive any such
writer.
- Out of scope: a transient fault in other pre-dispatch statements of a
run with a prepared remote workspace. There, workspace preservation
still stamps the pre-claim code before `failRun`. The pre-claim
fresh-continuation fallback then continues the task.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, embedded PostgreSQL tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20561 (pre-dispatch transient retry), #20682 (provider-pack
admission lock wait).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:23:33 -06:00
DottaandClaude Opus 5.5 c38a471042 perf(server): download stale agent files before the agent row lock (#21694)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Durable agent files live in object storage. Each server keeps a
local copy (the projection), marked with the head generation it matches
> - When the projection is stale, the next locked read or write restores
it: it downloads every file from object storage. This happens for the
first run of each agent on a new server (every deploy on hosted
ephemeral servers) and after another server wrote
> - The download ran inside the transaction that holds the `agents` row
lock (`FOR UPDATE`), on the main pool for the run-start read
(`readCommittedForRuntime`) and on the 2-connection file-storage pool
for the store. A download can take up to 30 s
> - After a deploy at 1,000 agents, this is one long row lock and one
pinned connection per agent, all at once, and every writer of that agent
waits behind it
> - This pull request downloads the files outside the transaction and
takes the lock again only to swap them in
> - The benefit is that the row lock is held for milliseconds instead of
the download time (0 ms observed instead of 321 ms with a 300 ms
download in the test), and writers do not wait for a download

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The restore of a stale local copy of an agent's durable files (the first
run of each agent on a new server, or after another server wrote).

**Subsystem affected**
Server: durable agent files (`agent-file-durability.ts`,
`agent-file-store.ts`, `agent-instruction-revisions.ts`).

**Current behavior**
The restore downloads every file from object storage inside the
transaction that holds the `agents` row lock (`FOR UPDATE`) and a pooled
connection. A download can take up to 30 s, and every writer of that
agent waits behind it.

**Proposed behavior**
Under the lock, detect that a download is needed and roll back. Download
the head's files, verified by digest, into a private staging directory
with no lock. Run the transaction again, and swap the staged files in
only when the head generation is unchanged. After two changed heads,
restore under the lock as before.

**Reason and benefit**
After a deploy at 1,000 agents, every agent's first run restores at
once. The row lock and connection are now held for milliseconds instead
of the download time, and writers do not wait for a download.

**Additional context**
Refs the 1,000-concurrent-agents plan
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`). The save path
had the same problem for uploads and was fixed earlier with uploads
before the lock (`uploadAgentFilesBeforeCommit`); this is the restore
side. Reading the head before every transaction was considered; it adds
a query to every call, while the deferral costs nothing when the
projection is current.

## What Changed

- `agent-file-durability.ts`: the download loop moves into
`downloadAgentFiles`, unchanged.
`withAgentFilesRestoredBeforeLock(work)` runs `work` in a scope. Inside
the scope, the first restore of an attempt that needs a download throws
a private deferral error, which rolls back the transaction. The wrapper
stages the download in `.agent-files-<uuid>` next to the root and runs
`work` again. `restoreDurableAgentFiles` uses a staged directory only
when its generation equals the head it reads under the lock. Staged
directories are removed when the scope ends.
- Only the first restore of each attempt can defer, so nested restores
of other agents keep their old behavior. Outside the scope, the behavior
is unchanged.
- `agent-file-store.ts` (`locked`, so every store read, write, apply and
working-copy preparation) and `agent-instruction-revisions.ts`
(`readCommittedForRuntime`, `readCurrent`, `materializeCurrent`) run
their transactions in the scope.

## Verification

- New test `server/src/__tests__/agent-file-restore-locking.test.ts` on
embedded PostgreSQL, with an object-storage fixture:
- Hold time, with a 300 ms download, through `readCommittedForRuntime`
(the run-start read). Before: the download ran with the agent row
locked, and the row was locked 321 ms of 322 ms. After: the download ran
with the row unlocked, and a sampler saw the row locked for 0 ms.
- Concurrency: a board write of the same file runs while the reader's
download is blocked. Before: the writer waited for the reader's download
(5 s timeout). After: the writer commits a new head during the download.
The reader then discards its stale staged copy and returns the writer's
content. The projection is marked at the writer's generation, and no
staging directory is left.
- Failure: when object storage fails, the read fails once with the
storage error, after one attempt made without the row lock, and leaves
no staging directory.
- Progress: when the head generation changes during each of two
downloads, the third attempt restores under the lock as before (3
downloads), and the projection matches the final head.
- `npx vitest run` for agent-file-checkpoint-locking,
agent-file-checkpoints, agent-file-durability,
agent-file-restore-locking, agent-instruction-revisions,
agent-instruction-working-copies, agent-instructions-service,
agent-directory-working-copies and agent-instructions-routes: 287
passed, 1 skipped.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.

## Risks

- A deferred attempt runs the transaction body again. The deferral
happens at the first restore, before the body writes files, and the
transaction rollback undoes database work, so the second attempt starts
from the same state.
- A staged download uses disk next to the agent root until the scope
ends, the same as the temporary directory of the old download.
- If the head changes during every download, the third attempt downloads
under the lock as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:14:47 -06:00
DottaandClaude Opus 5.5 cb9fe3f72d perf(server): capture a remote agent-file checkpoint before the directory lock (#21690)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote run saves its agent files at the end of every turn
(`checkpointWarm`)
> - On hosted (ephemeral) servers, this save holds the cross-controller
`agent-directory` advisory lock in an open transaction on a dedicated
database connection
> - The lock was held across the full remote capture: an asset upload,
the checkpoint script, the download and the scratch cleanup in the
sandbox. Each step is a sandbox round trip of hundreds of milliseconds
or more
> - At 1,000 concurrent agents, connection-seconds are the scarce
resource, and this hold occurs on every turn of every warm remote run.
The lock is per agent, so a second run of the same agent got "Agent
directory operation is already active" for the whole capture
> - This pull request captures the remote files before the lock, and
takes the lock only to commit
> - The benefit is that the lock and its connection are held for the
commit only (85 ms instead of 2.5 s in the test with a 200 ms sandbox),
and no sandbox call runs under the lock

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The per-turn save of agent files for a warm remote run
(`checkpointWarm`) on hosted servers.

**Subsystem affected**
Server: agent directory working copies
(`server/src/services/agent-directory-working-copies.ts`).

**Current behavior**
The save holds the per-agent `agent-directory` advisory lock, an open
transaction and a dedicated database connection across every sandbox
round trip of the remote capture. A second run of the same agent gets
"Agent directory operation is already active" for that whole time.

**Proposed behavior**
Capture the remote files with no lock. Under the lock, use that capture
only when the working-copy row is unchanged. Otherwise discard it and
capture again under the lock, as before.

**Reason and benefit**
At 1,000 concurrent agents, connection-seconds are the scarce resource,
and this hold occurs on every turn of every warm remote run. The lock is
now held for the commit only.

**Additional context**
Refs the 1,000-concurrent-agents plan
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`). A lease row in
place of the advisory lock was considered; it is a larger change, and
the optimistic capture removes the slow part without a new protocol.

## What Changed

- `agent-directory-working-copies.ts`: `checkpointWarm` captures the
remote checkpoint before `serial()` for a disposable remote copy (hosted
servers only), with the same inputs that the locked capture uses.
- Under the lock, `checkpoint()` reuses that capture on its first
attempt only when the current row equals the row the capture used
(`adoptionAuthority`, which excludes only `updatedAt` and
`nextAttemptAt`) and the target is the same. If the row changed, it
captures again under the lock, as before.
- If the early capture fails, the code logs a warning and captures under
the lock, as before. The early capture's scratch is removed after the
lock is released.
- Local copies and non-hosted servers are unchanged.

## Verification

- New test
`server/src/__tests__/agent-directory-remote-checkpoint-lock.test.ts` on
embedded PostgreSQL, with the real checkpoint transport and a fixture
sandbox runner that takes 200 ms per call:
- Before: 11 of 11 sandbox calls ran while the `agent-directory`
advisory lock was held. The lock was held 2,482 ms of a 2,515 ms
checkpoint.
- After: 0 of 11 sandbox calls ran under the lock. The lock was held 85
ms of 2,863 ms.
- Concurrency: a competing collector (`collectStopped` from a second
service instance) runs while the early capture is blocked in the
sandbox. Before: the collector failed with "Agent directory operation is
already active". After: the collector saves and releases the copy, and
the checkpoint discards its early capture and returns the collector's
row unchanged. The canonical file has the final turn's content once, and
no scratch directory is left.
- `npx vitest run src/__tests__/agent-directory-working-copies.test.ts
src/__tests__/agent-file-checkpoint-locking.test.ts
src/__tests__/heartbeat-agent-directory-busy.test.ts
src/__tests__/agent-directory-lock-crash-point.test.ts
src/__tests__/agent-instruction-working-copies.test.ts`: 189 passed, 1
skipped.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.

## Risks

- The capture now runs while another controller can operate on the same
copy. The capture only reads the remote directory into private scratch.
The commit uses it only when the row is unchanged, so the result is the
same as one of the orders the lock allowed before: the checkpoint first,
or the other operation first and then the old locked checkpoint.
- The agent can change remote files between the capture and the commit.
That was also true before, because the agent is not frozen during a
checkpoint.
- Changed bytes are still uploaded to object storage inside the lock
(before the agent row lock). Moving that upload out is a possible
follow-up.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:13:54 -06:00
DottaandClaude Fable 5.2 d6acbf50e6 refactor: cut DATABASE.md to setup and schema-change rules (#21700)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `AGENTS.md` tells every contributor and agent to read
`doc/DATABASE.md` before a schema or migration change
> - That file grew to 2,611 lines. About 1,950 of them were notes on the
tables of one feature, added one pull request at a time
> - The setup steps and the migration rules started at line 792, below
those notes
> - This pull request cuts `doc/DATABASE.md` to 377 lines: setup, and
the rules that apply to every schema change
> - The benefit is a guide that a person or an agent can read in full
before a schema change

## Linked Issues or Issue Description

Refs #21670 (the same cleanup for `doc/DEVELOPING.md`).

**Issue type**

Outdated or confusing documentation.

**Where is the issue?**

`doc/DATABASE.md`

**What's wrong?**

The file was 2,611 lines. It opened with a Better Auth upgrade note and
492 lines on task query performance. The schema change workflow started
at line 1367.

**Suggested fix**

Keep database setup and the general rules in `doc/DATABASE.md`. Remove
the notes on individual tables. The schema and its tests describe those.

## What Changed

- `doc/DATABASE.md` is now 377 lines. It keeps the three database modes,
client tuning, the schema change workflow, migration compatibility, the
migration authoring checklist, migration snapshots, the rule for retries
after a lost connection, transaction-aware work signals, plugin
namespaces, and backups. The wording of the kept text is unchanged.
- New `doc/database-client-internals.md` holds the response-size
diagnostic, the notes on the postgres.js patch, and the transaction
memory note. The text is moved without wording changes.
- "Experimental remote workspace snapshots" moved to
`doc/workspace-storage.md`, which links to it.
- "Execution admission accounting" moved to
`server/src/modules/README.md`, which links to it.
- Deleted about 1,950 lines of notes on one feature each. No other file
linked to them. The deleted sections are: Better Auth 1.7.16 upgrade,
Durable native provider backups, Workspace cohort admission proofs,
Original local Git sources for archived workspaces, Proactive agent chat
messages, Persistent case handoffs, Task query performance, Tool profile
reads, Execution identity row locks, Remote workspace row locks, Managed
adapter settings, Cloud configuration migration receipt, Cloud runtime
identity singleton, Unclaimed Cloud warm standby, Native finalization
history and idle sleep, Feedback history and idle sleep, Resource
membership tables, Decision training snapshot retention, Decision queues
and triage provenance, Native runner persistence, Chat communication
snapshots, Telegram private draft identities, Attachment upload
provenance, Question-response delivery receipts, Secret storage,
Persistent agent conversations, Resource lifecycle events, Legacy
controller ownership, Agent file persistence and legacy revisions, Large
API response snapshots, Runner API response reservations, Internal agent
commentary, Agent identity keys and backups, Slack app registration,
Slack manager workspace grants, Installed skill packages, Project
repository skill synchronization, Task workspace records, Execution
recovery query performance, Agent runtime synchronization journal,
Retired Pipelines storage, Prompt injection classification, Deferred
conversation requests, GitHub inherited review-wait reads, Sandbox
creation ownership, Object-backed run evidence, Cloud runtime
configuration singleton, Independent accounting and budget evaluation,
Standard workspace and Classic opt-in, Project environment rolling
compatibility.
- Also deleted from kept sections: the subscription-table note and the
issue-reference backfill note in the embedded mode section, the HTTP
access log note and the per-endpoint retry notes in the connection loss
section, and the notes on migrations `0355`, `0313`, and `0314` in the
authoring checklist.
- `AGENTS.md` now states what `doc/DATABASE.md` is for.

## Verification

- A script built the new files by line range from the old file. It
asserts the old line count and the first line of each range.
- A link check on every changed Markdown file, and on the four files
that link to a `DATABASE.md` anchor, found 0 missing files and 0 missing
anchors.
- `node scripts/check-maps.mjs` passes.
- `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail.
- `pnpm verify` was not run. This worktree has no installed
dependencies, and the change has no runtime code.

## Risks

- The deleted notes were not all read in full before deletion. The
decision was by section: a section about one feature's tables, with no
inbound link, was deleted. The text stays in git history.
- Two deleted sections describe procedures for an operator: the
duplicate-account check before the Better Auth 1.7.16 migration, and the
secret storage tables. Restore them into a focused doc if operators
still need them.
- No runtime behavior changes.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool
use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 20:01:28 -06:00
DottaandClaude Opus 5.5 958eda6f16 fix(native): wait for a live outgoing server before resuming its warm session in a deploy overlap (#21697)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap. The outgoing server still
serves while the incoming server starts new runs.
> - A warm agent session stays with the server that ran its last turn.
That server closes the session about 1 s after it leaves new run starts
to the replacement (#21627).
> - A follow-up that started within about 1 s of its predecessor's
finish read the session as suspended, while the outgoing server was
still closing it. The follow-up did not wait. It started a second Runner
with the wrong OpenCode launch files, and the Runner refused the launch.
> - The same-run retry then went to the outgoing server, which had
already left run starts. The run failed, and a blocked parent showed a
Retry during its scheduled resume.
> - This pull request makes the follow-up wait for the live outgoing
server, rebuilds the admitted launch files on the new Runner, keeps
same-run retries on the replacement, and shows the scheduled resume for
a blocked parent.
> - The benefit is that a follow-up during a deploy overlap finishes
without a failed run or a Retry banner.

## Linked Issues or Issue Description

Refs #21627

**What happened?**
A zero-downtime release-swap test (Gate B, wave B1 on master 890dcbdd66)
failed 2 of 8 cross-server hand-offs. In both, the follow-up started 1.0
s and 0.6 s after its predecessor finished on the outgoing server. The
other 6 hand-offs started 34 to 217 s later and passed.
- The incoming server took the resume path that skips
`inspectRemoteWarmSessionRecovery`, because the local authority state
read `suspended`. The live-owner wait from #21627 lives only in that
inspection, so it never ran (0 waits recorded).
- The new home had no harness backup, so the run had no provider
artifact receipt. `prepareRemoteRunner` returned early on
`legacyProviderContinuity`, and the launch profile named the provider
pack folder. Runner core refused it: "OpenCode run.prepare launch does
not match the runner-owned qualified profile".
- The same-run retry was claimed by the outgoing server after it left
run starts. Its predecessor retirement reported `not_suspended`. Both
runs failed with `native_continuation_requires_reconciliation`, and
recovery started fresh runs.
- For a parent already blocked on its subtasks, the task showed "failed"
with Retry for 31 s while the same-run resume was scheduled.

**Expected behavior**
The follow-up waits while the outgoing server may still hold the
session. A new Runner gets the same OpenCode launch files as the
session's first launch. Only the replacement claims same-run retries. A
blocked parent shows the scheduled resume and no Retry.

**Steps to reproduce**
1. Run the restart-chaos release-swap preset with fresh homes and a warm
OpenCode staging mix.
2. Let a run finish on the outgoing server after the traffic switch,
with a follow-up comment already queued.
3. See the follow-up start on the incoming server within about 1 s and
fail on `run.attach` with the qualified-profile mismatch.

**Paperclip version or commit**
master 890dcbdd66.

**Deployment mode**
Hosted deploy overlap (release swap) with Daytona sandboxes and S3
storage.

## What Changed

- `remote-warm-session-recovery.ts`: new `waitForLiveWarmSessionOwner`.
It reads the finished run's finalization owner. While that server is
alive inside the warm window, the follow-up defers with the existing
`native_controller_recovery_waiting_for_owner` wait. A harness backup
that holds the suspended state and was completed after the turn proves
that the owner finished its close, so the follow-up continues at once.
- `native-session-executor.ts`: calls that wait before the `suspended`
check, so every warm follow-up on another server's session passes
through it. The backup check runs only when the owner is alive.
- `native-session-executor.ts`: when a resume has a durable session but
no receipt from a harness backup or the admission, it reads the provider
artifact receipt recorded on the session's last run. The new Runner then
gets the admitted launch root, as a cold continuation already does.
- `heartbeat/recovery.ts` and `heartbeat.ts`: a server that yields run
starts to a replacement does not claim same-run retries, from the
recovery sweep or from its own resume timer. It re-checks every 3 s, so
it takes the retries back when the replacement stops.
- `task-interruption-presentation.ts`: a blocked task whose remote run
waits for its `exact_checkpoint_resume` keeps the resume time, so it
shows `retry_scheduled` with no Retry.
- `doc/execution-semantics.md`: documents these rules.

## Verification

Red on master, green with this change:
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "close
window waits for its live lease"`: "a follow-up that starts within the
outgoing server's close window waits for its live lease instead of
relaunching the Runner". Master proceeds and fails with
`runner_state_identity_mismatch`.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "remote
opencode-continuation-prior-receipt"`: a resume relaunch without an
adopted process uses the original OpenCode file paths. Master launches
from the provider pack folder and does not retain the receipt.
- `cd server && npx vitest run src/services/heartbeat/recovery.test.ts
-t "leaves run starts"`: "an outgoing server does not claim a same-run
retry once it leaves run starts". Master claims it.
- `cd server && npx vitest run
src/__tests__/task-interruption-query.test.ts -t "blocked parent whose
run awaits"`: "a blocked parent whose run awaits its exact same-run
resume shows retry_scheduled with no Retry". Master shows `failed` with
Retry.

Also new: "keeps a follow-up waiting for a live outgoing server until
that server has checkpointed the session after the turn" in
`native-controller-recovery.integration.test.ts` (the backup proof and
the owner exit end the wait).

Focused suites pass: native-session-executor,
remote-warm-session-recovery, native-controller-recovery (integration),
heartbeat recovery, task-interruption-query,
task-interruption-presentation, native-recovered-owner-followup,
heartbeat-native-predispatch-retry, run-start-handover. `npx tsc
--noEmit -p server/tsconfig.json` passes.

Not run here: the restart-chaos release-swap wave. The maintainer runs
it after merge.

## Risks

- A follow-up on a session that a live outgoing server ran can now wait
until that server exits or the warm window ends. This applies only when
no harness backup proves that the server finished its close after the
turn. Before, such a follow-up failed and needed recovery.
- The backup proof hashes the restored backup. It runs only when the
previous owner is alive in another process (a deploy overlap), not on a
normal turn.
- A resume without a backup receipt now restores the admitted artifacts
from storage, as a cold continuation does. A session whose last run
recorded no receipt keeps the legacy pack-folder launch.
- A yielding server leaves same-run retries to the replacement. If the
replacement stops, the yield ends with its lease, and the older server
claims them again within seconds.
- Previous-release state: no persisted field is added. Receipts,
finalization owners and `failureDetail.recoveryMode` are already saved
by the previous release.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 19:50:01 -06:00
DottaandClaude Opus 5.5 54cde3353b perf(recovery): back off assigned-work re-checks that find the same refused state (#21691)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The assigned-work sweep checks every open task assigned to an agent
(`todo`, `in_progress`, `blocked`). When a task has nothing running,
nothing pending and no one else's move, it claims one automatic
continuation and asks the wake queue to start it.
> - When the wake queue refuses that continuation, the claim is recorded
as "not started" and the task is checked again 90 seconds later. The
per-task spacing (2, 4, 8, then 15 minutes) lets it claim again. The
refusal is not counted toward the hourly cap, so this never ends.
> - Each cycle reads all of the task's facts, writes an activity entry
and saves a skipped wake row. For 10 such tasks this is about 5,000
statements and 50 claims per hour, forever, on an idle server.
> - This pull request stores a fingerprint of the task's facts with each
refused continuation. A later check that finds the same facts waits,
with reads only, on a backoff that doubles to a 6-hour ceiling.
> - The benefit is that a task that cannot progress costs almost nothing
while idle, and any real change on the task is still picked up within
seconds.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

`act` in `server/src/services/recovery/assigned-work-sweep.ts` repairs a
`no_pending_path` or `retry_offered` task with `deps.enqueueWakeup`.
When admission refuses (for example the assignee has `wakeOnDemand:
false`), `act` returns `enqueue_refused` with `retryAt` 90 seconds
later. The next check claims again as soon as `continuationAllowance`
allows: 2, 4, 8, then 15 minutes. Refused claims do not count toward
`CONTINUATION_CAP`, so the task is re-claimed forever. Each claim writes
an `issue.assigned_work_continued` activity entry ("could not start yet.
It will try again shortly."), and each wake attempt saves a skipped wake
row.

Census on a staging-shaped fixture (real PostgreSQL, real report): 60
tasks in review, 10 running, 5 assigned to a paused agent and 10 whose
continuation is refused. Only the 10 refused tasks are violations
(`retry_offered`). Tasks in review are not in the sweep's scope at all,
and a paused assignee is an "agent paused" wait. So on a hosted instance
where most open tasks are in review, only the tasks whose continuation
is refused pay this cost. One claim check costs 102 statements. A
backed-off re-check costs 34 statements and writes nothing.

**Expected behavior**

A re-check that finds the same state as the last refused continuation
stores nothing and backs off. A real change on the task (a run finishes,
a status change, an assignment, a budget or pause change, a new wake)
re-checks it promptly.

**Steps to reproduce**

1. Create 10 tasks in `in_progress`, assigned to an agent with
`runtimeConfig.heartbeat = { enabled: false, wakeOnDemand: false }`,
each with a failed run.
2. Start the issue-state recovery work (it registers the assigned-work
sweep).
3. Leave the instance idle. The sweep claims and is refused about every
2 to 15 minutes per task, forever.

**Paperclip version or commit**

master at 3eabdd6d10.

## What Changed

- `server/src/services/recovery/assigned-work-sweep.ts`:
- `act` computes a fingerprint of the task's facts and the violation
(`stateKey`, without the task's `updatedAt`, which a refused claim on a
blocked task rewrites). Before it claims, it reads the task's latest
continuation entries (`refusalsInState`, one indexed read). When the
newest entries are refused continuations with the same fingerprint, it
waits until the last refusal plus 2 minutes, doubled per refusal, up to
6 hours. That check returns the new outcome `waiting_unchanged` and
writes nothing.
- A refused continuation (the ordinary path and the stalled-run
replacement) stores `stateKey` and `refusedAt` in its activity entry.
From the second refusal in the same state, the entry says "Paperclip
still could not start this task. It will keep trying, less often each
time, and right away when something on the task changes."
- The agent fact now carries the agent's `updatedAt`, so a change to the
agent's configuration changes the fingerprint.
- `server/src/services/recovery/work.ts`: a task that waits on
`waiting_unchanged` is marked on its queue entry. When an event reloads
the task, the entry becomes due at once instead of keeping the later
deadline. A backstop reload keeps the deadline.
- `server/src/services/recovery/assigned-work-invariant.ts`: the new
activity text, and the optional `updatedAt` on the agent fact.
- `server/src/__tests__/assigned-work-invariant.test.ts`: the "too soon"
check after a refusal now reports `waiting_unchanged` instead of
`deferred_rate_limit`.
- `server/src/__tests__/assigned-work-recheck-idle.test.ts`: three new
tests on real PostgreSQL with the real coordinator, the real issue-state
recovery work and a fake clock.

What the user sees: the first refused continuation still says "could not
start yet. It will try again shortly." Later refusals in the same state
say Paperclip keeps trying less often. The task stays assigned, and any
Retry the task shows is unchanged.

About `retry_offered`: the request was to treat it as "a person was
offered Retry, so nothing re-checks until they act". The code does not
support that reading. `retry_offered` is defined as "the visible state
offers a Retry that only the system can act on", and the sweep continues
it so that the person does not need to press Retry. A rule that stops
after one refusal would turn every transient refusal into a task that
needs a person. So this pull request gives `retry_offered` the same
backoff as `no_pending_path`. A person's Retry or message is still an
event that re-checks the task at once.

## Verification

- `pnpm exec vitest run src/__tests__/assigned-work-recheck-idle.test.ts
src/__tests__/assigned-work-invariant.test.ts` (in `server/`), 35 tests
pass.
- `backs off while every check finds the same refused state`: 10 refused
tasks, measured on the fake clock.

| Window | master claims | master statements | this branch claims | this
branch statements |
    | --- | --- | --- | --- | --- |
    | first idle hour | 50 | 5,060 | 40 | 5,078 |
    | idle hours 1 to 6 | 1,460 | 147,752 | 21 | 2,866 |

The master numbers after the first hour are inflated by the test: the
cap and spacing read activity `created_at`, which comes from the
database clock, not the fake clock, so after one fake hour master spaces
claims by only 2 minutes. At the production spacing (15 minutes, about 6
claims per task per hour), master would do about 300 claims and 25,000
statements in the same 5 hours. The assertion (at most 30 claims and
fewer than 5,000 statements in hours 1 to 6) fails on master and passes
on this branch.
- `re-checks a backed-off task within two minutes when one of its runs
finishes`: after 3 idle hours, a new failed run of one task is notified.
Within 2 minutes (the 90-second settling grace) the sweep claims that
task again. Without the `work.ts` change, the event keeps the backed-off
deadline and this test fails.
- `leaves review work alone and re-checks a refused task with reads only
while it waits`: the census above, plus one claim check (102 statements,
writes an activity entry) and one backed-off re-check (34 statements, no
insert, update or delete).
- Neighbouring suites pass locally: `assigned-work-sweep-round-trips`,
`assigned-work-stalled-queued-run`,
`failed-continuation-continues-again`,
`automatic-run-brake-recovery-passes`,
`issue-recovery-recorded-decision-idle`, `recovery-wake-herd`,
`issue-recovery-signal-fanout`, `saved-work-recovery-overdue`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- A refusal whose cause clears without any change to the task's facts or
a reloading event (for example a scheduling suppression that lifts) is
retried on the backoff, at most 6 hours later, instead of within 15
minutes. Changes to the task, its runs, its wakes, the agent row, budget
or company pause change the fingerprint and claim at once.
- The fingerprint covers the full fact set. A fact that changes on every
check would turn the backoff off for that task and keep the master
behavior. The idle test shows that the refused fixture keeps a stable
fingerprint.
- An event that reloads a backed-off task now re-checks it at once. That
check is read-only while the state is the same (34 statements).
- No schema or API change. The new fields live in the existing activity
`details` JSON.
- Separate finding, not changed here: a task whose pending continuation
counts as its pending path is not re-checked when that continuation's
bound ends unless something reloads it. With continuation delivery
running this is rare, but it is a missed re-check rather than a loop.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21687, #21621, #21674, #21675, #21676, #21617.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 19:42:25 -06:00
DottaandClaude Opus 5.5 f4e025a1cf perf(recovery): wait for an assignee that cannot run instead of retrying its continuation every pass (#21687)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops and recovery decides to continue the task in a
fresh run, it records a pending continuation. The execution-continuation
worker delivers it with `heartbeat.wakeup`.
> - When the task's assignee cannot take work (paused, terminated,
waiting for hire approval, or an invalid reporting chain), the wake
refuses. The worker recorded only an "attempted at" time and tried again
on the next 15-second pass, forever.
> - Each refused wake also saved a new skipped wake row. An idle server
with one such task did about 13,000 statements and 390 wake attempts
every 2 hours.
> - This pull request makes delivery check the assignee with the same
rule the wake uses. When the assignee cannot run, the continuation waits
on a stored retry time and is re-admitted at once when the assignee can
run again.
> - The benefit is that an idle server does almost no work for these
tasks, and a resumed agent still gets its continuation within seconds.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

`deliverReconciledExecutions`
(`server/src/services/execution-recovery-resolution.ts`) offers each
pending continuation (`continuationDelivery: "pending"`) to
`heartbeat.wakeup`. When the assignee is not invokable, the wake throws
"Agent is not invokable in its current state" and saves a skipped
`agent_wakeup_requests` row. The worker catches the refusal, records
`continuationDeliveryAttemptedAt`, and the continuation schedule
(`executionControlSchedule.continuation`) asks for another pass in 15
seconds because a pending row exists. Nothing ever changes, so the loop
never ends.

Measured on real PostgreSQL with the real coordinator and the real wake,
on a fake clock, for one pending continuation over 2 idle hours:

| Assignee status | Wake attempts (master) | Statements (master) |
| --- | --- | --- |
| paused | 380 | about 12,980 |
| terminated | 376 | about 12,820 |
| pending_approval | 395 | about 13,000 |

An agent in `error` is not affected: it is invokable, and the wake
delivers the continuation on the first pass. Only the statuses and
conditions that make an agent not invokable cause the loop.

**Expected behavior**

A continuation that cannot be delivered because its assignee cannot run
does not try again every pass. It waits for a real event (the agent can
run again) or a spaced backstop time.

**Steps to reproduce**

1. Create a task in `in_progress` assigned to an agent, with a failed
run.
2. Record a pending continuation for that run, as the fresh-continuation
fallback does.
3. Pause, terminate or un-approve the agent.
4. Run the execution-continuation worker. It calls the wake about every
18 seconds and saves a skipped wake row each time.

**Paperclip version or commit**

master at dcba9628c7.

## What Changed

- `server/src/services/execution-recovery-resolution.ts`: before the
wake, delivery calls `deferForUnwakeableAssignee`. When the assignee
cannot run, delivery skips the wake. The pending-page query skips
continuations whose stored retry time is in the future. The file stays
under 1,000 lines (997).
- `server/src/services/recovery/continuation-delivery-attempt.ts`:
`deferForUnwakeableAssignee` reads the assignee `FOR SHARE` and
evaluates it with `evaluateAgentInvokabilityFromDb`, the rule
`heartbeat.wakeup` uses (including the setup exception). In the same
transaction it stores `continuationDeliveryRetryAt` and a wait count on
the recovery action. The wait is 1 minute and doubles per refusal up to
1 hour. The row lock orders the wait against a concurrent status change,
so a resume cannot be missed. `continuationDeliveryDue` is the shared
"due now" condition. `readmitAgentContinuations` notifies the
continuation worker, then clears the stored time for one agent's waiting
continuations.
- `server/src/services/execution-control-work.ts`: the continuation
schedule asks for a pass in 15 seconds only when a continuation is due.
Otherwise it returns the earliest stored retry time.
- `server/src/services/agent-lifecycle.ts`: when an agent's status
changes from not invokable to invokable (resume, hire approval), the
existing hook also calls `readmitAgentContinuations`.
-
`server/src/__tests__/continuation-delivery-unwakeable-agent-idle.test.ts`:
five new tests on real PostgreSQL.

What the user sees does not change. The continuation stays pending and
the task stays assigned. The stranded-assignment check and the
assigned-work invariant still own the visible outcome for a task whose
assignee cannot run (the existing "can't run right now" escalation, or
the "agent paused" wait). When the agent can run again, the continuation
is delivered as before.

## Verification

- `pnpm exec vitest run
src/__tests__/continuation-delivery-unwakeable-agent-idle.test.ts` (in
`server/`):
- `waits instead of retrying every pass while the assignee is paused /
terminated / pending_approval`: 2 idle hours after the first minute.
    - master: 380, 376 and 395 wake attempts (red).
- this branch: 0 wake attempts, 68 statements, no skipped wake rows, the
task stays `in_progress` and assigned (green).
- `delivers the continuation within seconds once the paused assignee is
resumed`: after 3 idle hours, the agent is resumed through the real
agent lifecycle. Within 20 seconds the worker calls the wake once and
the record says `continuationDelivery: "delivered"` (green). On master
this test fails because the wake was already called during the idle
hours.
- `does not store a wait when the assignee is resumed while delivery is
checking it`: a resume transaction holds the agent row while delivery
checks. With the row lock, delivery sees the resumed agent and stores no
wait (green). Without the lock, it stores a wait (red).
- Neighbouring suites pass locally:
`automatic-run-brake-recovery-passes`, `recovery-wake-herd`,
`execution-control-work`, `fresh-continuation-move-forward-offer`,
`fresh-continuation-retry-delivery`, `fresh-continuation-fallback`,
`issue-recovery-actions`, `resource-lifecycle-events` (345 tests).
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. Only a continuation whose assignee is not invokable by the
wake's own rule waits. Every other refusal keeps the current behavior.
- A repair that makes the agent invokable without a status change (for
example a fixed reporting chain) does not fire the re-admit hook. The
backstop picks it up within at most 1 hour.
- A task reassigned while its continuation waits is invalidated on the
next backstop attempt instead of the next 15-second pass. The new
assignee gets its own assignment wake, so nothing waits on that.
- No schema or API change. The new fields live in the existing
`evidence` JSON.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21621, #21674, #21675, #21676, #21617.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 19:41:35 -06:00
DottaandClaude Opus 5.5 c66530da63 fix(server): release settled native run process identity at graceful shutdown (#21689)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A server process records the local process of each native run in
`heartbeat_runs` (`process_pid`, `process_group_id`,
`process_started_at`, `process_host_hash`)
> - Only a server in the same PID space can prove that such a process
stopped. `proveLocalOwnerGone` does not accept a recorded process group
from another container
> - Each deploy replaces the server process. Settled native runs that
the old server ran keep their process fields, and no later server can
release them. Recovery sweeps had to learn to skip these rows one by one
> - The stopping server is the one process that can check its own PID
space. After its graceful drain, it can see that a settled run's process
exited
> - This pull request adds a graceful-shutdown step that releases the
process identity of those runs, with the same rules that the running
server already uses, in one candidate read and one update
> - The benefit is that a deploy no longer adds stranded process records
for the runs that the stopping server could prove stopped, and a
successor on the same host has nothing to release at startup

## Linked Issues or Issue Description

Refs #21590

**What happened?**
After a graceful shutdown, settled native runs (`failed`, `cancelled`,
`timed_out`, `interrupted`) that the stopping server ran keep
`process_pid` and `process_group_id`. The successor runs in a new
container with a different `process_host_hash`. It cannot prove that the
recorded process stopped, so the rows stay forever. A successor on the
same host releases them at startup with about 10 statements for each
run.

**Expected behavior**
The server that ran the process knows that the process exited. A
graceful shutdown should release these records before the server stops.
An abrupt stop should keep today's behavior.

**Steps to reproduce**
1. Run native work so that some runs settle while the server runs, or
settle during the shutdown drain.
2. Send SIGTERM to the server.
3. Count settled native runs with `process_pid` or `process_group_id`
set: the count does not go down after the deploy.

**Paperclip version or commit**
master at cf2a4221e3

**Additional context**
This change does not clear historical rows. It does not change
`proveLocalOwnerGone`.

## What Changed

- `releaseOwnedSettledNativeRunProcessesAtShutdown`
(`native-local-process-stop.ts`): one read selects at most 500 settled
native runs whose `process_host_hash` is this server's PID space and
that pass the existing release rules (settled coordinator phase, a
checkpoint or provider evidence, no sandbox lease, no pending local
lease, a matching `native.process_identity_recorded` event from this
host). The server checks that each PID and process group is absent. One
statement in one transaction then clears the fields and appends the
`native.local_process_stopped` receipt with the original identity. The
update checks every rule and the exact recorded identity again, so a
changed row is not released.
- The transaction sends the same work signals as the per-run release
(execution replacement, live-run recovery, retained session cleanup,
accounting, waiting instruction copies). It sets a 2 s
`statement_timeout`.
- `notifyInstructionCopyStopEvidenceForRuns`
(`instruction-collection-notifications.ts`): the batch form of
`notifyInstructionCopyStopEvidence`, with the same waiting-copy
predicate.
- `server/src/index.ts`: a bounded shutdown step
`settled_native_process_release` (3 s) runs after the heartbeat drain
and the finalizer drain, before the database closes. A timeout or
failure does not stop the shutdown.
- `doc/plans/2026-11-13-thousand-concurrent-agents.md`: the measured
result under "Background recovery sweeps".

## Verification

- `pnpm exec vitest run
src/__tests__/shutdown-settled-process-release.test.ts` (embedded
PostgreSQL):
- 40 own settled runs with an exited process, plus 6 runs that must
stay: a running (restart-suspended) run, a settled run with a live
process, a sandbox run without a termination receipt, a run from another
container, a run with an unsettled coordinator, and a run with pending
lease cleanup. Before: 45 stranded settled runs. After: 5 (the 5 settled
runs that must stay). The release used 7 statements for 40 runs. The 6
kept runs keep their exact fields and get no receipt. Each released run
has one receipt, and `withReleasedNativeRunProcess` restores its
identity.
- Successor on the same host: `discoverSettledNativeRunProcesses` used 1
statement after a graceful shutdown and 401 statements (40 releases)
after an abrupt stop.
  - Abrupt stop: no step runs, and all rows and receipts stay as today.
- A run whose recorded identity changes between the read and the update
is not released.
- Red check: with the release step changed to do nothing, 3 of 4 tests
fail.
- Regression: `settled-native-process-rechecks.test.ts` and
`heartbeat-native-restart-suspension.test.ts` (127 tests),
`agent-instruction-copy-retry-backoff.test.ts` (12 tests),
`task-retry.test.ts` and `heartbeat-process-recovery.test.ts` pass. `tsc
--noEmit` for `server` passes.
- No server was started.

## Risks

- Low risk. The step releases only what the same server would release in
its next recovery pass, with the same rules, and only in its own PID
space. It does not touch running runs, so a restart-suspended native run
keeps everything that the resume path reads.
- The step is bounded (500 runs, 3 s, 2 s statement timeout). The
release is one transaction. A kill in the middle leaves the rows as they
were.
- Runs still running at shutdown, sandbox runs, coordinator `observed`
rows, controller leases and remote workspace writer markers do not
change. The open decision about the PID space of a stopped container
stays open.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 19:18:39 -06:00
DottaandClaude Opus 5.5 3eabdd6d10 perf(server): size run-log uploads and object journal IO from open run logs (#21685)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every run of a workspace, and the target
is 1,000 concurrent agents on one server
> - Each native run writes its run log to object storage. The server
batches a run's log lines and uploads one batch per run each second
> - The server allowed only 4 log uploads and 8 object storage
operations at a time, for the whole process. These numbers were set when
50 agents was a lot
> - At 200 runs, the run logs fell about 8 seconds behind. At 1,000
runs, pending log bytes grow toward the 64 MiB process cap. Past that
cap, run events wait and log appends can fail
> - This pull request sizes the log upload slots and the object storage
operation slots from the number of open run logs, between the old floor
and a ceiling that comes from the storage client's socket budget
> - The benefit is that run logs stay about one flush interval behind at
200 runs, and fall 7.7 seconds behind instead of 44 seconds at 1,000
runs (p50), without more sockets than the storage client allows

## Linked Issues or Issue Description

Refs plan item 1g in
`doc/plans/2026-11-13-thousand-concurrent-agents.md` ("Size the shared
limiters ... from the active run count").

**What existing behavior does this improve?**
The shared run-log upload limiter and the shared object journal
operation limiter.

**Subsystem affected**
Server: object storage run logs (`object-run-log-store.ts`,
`object-byte-journal.ts`) and the S3 storage client.

**Current behavior**
The process allows 4 run-log uploads and 8 object journal operations at
a time, for any number of runs. Each upload is 3 to 4 sequential object
writes. At 200 runs and 50 ms per write, the run logs commit with a p50
delay of 8.0 s, and only 55% of the log lines offered in a 20 s window
commit inside it.

**Proposed behavior**
The upload slots are one per 10 open run logs, at least 4 and at most
21. The operation slots are the upload slots plus 4 for reads, at most
25. That is half of the 50 sockets that the S3 client may open, so
downloads and other storage calls keep the other half. The S3 client now
sets its 50-socket limit explicitly, from the same constant.

**Reason and benefit**
With the change, 200 runs commit their logs at the flush interval (p50
1.1 s), and 1,000 runs commit with a p50 delay of 7.7 s instead of 43.8
s. Fewer pending bytes stay in memory, so the 64 MiB pending cap and its
admission failure are farther away.

**Breaking changes**
None. With 40 or fewer open run logs the limits are the same as before
(4 uploads, 8 operations).

## What Changed

- `server/src/lib/active-run-slots.ts`: a small helper that gives a slot
count from an active run count, clamped between a floor and a ceiling.
- `server/src/storage/s3-client-limits.ts` and `s3-provider.ts`: the S3
client sets `maxSockets` to 50 (the SDK default) from a named constant,
so the budget is explicit.
- `object-byte-journal.ts`: the object operation limit is the upload
slots plus a read reserve of 4, from the number of open run-log writers.
Open writers are counted with `trackObjectJournalWriter()`. When the
limit grows, waiting operations start at once. When it shrinks, running
operations finish and no new ones start until the count is below the
limit.
- `object-byte-journal.ts` also owns the upload limiter
(`acquireObjectJournalUpload`), process-wide, next to the writer count.
When a run log opens, waiting uploads start at once. When run logs
close, running uploads finish and no new upload starts until the running
count is below the new limit.
- `object-run-log-store.ts`: uses the shared upload limiter. A run log
registers when it begins and releases when it is finalized. The waiting
queue stays first in, first out, and each run has at most one upload in
flight, so one run cannot starve another.
- Review fix: the first revision kept the upload limiter per store, so a
run log opened by another store did not wake uploads waiting in this one
(a lost wakeup on growth). The shared limiter fixes it; a new test
covers growth and shrinkage with queued work.
- Tests: limit growth with uploads waiting and shrinkage with uploads
running; helper bounds; the storage socket budget; 200 open runs upload
at least 20 logs at once (4 before); 1,000 open runs stay at the ceiling
and every upload finishes.

## Verification

- `cd server && npx vitest run src/lib/active-run-slots.test.ts
src/services/object-run-log-store.test.ts
src/services/object-byte-journal.test.ts`: all pass.
- Red/green: with the slot count forced back to the fixed floor of 4,
the two new scale tests fail (4 uploads in flight); with the change they
pass.
- Simulation (in-memory object store with 50 ms per write, each run
appends 1.5 KB at 0.7 records/s, the recorded native event rate, 20 s
window, flush every 1 s). Counts are stable on a loaded machine because
the latency is simulated:

| Runs | Build | Peak concurrent writes | Writes/s | Log lines committed
in window / offered | Commit delay p50 / p99 |
| --- | --- | --- | --- | --- | --- |
| 200 | before | 4 | 74 | 1,473 / 2,669 | 8.0 s / 13.5 s |
| 200 | after | 20 | 211 | 2,646 / 2,790 | 1.1 s / 1.3 s |
| 1,000 | before | 4 | 75 | 1,229 / 13,499 | 43.8 s / 62.3 s |
| 1,000 | after | 21 | 392 | 7,789 / 13,667 | 7.7 s / 12.6 s |

## Risks

- More object storage writes per second at high run counts (at 1,000
runs and 50 ms per write, about 390/s instead of about 75/s). This
raises object storage request cost in proportion. The ceiling bounds it.
- The S3 client now states its socket limit. The value is the SDK
default, so behavior does not change.
- The open-writer count is process-wide. A run log that is never
finalized keeps its count, as it already keeps its writer entry. The
ceiling bounds the effect.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:52:23 -06:00
DottaandClaude Opus 5.5 28761f9c25 perf(workspaces): read a downloaded snapshot archive once, in the encrypt pass (#21683)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote agents work in sandboxes, and the server saves each sandbox
workspace as an encrypted snapshot in object storage
> - One Node main thread on the server controls every run, so per-agent
byte work on that thread limits how many agents one server can hold
> - The save path read every downloaded archive twice: a main-thread
SHA-256 re-hash after the download, then a full read in the off-thread
encrypt pass that already computes the same digest
> - This pull request removes the main-thread re-hash and checks the
encrypt pass's digest and size against the sandbox receipt instead
> - The benefit is one full archive read per save instead of two, and no
archive hashing on the main thread, with the same integrity guarantee

## Linked Issues or Issue Description

Refs the 1,000-concurrent-agents plan in
`doc/plans/2026-11-13-thousand-concurrent-agents.md` (Phase 1,
"Workspace snapshot" row).

**What existing behavior does this improve?**
Saving a remote workspace snapshot after a run.

**Subsystem affected**
Server: remote workspace snapshot save (`remote-workspace-transport.ts`,
`remote-workspace-snapshot-storage.ts`, `remote-workspaces.ts`).

**Current behavior**
`captureSandboxWorkspaceArchive` streams the whole downloaded archive
through `createHash` on the main thread to compare it with the sandbox
receipt. Then `uploadWorkspaceSnapshot` reads the whole archive again on
a worker thread to encrypt it, and computes the same plaintext digest.

**Proposed behavior**
After the download, the server checks only the file size against the
receipt. The encrypt pass compares its plaintext digest and size with
the receipt. A mismatch fails with the same
`remote_workspace_capture_transfer_mismatch` error before the upload
attempt is recorded and before any object is stored.

**Reason and benefit**
One full read and one SHA-256 of each archive leave the server's main
thread (about 0.36 ms of main-thread time per MB on a loaded dev
machine; 37 ms for a 100 MB archive, at event-loop utilization 1.0 for
that time).

## What Changed

- `captureSandboxWorkspaceArchive` no longer re-hashes the downloaded
archive. It checks with `lstat` that the download is a file with the
receipt's byte size, and fails with
`remote_workspace_capture_transfer_mismatch` when it is not.
- `uploadWorkspaceSnapshot` accepts an optional `expectedArchive`
(`sha256`, `byteSize`). After the encrypt pass, it compares the sealed
plaintext digest and size with it. A mismatch throws
`remote_workspace_capture_transfer_mismatch` before the attempt enters
`metadata.uploads` and before `putFile`. The scratch ciphertext is
removed as before.
- `saveRemoteWorkspace` passes the sandbox receipt as `expectedArchive`
when it captured an archive.
- The save-level `verifyMs` timing field is removed, because that step
no longer exists. The storage-level `verifyMs` stays.
- `doc/run-log-events.md` says that the encrypt step is the only full
read and where a mismatch fails.

Callers that relied on the early hash keep their behavior:
- Unchanged-snapshot reuse (`findUnchangedWorkspaceSnapshot`) compares
the sandbox receipt, not the local hash. When it reuses the parent, the
local copy is used only to list directories, and
`readRemoteWorkspaceArchiveDirectories` verifies the manifest digest.
- Reuse of an earlier upload attempt or an already uploaded snapshot
does not read the archive. Those objects were verified when they were
stored.
- The read-only disk-full fallback builds its archive on the server, so
its receipt always matches. Its behavior does not change.
- The error message and the failure path do not change, so the save
failure code, retry and fallback stay the same.

## Verification

- Red/green: `remote-workspace-transport.test.ts` "leaves the downloaded
archive's single full read to the encrypting upload" counts main-thread
read streams of the downloaded archive during capture. On master it is 1
(fails). With this change it is 0 (passes). The encrypt pass is the one
remaining read.
- New: digest mismatch and size mismatch in `uploadWorkspaceSnapshot`
reject with the transfer-mismatch error, do not call `putFile`, and
leave the row `pending` with no recorded attempt. A retry with the right
receipt saves.
- New: an end-to-end `saveRemoteWorkspace` with a transfer that flips
one byte (same size) fails with the transfer-mismatch error, stores
nothing, keeps the writer and the previous snapshot, and a retry saves
and restores the right content.
- New: a download whose size differs from the receipt fails in capture.
- `cd server && npx vitest run
src/services/remote-workspace-snapshot-storage.test.ts
src/services/remote-workspace-transport.test.ts
src/services/remote-workspaces.integration.test.ts
src/services/remote-workspace-transfer-timeout.test.ts
src/services/remote-workspace-readonly-capture-budget.test.ts
src/services/sandbox-disk-full-fault.test.ts` passes: 6 files, 360
tests. `src/services/remote-workspace-snapshot-reuse.test.ts` passes: 32
tests.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. A same-size corrupt download is now detected in the encrypt
pass instead of right after the download. It is still detected before
any attempt is recorded or any object is stored.
- When the sandbox reports an unchanged workspace, a corrupt local copy
no longer fails the save. The save reuses the verified parent snapshot,
which matches the sandbox's content. The local copy is used only for its
manifest, and the manifest digest is verified.
- The save-level `saveTiming.verifyMs` field is gone. No code reads it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:41:29 -06:00
DottaandClaude Opus 5.5 cf2a4221e3 perf(server): skip the per-request stale instance-admin purge when no row exists (#21680)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Cloud-hosted instances authenticate each request through trusted
Cloud headers in the actor middleware
> - That path deletes stale `instance_admin` rows for the user on every
request, static files and live-events upgrades included
> - On an idle hosted instance this delete ran about 1.2 times a second,
which is a write on an instance-wide table for each request
> - Almost always there is no row to delete, so the statement is wasted
work and a database round trip
> - This pull request finds out if a stale row exists inside the
membership read that the request already does, and deletes only when a
row exists
> - The benefit is one read and no write for each unchanged
authenticated request, with the same purge on the very next request when
a stale row appears

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Cloud trusted-header authentication in the actor middleware.

**Subsystem affected**
Server authentication (`server/src/middleware/auth.ts`).

**Current behavior**
`resolveCloudTenantActor` in `server/src/middleware/auth.ts` sends
`delete from instance_user_roles where user_id = $1 and role =
'instance_admin'` for every trusted-header request. The actor middleware
runs before the static UI, so asset requests run it too. Each
live-events WebSocket upgrade runs it again through `resolveCloudActor`.
The retry wrapper runs it again after a transient connection error.

**Proposed behavior**
An unchanged request sends no write. A stale `instance_admin` row is
still removed on the next request after it appears.

**Reason and benefit**
Idle hosted instances send about one delete for each request. This costs
a round trip and a write statement on the main database for no change.

**Breaking changes**
None. The resolved actor and the authorization decisions do not change.

## What Changed

- The Cloud trusted-header path reads the user's active memberships
together with an `exists` check for an `instance_admin` row, in one
statement.
- The purge runs only when that check finds a row, when the user has no
active membership row yet (first request), or when the read fails.
- The session path and the Cloud path share one membership filter
function, so both still resolve the same access set.
- New test `server/src/__tests__/cloud-tenant-actor-statements.test.ts`
meters the statements of one authenticated request on the embedded test
database.

## Verification

- `cd server && npx vitest run
src/__tests__/cloud-tenant-actor-statements.test.ts
src/middleware/cloud-tenant-actor.test.ts
src/__tests__/cloud-tenant-actor-round-trips.test.ts`: 41 passed.
- Red/green: with the `origin/master` version of `auth.ts`, the new test
fails. An unchanged member request sends 2 statements, and one of them
is the delete. With this change it sends 1 read and 0 writes.
- The test inserts a stale `instance_admin` row between two requests.
The next request deletes it, and the request after that sends no write
again.
- The test changes the stack role from member to admin and back. The
membership role and the experimental-settings access change on the very
next request each time.

## Risks

- Low risk. The actor that the middleware returns is the same.
Authorization decisions do not change.
- The purge now runs after the membership read, not before the tenant
sync. The sync does not touch `instance_user_roles`, so the order does
not matter. The request still waits for the purge before it continues.
- A race between the read and a new row insert existed before too
(delete, then insert). The next request removes such a row.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:34:30 -06:00
DottaandClaude Opus 5.5 dcba9628c7 perf(recovery): keep a finished task's settled attention settled and halve graph hint fan-out (#21676)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Issue recovery records "recovery actions" on tasks. A sweep resolves
the active action of a task that is already done or cancelled.
> - On an idle hosted instance, a debug-log run (from #21640) named a
ping-pong. A native worker re-surfaces a board attention for 13 finished
tasks. Recovery resolves it as `source_terminal`. The unique "one active
action" index then lets the worker insert it again, about every 26
seconds per task. That built 45,008 rows since 2026-10-21. Each insert
and each resolve also signals run dispatch and the task's recovery
graph, which wakes both sides again.
> - Separately, every task-graph recovery hint is published twice per
task queue: once as the graph hint and once as a `{companyId, backstop:
true}` twin. That is half of all idle signals, for a backstop that is
time-gated anyway.
> - This pull request breaks the ping-pong from the recovery side and
removes the backstop twin.
> - The benefit is that a finished task's settled attention stays
settled and wakes nothing. Each graph hint costs half the signals.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

With zero runs active, recovery and native workers kept re-creating and
re-resolving the same board attention on 13 done or cancelled tasks. The
rows are `active_run_watchdog`, owner `board`, cause
`native_restart_recovery_blocked`. The loop added about one row per task
every ~26 s and about 2.6 work-signal commits per second.

**Expected behavior**

Once recovery settles a finished task's attention, the same attention
identity is not surfaced again, and settling it wakes nothing that a
finished task cannot use.

**Steps to reproduce**

1. Create a cancelled task.
2. Surface the native "controller process space unverified" attention
for it (same cause and fingerprint each time), then run
`reconcileActiveRecoveryActions`.
3. Repeat step 2. On master each cycle inserts a new active row and
resolves it, and each resolve signals the task queues and run dispatch.

**Paperclip version or commit**

master at 36101ebcb9.

## What Changed

- `server/src/services/issue-recovery-actions.ts`:
- **No-match skip.** `resolveActiveForIssue` first reads whether any
action matches. When none does, it returns `null` with no transaction
and no signal. Before, `writeExecutionControl` always published
run-dispatch, graph-recovery and execution hints first.
- **Settled-attention guard.** `upsertSourceScoped` returns the existing
resolved action instead of inserting when ALL of these hold:
    - the task is done or cancelled;
    - it has no active action;
- an action with the same cause and fingerprint was already resolved
with `source_terminal`.
The guard writes no row and sends no signal, and the caller gets that
resolved action. It covers only the exact identity that recovery itself
settled because the task finished. A new fingerprint (renewed controller
authority), a different cause, or a reopened task still creates an
action.
- **Quiet terminal resolve.** A new `sourceTerminal` resolve mode. It
updates the row only while the task is still done or cancelled. It still
notifies native recovery notices, unsafe-workspace export recovery, and
a preserved native finalization waiter. It does NOT signal run dispatch,
the task's recovery graph, or the execution disposition/continuation
queues: a finished task has no run to dispatch and no assignment to
recover.
- `server/src/services/recovery/service.ts`: the `source_terminal`
branch of `reconcileActiveRecoveryActions` uses `sourceTerminal: true`.
- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery` publishes one graph hint per task queue and
no longer adds a `{companyId, backstop: true}` hint per queue. A new
`issueRecoveryBackstopHint(companyId)` builds that hint string.
- `server/src/services/recovery/work.ts`: when a task queue loads a
graph scope, it arms its own backstop in process
(`worker.invalidate(issueRecoveryBackstopHint(companyId))`). The #21552
guarantee stays: any graph activity arms the company backstop, which
runs at most once per backstop interval after the last full pass. It no
longer costs one signal per queue per hint.
- `server/src/__tests__/issue-recovery-actions.test.ts`: the fake-DB
unit tests accept the guard's extra read.
- `server/src/__tests__/issue-recovery-signal-fanout.test.ts` (new, real
PostgreSQL), six tests:
  - one graph hint per task queue and no backstop hint;
  - a resolve with no matching action returns null and signals nothing;
- an escalated action on a finished task is still resolved and still
signals;
- the ping-pong cycle: surface then reconcile, five times, leaves one
row and no stranded or active-recovery signals;
- settling a finished task's attention sends no task-queue, run-dispatch
or active-recovery signals;
- a renewed authority and a reopened task still get a new attention,
which stays active on a reopened task.

## Verification

- New test file:
  - master: 4 of 6 fail (red).
    - 8 task-queue hints per graph hint instead of 4.
    - A no-match resolve publishes 18 settled signals.
- Cycle 2 inserts a new active row; master builds a new row every cycle.
- Settling a finished task's attention signals the task queues and run
dispatch.
  - This branch: 6 of 6 pass (green).
- Regression on real PostgreSQL:
- heartbeat-native-restart-suspension, native-recovery-notice-work,
issue-recovery-deadlines (including the #21552 backstop tests),
assigned-work-invariant, execution-control-work, issue-recovery-actions,
issue-recovery-work-lifecycle, issue-recovery-assignments,
execution-recovery-resolution, automatic-run-brake-recovery-passes: all
pass (381 tests).
- Four unit tests in issue-recovery-actions use a hand-written fake DB.
They needed `innerJoin` on the fake query and an expected count of two
selects, because the guard adds one read.
  - recovery-deadline-acceptance: passes.
- `tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- The guard returns a resolved action to a caller that asked to surface
one. I checked the callers: the one that loops
(`surfaceNativeControllerNamespaceAttention`) ignores the return value.
The others only reach the guard for a finished task whose identical
attention recovery already settled, where re-surfacing it would be
resolved again within one sweep.
- The terminal resolve no longer wakes run dispatch or the task's graph.
Completing or cancelling a task already sends those signals itself.
- This does not stop the native worker from trying to surface attention
for finished tasks. That fix belongs in the native owner-release code,
which another change owns.
- The ~45k historical resolved rows on the hosted instance stay as they
are. A cleanup would need a migration or an operator script that deletes
resolved `native_restart_recovery_blocked` rows whose source is done or
cancelled, keeping the newest per (task, fingerprint) so the guard still
matches. That is not part of this change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21552, #21621, #21640.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:06:38 -06:00
DottaandClaude Opus 5.5 97a167093d fix(queue): send a message into a starting run without a cancel or recovery errors (#21661)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can queue a message while an agent works, then send it into
the running turn
> - A run that is still starting takes queued messages into its first
turn, and Interrupt hands off to them
> - Both rules only accepted a queue that a user started
> - A user's message can merge into an earlier automatic wake that waits
behind the task, and that wake keeps its system requester
> - Then the starting run did not take the message, Interrupt cancelled
the boot, and the task showed "Couldn't finish this attempt" with Retry
and "Waiting for execution recovery" under the composer until the
cancelled boot settled
> - This pull request makes both rules judge the messages and the
recorded Interrupt intent, not the wake's requester, and shows an
Interrupt hand-off as sent
> - The benefit is that sending a queued message feels immediate and
shows no error

## Linked Issues or Issue Description

**What happened?**
A run started from a resolved blocker. The user queued a message 20
seconds later, while the runner was still starting. The message merged
into an older automatic deferred wake. The queue offered Interrupt
instead of sending it. Interrupt cancelled the starting run. For about 7
seconds the task showed "Couldn't finish this attempt. Your message is
saved." with Retry, and the composer showed "Waiting for execution
recovery. Your message is saved." Then the message was delivered.

**Expected behavior**
The starting run takes the message into its first turn. When an
Interrupt does stop a run, the message shows as sent, with no error
banner, Retry, or recovery text.

**Steps to reproduce**
1. Have an automatic deferred wake for a task (for example from an
earlier blocked execution).
2. Start a run for the task with a Paperclip runner agent.
3. While the runner is still starting, queue a message and click the
send control.

## What Changed

- `deferredWakeJoinsFirstTurn`: a system or unattributed wake qualifies.
Each message must still be the responsible user's own
(`firstTurnEligibleComment`). A wake from another user or an agent keeps
its own delivery.
- Task interruption presentation: find the Interrupt queue by its
recorded `queuedCommentInterrupt` intent, not by `requestedByActorType =
user`. The hand-off now shows as working for a merged queue too.
- Queued-comment read: while a deferred queue carries an Interrupt
intent, report an `interrupt_handoff` wait instead of the recovery
message.
- UI (`TaskChatQueuedMessages`, `IssueDetail`): an `interrupt_handoff`
response counts as sent. The messages stay out of the queue and no
status or error line shows.

## Verification

Red/green tests. Each failed before the fix and passes after it:
- `server/src/__tests__/issue-queued-comments-routes.test.ts`: "gives a
booting run's first turn the user's message merged into an earlier
automatic wake"; the Interrupt fallback case now expects
`interrupt_handoff` on the response and on a poll. 144/144 pass.
- `server/src/__tests__/task-interruption-query.test.ts`: the hand-off
case runs for a user-started and a system-started queue. 32/32 pass.
- `ui/src/components/task-chat/TaskChatQueuedMessages.test.tsx`: "shows
an Interrupt as sent while the stopped turn hands off". 43/43 pass.
- `first-turn-queued-input.test.ts` 8/8, `issue-queued-comment-queue`
7/7. Server and UI typecheck pass.

Root cause from the staging wake diagnostics: the run started at
20:26:54, the message was coalesced at 20:27:17 into a deferred
`issue_execution_deferred` automation wake from the previous day, the
Interrupt at 20:27:20 recorded `execution_recovery`, and the
false-positive hold resolved at 20:27:26.

Not run: local runner E2E steering cells. The OpenCode chat-features
cells require the trusted CI host. The Codex and Claude
`revise-while-running` chat cells failed before steering on a fixture
workspace path (`workspaces/` vs `isolated-workspaces/`), which this
change does not touch. Staging verification follows the deploy.

## Risks

- Low. A wider set of deferred wakes can join a starting run's first
turn. Message eligibility, the claim receipt, and exactly-once delivery
are unchanged.
- An Interrupt intent that never delivers shows "sent" in the composer.
The task notice still shows a failure with Retry after its existing
grace period.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:06:15 -06:00
DottaandClaude Opus 5.5 648b238be1 perf(native-runtime): stop re-finalizing native results whose finalization error is recorded (#21674)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime releases owners of failed runs. When the owner's
controller is in a process space that Paperclip cannot verify, it shows
controller namespace attention on the task instead.
> - On a hosted instance with zero active runs, this writer made about
21% of all work signals. It inserted a new active attention row for the
same task about every 26 seconds.
> - The tasks were already done or cancelled. Recovery resolved each row
as `source_terminal`. The active slot was then free, so the next
owner-release pass inserted a new row and published a graph recovery
signal again. The loop did not stop.
> - A live task had the same defect: after the fresh-continuation
fallback resolved its attention, the next pass made the same incident
again.
> - This pull request stops the writer from surfacing attention for
finished tasks. It also stops the writer from making an incident again
when that run authority was resolved after its lease expired. Those
owners are no longer release candidates and do not set the release
deadline.
> - The benefit is that an idle instance does not write or publish
anything for these runs. Attention still shows once for a live task.

## Linked Issues or Issue Description

**What happened?**
The failed-run owner release (`settleFailedRunOwner`, expired-owner
path) called `surfaceNativeControllerNamespaceAttention` for every
expired foreign owner that had no active attention. It did this even
when the source task was `done` or `cancelled`. Recovery resolved that
attention as `source_terminal`. The next pass found no active attention
and inserted a new board row. Each insert published
`notifyIssueGraphRecovery` and the other execution-control signals. The
stranded-task check then ran again for the task graph. A hosted instance
collected about 45,000 rows on 13 finished tasks in this way.

**Expected behavior**
A pass that changes nothing writes nothing and publishes nothing. A
finished task never gets controller namespace attention. A live task
gets one incident for each exact run authority. A resolved incident
stays resolved until the authority changes.

**Steps to reproduce**
1. Make a native run that failed, with an `observed` coordinator whose
foreign lease has expired and whose controller host is not recorded.
2. Mark the task done or cancelled, or let the fresh-continuation
fallback resolve the attention of a live task.
3. Run the execution-finalization worker with the issue recovery workers
for 30 minutes.
4. On master, each task gets about 40 to 60 new attention rows. The
writer publishes thousands of signals.

## What Changed

- `surfaceNativeControllerNamespaceAttention` (shared helper): before it
creates attention, it checks for a resolved incident for this run that
was resolved at or after the owner's current lease expiry. If one
exists, it writes nothing.
- New `settledNativeControllerNamespaceAttention` SQL predicate. It is
shared by the helper, the release candidate page, and the release
deadline query, so all three use the same rule.
- `settleFailedRunOwner` (expired-owner path) does not surface attention
when the task is `done` or `cancelled`. The restart-recovery claim path
has the same guard.
- `releaseAbandonedNativeOwners` and `nextAbandonedNativeOwnerReleaseAt`
skip foreign owners whose task is finished or whose incident is settled.
A pass that has nothing to do selects nothing, and no immediate deadline
is set again. Local owners keep the normal proof-based release.
- The owner release proofs did not change. A foreign owner is still
never released without proof.
- Tests:
`server/src/__tests__/native-failed-run-owner-release-idle.test.ts`
(real PostgreSQL, real coordinator and workers, fake clock, statement
and signal counters) and three new cases in
`native-failed-run-owner-release.test.ts`.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/native-failed-run-owner-release-idle.test.ts
src/services/native-runtime/native-failed-run-owner-release.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/services/native-runtime/native-restart-recovery-isolation.test.ts`:
80 tests pass.
- Red on master, green with this change. The test seeds one done task
and one cancelled task (each with an earlier resolved `source_terminal`
incident) and one live task, then runs 30 idle minutes:
- master: 126 attention inserts, 249 attention writes, 7,024 signals,
8,025 statements.
- this branch: 1 attention insert (the live task, surfaced once), 2
attention writes (that insert and the fallback's resolution), 36
signals, 350 statements. All of this is startup work. Nothing repeats.
- After the fresh continuation settles a live task's attention, six
unrelated finalization wakes over 30 minutes:
- master: 3 attention inserts, 90 signals, 551 statements, and the
release deadline stays due.
- this branch: 0 attention writes, 0 run writes, 6 signals (the test's
own wakes), 153 statements, and the release deadline is null.
- New unit cases: a done or cancelled task never surfaces, and surfaces
once when it is reopened. A settled incident stays settled over repeated
passes. A renewed authority (a new lease expiry) surfaces one new
incident.
- The three new unit cases fail on master and pass on this branch.
- `tsc --noEmit` for `server` passes.

## Risks

- Low risk. The change only removes writes. It does not weaken a release
proof, an authorization, a budget, a pause, or an approval check.
- An incident that a person resolved or dismissed after the lease
expired is not made again for the same authority. A renewed authority (a
later lease expiry) still makes a new incident.
- A superseded incident (status `cancelled`) still shows again when the
superseding action clears. This keeps the current behavior.
- The historical rows that the loop made are not removed. This change
only stops new rows.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21621, #21616, #21617

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:50:45 -06:00
DottaandClaude Opus 5.5 64c7122dae perf(native-runtime): stop re-surfacing settled controller namespace attention (#21675)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime releases owners of failed runs. When the owner's
controller is in a process space that Paperclip cannot verify, it shows
controller namespace attention on the task instead.
> - On a hosted instance with zero active runs, this writer made about
21% of all work signals. It inserted a new active attention row for the
same task about every 26 seconds.
> - The tasks were already done or cancelled. Recovery resolved each row
as `source_terminal`. The active slot was then free, so the next
owner-release pass inserted a new row and published a graph recovery
signal again. The loop did not stop.
> - A live task had the same defect: after the fresh-continuation
fallback resolved its attention, the next pass made the same incident
again.
> - This pull request stops the writer from surfacing attention for
finished tasks. It also stops the writer from making an incident again
when that run authority was resolved after its lease expired. Those
owners are no longer release candidates and do not set the release
deadline.
> - The benefit is that an idle instance does not write or publish
anything for these runs. Attention still shows once for a live task.

## Linked Issues or Issue Description

**What happened?**
The failed-run owner release (`settleFailedRunOwner`, expired-owner
path) called `surfaceNativeControllerNamespaceAttention` for every
expired foreign owner that had no active attention. It did this even
when the source task was `done` or `cancelled`. Recovery resolved that
attention as `source_terminal`. The next pass found no active attention
and inserted a new board row. Each insert published
`notifyIssueGraphRecovery` and the other execution-control signals. The
stranded-task check then ran again for the task graph. A hosted instance
collected about 45,000 rows on 13 finished tasks in this way.

**Expected behavior**
A pass that changes nothing writes nothing and publishes nothing. A
finished task never gets controller namespace attention. A live task
gets one incident for each exact run authority. A resolved incident
stays resolved until the authority changes.

**Steps to reproduce**
1. Make a native run that failed, with an `observed` coordinator whose
foreign lease has expired and whose controller host is not recorded.
2. Mark the task done or cancelled, or let the fresh-continuation
fallback resolve the attention of a live task.
3. Run the execution-finalization worker with the issue recovery workers
for 30 minutes.
4. On master, each task gets about 40 to 60 new attention rows. The
writer publishes thousands of signals.

## What Changed

- `surfaceNativeControllerNamespaceAttention` (shared helper): before it
creates attention, it checks for a resolved incident for this run that
was resolved at or after the owner's current lease expiry. If one
exists, it writes nothing.
- New `settledNativeControllerNamespaceAttention` SQL predicate. It is
shared by the helper, the release candidate page, and the release
deadline query, so all three use the same rule.
- `settleFailedRunOwner` (expired-owner path) does not surface attention
when the task is `done` or `cancelled`. The restart-recovery claim path
has the same guard.
- `releaseAbandonedNativeOwners` and `nextAbandonedNativeOwnerReleaseAt`
skip foreign owners whose task is finished or whose incident is settled.
A pass that has nothing to do selects nothing, and no immediate deadline
is set again. Local owners keep the normal proof-based release.
- The owner release proofs did not change. A foreign owner is still
never released without proof.
- Tests:
`server/src/__tests__/native-failed-run-owner-release-idle.test.ts`
(real PostgreSQL, real coordinator and workers, fake clock, statement
and signal counters) and three new cases in
`native-failed-run-owner-release.test.ts`.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/native-failed-run-owner-release-idle.test.ts
src/services/native-runtime/native-failed-run-owner-release.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/services/native-runtime/native-restart-recovery-isolation.test.ts`:
80 tests pass.
- Red on master, green with this change. The test seeds one done task
and one cancelled task (each with an earlier resolved `source_terminal`
incident) and one live task, then runs 30 idle minutes:
- master: 126 attention inserts, 249 attention writes, 7,024 signals,
8,025 statements.
- this branch: 1 attention insert (the live task, surfaced once), 2
attention writes (that insert and the fallback's resolution), 36
signals, 350 statements. All of this is startup work. Nothing repeats.
- After the fresh continuation settles a live task's attention, six
unrelated finalization wakes over 30 minutes:
- master: 3 attention inserts, 90 signals, 551 statements, and the
release deadline stays due.
- this branch: 0 attention writes, 0 run writes, 6 signals (the test's
own wakes), 153 statements, and the release deadline is null.
- New unit cases: a done or cancelled task never surfaces, and surfaces
once when it is reopened. A settled incident stays settled over repeated
passes. A renewed authority (a new lease expiry) surfaces one new
incident.
- The three new unit cases fail on master and pass on this branch.
- `tsc --noEmit` for `server` passes.

## Risks

- Low risk. The change only removes writes. It does not weaken a release
proof, an authorization, a budget, a pause, or an approval check.
- An incident that a person resolved or dismissed after the lease
expired is not made again for the same authority. A renewed authority (a
later lease expiry) still makes a new incident.
- A superseded incident (status `cancelled`) still shows again when the
superseding action clears. This keeps the current behavior.
- The historical rows that the loop made are not removed. This change
only stops new rows.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21621, #21616, #21617

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:45:02 -06:00
DottaandClaude Opus 5.5 172c267338 fix(runner): count returned validation rejections and give a copyable call in repeated-failure corrections (#21666)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents change Paperclip records through the `call_api` tool, and the
native runtime runs each tool call through a repeated-identical-failure
guard (#21360, #21398)
> - Some model hosts drop arguments. A model then sends the same failing
call again and again, and each attempt costs a model turn
> - In chat-eval cohort #14 (OpenCode, `deepseek-v4.1-flash`), two cases
sent a body-less `PATCH /api/issues/{id}` 7 and 10 times with one input
hash
> - The guard counted these refusals and stopped running the call after
the second failure. But its correction showed only a `"body": {...}`
placeholder, so the model had no concrete call to copy. Also, the guard
did not count a route's own 400/422 when it came back as an `ok: false`
result
> - This pull request puts a copyable call in each correction and counts
returned 400/422 rejections. It never counts transient failures
> - The benefit is that a model gets the exact call to send on the
second failure, and the third identical validation failure does not run,
whether the rejection is thrown or returned

## Linked Issues or Issue Description

Refs #21360, #21398

**What happened?**

In chat-eval cohort #14, `questions-and-approvals.review-plan` (run
470002dc) and `tasks.private-task` (run bb76a832) sent `call_api PATCH
/api/issues/{id}` with no `body` 7 and 10 times, all with the same input
hash. The pre-dispatch check (`assertBodyPresent`) refused each call and
returned HTTP 400. The guard corrected the second failure and refused
the rest. But the correction said only `Resend with a top-level body
argument: {"operationId": ..., "pathParams": {...}, "body": {...}}`. The
model kept sending the identical call, and each refusal cost another
model turn: 2 min 18 s in review-plan and 24 s in private-task. Also,
when the route itself returns a 400/422 validation response, `call_api`
gives back `{ ok: false, status: 422, ... }` as a result and does not
throw. The guard did not count that, so an identical invalid request
reached the route every time.

**Expected behavior**

On the second identical deterministic failure, the agent gets a
correction with a call it can copy that passes validation. On the third,
Paperclip refuses to run the call and tells the agent to change the
arguments or to stop and report. Transient failures (5xx, timeouts, rate
limits, conflicts) always run again.

**Steps to reproduce**

1. Run a native-runtime agent on a host that drops the `body` argument.
2. Have it update a task with `call_api PATCH /api/issues/{id}`.
3. Look at the tool results. They contain only a `{...}` placeholder. A
route's 422 returned as `ok: false` is never counted.

**Paperclip version or commit**

master d4498ae7ac (cohort #14), reproduced against ecdeed5e7e.

## What Changed

- `runner-api-client.ts`: Add `runnerApiCallExample`. It builds a
copyable call from the operation's catalog schema: the path parameters
the model sent, plus a body with the required fields. If no field is
required, the body has one free-text field, for example
`{"operationId":"PATCH /api/issues/{id}","pathParams":{"id":"<sent
id>"},"body":{"title":"<title>"}}`. The "requires body" correction and
the "body must be a JSON object, not a JSON-encoded string" correction
now include this example in place of the `{...}` placeholder.
- `repeated-tool-failure-guard.ts`: Count a result with `ok: false` and
HTTP 400/422 as a deterministic failure, in the same way as a thrown
400/422. The first returned rejection passes through unchanged. The
second gets a `guidance` note that says the call will not run again. The
third and later identical calls are refused without running, and the
refusal quotes the response.
- Never count transient results: 5xx, 408, 429, 409, 404, and a missing
status (timeouts or unknown outcome).
- A route can reject a request because of the state it found. So the
guard forgets a returned rejection after a different call succeeds. A
refusal before sending depends only on the arguments, and the guard
keeps it for the run, as before.
- Tests: the guard tests cover returned 422 (correct, then refuse), a
changed call that still runs, returned 500/503/429/408/409/404/no-status
and thrown 429/409/timeout that keep running, and a returned rejection
forgotten after a state change. The runner API tests check that the
example in each recorded body-less correction is sent when copied as is,
that the issue-update example passes `updateIssueSchema`, and that every
catalog write with a required body (more than 200) has an example its
request schema accepts.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/services/native-runtime/runner-api.test.ts`: 2190 passed.
- Red/green: with the master guard, the 2 returned-rejection tests fail.
With the master client, 6 example tests fail. With this change, all
pass.
- `node scripts/typecheck-layers.mjs` (server): no errors.
- Per-call cost (median of 5 × 20,000 calls, local M-series): guard on a
successful call 2.13 µs before, 2.19 µs after. Transient-failure path
2.12 µs. Building an example takes 0.67 µs and happens only on a refused
call. Nothing is added before the turn starts. The guard is created with
the run's tool authority, as before.
- Paid narrow run: see the comment below.

## Risks

- Low. The change affects only calls that fail validation twice with
identical arguments.
- A route that returns 400/422 because of state can succeed later with
the same arguments. To cover that, the guard forgets a returned
rejection after any different call succeeds. A model that retries with
no call in between is refused after two attempts and must change the
arguments or report.
- The example uses `<field>` placeholders for free text and for ids that
the model must look up. If a model copies a placeholder verbatim, the
route stores that text. The correction says to send the fields the model
means to change.
- This does not make a model stop resending a refused call. It only
gives better guidance and keeps the refused call from running.

## Model Used

- Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use,
code execution, and extended reasoning).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 15:55:09 -06:00
Nicky LeachandClaude Opus 5.5 c603f591dc fix(runner): bound ACPX instructions only where the sandbox delivers them (#21646)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner prepares a private sandbox for each ACPX agent before it
opens the provider session
> - `validateProviderPolicy` in that sandbox stage refuses system
instructions over 32 KiB for every ACPX agent
> - Only Pi gets the text from the sandbox stage. Claude, Codex, Grok
and Antigravity do not use it there. The host accepts 256 KiB
> - An agent with a large instruction bundle fails every run at
`session.open`. The error says only `classification=admission_sandbox`,
so the operator cannot see the cause
> - This pull request applies the 32 KiB sandbox bound only to Pi, and
gives every instructions rejection its own closed admission code
> - The benefit is that Claude and Codex agents with large instructions
can start, and an over-bound rejection names its real cause

## Linked Issues or Issue Description

Fixes: #20814

Refs #14430 (this pull request added the 32 KiB check, with no stated
reason for the other providers)

Related: #21644 (also fixes #20814, but only for Claude, and does not
change the classification)

**What happened?**
Every run of a Claude ACPX agent failed with
`provider_session_open_rejected`. The message was `ACPX sidecar command
session.open was rejected (retryable=false,
classification=admission_sandbox)`. A reproduction in a sandbox from the
same image showed that all the filesystem steps of
`prepareAcpxRuntimeSandbox` succeed. The real cause: the agent's
instructions entry file was 54,734 bytes. That is more than the 32 KiB
check in `validateProviderPolicy`. The host bound (`boundedInstructions`
in `runtime-host.ts`) is 256 KiB. Agents with smaller instructions (12.7
KB) work.

**Expected behavior**
The sandbox stage must refuse only the text that it delivers. #14430
added the check when only Pi consumed `systemInstructions` in the
sandbox (as the `PAPERCLIP_PI_SYSTEM_INSTRUCTIONS` env var). The check
runs before any per-provider branch, so it also refuses Claude, Codex,
Grok and Antigravity. Those providers get the text in the session
request. When a bound refuses the text, the error must name the
instructions, not the sandbox.

**Steps to reproduce**
1. Create a Claude ACPX agent whose composed instructions are more than
32 KiB (for example, an `AGENTS.md` of 54 KB).
2. Start a run of the agent on the Paperclip Runner.
3. See that `session.open` fails with
`classification=admission_sandbox`.

**Paperclip version/commit**
`master` at 163d9f323c.

**Deployment mode**
Paperclip Runner with ACPX providers (any deployment).

Where each provider consumes `systemInstructions` today:

| Provider | Delivery | Bound |
| --- | --- | --- |
| Pi | `PAPERCLIP_PI_SYSTEM_INSTRUCTIONS` env var (sandbox) | 32 KiB in
`validateProviderPolicy` (kept) |
| Cursor | `PAPERCLIP_CURSOR_INSTRUCTIONS` env var (sandbox) | 32 KiB in
`cursorInstructionBinding` |
| Kimi | `PAPERCLIP_KIMI_INSTRUCTIONS` env var (sandbox) | 32 KiB in
`kimiInstructionBinding` |
| Copilot | `copilot-instructions.md` (host, after the sandbox stage) |
32 KiB in `composeCopilotSystemInstructions` and
`refreshCopilotSystemInstructions` |
| Claude, Codex, Grok, Antigravity | session request
(`systemPrompt.append` or the provider's own session admission) | host
bound, 256 KiB |

## What Changed

- `runtime-sandbox.ts`: `validateProviderPolicy` gets the agent. It
applies the 32 KiB bound only to Pi. The type and NUL checks stay for
all agents. Cursor, Kimi and Copilot keep their own 32 KiB checks.
- `admission-diagnostics.ts`: new closed admission code
`ACPX_ADMISSION_INSTRUCTIONS`, and `acpxInstructionsAdmissionError`,
which tags the error with that code. The tag wins over the stage code
that the host records later.
- The instructions bound and NUL rejections in `runtime-sandbox.ts`,
`runtime-host.ts` (`boundedInstructions`), `cursor-instructions.ts`,
`kimi-instructions.ts` and `copilot-profile.ts` use the new tagged
error. The messages contain no instruction text and no provider output.
- `acpx_sidecar_transport.rs`: `ACPX_ADMISSION_INSTRUCTIONS` maps to
`classification=admission_instructions`. The classification set stays
closed.
- `antigravity-connection-probe.ts`: the closed classification pattern
accepts `admission_instructions`.
- `copilot-installation.test.ts`: the v70 identity test compared two
historical source hashes with the live files. Since #18325, Runner
source changes do not advance profile identities. The test now pins the
hashes that the v70 record holds.

## Verification

- Red-green: the new tests failed on `master` (11 failed) and pass with
the fix.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/acpx/runtime-sandbox.test.ts
src/drivers/acpx/runtime-host.test.ts
src/drivers/acpx/kimi-instructions.test.ts
src/drivers/acpx/cursor-instructions.test.ts
src/drivers/acpx/copilot-profile.test.ts
src/drivers/acpx/copilot-runtime-host.test.ts
src/drivers/acpx/pi-acp-runtime.test.ts src/cli`: 29 files, 489 tests
pass.
- New tests:
- The sandbox admits 54,734-byte instructions for Claude, Codex, Grok
and Antigravity, and does not put them in the launch environment.
- The host opens a Claude runtime with 54,734-byte instructions and
passes the full text to the provider.
- Pi and Cursor sandboxes refuse more than 32 KiB with
`ACPX_ADMISSION_INSTRUCTIONS`. NUL also gives that code.
- Through the host and the sidecar error mapping
(`acpxSidecarErrorCode`), oversized Pi (32 KiB + 1), Copilot (32 KiB +
1) and Claude (256 KiB + 1) instructions give
`ACPX_ADMISSION_INSTRUCTIONS` before the provider launches.
- The Rust classification test maps `ACPX_ADMISSION_INSTRUCTIONS` to
`admission_instructions`.
- The Antigravity probe logs `admission_instructions` as a closed
classification.
- `cargo test -p paperclip-runner-core --lib
classifies_only_allowlisted_internal_sidecar_failures` passes. `cargo
fmt --check` passes.
- `tsc -p packages/paperclip-runner/tsconfig.json --noEmit` passes.
- `cd server && npx vitest run
src/__tests__/antigravity-connection-probe.test.ts`: 11 tests pass.
- I did not do a live provider run. I did not run the full local suite.
CI supplies that coverage.

## Risks

- Low risk. The bound gets less strict only for providers that do not
get the text from the sandbox stage. Those providers stay under the host
bound (256 KiB). Pi, Cursor, Kimi and Copilot keep 32 KiB.
- An over-bound rejection now reports `admission_instructions` and not
`admission_sandbox` (or `admission_configuration` for Copilot). A
dashboard or alert that matches the old classification for this case
must also match the new one. Server code that checks a specific
classification does not check these values, except the Antigravity
probe, which this pull request updates.
- The NUL rejection message in `validateProviderPolicy` changes from
"exceed their bounded size" to "are invalid".

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:40:20 -08:00
Nicky LeachandClaude Fable 5.2 0ab73d4e30 feat(native-runtime): ask the Runner for provider-started turns on native ACPX Claude runs (#21456)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner can keep Claude while a background shell is running and
give the cycle of Claude to the controller as a turn that the provider
started. Core can wait for that turn. The transport can ask for it
(#21607), and a controller restart during it is safe (#21612).
> - No run does this yet, because Core does not ask the Runner for it.
> - This pull request makes the native runtime of Core ask, for ACPX
Claude. It is the only part that changes what a production run does.
> - The benefit is the goal of the whole work: a run with a background
command that is still running continues when the command finishes, and
does not fail with `native_finalization_missing`.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request turned the behavior on and had grown too large to
review. It is now three stacked pull requests:

1. #21607 (merged): the transport can ask the Runner for
provider-started turns. No caller asks.
2. #21612: the recovery after a controller restart during a wait or
during a provider-started turn.
3. **This pull request:** Core sets the option for native ACPX Claude
runs.

This is part 3. Its base branch is `fix/provider-turn-restart-recovery`.
Refs #19912 (design).

**What existing behavior does this improve?**

What a native ACPX Claude run does after a turn that ends with no result
while a background command is still running.

**Subsystem affected**

Core, the native runtime:
`server/src/services/native-runtime/native-session-executor.ts` and
`native-provider-turn-wait.ts`. ACPX Claude only.

**Current behavior**

- The run ends at the end of the turn. The Runner restarts Claude, the
background command stops, and the run fails with
`native_finalization_missing`.

**Proposed behavior**

- The native runtime sets `providerStartedTurns: true` in the transport
options for ACPX Claude, at the two places that build those options. One
function, `nativeProviderStartedTurns`, decides this and also decides if
the consumer gets the wait options, so the two cannot be different.
- A turn that ends with no result while a background command is still
running now waits, for a maximum of 10 minutes, for the turn that Claude
starts by itself. That turn is consumed as the next turn of the same
run, and its result settles the run.
- Other providers and other ACPX agents do the same as today.
- Limit of the Runner (see #21421): after a follow-up turn, the Runner
waits again only while a background command is still running. A command
that finishes while Claude reports on an earlier command gets no
follow-up.

**Reason and benefit**

- The opt-in is in the last part, so each earlier part can merge with no
change for a production run.

## What Changed

- `server/src/services/native-runtime/native-provider-turn-wait.ts` (5
lines added, 1 removed): `nativeProviderStartedTurns`.
- `server/src/services/native-runtime/native-session-executor.ts` (4
added, 2 removed): the option at the two places that build the transport
options, and the same function for the wait options.
- `server/src/services/native-runtime/native-session-executor.test.ts`
(4 added, 2 removed): the two native paths set the option for ACPX
Claude and not for other providers.

## State saved by the previous release

- No persisted state changes in this part.
- **A Runner that does not know the field** ignores it, and the run does
the same as today.
- **A deploy during a run:** a turn that started before the deploy has
no opt-in and ends as today. The next turn has the opt-in.
- **Rollback of Core:** the previous Core does not set the option, so
the Runner does not wait.

## Verification

The items below were run on the tree that was verified before the split
into three parts (commit `1b15ded1c4` has that tree, byte for byte). The
head of this branch (`23e2c13ad3`) is that commit on the new head of
#21612; its three server files are the same.

- `pnpm typecheck` and `node
scripts/verify-runner-vendor-dependencies.mjs` in `server`; `npx tsc
--noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`,
`pnpm run typecheck:typescript` and `pnpm run build:typescript` in
`packages/paperclip-runner`: all passed.
- `server/src/services/native-runtime/native-session-executor.test.ts`
and `native-runner-restart-recovery.integration.test.ts`: 1,732 passed,
21 skipped.
- The end-to-end behavior is tested in parts 1 and 2 with the real
`paperclip-runnerd`, where the test sets the option in the way that this
part does.
- Live run with a real Claude account, not in CI, at the adapter level
of the Runner (see #21316): the cycle of Claude was given as a turn with
its own usage, and each Paperclip tool call was admitted.

Not run: a live run through Core, `paperclip-runnerd` and the real
sidecar with a real Claude account (that path needs an API key, not a
local sign-in); a remote sandbox; the full suite. CI supplies the broad
coverage.

## Risks

- **This part turns the behavior on for each native ACPX Claude run.** A
run with no background command is not changed: the Runner waits only
when a shell is still running at the end of a turn that has no result.
- **This part must not merge before #21612.** Without #21612, a
controller restart during a wait or during a provider-started turn fails
the run.
- **A run can now stay open up to 10 minutes longer** when the
background command does not finish or Claude does not start a turn. The
agent computer stays active for that time.
- **A waiting run can show as "suspicious" after 5 minutes of silence**
in the run watchdog. The watchdog only labels a native run; it does not
stop it.
- **Follow-ups that the Runner cannot wait for are lost,** as the limit
above says.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 13:10:20 -08:00
DottaandClaude Opus 5.5 6ba46fd080 fix(github): read only the webhook secrets each delivery step uses (#21651)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents receive GitHub work through a GitHub App chat channel. GitHub
sends each event as a signed webhook delivery.
> - Each delivery passes three steps: durable staging, the durable
replay worker, and the replayed request's preflight under the credential
lease.
> - Each step resolved every stored GitHub App secret. A GitHub App
connection stores seven secrets, so each delivery decrypted and audited
about 21 secrets. The steps use only the signing secret, the bound
installation id and the App id.
> - On our staging instance, these reads are now the largest source of
secret reads and `secret_access_events` rows.
> - This pull request makes each webhook step read only its own keys,
and only when it needs them.
> - The benefit is fewer vault reads and audit inserts per delivery, and
the App private key and OAuth client secret stay sealed on the webhook
path.

## Linked Issues or Issue Description

Refs #21641 (fixed the other source: review-check polls now reuse cached
installation tokens).

**What happened?**
On our staging instance (serving 163d9f323c), from 2026-11-14 20:23:52Z
to 20:32Z, 32 `github_webhook_ingress` chat actions arrived. In the same
window, `secret_access_events` recorded 97 resolutions of
`credentials.webhookSecret`. It recorded the same count for
`privateKey`, `appId`, `installationId`, `clientId`, `clientSecret` and
`slug`. That is about 3 full credential resolutions per delivery, and
about 21 audit rows per delivery (roughly 70–80 rows per minute). In the
same window, 84 review-check polls caused no secret reads, because
#21641 fixed that source.

**What did you expect?**
A delivery reads only the secrets it uses. An unsigned or forged
delivery reads only the signing secret.

**Steps to reproduce**
1. Configure a GitHub App chat endpoint.
2. Send one signed `issue_comment` delivery for the bound installation.
3. Count the new `secret_access_events` rows for the connection. Before
this change, each stored key has 3 rows. A bad signature still reads
every key once.

The three call sites per delivery:
1. `server/src/services/chat-channels/github/webhook-ingress.ts`
`stageGitHubWebhookIngress` (`resolveCredentialRefs` with all refs).
2. `server/src/services/chat-channels/github/webhook-ingress.ts`
`processGitHubWebhookIngress` (`resolveCredentialRefs` with all refs, to
re-sign the retained body). This was the unattributed third resolution.
3. `server/src/services/chat-channels/github/webhook-authentication.ts`
preflight and ping (`resolveCredentials` with no key filter).

## What Changed

- Added `githubCredentialRefsFor(refs, keys)` in the GitHub webhook
ingress module. It selects the secret refs for named keys.
- Staging reads `webhookSecret` first. After the signature holds, it
reads `installationId` (only when the event names an installation and is
not an `installation` event) and `appId` (it scopes the recovery-receipt
denial query).
- The replay worker reads `webhookSecret` to re-sign the body. It reads
`appId`, `privateKey` and `installationId` only for a recovered delivery
that needs a source recheck.
- The preflight and ping read `webhookSecret` through the existing
`resolveCredentials` key filter. The preflight reads `installationId`
only after the signature holds and only for an event that names an
installation.
- The failed-delivery recovery scan reads only `appId`, `privateKey` and
`installationId`.
- Unchanged: the timing-safe compare, the credential-mutation lease, the
re-read of current endpoint state under the lease, the status and fence
checks, and the superseded-credential retry. No cache of decrypted
values.
- Per valid delivery, secret reads go from 7 keys × 3 (21 on staging) to
6: `webhookSecret` × 3, `installationId` × 2, `appId` × 1. A bad
signature goes from all keys to 1 read.
- `webhook-ingress.ts` was already above 1,000 lines. It grows by about
30 lines because the key selection belongs with the steps that use it.
`chat-channels.integration.test.ts` holds the GitHub webhook fixtures,
so the new test goes beside the existing signature tests.

## Verification

- Red on master, green with this change: `pnpm exec vitest run
server/src/__tests__/chat-channels.integration.test.ts -t "resolves only
the secrets a"`.
- `current`: before, `{webhookSecret: 3, installationId: 3, appId: 3,
privateKey: 3}` (the fixture stores 4 keys). After, `{webhookSecret: 3,
installationId: 2, appId: 1}`.
- `invalid_signature`: before, every key once. After, `{webhookSecret:
1}`.
- Regression: GitHub webhook, signature, rotation, ping, recovery,
installation, paused and revoked tests in
`chat-channels.integration.test.ts` (filter `-t
"[Ww]ebhook|signature|rotat|[Pp]ing|recovery|installation|GitHub
callback|revoked|paused"`): 304 passed, 1 failed. The failure is a Slack
test (`materializes direct external-chat finals once…`, matched by the
`ping` filter in "bookkeeping"). It passes when run alone, so it is a
load flake and not related to this change.
- `pnpm exec vitest run
server/src/services/chat-channels/github/webhook-ingress.test.ts
server/src/services/chat-github-provider-stress.test.ts
server/src/services/chat-github-webhook-config.test.ts`: 105 passed.
- `tsc --noEmit -p server/tsconfig.json`: passed.
- `pnpm verify` selects 2,726 of 3,554 Vitest files for this change (the
chat integration file pulls in a wide dependency set). I did not run
that broad set locally. Required CI supplies that coverage for the final
revision.

## Risks

- Low risk. Each step reads a subset of the same refs from the same
fenced record, so the values it uses do not change.
- A missing `webhookSecret` ref still fails the signature check as
before. A missing `installationId` ref still makes an
installation-bearing event "ignored" as before.
- The preflight now resolves `webhookSecret` and `installationId` in two
reads under the same credential lease. The lease blocks credential
mutation between them.
- Installation lifecycle events (`installation-lifecycle.ts`) still
resolve the full set, because they re-verify and re-persist all App
credentials. They are rare.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run in Claude Code
with tool use (shell, file edits, test execution) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 15:06:29 -06:00
DottaandClaude Opus 5.5 d4498ae7ac fix(runner): settle every recorded tool result and free sessions blocked by a closed turn's pending call (#21490)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call Paperclip tools through the Runner. The Runner records
each call and returns the controller's result to the provider (here
OpenCode)
> - In chat-eval cohort 13, a `list_workflows` result was recorded but
never reached OpenCode. OpenCode waited about 290 s, and the call stayed
pending
> - The next run's `run.attach` then failed with
`durable_pending_tool_calls`, so the person's follow-up question never
got an answer
> - This pull request fixes the result shape that caused the rejection,
makes the Runner settle every recorded result, and lets the next run
settle a call that a closed turn left pending
> - The benefit is that a tool result always reaches the agent, and a
stuck call can no longer block every later turn

## Linked Issues or Issue Description

**What happened?**
Chat-eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`). In
`cases.case-history` and `cases.create-case`, the CEO called
`list_documents` (or `list_projects`) and `list_workflows` (call id 5)
together. The server answered `GET /companies/{id}/workflows` with 200
in 584 ms. The OpenCode MCP bridge diagnostic for call 5 says `outcome:
"unanswered"`, `written: false`, `handlerMs: 289038` (290723 in
create-case). The sibling call was answered in about 1 s. The run ended
with `runner_settlement_incomplete`: `semanticTools.pending` held call 5
and its `semantic_tool.result` delivery command had status `failed`. The
next question started a run whose `run.attach` was refused: `run.attach
requires a settled Codex provider session: durable_pending_tool_calls`.
The run failed in 17 s, and the question was not answered.

Cause: `executeNativeWorkflowTool` returned the Runner API result
unchanged: `{ ok: true, status: 200, operationId: "GET
/api/companies/{companyId}/workflows", ... }`. Every other API-backed
tool renames that field to `apiOperationId`. The Rust tool bridge
(`semantic_response_value`) reads an object with a boolean `ok` and an
`operationId` as a semantic envelope that must name this tool and call.
The REST route did not match, so `apply_result` failed with `semantic
result envelope does not match its provider call`. Nothing was sent to
the provider, and the call stayed pending forever. The red test
reproduces this exact error. The parallel call is not the trigger: the
sibling call settled normally, and the OpenCode MCP bridge answered
other parallel pairs in the same runs. `list_workflows` first appears in
cohort 13, so cohorts 11 and 12 did not hit it. All four workflow tools
had this shape.

**Expected behavior**
Every recorded tool result reaches the provider. When a recorded result
cannot be delivered, the provider gets an error at once and the call
settles. A call that a closed turn left pending does not block the next
run.

**Steps to reproduce**
1. Run an OpenCode agent with Cases enabled on the durable Runner.
2. Ask a question that makes the agent call `list_workflows`.
3. See the call wait about 290 s. Then ask a follow-up question and see
`run.attach` fail with `durable_pending_tool_calls`.

**Paperclip version or commit**
`master` at `df1b12ad8e`.

## What Changed

- `server/src/services/native-runtime/native-workflow-tools.ts`:
workflow tool results name the route `apiOperationId`, like case tools
and `call_api`.
- `provider_bridge.rs`: add `settle_undeliverable_result`. When the
bridge rejects a recorded result for a call it still holds, the call
settles with a bounded, non-retryable runner error. Conflicting or
unknown results still fail as before.
- `provider_backend.rs` (`deliver_semantic_result`): use that
settlement, deliver the error to the provider, and record a
`semantic_tool_result_undeliverable` diagnostic with the call,
operation, reason and the rejected result's digest.
- `provider_bridge.rs` and `provider_backend.rs` (`attach_run`): add
`expire_closed_turn_calls`. When nothing is live (no active or ambiguous
turn, no provider request waiting), `run.attach` settles calls that a
closed turn left pending as an expired failure with an unknown outcome,
and records `semantic_tool_result_expired`. The provider stopped waiting
when its turn ended, and the new run's controller never delivers the old
run's results. A call that a live turn still waits on still blocks
attachment. This follows the existing `tool_call_interrupted_by_restart`
rule: never execute the operation again, and record an unknown outcome
when no result can ever arrive.
- Docs: `doc/run-log-events.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe both
diagnostics.

## Verification

- Red/green, Rust (`cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --test codex_provider --
a_rejected_recorded_result run_attach_expires`):
- Before:
`a_rejected_recorded_result_still_answers_the_provider_and_frees_the_session`
fails with `semantic result envelope does not match its provider call`
(the production error).
`run_attach_expires_a_call_left_pending_by_a_closed_turn` fails with
`run.attach requires a settled Codex provider session:
durable_pending_tool_calls` (the production refusal).
- After: both pass. The fake provider logs `tool-response:failure`, so
it got an answer at once. The next `run.attach` succeeds.
- Parallel calls (`cargo test -p paperclip-runner-core --lib --
provider_bridge`):
`a_rejected_result_in_a_parallel_pair_still_settles_its_own_call` sends
the cohort 13 pair (`list_documents` + `list_workflows`). The rejected
call settles, its sibling keeps its own result, and the next turn
starts. `only_a_closed_turn_expires_its_pending_calls` checks that an
open turn expires nothing.
- Upgrade and mid-run cutover:
`run_attach_expires_a_call_left_pending_by_a_closed_turn` saves state
with a pending call after the turn ends, stops that Runner, and attaches
with a new Runner process from the saved state.
`durable_backend_settles_pending_tools_when_recovery_finds_the_turn_ended`
(unchanged) still passes: a late result in the same run still settles as
`settled_after_turn`.
- Red/green, server (`cd server && npx vitest run
src/services/native-runtime/native-case-tools.integration.test.ts -t
workflows`): before, `expected { ok: true, status: 201, … } to not have
property "operationId"`; after, 2 passed.
- `cargo test -p paperclip-runner-core`: 643 lib tests pass. One lib
test (`fenced_receipt_output_keeps_prefix_when_stdout_never_closes`)
failed once with `EPERM` while signaling a process group under parallel
load. It passes alone 3 of 3 and does not touch this change. The
integration test binaries pass.
- Paid narrow run for `cases.case-history` and `cases.create-case`: see
the comment below.

## Risks

- Previous-release story: no persisted shape changes. State saved by the
previous Runner with a call pending after its turn closed carries
forward. The new Runner settles that call at the next `run.attach`, and
the session continues (tested from saved state with a new process). A
previous-release Runner that receives the corrected `apiOperationId`
result delivers it normally, because the result is no longer read as an
envelope.
- Behavior change: a call left by a closed turn is now settled at the
next run's attachment, not kept forever. Its outcome is recorded as
unknown, and the operation is never run again. A provider turn that is
still live keeps the old refusal (`durable_pending_tool_calls`,
`provider_side_effect_pending`), and the controller still never replaces
the session.
- A recorded result that the bridge rejects now becomes an error for the
agent, not an endless wait. The agent is told the action may have
completed and must check the current state before repeating it.
- Latency: nothing is added before turn start. The attach-time check
walks the in-memory pending calls (usually none) and adds no I/O beyond
the existing state save. The rejected-delivery path now answers in the
same command instead of after the provider's ~290 s client timeout, and
the follow-up run no longer fails after 17 s.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21398 (first release where `list_workflows` was called in this
eval), #19733 (the attach refusal), #20886 (earlier OpenCode delivery
fix).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:21 -06:00
DottaandClaude Opus 5.5 9b581d689b fix(chat): report an outcome once and create no run when an earlier reply gave it (#21489)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In agent chat, a finished delegated task or a declined tool action
wakes the chat agent so it tells the person
> - #21401 stopped a second completion report when an earlier reply
already gave it
> - But the suppressed completion still created a zero-second heartbeat
run, and declined tool actions still got a second report
> - Run accounting must count only real turns, and the person must get
each outcome once
> - This pull request settles an already-reported outcome before it
becomes a run, and applies the same rule to declined tool actions
> - The benefit is one report per outcome, no phantom run, and no extra
model call

## Linked Issues or Issue Description

Refs #21401

**What happened?**
Chat eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`)
showed three problems after #21401.
1. `questions-and-approvals.duplicate-answer`: the completion wake was
suppressed, but it still created a 0-second run (`runtimeMode: legacy`,
no usage). The bounded-native-runs check rejected that run.
2. `skills.studio-test`: the CEO's single reply reported the Studio
result after it read the test task done (0 passed, 1 failed, links
COM-2). The completion was then suppressed, but the eval waited for a
separate completion run and timed out after 240 s.
3. `questions-and-approvals.decline-tool`: the person declined the tool
review one second after the turn for their message started. That turn
read the declined action and said so. The decline continuation, which
waited behind that turn, then posted the same declined outcome again.

**Expected behavior**
Each finished task or declined tool action is reported once in the
conversation. An outcome that an earlier reply already reported creates
no heartbeat run.

**Steps to reproduce**
1. In chat, delegate a short task, or start a governed tool read that
needs approval.
2. While it waits, send a message that wakes the chat agent.
3. Let the task finish, or decline the tool review, while that turn
runs. The turn reads the outcome and reports it.
4. See the completion wake become a 0-second run, or see a second
decline report from the decline continuation.

**Paperclip version or commit**
master `df1b12ad8e`

## What Changed

- `server/src/services/chat-completion-delivery.ts`:
`settleReportedChatCompletions` marks each undispatched completion that
an earlier reply reported as delivered, with that reply as its response.
It returns true only when nothing else is owed. The outbox `deliver`
also checks this before it requests a wake, so no wake and no run exist
for an already-reported completion.
- `server/src/modules/wake-queue`: a new transaction port,
`skipAlreadyReportedChatWake`, runs in the release drain before a
deferred wake is promoted. When the earlier reply already reported every
outcome of the wake, the wake ends `skipped` ("Already reported by an
earlier reply in this conversation") and no run is created. Wakes that
carry queued comments or a failed-chat retry are never skipped.
- `server/src/services/chat-interaction-outcome-report.ts` (new): the
same structural rule for declined tool actions. A conversation agent run
that lists its interactions records `chat_interaction.declined_read` for
each declined tool action. The decline continuation is settled when the
same agent replied in the conversation after the decline, from a run
with that receipt, posted after the read. The rule covers only declined
tool-action confirmations. Answers, plan reviews and accepted actions
keep their own turn.
- `server/src/routes/issues.ts`: `GET /issues/:id/interactions` calls
the recorder for agent runs.
- The run-start check from #21401 stays as a fallback for a completion
turn that was admitted before the earlier reply landed.
- `tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.ts`:
the Studio oracle (v5) accepts either the completion turn's reply or an
earlier CEO reply as the delivery. The earlier reply counts only from
saved state: no completion run exists, the replying run has a
`chat_handoff.completed_read` receipt for the test task, the reply was
posted after that read, it is the only such reply, and it links the
saved result. No wording check is added.
- `doc/run-log-events.md`: documents both read receipts and the skipped
wake outcome.

## Verification

- `cd server && npx vitest run
src/__tests__/chat-completion-delivery.test.ts
src/modules/wake-queue/adapters/chat-declined-outcome-reported.test.ts
src/modules/wake-queue/application/use-cases.test.ts`: all pass.
- Red: with the drain check and the outbox check disabled, 5 tests fail:
"starts no completion wake or run ...", "settles a deferred completion
wake without a run ... (true)", "settles the decline continuation with
no run ...", and both "settles an already-reported chat wake without a
run" cases.
- Negative controls stay green: a linking reply without a done read, a
decline the earlier turn never read, a non-tool-action card, and an
accepted card all still run their continuation.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.test.ts`:
34 pass. Red on the old oracle: "accepts the earlier reply the server
took as the completion report" fails. New negative cases reject no done
read, a read of another task, a reply before the read, a reply from
another run, two replies, an extra completion run, and a missing task
link.
- A narrow paid eval run of `duplicate-answer`, `studio-test` and
`decline-tool` on this branch follows in a comment.

## Risks

- A completion turn admitted before the earlier reply landed (a narrow
race) still ends as a short succeeded run with no model call, as in
#21401.
- If an earlier reply read the outcome but did not mention it, the
person does not get a second message. The read receipt, the
reply-after-read order and, for completions, the task link make this
unlikely.
- The decline rule covers only declined tool-action confirmations. Other
interaction outcomes are unchanged.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:16 -06:00
DottaandClaude Opus 5.5 8a97037bd2 fix(runs): say who started a retry run in the issue run read (#21486)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat agent answers questions about failed runs by reading the
task's run list (`GET /api/issues/{id}/runs`)
> - For a retry run, that read returned only raw invocation fields
(`on_demand`, `manual`) and an owner id. It did not say who started the
retry
> - In chat-eval cohort 13, the agent told the person "Paperclip retried
automatically" about a run that the board started with Retry
> - This pull request adds a plain-words `retriedBy` field to each retry
run, and tells the chat agent to use it
> - The benefit is that people get a true account of who retried their
work, with a name and no raw ids

## Linked Issues or Issue Description

**What happened?**
Chat-eval cohort 13 ran OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017` on master `df1b12ad8e`.
In `runs-adapters.inspect-failed-run`, the reply said "Paperclip retried
automatically on a supported model". The successful run was a person's
manual Retry: `invocationSource: on_demand`, `triggerDetail: manual`,
wake reason `retry_failed_run`, and the responsible user was the board.
The agent's own reasoning hedged ("probably automatic"), because the run
read did not say who started the retry.

**Expected behavior**
The run reads that the agent uses say who started each retry: a person,
by display name, or Paperclip's automatic recovery, in plain words and
with no raw ids.

**Steps to reproduce**
1. Let a run fail on a task.
2. Press Retry on the failed run as a board user.
3. Read `GET /api/issues/{id}/runs`. Before this change, nothing in the
retry row says that a person pressed Retry.

## What Changed

- `runsForIssue` (`server/src/services/activity.ts`) reads the wake
request of each retry run. It adds `retriedBy` to each row:
- `{ kind: "person", personName, summary }` when a user wake with reason
`retry_failed_run` or `task_retry` started the run. The name comes from
the user record, or "Board" for the local board, with the same fallback
as activity actor names.
  - `{ kind: "automatic", summary }` for an `automation` retry run.
- `null` for a run that is not a retry, or a retry whose starter is not
recorded. The read does not guess. It does not use `responsibleUserId`,
because automatic retries keep the original responsible user.
- The OpenAPI description of `GET /api/issues/{id}/runs` and the chat
agent directive tell the agent to use `retriedBy` and not to call a
retry automatic unless `retriedBy` says so.
- The UI `RunForIssue` type has the new optional field.

## Verification

- `cd server && npx vitest run src/__tests__/activity-service.test.ts
src/__tests__/agent-conversations.test.ts
src/__tests__/activity-routes.test.ts
src/__tests__/openapi-routes.test.ts` passes.
- Red/green: the new test "says who started each retry run" fails
without the service change (`expected undefined to be null`) and passes
with it. It covers a board Retry, a named member's Retry, an automatic
retry that keeps the board as responsible user, a retry with no recorded
wake, and no raw ids in the output.
- `npx tsc --noEmit -p server` passes.
- A narrow paid run of `runs-adapters.inspect-failed-run` (with the two
sibling cohort-13 PRs) ran on a combined branch. The results are in a PR
comment.

## Risks

- Low risk. The change adds one read-only field and one bounded query by
primary key on the wake requests of retry runs in the page.
- No persisted state changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:04 -06:00
DottaandClaude Opus 5.5 95a6ab31e6 feat(db): attribute work signals by caller behind a debug flag (#21640)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Background queues wake from work signals. A transaction calls
`signalDatabaseWork(topic, hint)`, and the delivery coordinator wakes
the queue for that topic with that hint.
> - On a hosted instance with zero runs, the stranded-assignment
recovery queue re-checks tasks about 7 times a second. About 2.6 hinting
commits per second drive it. HTTP traffic, chat review polls and local
fixtures with all heartbeat workers do not explain the writer.
> - Statement statistics show what runs, but not who signalled it.
Nothing records the caller of a work signal or why a queue woke.
> - This pull request adds an opt-in attribution log for work signals
and recovery-queue wakes, off by default.
> - The benefit is that one short run on an affected instance names the
writer of any idle signal loop, without code changes.

## Linked Issues or Issue Description

No issue exists. This describes the gap with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

Observability of `signalDatabaseWork`
(`packages/db/src/work-signals.ts`) and of the issue recovery queues
(`server/src/services/recovery/work.ts`).

**Current behavior**

A queue that wakes several times a second at idle shows up only as
repeated statements. Nothing tells you which caller published the hint,
whether the wake came from another process, a reconnect, a direct
settled publish or a stored deadline, or which load and check it caused.

**Proposed behavior**

An environment variable, `PAPERCLIP_DEBUG_WORK_SIGNALS`, read once when
the module loads. It holds a comma-separated topic allowlist, and each
topic can carry `:N` for its stack sampling rate. When it is unset or
empty, the signal path only checks one boolean and does nothing else.

**Reason and benefit**

Several idle-load loops this week took hours of statement analysis
before the writer was found. Attribution by caller turns that into one
log query.

## What Changed

- `packages/db/src/work-signal-debug.ts` (new):
- `parseWorkSignalDebugSpec` reads `topic[:N],…`. An optional numeric
suffix is parsed from the right, so topics that contain colons (for
example `live:<companyId>`) work with or without `:N`. N defaults to 20;
malformed entries are ignored.
- `createWorkSignalDebug` returns `enabled`, `allows(topic)`,
`signal(topic, hint, entry)`, `event(topic, name, fields, stack, entry)`
and `flush()`. A disabled instance constructs no `Error`, captures no
stack, builds no line and starts no timer.
- Stack capture (`captureWorkSignalStack(entry)`) raises
`Error.stackTraceLimit` to 40 and restores it in a `finally`. It calls
`Error.captureStackTrace(holder, entry)`, so V8 cuts every frame above
the public entry (`signalDatabaseWork`, `publishDatabaseWorkSettled`, or
the recovery `wake()`).
- Frames are dropped by file (`node_modules`, `node:internal`, the
signal wrapper files) and by function name (`signalDatabaseWork`,
`notifyDeliveryWork`, …), so trimming works on the bundled `index.mjs`.
Positions are kept raw (`index.mjs:L:C`) for offline source-map lookup.
At most 10 frames are kept.
- Each (topic, event) pair has its own budget of 10 lines per second,
checked BEFORE any stack is captured. A `suppressed` line names the
topic, the dropped event and the count. Frequent `check` lines therefore
never drop `signal` or `direct_wake` lines.
- Stack sampling (1 in N) counts per (topic, event), so `check` and
`load` do not shift the phase of `signal` sampling. `direct_wake` always
captures its caller's stack.
- Lines go to an in-memory buffer that an unref'd 1-second timer flushes
to stderr. The signal path does no I/O and no await.
- `workSignalDebug` is the process instance, built from
`process.env.PAPERCLIP_DEBUG_WORK_SIGNALS` once at module load.
- `packages/db/src/work-signals.ts`: when enabled, it records:
  - `signal` in `signalDatabaseWork`;
  - `remote_notification` and `notification_reconnect`;
- `settled_publish` (with a sampled stack) for
`publishDatabaseWorkSettled`;
  - `uncertain_settled` when an uncertain transaction resolves.
- `server/src/services/recovery/work.ts`: under the same flag:
  - each task-queue `load` logs its scope key;
- each `check` logs the issue and company id, `dueAt`, and `wake:
"deadline"` or `"load"`. The marker rides on the queued entry itself,
and only while the flag is on, so nothing is tracked otherwise and
nothing outlives the entry;
- `wake()` logs ONE `direct_wake` line listing the queues it actually
invalidates (assigned-work only when registered), under the first
allowlisted one, with its caller's stack.
- **No request route is logged.** The server has no request async
context to read one synchronously, and this change does not add one.
- `docs/deploy/environment-variables.md`: documents the flag, internal
ids only, raw positions, and that enabling or changing it requires a
restart.
- Tests:
  - `packages/db/src/work-signal-debug.test.ts`, 12 tests:
- strict no-op when unset or empty (no `Error`, no `captureStackTrace`,
no stack capture, no sink);
    - the process instance is disabled without the env;
    - allowlist filtering, and spec parsing including colon topics;
    - per-(topic, event) sampling, and always-on direct-wake stacks;
- source-tree trimming, and a synthetic single-file bundle stack with
raw positions kept and wrappers dropped by name;
    - real capture below the entry with the limit raised and restored;
- checks at about 8/s alongside signals at about 3/s for 5 seconds, with
every signal line kept;
    - suppressed records never capture a stack;
    - buffered, timer-driven output.
- `packages/db/src/work-signals.test.ts`: on real PostgreSQL, the signal
and settled-publish paths never call the debug log while the flag is
off.
- `server/src/__tests__/issue-recovery-work-debug-attribution.test.ts`,
on the recovery fixture with real PostgreSQL:
    - no recovery log calls while the flag is off;
- with the flag on, a run graph hint logs its signal, the graph-scoped
load and the check;
- `wake()` logs exactly one `direct_wake` that lists the queues and
carries the caller's stack.

## Verification

- `packages/db`: `vitest run src/work-signal-debug.test.ts
src/work-signals.test.ts`: 39 passed. `tsc --noEmit` passes.
- `server`: `vitest run
src/__tests__/issue-recovery-work-debug-attribution.test.ts
src/__tests__/issue-recovery-deadlines.test.ts`: 17 passed. `tsc
--noEmit` passes.
- Staging values:
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:10`, then restart.
- Sample output from the synthetic single-file bundle test:

```
{"msg":"paperclip work signal debug","at":1000000,"event":"signal","topic":"stranded-assignments","hint":"{\"companyId\":\"c\",\"graphRunId\":\"r\"}","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","at async notifyRunDispatch (file:///app/server/dist-runtime/index.mjs:120501:3)","at async PostgresJsDatabase.connection.transaction [as transaction] (file:///app/server/dist-runtime/index.mjs:4021:22)","at async Object.settleRunProjection (file:///app/server/dist-runtime/index.mjs:300777:5)","at async file:///app/server/dist-runtime/index.mjs:310002:17"]}
{"msg":"paperclip work signal debug","at":1000000,"event":"check","topic":"stranded-assignments","issueId":"i","companyId":"c","wake":"load","dueAt":1}
{"msg":"paperclip work signal debug","at":1000000,"event":"direct_wake","topic":"stranded-assignments","queues":"stranded-assignments,orphan-blockers","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","…"]}
{"msg":"paperclip work signal debug","at":1001000,"event":"suppressed","topic":"stranded-assignments","suppressedEvent":"check","count":17}
```

## Risks

- Low risk. It is off by default. While off, each signal costs one
property read and one boolean check.
- While on, the log can show internal ids from hints (company, issue,
run). It holds no secrets and no user identifiers, and the topic
allowlist limits what is logged.
- Stack capture costs CPU. Sampling (default 1 in 20) and the 10 lines
per second cap bound it.
- The file owner of `packages/db/src/work-signals.ts` reviews this
before merge.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21621.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:54:00 -06:00
DottaandClaude Opus 5.5 163d9f323c perf(github): reuse repository installation tokens until shortly before expiry (#21641)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel lets an agent review pull requests and
answer comments through a GitHub App
> - Each GitHub call gets a repository-scoped installation token from
`githubBotRepositoryToken`
> - That function minted a new token on every call. To mint, it resolved
every App secret through the secret service, and each resolution wrote
one audit row
> - The review-check head poll calls it about every 5 minutes for each
waiting check. On staging this made the secret audit insert one of the
busiest statements in the database
> - This pull request keeps a minted token in memory until shortly
before it expires, and checks the connection's current state on every
call
> - The benefit is far fewer database writes and reads, fewer GitHub
token mints, and an audit log that records real secret use

## Linked Issues or Issue Description

**What happened?**
On paperclip.staging (commit a18479bbba, measured at 19:05Z on
2026-11-14), `secret_access_events` got about 285 rows in 60 s. These
were 44 resolutions each of the GitHub App connection's
`credentials.appId`, `privateKey`, `installationId`, `webhookSecret`,
`clientId` and `clientSecret` (+21 `slug`), with `consumer_type`
`tool_connection` and actor `system`. That is about 4.7 audit inserts/s.
The matching `secret_versions` and binding reads ran at about 3.6/s each
in Neon. These were among the busiest statements on the database.

**Expected behavior**
A GitHub installation token is valid for one hour. Paperclip should
reuse it and resolve the App secrets only when it mints a new token.

**Steps to reproduce**
1. Connect a GitHub App chat endpoint and get pull request reviews into
the `received` review-check state (staging had 84 `github_review_check`
actions).
2. Let the review-check worker poll the pull request head (23 polls/min
measured on staging).
3. Count `secret_access_events` rows per minute. Each poll adds one row
per App secret and mints a new token at GitHub.

**Paperclip version or commit**
master at a18479bbba (the commit serving paperclip.staging at the time
of measurement).

**Where in the code**
`githubBotRepositoryToken` in
`server/src/services/chat-github-client.ts`. Callers:
`chat-github-checks.ts` (review-check head poll),
`chat-github-response-comments.ts` and `chat-github-reviews.ts`.

## What Changed

- `githubBotRepositoryToken` keeps a minted token in a per-process,
in-memory cache (at most 512 entries, least recently used first out). It
never persists or logs tokens.
- The cache uses the token until 5 minutes before GitHub's `expires_at`
(capped at one hour). If GitHub does not give a usable expiry, the token
is not kept.
- The cache key holds the company, endpoint, repository id, the exact
permission set, and an authority revision. The revision holds the
connection id, the endpoint's runtime generation (pause, revoke and
availability changes advance it), and for each mint credential its
secret ref, version selector, secret status, current version,
`updatedAt`, and binding. The installation id is a secret, so its secret
ref and version identify it in the key.
- Each call reads the current authority from the database in two narrow
statements: one row from the endpoint and connection join (status,
enabled, refs, runtime generation), and the three mint secrets with
their bindings (status and version columns only, no secret values). The
endpoint and connection status checks run on every call, as before.
- If a status check fails, all cached tokens for that endpoint are
dropped. A token cannot survive a revoke, pause, disable or credential
change.
- Secrets are resolved only when a token is minted, and only the three
that minting uses (`appId`, `privateKey`, `installationId`). So the
audit log gets three rows per mint instead of one row per App secret per
call.
- New `githubBotRepositoryClient` sends repository requests. If GitHub
returns 401 for a cached token, it drops that token, mints a new one,
and sends the request once more. A 401 for a newly minted token is not
retried. The review-check, response-comment and review paths use this
client.
- The token policy stays in the GitHub chat client module
(`rules/module-boundaries.mdc`). The callers only change how they get a
request function.

## Verification

- New test file
`server/src/__tests__/github-repository-token-cache.test.ts` (embedded
Postgres, real secret service, mocked GitHub fetch):
- `mints once and resolves the App's secrets only when it mints`: red on
master (second call minted `installation-token-2`), green now. It also
checks that only the three mint secrets are audited.
- `refreshes a token shortly before GitHub's expiry`: reused at +54 min,
minted again at +56 min.
  - `never keeps a token GitHub issues for less than the safety margin`
- `mints again after a credential rotation and uses the new credential`:
rotates the installation id secret and checks that the new mint uses it.
  - `mints again after the connection's credential refs change`
- `never serves a cached token across a pause, revoke or disabled
connection`: includes a pause and resume with no call in between.
  - `mints a separate token for each repository`
- `replaces a cached token GitHub rejects and retries the request once`:
also checks that a 401 for a new token is not retried.
- On master, 5 of the 8 tests fail. All 8 pass with this change.
- Focused suites pass locally: every server test file that matches
`github`, plus `chat-sdk-runtime.adapters`, `chat-provider-inventory`,
`idle-completed-history` and `connection-management.integration` (41 of
42 files, 955 of 956 tests). The one failure,
`github-connection-enrollment-work.test.ts`, also fails on unchanged
master (`expected +0 to be 1`), so this change did not cause it.
- `pnpm --filter @paperclipai/server typecheck` passes.
- CI supplies the full typecheck, build and test coverage for the final
revision.

## Risks

- Low risk. A cached token is used only while the endpoint, connection
and mint secrets are unchanged, and the status checks still read current
database state on every call.
- A GitHub-side revoke of a token (for example, an App uninstall) makes
GitHub return 401. The client then mints again once. If the installation
is gone, the mint fails as before.
- The cache is per process. Each server process mints its own token, at
most once per hour per repository and permission set.
- Two concurrent first calls for the same key can both mint. This is the
same as before and is rare.
- No schema, migration or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:42:06 -06:00
DottaandClaude Opus 5.5 a18479bbba perf(workspaces): start a full terminal workspace pass only at startup and on the hourly backstop (#21630)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep walks every candidate workspace in a
full pass. On staging that is about 4,000 finished remote workspaces,
and each one costs a persistence probe and a work product read.
> - #21617 limited full passes to startup and an hourly backstop that
only activity arms. But three other paths in the worker still started a
full pass at once:
>   - a wake whose hints do not parse to workspace or task targets;
>   - more than 50 pending targets;
>   - any run that finds no targets.
> - Across one staging deploy the probe count rose by about 2.8 full
passes in 26 minutes. At 1,000 concurrent agents, more than 50
run-finish hints between passes would be normal, so the overflow path
alone would bring back passes proportional to history.
> - This pull request works through targets in batches of 50 and turns
hintless wakes into the backstop. It gives the one caller that needs a
broad look, a workspace provider plugin becoming ready, its own hint
that inspects only plugin workspaces.
> - The benefit is that a full pass happens only at startup and at most
once per backstop hour, whatever the activity.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On staging, the hourly steady state holds (one full pass per hour, no
probes between). But across a deploy the probe count rose by 11,351 in
under 26 minutes, about 2.8 full passes over the roughly 4,000 finished
remote workspaces.

**Steps to reproduce**

The new test file
`server/src/__tests__/terminal-workspace-rotation-limits.test.ts` runs
the real coordinator and worker against real PostgreSQL on a fake clock,
with finished remote workspaces that cannot be archived (the staging
state). These are the counts on master. The numbers are workspaces
checked by full passes; a pass over N workspaces checks N:

| Scenario | Full-pass checks on master | Expected |
| --- | --- | --- |
| A burst of 120 run-finish `issue:` hints, 300 workspaces | 600 (2
extra passes) | 0 |
| One hintless wake and one unparseable hint, 120 workspaces, 30 minutes
| 1,490 (about 12 passes) | 0, then 1 at the backstop |
| A workspace provider plugin becomes ready (hintless notify from
`plugin-registry`), 123 workspaces | 123 (1 pass) | 0; only the 3 plugin
workspaces |
| Restart: old instance drains while hints arrive, new instance starts
and gets the plugin hint and 120 run-finish hints, 202 workspaces, 30
minutes | 2,424 (12 passes) | 202 (the startup pass) |

**Cause of the passes across a deploy:**
- The new instance runs its startup pass.
- Each workspace provider plugin that turns `ready` sends a hintless
notification (`plugin-registry.ts`), which started a full pass.
- Run recovery at startup finishes many runs, and their `issue:` hints
overflow 50, which started another full pass.
- Any later wake without targets also started a full pass, including the
coordinator's 5-minute retry timer. In the fixture this produced two
back-to-back passes every 5 minutes.
- The draining old instance starts nothing, because the worker's
`canRun` is false during drain. The test confirms this.

**Expected behavior**

A full pass runs only at startup and when the activity-armed backstop is
due, at most once per hour. Targets are worked through in targeted
batches.

## What Changed

- `server/src/services/terminal-workspace-work.ts`:
- **Pending targets.** Workspace and task targets go into pending sets
with no cap. Each run inspects at most 50 of them, and the rest stay
queued for the next run, which is scheduled at once. A failed targeted
run puts its targets back.
  - **Hintless or unparseable wakes** only arm the hourly backstop.
  - **No targets, nothing due:** the run does nothing.
- **Due deadlines:** the due workspaces are queued as targets, including
isolated re-home and nested clone deadlines keyed by a workspace. A due
non-workspace deadline (the isolated source orphan sweep, task directory
work) runs a targeted sweep for its prelude and arms the backstop.
- **A failed plugin workspace listing** restores the plugin refresh
flag, so the coordinator's retry lists again.
- **Full passes** happen only at startup, when the backstop is due, and
on continuation pages of a pass already under way.
- **New hint `plugin-workspaces`** (`TERMINAL_WORKSPACE_PLUGIN_HINT`):
it queues the ids of plugin-provider candidate workspaces from
`listPluginTerminalWorkspaceIds`.
- The worker now always tells the service `targeted: true`, so a hint
during a pass no longer requests a rescan lap.
- `server/src/services/execution-workspaces.ts`:
- New `listPluginTerminalWorkspaceIds` reads the plugin-provider
candidates.
- A targeted sweep reads all matches of its at most 50 targets, with no
page limit and no rescan request.
- A failed targeted sweep no longer requests a rescan lap, because the
worker re-queues its targets.
- `server/src/services/plugin-registry.ts`: the two workspace-provider
notifications send `plugin-workspaces` instead of no hint.
- `server/src/__tests__/terminal-workspace-work.test.ts`: a new test,
`retries the plugin workspace listing after it fails instead of waiting
for the backstop`, fails without the retry fix. Three existing tests now
describe the new behaviour:
  - unscoped commits go to the backstop;
  - more than 50 hinted workspaces are inspected in batches of 50 and 1;
- a due deadline on a service without targeting still runs a full pass.
- `server/src/__tests__/terminal-workspace-rotation-limits.test.ts`:
four new tests, the scenarios above.

The startup pass stays. The 1,000-workspace test in
`terminal-workspace-deadlines.test.ts` shows that durable deadlines keep
an idle instance quiet after startup. But only the startup pass
reconciles workspaces whose hints were lost while no instance was
running, so removing it would need a stored "last full pass" marker
first.

## Verification

- The four new tests fail on master with the counts in the table and
pass after the change:
- 0 full-pass checks for the burst, worked as targeted batches of at
most 50 (120 checks in total);
- 0 full-pass checks for 30 minutes after hintless and unparseable
wakes, then 1 pass at the backstop;
  - only the 3 plugin workspaces are inspected after the plugin hint;
- exactly 1 full pass across the restart: none on the draining instance,
only the startup pass on the new one.
- Prompt archiving and the missed-hint backstop keep their existing
tests in `terminal-workspace-deadlines.test.ts`:
  - the release of a holding run inspects its workspace at once;
- a run made terminal outside `writeRunStatus` is archived within 1 hour
1 minute;
  - 1,000 finished remote workspaces stay unread at idle.
- `npx vitest run` in `server/` on `terminal-workspace-rotation-limits`,
`terminal-workspace-work`, `terminal-workspace-deadlines`,
`terminal-workspace-idle`, `terminal-workspace-page-batch`,
`execution-workspaces-service`, `plugin-workspaces.integration`,
`workspace-operations-reconciliation`, `execution-workspace-reopen`,
`isolated-task-directory-retention`, `git-workspace-expiration` and
`workspace-delivery-cleanup.integration`.
- `pnpm check:module-boundaries` passes.

## Risks

- **A notification that relied on an immediate broad pass now waits up
to an hour.** The only caller I found that sent such a notification on
purpose is the workspace provider readiness in `plugin-registry.ts`, and
it now has its own targeted hint. Other hintless notifications arm the
backstop.
- **The isolated source orphan sweep** runs only at the start of a full
pass. Its due deadline now arms the backstop instead of starting a pass
at once. This only matters when the isolated source re-home feature is
enabled.
- **A targeted sweep has no page limit.** Its at most 50 targets bound
the matched workspaces to those tasks, their ancestors and their
execution workspaces.
- **A burst of N hints costs N/50 targeted runs, back to back.** The
cost scales with activity, not with history.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 12:19:43 -06:00