Commit Graph
8146 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 93111b175b fix(wake-queue): keep the child-completion reason when a person's comment merges into its held wake (#21918)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The wake queue holds new work for a task while an agent run is
active. A later wake for the same task merges into the held wake
> - A parent task must not finish while a finished child's result is
still waiting for a parent turn. The finish gate checks for this pending
child result
> - The gate finds the held child-completion wake by its context wake
reason
> - When a person's comment merged into that held wake, the merge
replaced the wake reason with `issue_commented`. The gate then did not
see the pending child result
> - This pull request keeps every merged reason in the held wake and
makes the gate read that list
> - The benefit is that a merge keeps the meaning of both wakes: the
comment is delivered, and the pending child result still blocks a
premature finish

## Linked Issues or Issue Description

Refs #21914

**What happened?**

A restart test held a child-completion wake behind a running parent run.
A person then posted a comment. The comment wake merged into the held
wake. After the merge, the held wake's
`_paperclipWakeContext.wakeReason` was `issue_commented`, not
`issue_children_completed`. `hasPendingNativeChildCompletion` matches
held wakes by that field only. So the finish gate could no longer see
the pending child result.

**Expected behavior**

The held wake keeps both reasons. The finish gate still sees the pending
child result. The next run still receives the person's comment.

**Steps to reproduce**

1. Start a run for a parent task.
2. Hold a child-completion wake behind the run.
3. Post a person's comment on the parent. The comment wake merges into
the held wake.
4. Call `hasPendingNativeChildCompletion` for the running parent run. It
returns `false`.

**Paperclip version or commit**

`master` at 20b0d4e5a5.

**Relevant logs or output**

The suspected effect on finish timing in the test run was not real. The
two rejected finish calls failed for other reasons: the first had
invalid tool input (a missing `remainingWork` field) and the second had
no accessible deliverable. Both happened before the comment merge. Also,
the run had already read the finished child before the merge, so the
gate was already satisfied for that child. The gap is real for a run
that has not read the child yet. The new test proves it.

## What Changed

- `admitWakeBehindIssueExecution` records `wakeReasons` in the merged
held-wake context when two different reasons merge. `wakeReason` keeps
the latest reason, as before, so comment delivery and person-wake
handling do not change.
- New `withMergedWakeReasons` helper in the wake-queue domain context
module.
- `hasPendingNativeChildCompletion` matches a held wake when
`wakeReason` is `issue_children_completed` or when `wakeReasons`
contains it.
- New embedded-Postgres test in
`heartbeat-comment-wake-batching.test.ts`: a child-completion wake is
held behind a running parent, then a person's comment merges in. The
gate still sees the pending child result, and the next run receives the
comment.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-comment-wake-batching.test.ts
src/__tests__/native-child-task-read-routes.test.ts
src/modules/wake-queue` passes (16 files, 400 tests). This includes both
#21914 reopen tests.
- `cd server && npx vitest run
src/__tests__/native-status-arbiter-corpus.test.ts
src/services/workspace-recovery-admission.test.ts
src/__tests__/heartbeat-held-comment-wake.test.ts` passes (343 tests).
- `cd server && npx tsc --noEmit` passes.
- Red check: with only the source change reverted, the new test fails.
The merged context has no `wakeReasons`, and the gate check after the
merge fails with `expected false to be true`.
- `pnpm verify` exits 2 (an opaque setup dependency). Required CI
supplies broad coverage.

## Risks

- Low risk. The change adds one context key on merged held wakes and
widens one gate query. `wakeReason` does not change.
- A wider gate can hold a parent finish for a child result that a merge
used to hide. That is the intended behavior. The gate still ignores a
child result that the run already read.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1116.0-canary.10
2026-11-16 10:37:46 -06:00
DottaandClaude Opus 5.5 4da6902cb3 fix(dispatch): start work on a task tree a person created in the person tier (#21916)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue chooses which queued run starts when an agent at
its concurrency limit frees a slot
> - The queue puts runs a person asked for in a "person" tier ahead of
agent "work"
> - Subtasks of a person's task, and the parent's resume after its
children complete, ranked as agent work
> - Under load these runs started only through the 60-minute oldest-run
share, so a half-finished task waited about 55 minutes per step behind
newer tasks
> - This pull request puts every non-recovery run on a task tree a
person created in the person tier
> - The benefit is that a person's started work finishes before newer
work starts, and the fairness bounds stay in place

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The order in which an agent at its concurrency limit starts queued runs.

**Subsystem affected**
Server heartbeat queue:
`server/src/services/heartbeat/queued-run-dispatch-order.ts` and
`queued-run-line.ts`.

**Current behavior**
A board user's new assignment ranks as person work. An agent's subtask
of that person's task, and the parent's `issue_children_completed`
resume, rank as agent work. In a restart chaos run with agents at
capacity, the median queue wait of a parent resume was about 1,640 s.
The median wait of a new assignment was about 190 to 290 s. One parent
waited while 14 newer tasks started first. Subtasks and parent resumes
started only near the 60-minute aging bound.

**Proposed behavior**
A non-recovery run on a task tree that a person created goes in the
person tier. A task is in such a tree when the task, or any task above
it, has `createdByUserId`. In the tier, runs keep the existing order:
readiness, then priority, then age (oldest first).

**Reason and benefit**
A person waits on the whole tree, not only on the top task. Work that a
person started now finishes before newer tasks take its slots.

**Breaking changes**
None. There is no schema or API change.

## What Changed

- `queuedRunDispatchClass` puts a run with `personTask` in the person
tier. System recovery stays in the recovery tiers, also on a person's
tree.
- `personTask` now means "the task, or a task above it, was created by a
person". The existing `createdByUserId` marker is the source.
- New `listPersonTreeIssueIds`: one recursive query per dispatch pass.
It walks each queued task's parent chain through the `(company_id, id)`
unique index. It stops at the first person-created task, at most 32
levels. It runs only when the agent has more than one queued run. The
start-history read uses the same query in place of its old direct
`createdByUserId` read, so the query count does not change there.
- Recovery's 1-in-4 share is held only while a run a person asked for
(`isPersonRequestedRun`) is ready, not for the rest of a person's tree.
So a deep tree cannot hold recovery indefinitely. The dispatcher and the
place-in-line replay use the same rule.
- The claim backoff for a declined claim keeps the short person backoff
only for runs a person asked for (unchanged behavior for tree runs).
- `doc/heartbeat-internals.md`: new "Queued run dispatch order" section
with the tiers, the order and the fairness bounds.

Fairness bound: the oldest-run share is unchanged. Every 4th start goes
to the longest-waiting ready run past 60 minutes, preferring runs people
wait on. So another person's new task starts within 60 minutes plus 4 x
(aged person-tier runs queued before it + 1) starts, even while a deep
tree keeps adding in-progress or higher-priority work. A newer tree run
never passes an older, equally ranked person task. Queues are per agent,
and an agent belongs to one company, so no agent or company gets ahead
of another.

## Verification

- Red on master, green with this change:
- `queued-run-dispatch-order.test.ts`: an older parent resume
(`issue_children_completed`, person tree) starts before a newer person
assignment. The place-in-line replay agrees.
- `queued-run-dispatch-order.test.ts`: an older subtask of a person's
task starts before a newer person assignment.
- `queued-run-dispatch-order.test.ts`: fairness simulation. A deep tree
adds two in-progress, higher-priority runs per start. Another person's
new task starts within `QUEUED_RUN_AGING_MS` + `OLDEST_DISPATCH_SHARE`
starts.
- `execution-projection-agent-queue.test.ts` (embedded Postgres, real
SQL): line is subtask (grandchild of a person's task), parent resume,
newer person assignment, then work on an agent-created tree. Master
gives the person assignment first and the tree runs last.
- Negative and guard tests (green on both): work on an agent-created
tree stays in the work tier behind a newer person assignment; system
recovery on a person's tree stays in the recovery tier; newer tree work
does not pass an older equally ranked person task; recovery still gets
its share while only tree work waits.
- Query plan on embedded Postgres with 300,000 issues and 200 queued
tasks six levels deep: every step is an index scan on
`issues_company_id_uq`, 1.5 ms execution.
- `pnpm verify` exited 2 (a doc is a global CI input). Focused run:
`queued-run-dispatch-order`, `queued-run-line`,
`execution-projection-agent-queue`, `fresh-continuation-retry-delivery`,
`assigned-work-stalled-queued-run`, `automatic-run-start-gate`,
`automatic-run-brake-recovery-passes`,
`heartbeat-stale-queue-invalidation`: 8 files, 173 tests passed. `tsc
--noEmit` for the server passed.

## Risks

- Behavior shift: in a company where people create most top-level tasks,
most issue runs are now person tier. The "work" tier then holds
agent-started trees, routines and runs without a task. Recovery keeps
its share because only runs a person asked for hold it.
- An in-progress run on a person's tree now ranks ahead of a new person
task that is not in progress (readiness rank). The oldest-run share
bounds that wait at about 60 minutes plus a few starts, the same bound
agent work had before.
- A tree deeper than 32 levels above a queued task is treated as not a
person tree. It falls back to the work tier and the aging share.
- The extra query runs only when an agent has more than one queued run,
and only for queued tasks without `createdByUserId`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 10:35:42 -06:00
Nicky LeachandClaude Opus 5.5 db73178bcf fix(heartbeat): keep saved sessions across config edits the provider thread cannot see (#21862)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A heartbeat run resumes the agent's saved task session, which holds
the provider thread and its history.
> - Before resume, Paperclip compares a fingerprint of the effective run
configuration with the fingerprint that the session saved. A difference
discards the thread.
> - The fingerprint hashed the agent config revision row (id, changed
keys, time). Every saved agent edit therefore discarded the thread, also
an edit that set a value back to what it was.
> - The fingerprint also hashed settings that never reach the thread or
the native execution input, for example turn timeouts, wake scheduling,
and the environment row's update time.
> - This pull request removes those inputs from the fingerprint,
classifies every input with its reason, and adds backstop tests so that
a new input cannot join or leave the fingerprint without a decision.
> - The benefit is that agents keep their thread history across edits
that cannot change what the thread saw or may do, while real changes
still start a new thread.

## Linked Issues or Issue Description

Related: #9977 (semantic task-session fingerprints, open) and #8797
(added the effective run config fingerprint).

**What happened?**
An operator changed an agent's warm idle window from 60000 to unset, and
then back to 60000, between two turns on the same issue. The next run
logged `Skipping saved session resume for task "<issue>" because
effective run configuration changed: adapter.` Only the `adapter`
category changed, because the new config revision row was in it. The
adapter config itself was identical. The run started a new provider
thread. The environment lease was reused, so the run stopped the
retained remote task before it started a new one. That stop raced the
task's exit, and the run failed with `Plugin Runner termination is
unconfirmed; reconcile the saved task`. An automatic retry then
succeeded on a fresh environment.

**Expected behavior**
An edit that cannot change what the provider thread saw or may do, and
cannot change the native execution input, keeps the saved session. An
edit to the model, provider, active permission mode, lifecycle mode or
warm idle window, instructions, skills, environment, secrets, or another
thread input still starts a new session.

**Steps to reproduce**
1. Run an agent turn on an issue so that the task session saves its
fingerprint.
2. Save an agent config edit that does not change a thread input, for
example set `adapterConfig.idleTimeoutMs` to another value and back, or
rename the agent.
3. Wake the agent on the same issue. On master the run log shows
"effective run configuration changed: adapter" and the run starts a new
thread.

## What Changed

- New `server/src/services/heartbeat/session-fingerprint-inputs.ts`. It
classifies each fingerprint input as kept or excluded, with the reason:
  - agent config revision fields (`CONFIG_REVISION_FIELDS`);
  - `paperclip_runner` adapter config keys;
  - top-level environment driver config keys;
  - agent runtime config keys;
  - selected-environment and environment-selection fields.
- The session fingerprint no longer hashes these inputs:
  - the config revision row (id, changed keys, time);
- for `paperclip_runner`: `timeoutSec`, `graceSec`, the permission keys
of providers that are not selected, and `search` when the provider is
not Codex;
- `runtimeConfig.heartbeat` (wake scheduling, concurrency, daily caps,
max-turn continuation);
- the selected environment's `configRevisionAt` (row update time) and
the `selectionSource`;
- environment config `archiveOnRelease`, `streamRunLogs`, and
`timeoutMs` (sandbox RPC timeout).
- Unknown adapter and runtime config keys are still hashed. A missing
classification can only keep the old reset behavior; it cannot skip a
reset.
- Lifecycle inputs stay in the fingerprint: `lifecycleMode`,
`idleTimeoutMs`, and the environment's `runnerLifecycleMode`,
`runnerIdleTimeoutMs`, and `reuseLease`. They become
`session.lifecyclePolicy`, which the native session digest compares
before a follow-up turn attaches to its warm Runner. A kept session with
a different policy would be refused
(`warm_recovery_refused=execution_mismatch`).
- Session preparation accepts a saved fingerprint built with the
previous composition (`legacyComposition`) for the same inputs. The
upgrade therefore does not discard every saved session once. The run
then saves the new fingerprint.
- Backstop tests in `session-fingerprint-inputs.test.ts`:
- every agent revision field, runner config schema field, provider
permission key, and environment driver config key has a classification;
- the shape of each hashed category is pinned, so a new field cannot
reach the fingerprint without a decision;
- every runner key classified as excluded leaves the native provider
input and the lifecycle policy unchanged (this test fails if
`idleTimeoutMs` is reclassified as excluded);
- behavior tests for each exclusion and for the thread and lifecycle
inputs that must still reset;
- a pinned fingerprint proves that the legacy composition reproduces the
master fingerprint exactly.
- `ai-connection-session.test.ts`: two tests used a rename or a
`heartbeat` edit as the "unrelated change". They now use a model change
or an unclassified runtime setting, and two new tests pin that a rename
and a scheduling change keep the session.
- `doc/heartbeat-internals.md` (Config Freshness) describes the rule and
the backstop. The PR adds no code comments, as `CONTRIBUTING.md`
requires; the reasons live in the classification tables and the doc.

## Verification

- Red/green: on master, the exclusion cases, the category-shape pin, and
the legacy acceptance case fail. With the change, all pass.
- Backstop check: with `idleTimeoutMs` temporarily reclassified as
excluded, the native-input test and the idle-window reset test fail.
- Legacy bridge: I computed the fingerprint for one runner fixture and
one `codex_local` fixture with master's `workspaces.ts` and with
`legacyComposition: true` on this branch. The output files were
byte-identical, category fingerprints included. The pinned test uses
those values.
- `cd server && npx vitest run
src/services/heartbeat/session-fingerprint-inputs.test.ts
src/__tests__/ai-connection-session.test.ts
src/__tests__/heartbeat-workspace-session.test.ts
src/__tests__/heartbeat-execution-configuration.test.ts
src/__tests__/ai-connection-router.test.ts
src/__tests__/native-session-resumption.test.ts
src/__tests__/environment-service.test.ts`: 7 files, 397 tests, all
pass.
- `npx tsc --noEmit -p server`: no errors.
- `pnpm verify` exited 2 (broader coverage needed), so the focused
checks above and required CI give the coverage.

## Risks

- Behavior change: an agent rename, role, title, icon, appearance,
reports-to, capabilities, budget, or metadata edit no longer starts a
new thread. Agent tools read these fields live at call time
(`runner-semantic-authority.ts`, `paperclip-runner-tool-authority.ts`),
so the thread never holds a frozen copy.
- Behavior change: wake-scheduling and turn-timeout edits no longer
start a new thread. Neither value is part of the native execution input,
so warm Runner attachment is unaffected.
- A warm idle window edit still starts a new thread. Setting it and
setting it back between turns no longer does, because the revision row
is not hashed.
- The legacy bridge computes extra candidate fingerprints only when the
saved fingerprint differs. It accepts a legacy fingerprint only for the
same current inputs; it never widens what a reset protects.
- During the transition, a real change on a session that still holds a
legacy fingerprint can list extra categories in the reset reason. The
reset decision is correct.
- No schema or API change. The fingerprint version stays 1.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1116.0-canary.9
2026-11-16 08:24:47 -08:00
DottaandClaude Opus 5.5 20b0d4e5a5 fix(wake-queue): reopen a finished task for a person's comment merged into a system wake (#21914)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The wake queue holds new input for a task while an agent run is
active, and releases it when the run ends
> - A person's comment on a task always reopens the task when the run
that closed it did not receive the comment
> - A restart test found a person's comment that was posted while the
run was finishing. The run marked the task done and no later run
started. The comment was lost
> - The comment joined a deferred child-completion wake that a system
actor requested. The release step only looked at the wake requester,
which was "system", so it did not see the person
> - This pull request makes the release step look at who wrote the
undelivered comments in the wake
> - The benefit is that a person's comment is never dropped because it
shares a queued wake with system work

## Linked Issues or Issue Description

**What happened?**

A person posted a comment on a task while the assigned agent's run was
active. The run then marked the task done. The task stayed done, and no
run received the comment. The deferred wake that held the comment was
cancelled with "Deferred execution wake no longer applies to a terminal
task".

**Expected behavior**

The task reopens and a new run receives the person's comment.

**Steps to reproduce**

1. Start a run for a task.
2. While the run is active, let the system defer a wake for the same
task (for example, a child task completes).
3. Post a comment as a person. The comment wake merges into the deferred
system wake.
4. Let the run mark the task done.
5. The task stays done. No run receives the comment.

**Paperclip version or commit**

`master` at b4fa1c7fb1 (first seen at c5218f5ec4).

**Relevant logs or output**

The merged deferred wake had `reason = issue_execution_deferred` and
`requestedByActorType = system`. Its payload held the person's comment
id. The coalesced comment receipt had `requestedByActorType = user`.
Release cancelled the merged wake as terminal-task work.

## What Changed

- `getCommentSelfAuthorship` in the wake-queue Postgres adapter also
returns `includesPersonComment`. This value is true when one of the
wake's undelivered comments was written by a person and not by an agent
run.
- `promoteDeferredWake` reopens a done or cancelled task when
`includesPersonComment` is true, also when the wake requester is the
system.
- The delivered-comment filter is not changed. A comment that a run
already received does not reopen the task.
- New embedded-Postgres tests in
`heartbeat-comment-wake-batching.test.ts`:
- A person's comment merges into a deferred child-completion wake during
a run. The run closes the task. The task reopens and the next run
receives the comment. This test is red on `master` with the exact
production cancel reason.
- Negative: the person's comment in the merged wake is the comment that
the closing run received as its input. The task stays done and no new
run starts.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-comment-wake-batching.test.ts
src/modules/wake-queue` passes (15 files, 391 tests).
- `cd server && npx vitest run
src/__tests__/heartbeat-held-comment-wake.test.ts
src/__tests__/heartbeat-issue-rewake-throttle.test.ts
src/__tests__/heartbeat-answer-behind-settling-run.test.ts` passes (57
tests).
- `cd server && npx tsc --noEmit` passes.
- Red check: with only the source change reverted, the new positive test
fails with `expected 'Deferred execution wake no longer app…' to be
null`.
- `pnpm verify` exits 2 (an opaque setup dependency). Required CI
supplies broad coverage.

## Risks

- Low risk. The change only adds a reopen reason for a done or cancelled
task. The comment must be live, undelivered, not written by the
finishing run, and written by a person.
- A comment from a local CLI agent can carry a user id. The check
excludes comments that have a creating run or an agent author, so these
comments do not count as person comments.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 10:10:09 -06:00
DottaandClaude Opus 5.5 77ddf260bf fix(runner-e2e): keep the supervisor off the heartbeat graph and ship the plugin SDK build (#21860)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The paid Runner Full-Stack E2E workflow proves chat features against
a real source server
> - Since master f7a3497e9d, every paid cell fails before any test runs:
the source server cannot find `@paperclipai/plugin-sdk/dist/index.js`
> - The supervisor builds the plugin SDK before it starts Paperclip, but
its own static imports now load the SDK first
> - This pull request removes the heavy import edge, adds a regression
test for that class, and builds and checks the SDK once per campaign
> - The benefit is that paid campaigns run again, and a missing
workspace build fails once in the build job instead of in 199 paid cells

## Linked Issues or Issue Description

Refs #21791

**What happened?**
Chat-eval cohort 17 (master f7a3497e9d) failed in all 199 cells. Each
cell exited in the "Run paid cell" step. The private Playwright log
shows `ERR_MODULE_NOT_FOUND` for
`server/node_modules/@paperclipai/plugin-sdk/dist/index.js`, imported
from `server/src/services/reusable-sandbox-resume-error.ts`, and then
`Process from config.webServer was not able to start`. Cohort 16 (master
819f61f1fd) passed 169/199 with the same workflow steps.

**Root cause**
#21791 (df9914dbd9) added `import { NATIVE_RETAINED_OWNER_WAIT } from
"./native-controller-recovery.js"` to `native-runner-checkpoint.ts`.
`tests/runner-e2e/server.ts` statically imports
`controller-recovery-observer.ts`, which imports the checkpoint reader.
The new edge connects it to the full heartbeat graph:
`native-runner-checkpoint -> native-controller-recovery ->
native-finalization-notifications -> external-agents -> muse ->
heartbeat -> heartbeat/run-completion -> reusable-sandbox-resume-error
-> @paperclipai/plugin-sdk`. The plugin SDK exports only its generated
`dist`. The supervisor builds that `dist` in
`prepareSourceServerDependencies`, but ES static imports load before
that call. The workflow installs with `--ignore-scripts` and never built
the SDK, so the supervisor crashes at load. #21554 is not involved: it
changes only release workflows.

**Expected behavior**
The supervisor loads without generated SDK output. The campaign build
job builds and checks the SDK once, and each cell restores it.

**Steps to reproduce**
1. Remove `packages/plugins/sdk/dist`.
2. Run `node cli/node_modules/tsx/dist/cli.mjs
tests/runner-e2e/server.ts`.
3. Before this change: `ERR_MODULE_NOT_FOUND` for the SDK. After this
change: the normal `PAPERCLIP_RUNNER_E2E_SERVER_LOG is required` error,
which shows that all static imports loaded.

## What Changed

- Moved `NATIVE_RETAINED_OWNER_WAIT` and
`NATIVE_AGENT_DIRECTORY_BUSY_WAIT` to a new leaf module
`native-recovery-wait-codes.ts`. `native-controller-recovery.ts`
re-exports them, so other importers do not change.
`native-runner-checkpoint.ts` imports the leaf. This also removes a
checkpoint/controller-recovery import cycle.
- Added `tests/runner-e2e/server-startup-import-graph.test.ts`. It uses
esbuild to walk the static import graph of `tests/runner-e2e/server.ts`
and fails with the full chain if it reaches `@paperclipai/plugin-sdk`.
- `runner-full-stack-e2e.yml`: the "Build reusable runner campaign
artifacts" job runs `scripts/ensure-plugin-build-deps.mjs` (the same
certified helper the cell uses), checks that `shared` and `plugin-sdk`
dist and their completion markers exist with a clear `::error::`, and
ships both dist trees in the campaign bundle. The cell restore step
checks both entry points with a clear `::error::`. The helper's
fingerprint receipt lets the cell reuse the shipped output instead of a
rebuild.
- `workflow-security.test.ts` pins the build, packaging and guard.

## Verification

- Red/green: `npx vitest run --config tests/runner-e2e/vitest.config.ts
server-startup-import-graph` fails with the old import and passes with
the fix.
- Local reproduction above: fails before, loads after.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
workflow-security server-startup-import-graph native-completion`: 137
passed.
- `node --test tests/runner-e2e/native-completion-git-source.test.mjs`:
36 passed.
- Server `tsc --noEmit`: clean.
- Paid proof of the source fix: one cell
(`chat-features.runner-opencode.local.access.private-resource`),
dispatched with `--ref master -f
target_branch=fix/runner-e2e-plugin-sdk-dist`:
https://github.com/paperclipai/paperclip/actions/runs/43804619166 . That
run uses the master workflow and this branch's source, so it proves the
import-graph fix alone restores the cells.
- The workflow changes (SDK build, bundle and guards) cannot run in a
paid campaign before merge, because paid runs use the default-branch
workflow. `workflow-security.test.ts` pins them. A 1-case paid check
from master follows the merge.
- The failed "Authorize paid campaign" check on this commit is from a
dispatch with a branch `--ref`. The default-branch gate rejected it as
designed. It is not a PR-event check.

## Risks

- Low risk. The constants keep their values and are re-exported from the
old module.
- The campaign bundle grows by the `shared` and `plugin-sdk` dist trees
(small). If the fingerprints do not match in a cell, the helper
rebuilds, as before.

## Model Used

- Claude, `claude-opus-5-5`, via Claude Code with tool use (shell, file
edit, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 10:08:11 -06:00
DottaandClaude Opus 5.5 9de343b3fe perf(recovery): keep a repeatedly deferring task's saved row; stop agent settlement from re-checking its backlog (#21905)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The task recovery queues check assigned tasks for work that has no
live run and no queued wake. Since #21872 they load only changed rows,
and they save pending deferrals so a replaced server keeps their
deadlines
> - Two costs remained. A running task's check defers to now + 30 s, and
each new deadline rewrote its saved deferral row: about two writes a
minute per running task per queue
> - Also, two writers that settle an agent after its runs end set the
agent's status and `updated_at` but not `last_heartbeat_at`. The
changed-rows agent rule then reads the write as a pause or resume and
re-checks all of that agent's tasks for about 5 minutes
> - This pull request keeps a repeatedly deferring task's saved row
until its deadline is well past, and makes both writers record a
heartbeat as `finalizeAgentStatus` does
> - The benefit is near-zero deferral writes for running tasks and no
whole-backlog re-check after an abandoned or stranded run settles an
agent

## Linked Issues or Issue Description

Refs #21872, #21861

**What happened?**
1. In `recovery/work.ts`, a deferring check upserted its
`issue_recovery_deferrals` row whenever the deadline changed. A deferral
to now + 30 s changes on every check, so a task deferring for 20 minutes
wrote its row 41 times. At 1,000 concurrent agents that is roughly 30 to
70 writes a second.
2. `execution-control-reconciliation.ts` (abandoned run past its
execution-control deadline) and `heartbeat/agent-status.ts`
`reconcileStrandedAgentStatuses` (agent left "running" with no live run)
set the agent to idle with `updated_at` but without `last_heartbeat_at`.
The agent clause of `changedTaskCondition` matches an agent that is not
running and whose `updated_at` is newer than its last heartbeat. So each
such write made the backstop re-check every task of that agent for about
5 minutes.

**Expected behavior**
A task that keeps deferring writes its row a small fixed number of
times. Settling an agent after its runs end does not re-check the
agent's other tasks; the run's own task is still checked promptly.

**Steps to reproduce**
1. Defer one task's check to now + 30 s on every check for 20 simulated
minutes and count inserts into `issue_recovery_deferrals`: 41.
2. Seed an agent left "running" with 140 settled reviews, settle it with
`reconcileAbandonedExecutionControl` or
`reconcileStrandedAgentStatuses`, and count stranded checks of those
reviews over the next backstop: 280 per agent.

**Paperclip version or commit**
`master` at b4fa1c7fb1.

## What Changed

- `recovery/work.ts`: a deferring check skips the upsert while this
server knows a saved row for the task whose deadline is less than
`DEFERRAL_REFRESH_MS` (5 minutes) past. It refreshes the row after that.
The refresh comes before the 10-minute stale prune margin, so a
still-deferring task's row never looks stale. A boot re-checks every
task with a row whatever its deadline, so the exact deadline only
matters to the conditional deletes and the stale prune, which still
match the value this server read or wrote. A cleared row, a failed write
or a reload still drop the server's memory of the row, so the next
deferral writes it again.
- `execution-control-reconciliation.ts` and `heartbeat/agent-status.ts`:
settling an agent after its runs end sets `lastHeartbeatAt` with
`updatedAt`, as `finalizeAgentStatus` already does, through a new
`settledHeartbeat(now)`. It records the settlement time only when the
agent already has a heartbeat; an agent with none keeps none, so the
first real heartbeat stays first (the existing "repairs a shutdown flag
without inventing a heartbeat" test still passes).
- `recovery/changed-tasks.ts`: the comment now states the contract:
every write that settles an agent's status after its runs end sets
`last_heartbeat_at`.
- Tests: a write-rate census in
`issue-recovery-saved-deferrals.test.ts`, and an agent-settle census and
guard in `issue-recovery-company-reload-census.test.ts`.

Why fix the writers, not the rule: the agent rule's discriminator
already is `last_heartbeat_at` against `updated_at` ("a run's finish
records a heartbeat"). Pauses, resumes, holds and configuration writes
leave `last_heartbeat_at` alone; run settlement records one. No other
column tells a settle-to-idle apart from a resume to idle, because both
end with status idle and a new `updated_at`. Since #21872, lifecycle
transitions also send their own agent hint, but configuration writes to
an idle agent still rely on this rule (the #21861 guard). So the sound
fix is for the two run-settling writers to follow the same contract as
`finalizeAgentStatus`. Recording the heartbeat at settlement has the
same effect on agent timers as a normal run finish.

## Verification

Red/green:

| Census | Before (master code) | After |
| --- | --- | --- |
| One task deferring every 30 s for 20 minutes: deferral writes | 41 (41
checks) | 4 (41 checks) |
| Same task: saved deadline behind now after 20 minutes | 0 (rewritten
each check) | 3 min (refreshed; the test fails above 6 min, and fails
with the refresh removed: 1 write, 19.5 min) |
| Two agents (140 settled reviews each) settled by an abandoned-run
reconcile and a stranded-status reconcile: backlog checks over the next
backstops | 560 | 0 |
| Same census: statements | 12,570 | 2,507 |

Guards:
- The abandoned run's task is checked within 1 minute of the reconcile
(passes before and after).
- The three deferral tests from #21872 still pass: replacement mid-wait
checks at the original deadline, the two-server interleaving keeps
server B's deadline, and the stale row of a task that left the queue is
removed.

Focused suites, each file run alone in the foreground:
`issue-recovery-company-reload-census.test.ts` 8 passed (38 s),
`issue-recovery-saved-deferrals.test.ts` 6 passed (14 s),
`heartbeat/agent-status.test.ts` and
`execution-control-reconciliation.test.ts` 60 passed,
`issue-recovery-idle-backstop.test.ts` 4 passed. An 18-file run earlier:
516 passed; the 4 failures were 1 test that this PR then fixed
(heartbeat for agents with none) and 3 in
`issue-recovery-idle-backstop.test.ts`, which pass alone (see Risks).
Server `tsc --noEmit`: passed. `pnpm verify`: not run; required CI
covers the broad suites.

## Risks

- A task that keeps deferring may have a saved deadline up to 5 minutes
old. Only the stale prune reads it, and it deletes a row only after the
task left the queue and the deadline is 10 minutes past, so a
still-deferring task's row is refreshed first.
- If another server deletes the row (its check stopped deferring and it
matched the same deadline) while this server keeps deferring, this
server rewrites the row at its next refresh (at most about 5 minutes)
instead of at its next check. Within that window a crash of this server
would leave the task to its successor's next changed-rows or full pass.
This needs the two servers to disagree about the same task, which only
happens through facts held in one server's memory.
- `last_heartbeat_at` now also moves when an abandoned or stranded run
settles the agent. Agent timers then count from that time, as they do
after a normal run finish.
- An agent that never recorded a heartbeat still has its whole backlog
re-checked once when such a writer settles it. This is rare: that agent
has not finished a run yet.
- Not from this PR: `issue-recovery-idle-backstop.test.ts` > "picks up a
finished run, a user message, a raised budget and a resumed agent within
a minute" hung once and failed 3 tests when run with other recovery
files in one process. It passes alone. Its waits have no time limits,
and its fixture agents can take on-demand wakes, which is the hang
pattern that #21867 removed from the agent-row census. It needs the same
treatment in a separate change.
- No schema change. Nothing on the run-start path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 09:51:09 -06:00
github-actions[bot]andlockfile-bot ac22f7f42d chore(lockfile): refresh pnpm lockfiles (#21915)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.
- Resync the SDK public source proof's lock pins from the refreshed
lock.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when every added
package version is at least 3 days old on npm. Otherwise the PR waits
for a maintainer, and the daily refresh tries again. The bot merges only
the exact commit this workflow generated from manifests already reviewed
on master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
2026-11-16 15:44:28 +00:00
DottaandClaude Opus 5.5 5561a4068c fix(recovery): keep a running run visible when a dropped continuation is newer (#21907)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task shows a Retry banner when its latest run failed and nothing
else continues the task
> - The banner reads "the latest run" by creation time only
> - The queue cancels a queued system continuation before it starts when
the task no longer needs it. That run is newer than the run that is
really working
> - So a task showed "failed" with Retry for about 79 seconds while its
run was working
> - The stranded-work sweep also queued one of those extra
continuations, because it did not see a native status wake that was
still waiting for dispatch
> - This pull request ignores never-started, superseded continuations in
the banner, and makes the sweep see a pending native status wake
> - The benefit is no false Retry offer while an agent works, and no
duplicate continuation after a slow finalization

## Linked Issues or Issue Description

**What happened?**

In a restart-chaos wave (6 rolling deploys, 207 tasks), one task showed
state `failed` with Retry for 75–79 seconds while its run was running.
Two queued system continuations for that task were cancelled before they
started with `recovery_no_longer_needed`. That cancel is correct. But
the newest cancelled run became the "latest run" in the interruption
projection. The projection then fell through to `{ state: "failed",
canRetry: true }`.

The first of the two extra continuations came from a double queue. The
finishing run asked for its own continuation through a native status
wake (`agent_tool_call_loop`). Its finalization did not apply at once
("native result persisted but finalization did not apply; the
reconciliation loop will retry"), so the dispatch of that wake waited
about 17 seconds. In that window, the stranded-work sweep saw an
`in_progress` task with a succeeded run and no live path, and queued its
own continuation (`issue.productive_terminal_continuation_recovery`).
The sweep's live-path check counted only runs and
`deferred_issue_execution` wakes. It did not count a native status wake
that was still `queued` or `claimed` without a run. When the native
status wake was dispatched later, the queue correctly dropped it. The
dropped wake also carried the better instruction (the tool-loop
guidance).

**Expected behavior**

A task with a running run shows "working" with no Retry. One finished
run produces one continuation.

**Steps to reproduce**

1. Give a task a running run R1.
2. Add a newer run R2 for the task with status `cancelled`, no
`startedAt`, and error code `recovery_no_longer_needed`.
3. Read the task interruption. Before this change: `failed`, `canRetry:
true`, observed run R2.

For the sweep: give an `in_progress` task a succeeded native run and a
`native-status-committer` wake in `queued` or `claimed` with no run. Run
`reconcileStrandedAssignedIssues`. Before this change it queues a second
continuation.

**Paperclip version or commit**

master `c5218f5ec4` (wave), reproduced on `b4fa1c7fb1`.

## What Changed

- `task-interruption-presentation.ts`: the latest-run candidates skip a
run that is `cancelled`, never started (`startedAt` is null), and has
error code `recovery_no_longer_needed` or `issue_closed_before_start`.
The filter is in each per-arm `limit 1` index scan, so the query shape
and cost stay the same.
- `recovery/service.ts`: `hasActiveExecutionPath` also counts a
`native-status-committer` wake in `queued` or `claimed` with no run for
the issue. The native status wake dispatcher owns that wake, so the
sweep does not queue a second continuation and does not escalate the
task while it waits.
- Tests (red on master, green here):
- `task-interruption-query.test.ts`: a running R1 plus a newer
never-started R2 cancelled with each code shows `working`, `canRetry:
false`, observed R1. Negatives keep their behavior: a cancelled run that
had started, and a run a person cancelled, still show `failed` with
Retry.
- `heartbeat-process-recovery.test.ts`: a succeeded native run with its
own status wake still `queued` or `claimed` gets no second continuation
and no escalation.

## Verification

- `pnpm verify` exited 2 (opaque setup dependency), so I ran focused
suites:
- `server/src/__tests__/task-interruption-query.test.ts`,
`task-interruption-automatic-path.test.ts`,
`issue-confirm-run-stopped-routes.test.ts`: 83/83 pass. This includes
the query-plan test that bounds `heartbeat_runs` rows read.
- `server/src/__tests__/heartbeat-process-recovery.test.ts`,
`server/src/services/recovery/*.test.ts`, the interruption suites and
`task-interruption-presentation.test.ts` together: 18 files, 871/871
pass.
- Red/green: the new interruption tests fail on master with `state:
"failed", canRetry: true` and the dropped run as observed run. The new
sweep tests fail on master with `continuationRequeued: 1`.
- `tsc --noEmit` for `server` passes.

## Risks

- Low. The banner change only hides runs that never started and were
dropped as superseded. Person-cancelled runs, started runs, and other
cancel codes behave as before.
- The live-path change makes the sweep wait while a native status wake
is pending. The dispatcher always settles such a wake: it dispatches it,
or marks it skipped. A throttled wake stays queued only until its
throttle time, so the sweep waits at most that long.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 09:43:36 -06:00
DottaandPaperclip 698c371b1b fix(ui): keep new Recents tasks ready to open from cache (#21904)
Track warmed tasks per query client, company, and account so tasks added to the top Recents rows receive the existing idle preload. Preserve the batch limit, cache ownership, and retry-on-revisit behavior.

Add unit and browser regressions for later additions, held API responses, editor retention, and message position. Extend controlled performance coverage and document timing results.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
canary/v2026.1116.0-canary.8
2026-11-16 09:43:29 -06:00
DottaandPaperclip 991f773d7f fix(ui): align recent task status icons and move fold controls right (#21859)
## Thinking Path

> - Paperclip helps people manage AI agents for work.
> - The sidebar shows recent tasks and their subtasks.
> - A left fold control moves parent tasks away from the status column.
> - Status icons must stay in one column so people can scan tasks.
> - This pull request moves the fold control to the right.
> - The child guide now starts below the parent status icon.

## Linked Issues or Issue Description

Follow-up to #21649. Related sidebar alignment work: #20926. Related
experimental Recents work: #12314. This change corrects the status
column in the shipped subtask group.

**What happened?**

Parent tasks with subtasks moved 20 pixels to the right. Their status
icons did not align with tasks that had no subtasks.

**Expected behavior**

All top-level task status icons align. The child guide is centered below
the parent status icon. The fold control is on the right.

**Steps to reproduce**

1. Open a sidebar with a recent parent task and a task that has no
subtasks.
2. Compare their status icons.
3. Expand the parent and compare the child guide with its status icon.

## What Changed

- Remove the left offset from parent task rows.
- Put the fold control in a separate right slot.
- Center the child guide below the parent status slot.
- Add browser checks for alignment, narrow rows, unread parents, both
themes, and separate Classic menu and fold controls.

## Verification

- The browser regression failed before the fix: the status icons
differed by 20 pixels.
- Final browser run: 44 checks passed. It covers 20 states in both
themes, folding, paging, unread state, light and dark alignment, and the
Classic action menu.
- The focused component and request-performance suites passed: 46 tests.
- Final source is based on master `29903bed59b1`. UI typecheck and the
focused browser-test typecheck passed. The published head is
`b8287aec1f55aa182b44622d72b20fe7ed7ce0e6`. Its tree matches the tested
local tree `958e8d901ed1fec9ae17c164271eda4d25edca45`.
- `pnpm verify --base origin/master` returned exit 2. It requires broad
CI because the browser suite changed. Focused checks pass. [CI
43803650911](https://github.com/paperclipai/paperclip/actions/runs/43803650911)
passes typecheck, build, all eight E2E shards, policy, Runner checks,
all nine serialized server suites, all twelve general server shards, and
[final
verification](https://github.com/paperclipai/paperclip/actions/runs/43803650911/job/133095032900).
All 62 check runs are clean: 57 success, 3 skipped, and 2 neutral. No
checks are pending or failed. The legacy Snyk status also passes.
- [Greptile
review](https://github.com/paperclipai/paperclip/pull/21859#issuecomment-6907272673)
is 5/5 on head `b8287aec1f55aa182b44622d72b20fe7ed7ce0e6`. The check
passed. There are no review threads or actionable findings. The separate
source review also reports 5/5.
- Screenshots use production components with HTTP and session fixtures
in Chromium on Linux. No deployed-instance or image-baseline comparison
is claimed.
- No database or API behavior changes.

## Risks

Low risk. The change adjusts layout in the shared task group. The fold
button remains separate from the task link and the Classic action menu.
Exact unread counts and request limits keep their current behavior.

## Model Used

OpenAI Codex, GPT-6. Exact deployment model ID was not supplied by the
runtime.

## Checklist

- [x] I have searched GitHub for duplicate or related PRs and linked
them above

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work (this is a sidebar layout bug fix)
- [x] I have linked public issues or described the issue in this PR
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and has no internal task ID
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
command or subsystem behavior change)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-16 09:22:27 -06:00
DottaandClaude Opus 5.5 b4fa1c7fb1 perf(recovery): load changed rows, not every task, for company hints, boots and reconnects (#21872)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The task recovery queues (stranded assignments, assigned-work
invariant, resolved dependencies, stale issue locks) check tasks to find
work that has no live run and no queued wake
> - A plain company hint, or an unscoped `"*"` reload, made each queue
load and check every eligible task of the company, at about 22 read-only
statements per task
> - Many writers send that hint (issue creation, documents, budgets,
work intents, agent status changes), every server boot reloads every
task, and the notification connection publishes a `"*"` reload each time
it opens or reconnects
> - On staging this shows as bursts of about 1,400 to 1,550 checks for
one company roughly every 9 minutes, with almost no real work
> - #21861 removed one writer. This pull request fixes the class: a
company hint, a boot and a reconnect now load only changed rows since
the queue last looked, and the hourly full pass is saved so a replaced
server keeps its schedule
> - The benefit is that settled tasks are checked by the hourly full
pass and by real changes, not by every write, deploy or reconnect

## Linked Issues or Issue Description

Refs #21861, #21831, #21867

**What happened?**
A company hint `{ companyId }` on a task queue loaded every eligible
task of the company with no changed-rows filter. The same happened for a
`"*"` reload. Three sources send these all the time:
1. Each server boot starts every task queue with a `"*"` reload.
2. `reconcileDatabaseWorkNotifications` publishes a `"*"` reload to
every queue each time the deployment notification connection opens or
reconnects. The connection opens at boot and again whenever a native
execution starts after an idle period.
3. Many writers send a bare company hint: issue creation, document lock,
unlock, restore and delete, budget pause and resume, work intent
promotion, agent status changes, watchdog writes, and others.

In a staging-shaped fixture each server replacement caused two full
reloads (boot and connection open).

**Expected behavior**
A company hint, a boot or a reconnect re-checks the tasks whose rows
changed. A change that makes tasks eligible without writing any of their
rows (a company or project resume, a tree hold release) still re-checks
every task of the company. An agent status change re-checks that agent's
tasks. The hourly full pass stays the safety net, also across server
replacements.

**Steps to reproduce**
1. Seed one agent with settled in-review tasks and a live native run.
2. Start the recovery workers with
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:1`.
3. Restart the workers, call `reconcileDatabaseWorkNotifications(db)`,
or create an issue.
4. See one `check` event per settled task with `wake: "reload"` or
`wake: "company"`.

**Paperclip version or commit**
`master` at 278660149c.

## What Changed

- `recovery/work.ts`: the full-pass bookkeeping now decides a row filter
per load.
- A plain company hint and the changed-rows backstop load the rows
changed since the queue last looked at the company
(`changedTaskCondition`, minus the existing 5-minute margin). They load
every task only when no full pass is recent.
- A boot and a reconnect use a new reload scope. One statement loads,
per company, every task when that company has no saved full pass within
the hour, and otherwise the rows changed since that saved pass (or since
this server's last reload).
- An agent scope resolves the tasks the agent is assigned, currently
reviews, or holds a run lock on, and checks only those.
- The graph fallback for a graph too large to list now sends the
explicit `everyTask` hint, so it keeps its old meaning.
- When scheduling is suppressed, the queue forgets its passes and
ignores older saved ones, so the first pass after suppression checks
every task (entries were dropped while suppressed).
- New table `issue_recovery_deferrals` (queue, issue, company, due_at),
in the same migration. A row exists while a task's check has deferred
itself (grace period, back-off, live owner lease). A boot or reconnect
reload also loads these tasks and checks them again, which rebuilds each
deadline. Deferrals cannot be rebuilt from changed rows: several defer
with no limit while none of the task's rows change (see the table
below). The store is built so that any race or lost write costs at most
one extra check, never a missed one:
- A deferring check upserts the row with its deadline when the deadline
differs from the one this server last read or wrote (one statement; none
when unchanged). A failed write keeps the full pass unsaved, so a
successor's full pass rebuilds the deadline, and drops the server's
memory of the row.
- A check that no longer defers deletes the row only if `due_at` still
equals the deadline this server read or wrote (one statement, only when
it knew of a row). A row another server rewrote since stays.
- A reload deletes the row of a task it did not load (the task left the
queue, for any reason) when, in the same statement, `due_at` still
equals the value the reload read and either the task is closed or that
deadline passed more than 10 minutes ago. This bounds the table without
listing each queue's eligibility rules.
  - A check that does not defer and knows of no row adds no statement.
- Two overlapping servers both keep the row current; a crash leaves it
in place.
- New table `issue_recovery_full_passes` (company, queue, started_at),
migration `0488`. A full pass writes it only after every task it loaded
has been checked (a check that fails keeps it unsaved), so a server
replaced mid-pass leaves no record and its replacement runs the full
pass again; a server with no memory of its own reads it. This keeps the
hourly full pass on one schedule for replaced and overlapping servers.
Reads and writes are one statement each and only happen when a full pass
is due or done. A failed read means a full pass; a failed write means
one extra full pass later.
- `issue-recovery-notifications.ts`: new scope fields `everyTask`,
`agentId` and `reload`, `notifyAgentRecovery`, `resolveAgentIssues`, and
the reload hint.
- `run-dispatch-notifications.ts`: `RunDispatchScope.everyTask` and
`notifyAgentStatusDispatch`. The agent lifecycle status-change effect
uses it instead of a company hint.
- `work-signals.ts`: `setDatabaseWorkReconnectHint` lets a topic declare
the hint it gets on reconnect instead of an unscoped reload. The task
queues declare the reload hint. Other topics keep the unscoped reload.
- `scoped-deadline-work.ts`: optional `initialHint` for the first pass
after registration. The task queues start with the reload hint.
- Writers that make tasks eligible without changing their rows now send
`everyTask`: company resume (`companies.ts`), project and company budget
resume (`budgets.ts`), tree hold resume and hold creation
(`issue-tree-control.ts`), conversation reset that releases a tree hold
(`agent-conversations.ts`), work intent promotion for `pause_lifted` and
`tree_hold_released`, task reparenting and task deletion (`issues.ts`),
company import relations (`company-portability.ts`), and company-wide
chat milestones.
- Writers with a known task now send a task or graph hint instead of a
company hint: document lock, unlock, restore and delete
(`documents.ts`), blocker edits and attachment delete (`issues.ts`),
watchdog writes (`task-watchdogs.ts`).
- Native run writers that sent a company hint now send a graph hint for
the run and its task: Runner file handoff
(`native-runner-file-handoff.ts`) and chat attachment reuse
(`chat-attachment-reuse.ts`).
- Agent-scoped hint: only the agent lifecycle status-change effect sends
it, for transitions that go through the lifecycle (pause, resume, hold
and release, terminate, error recovery, environment pause, and budget
holds through `applyAgentBudgetHoldInTransaction`). A run starting or
finishing writes the agent row directly (execution start,
`finalizeAgentStatus`) and sends only run-scoped hints, which re-check
the run's task and its graph. Queued-run dispatch after a run finishes
is unchanged (its own dispatch queues).
- `assigned-work-sweep.ts`: `listSuspects` takes the row filter as SQL.
- Tests: new census `issue-recovery-company-reload-census.test.ts`, new
`issue-recovery-saved-deferrals.test.ts` (real test database, recorded
checks), a `restart` helper in the recovery fixture, and a work-signals
test for the reconnect hint.

## Verification

Census (`issue-recovery-company-reload-census.test.ts`, real workers,
fake clock, 28 simulated minutes, 140 + 20 settled reviews, an agent
with a live native run, no fixture agent can start a run). Steps: server
replacement plus connection open, issue creation, document lock, project
budget hard stop, agent pause, agent resume, second server replacement
plus connection open.

| | Settled-task checks | Stranded checks by wake | Statements |
| --- | --- | --- | --- |
| Before (master code) | 1,440 (320 per replacement, 160 per other step)
| company 830, reload 662, hint 163 | 33,485 |
| After | 0 | hint 168, reconnect 12, backstop 7, agent 5, company 5 |
4,293 |

An earlier run of the same census at staging size (700 + 80 settled
reviews, 35 minutes) gave 8,028 settled checks and 165,882 statements
before, and 240 settled checks (only the paused agent's own tasks) and
9,714 statements after.

Guard tests in the same file:
- A task stranded by a write that sends no hint, right before each step,
is checked within 1 minute (replacements, issue creation, budget stop,
pause, resume) or within the 5-minute backstop plus 2 minutes (document
lock, which now re-checks only its own task). Passes before and after.
- A server replaced after its first full pass loaded every task but
before it checked them all: the replacement checks every unchecked task
within 2 minutes. Fails before the deferred save (112 tasks never
checked), passes after.
- Run start and finish (same census file): an agent with a settled
backlog runs start, finish, start, finish through the run row, the
execution-start agent write, the execution-run hints and
`finalizeAgentStatus` (no process runs). 0 settled-task checks on this
branch and on master; only the run's task is checked. A run that fails
has its task checked within 1 minute of the run ending.
- `issue-recovery-saved-deferrals.test.ts`: a task's check defers 20
minutes, the full pass is saved, the server is replaced 5 minutes later
without shutdown work. The replacement checks the task within 1 minute
of the original deadline. Fails at the previous head (never checked
before the next full pass), passes after.
- Same file: server B saves a deadline; server A boots and reads it; B
clears it and defers again to a new deadline; A's check stops deferring
and clears the row it read; B is replaced. The successor checks at B's
deadline. Fails at the previous head (A deleted B's row by id), passes
after.
- Same file: a saved deferral is removed at the next reload once its
task closed.
- Same file: an open task that leaves the queue (taken over by a user)
while deferred has its row removed by the first reload after its
deadline is 10 minutes past. Fails at the previous head (row kept
forever), passes after.
- Same file: after scheduling suppression, a plain company hint checks
every task again even though a recent full pass is saved.
- A change that no changed-rows clause sees is not checked by either
server replacement, and is checked by the next full pass on the saved
schedule. Fails before (each replacement did a full pass), passes after.

The file takes about 30 s locally. Every wait has a 20 s real-time limit
and each test a 2-minute limit, as in #21867.

Focused suites (all `issue-recovery-*` suites and censuses,
assigned-work, scoped deadline work, deployment notifications, budgets,
companies, run dispatch notifications, and related idle tests) on the
final head (34 files with the new deferral test): 288 passed, 1 skipped.
The #21867 agent-row census still reports 0 settled checks.
`packages/db` `work-signals.test.ts`: 34 passed. Server `tsc --noEmit`:
passed. `pnpm --filter @paperclipai/db check:migrations`: passed. `pnpm
verify` returned exit 2 (broad input: the new migration), so required CI
supplies broad coverage.

## Risks

Time bound that changes:
- A task stranded by a change that no changed-rows rule sees was often
found sooner by an unrelated company hint, boot or reconnect. It is now
found by the next full pass: at the next backstop (needs activity in the
company), boot or reconnect after one hour from the last saved full
pass. This was already the documented worst case.

Row changes the changed-rows rules see (`recovery/changed-tasks.ts`),
for auditing completeness. A task is loaded when, since the window
start:
- its own `issues.updated_at` moved;
- a `heartbeat_runs` row whose `context_snapshot.issueId` is the task
has `updated_at` in the window;
- an `issue_recovery_actions` row with the task as `source_issue_id` has
`updated_at` in the window;
- an `agent_wakeup_requests` row whose `payload.issueId` is the task has
`updated_at` in the window;
- an `issue_thread_interactions` row of the task has `updated_at` in the
window;
- an `issue_comments` row of the task was created in the window;
- its assignee agent row has `updated_at` in the window, is not
`running`, and has no heartbeat since that update (pause, resume, hold,
termination, configuration).
Not seen: company, project, tree-hold and budget rows (their resumes
send `everyTask`), the activity log, other agents' rows, lease renewals
that do not move `updated_at`, and time passing. Deferrals are covered
separately by the saved deferral rows.

Rebuilt across a replacement (no change): every pending deferral, by the
saved deferral rows. Every way the four checks defer, from tracing each
`onDeferredDeadline` call:

| Queue | Deferral | Deadline | Bounded after a seen row change? |
| --- | --- | --- | --- |
| stranded | terminal run still settling | run lease, or now + 30 s
while the run executes in memory | no (repeats while it executes) |
| stranded | chat agent daily limit | next 00:00 UTC | < 24 h |
| stranded | accepted interaction, no run since | interaction resolved +
2 min | yes |
| stranded | transient continuation back-off | run finished + up to 16
min | yes |
| dependencies | refused or failed stall wake | last attempt + up to 20
min | yes |
| dependencies | waiter's workspace hold | now + 1 h | no (repeats while
held) |
| dependencies | waiter's agent takes no wakes | now + 10 min | no
(another agent's row) |
| locks | live legacy controller lease | lease expiry | no (renewal does
not move `updated_at`) |
| locks | lapsed lease, owner proven alive | lease + up to 5 min | yes |
| locks | run updated less than 5 min ago | run updated + 5 min | yes |
| locks | adopted or lost child process still alive | now + 30 s | no
(repeats while it lives) |
| locks | prior-boot setup | about run updated + 5 min | yes |
| assigned work | settling, saved-work due, reattach | row time + up to
10 min, or now + 2 s during reattach | yes |
| assigned work | repair budget, company run brake, stalled queued run,
saved work busy | now + 60 s to 1 h | no (repeats while the condition
lasts) |
| assigned work | `waiting_unchanged` (same refusal state) | last
refusal + up to 6 h (activity log) | no |
| assigned work | continuation cap, spacing, fallback back-off | first
attempt + up to 1 h | no (activity log) |

`unchanged` (assigned work only) changes queue priority, not whether a
deferral is pending.
- Unchanged: the 5-minute changed-rows backstop, the 1-minute bound for
paused, resumed, held and terminated agents (covered by the
#21861/#21867 guards and the new agent scope), and whole-company
re-checks for company and project resumes and tree hold releases.

Other risks:
- During the deploy that ships this change, the old build still sends a
bare company hint for a company or budget resume. The new build reads
that as changed rows only. The old build re-checks every task for its
own hint while its workers run; otherwise the next full pass does. After
this deploy both builds send `everyTask`.
- If another server's `everyTask` hint is lost while this server's
notification connection is closed, the reconnect no longer re-checks
every task. The writing server checks its own hint; otherwise the next
full pass does.
- The first boot after this deploy has no saved full passes, so it runs
one full pass per company (as every boot did before).
- Nothing is added before a run's first turn or on the run-start path.
- Schema change: two new company-scoped tables with cascade foreign
keys, no change to existing tables. The old build ignores them. Neither
is a timer: no deadline is stored or scheduled from them, they only tell
a boot which tasks to check again.
- Extra statements: per deferring check, one upsert when its deadline
changed (a deferral to `now + 30 s` changes each time, so about two
writes a minute per such task) and none otherwise; one delete when a
check stops deferring; one read of the queue's deferral rows and at most
one prune per reload; one write per completed full pass.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1116.0-canary.7
2026-11-16 05:52:57 -06:00
DottaandClaude Opus 5.5 278660149c test(recovery): bound every wait in the agent-row census and stop it starting real runs (#21867)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The recovery census tests prove that recovery queues do not re-check
settled tasks without a reason
> - The agent-row census from #21861 stopped producing output in a
server test shard and ran until the 20-minute job limit
> - Its tests let real agent runs start while `setTimeout` was faked,
and no wait in the file had a bound (the test timeout was 30 minutes)
> - This pull request stops the census from starting real runs, bounds
every wait with a real timer, and makes the file much faster
> - The benefit is that the file cannot hang a shard, and a stuck step
fails in seconds with a message that names the step

## Linked Issues or Issue Description

Refs #21861

**What happened?**
In a 10-shard server test run,
`server/src/__tests__/issue-recovery-agent-row-write-census.test.ts`
finished its third test and then wrote no more output until the job
reached its 20-minute limit. The file had a 30-minute timeout per test,
so the hang never failed as a test. Several tests let real agent runs
start while `setTimeout` was faked:
- the idle-agent guard marked a live run succeeded while its task stayed
in progress, so recovery continued the task;
- the hold-release guard released an agent that had a `todo` task;
- the pause and terminate guards stopped agents with live native runs.

A run execution that waits on a faked timer, or holds a lock that a
delivery pass then waits on, does not end. A local run of the old file
showed 4 run executions ("heartbeat execution setup failed").

**Expected behavior**
The census never starts a real run. Every wait has a bound and a message
that names what it waited for. The file finishes well under 60 seconds.

**Steps to reproduce**
1. Run the old file and search its output for "heartbeat execution setup
failed": 4 run executions.
2. Run it in a slow, shared CI shard: the file can stop with no output
until the job limit.

**Paperclip version or commit**
`master` at 94cb7ad359.

## What Changed

- Fixture agents take no on-demand wakes (`wakeOnDemand: false`), so a
recovery check cannot dispatch a run. The census counts queue checks, so
this does not change what the tests measure.
- The idle-agent guard writes the configuration of an agent that never
ran, instead of ending the second agent's live run.
- The held agent's waiting task is a settled review, not a `todo` task.
- The pause and terminate guards use two idle agents without live runs.
- Each step (run-event tick, fake-timer advance, pass settlement,
writes, lifecycle commands) has a real 20-second bound that names what
it waited for. The bound uses the real `setTimeout`, captured before the
timers are faked. Each test has a 2-minute timeout instead of 30
minutes.
- The fixture is smaller (140 + 20 settled reviews instead of 700 + 80).
Warm-up is 6 simulated minutes instead of 12, and each write window is 6
minutes instead of 7.
- The full-pass guard uses a 15-minute full pass (`fullBackstopMs`)
instead of the hourly default. The fixture's `start` accepts
`fullBackstopMs` for this.

## Verification

- Old file, 4 copies at the same time (local CPU contention): all
passed, 190–211 s per file, 172 s of test time, 4 real run executions.
- New file, 4 copies at the same time: all passed, 48.5–49.1 s per file
(about 30 s of test time; each test 4–6 s), 0 run executions.
- The census still fails without the #21861 fix. With
`notifyRunDispatch` put back in `agent-configuration.ts`, the census
test fails with 480 settled checks (three whole-company reloads of 160)
and 519 `company` checks. With the fix, it has 0 settled checks.
- Server `tsc --noEmit` passed.

## Risks

- Test-only change. No product code changed.
- The guards now use idle agents and settled tasks rather than agents
with live runs, so they do not cover a pause or termination that must
stop a live run. Those flows have their own lifecycle tests. The
live-run census (`issue-recovery-live-run-agent-census.test.ts`) still
covers the hourly full pass with the real interval.
- The fixture is smaller, but the census still compares whole-company
reloads against zero.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1116.0-canary.6 nightly/v2026.1116.0-nightly.0
2026-11-16 01:13:19 -06:00
DottaandClaude Opus 5.5 94cb7ad359 fix(recovery): stop an agent configuration write from re-checking every task of the company (#21861)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The recovery queues check each assigned task to find work that has
no live run and no queued wake
> - On a nearly idle hosted instance, these queues still re-check almost
every task of a company in bursts, at about 22 read-only statements per
task
> - #21831 stopped a run start from re-checking all of its agent's
tasks, but bursts continued after writes to an agent row during a long
run
> - The cause is not the changed-rows backstop. An agent configuration
write publishes the company-wide task recovery hint, so every task queue
reloads and checks every assigned task of the company
> - This pull request makes an agent configuration write signal only the
company-level recovery queues and the run dispatch queues, not the
per-task queues
> - The benefit is that a configuration write during a long run no
longer re-checks hundreds of settled tasks

## Linked Issues or Issue Description

Refs #21831

**What happened?**
A write to an agent's configuration (for example `PATCH /agents/:id`, an
instructions save that rebinds the entry file, a skill or connection
change) calls `notifyRunDispatch({ companyId })`. That function sends
the company hint to the task queues (`stranded-assignments`,
`assigned-work-invariant`, `resolved-dependencies`,
`stale-issue-locks`). A company hint is not a backstop hint, so the
queue loads every eligible task of the company with no changed-rows
filter and checks each one. In a staging-shaped fixture (780 settled
reviews), each configuration write caused 780 checks and about 16,000
statements. The agent row keeps status `running` and its old
`last_heartbeat_at`, which matches the writes seen on staging.

**Expected behavior**
A configuration write does not re-check tasks. Only a change that can
strand a task (pause, resume, hold release, termination) re-checks
tasks, and those changes keep their own lifecycle hint.

**Steps to reproduce**
1. Seed one agent with 700 settled in-review tasks and a second agent
with 80, each with one live native run.
2. Start the recovery workers with the debug flag
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:1`.
3. Update the agent's `runtimeConfig` or `adapterConfig` through
`agentService(db).update`.
4. See 780 `check` events with `wake: "company"`.

**Paperclip version or commit**
`master` at c5218f5ec4.

## What Changed

- `run-dispatch-notifications.ts`: split the dispatch queue hints out of
`notifyRunDispatch` and add `notifyAgentConfigurationDispatch`. It sends
the company-level recovery hints and the dispatch hints, but no
task-queue hint.
- `issue-recovery-notifications.ts`: add `notifyCompanyRecovery`, which
signals only the company-level recovery queues (orphan blockers, active
recovery actions, task watchdogs).
- `agent-configuration.ts`: a configuration write uses
`notifyAgentConfigurationDispatch` instead of `notifyRunDispatch`.
- New census test `issue-recovery-agent-row-write-census.test.ts`. It
uses real workers, a fake clock and a staging-shaped fixture. It drives
five agent-row writers through their real code paths over 35 simulated
minutes, and it adds guard tests.

## Verification

Red/green census (`issue-recovery-agent-row-write-census.test.ts`, 35
simulated minutes, five writes: runtime config on the big agent, adapter
config on the second agent, execution start rewrite, permissions,
adapter config on the big agent):

| | Settled-task checks | Stranded checks by wake | Statements |
| --- | --- | --- | --- |
| Before | 2,340 | company 2,346, hint 204, backstop 6 | 51,497 |
| After | 0 | hint 210, backstop 7 | 4,580 |

The execution start rewrite and the permissions write caused 0 settled
checks before and after. Only the three configuration writes caused the
whole-company reloads.

Guard tests (pass before and after):
- A task of the running agent that becomes stranded after a
configuration write is checked within the backstop bound.
- An idle agent's tasks are still checked within the backstop bound
after its configuration is written (the agent clause of the changed-rows
backstop).
- A hold release through the lifecycle status-change effect checks the
waiting task within one minute.
- Pause and termination through the lifecycle check the agent's tasks
within one minute.
- The hourly full pass still checks every settled task.

Focused suites: `cd server && npx vitest run` on 19 files (recovery
work, census, backstop, debug attribution, signal fanout, run dispatch,
agent configuration, budget configuration, agent timers, lifecycle): 278
passed, 1 skipped. `pnpm verify` returned exit 2 (opaque setup
dependency), so required CI supplies broad coverage. Server `tsc
--noEmit` passed.

## Risks

- A configuration write no longer re-checks the company's tasks at once.
Agent status changes still send the company hint through the lifecycle
effect. For a non-running agent, the changed-rows backstop still picks
up its tasks within the existing 5-minute bound. The hourly full pass is
unchanged.
- The stranded check reads agent status, holds and budget, not
configuration, so a configuration change cannot strand a task by itself.
- Low risk otherwise. No schema change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1116.0-canary.5
2026-11-15 23:46:49 -06:00
DottaandPaperclip 29903bed59 fix(ui): add a space after composer autocomplete (#21754)
Preserve an ordinary space after composer autocomplete selections and place the caret after it. Cover all seven completion types and queries with trailing spaces.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
canary/v2026.1116.0-canary.4
2026-11-15 22:15:00 -06:00
DottaandClaude Opus 5.5 f7a3497e9d fix(evals): compare browser zones by clock, wait for the reloaded profile name, bind accounting reads by returned run (#21760)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature eval catalog runs real agent conversations and
grades the saved outcome
> - Chat-eval cohort 16 (OpenCode with a DeepSeek model) had three
failures that were not product failures of the chat agent
> - One setup check compared time zone names, one observer read a form
before it filled, and one grader accepted only exact query digests
> - An eval that fails valid behavior hides real regressions behind
noise, so each check must grade the outcome, not an incidental detail
> - This pull request fixes the three checks, and it fixes a first-frame
empty display name on the Profile page
> - The benefit is that these three cases now fail only for the behavior
they test

## Linked Issues or Issue Description

**What happened?**
Three chat-eval cases failed in cohort 16 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`) on correct behavior. (1)
`activity.routine-audit`: setup threw "Browser timezone override did not
apply (America/Buenos_Aires)". At 11:00-11:59 UTC the picker selects
`America/Argentina/Buenos_Aires`, and Chrome reports that zone by its
alias `America/Buenos_Aires`. The check compared names. (2)
`navigation-preferences.edit-profile`: the name saved correctly, but the
observer read the display-name field once, right after reload, and got
an empty value. (3) `budgets-costs.spend-breakdown`: the answer was
correct, but the grader accepted company cost reports only for
`period=all` or `period=month`. The agent read `costs/by-agent` and
`costs/summary` for the requested week, and that range covers the
unpriced run.

**Expected behavior**
Each case passes when the product and the agent do the requested work.
Each case fails when they do not.

**Steps to reproduce**

1. Run `chat-features.runner-opencode.local.activity.routine-audit`
between 11:00 and 11:59 UTC.
2. Run
`chat-features.runner-opencode.local.navigation-preferences.edit-profile`
and look at `profileAfterReload`.
3. Grade the retained cohort 16 observation of
`budgets-costs.spend-breakdown` with the old grader.

**Additional context**
The run tool inputs are saved only as SHA-256 digests, so a grader binds
a read by recomputing the digest of inputs it can check. For the cohort
16 spend-breakdown reads, the inputs were recovered this way:
`from=2026-11-09T00:00:00.000Z`, `to=2026-11-16T00:00:00.000Z`.

## What Changed

- `internal-routine-audit.ts`: the browser override check compares the
UTC offset of the requested and reported zones, now and 182 days later,
instead of the zone names. An unknown zone or a different clock still
fails setup.
- `identity-profile.ts`: after reload, `settledFieldValue` waits up to
15 s for the display name to reach the saved value, then records what
the person sees. The grade still fails a name that does not arrive.
- `ProfileSettings.tsx`: unedited fields show the session's saved name
and image from the first frame. Before, a session restored from the
saved read cache rendered an empty name until an effect filled it.
- `identity-accounting.ts`: `readSavedAccounting` also accepts a company
cost report whose checked `from`/`to` covers the unpriced run's cost
event. The run log keeps only the input digest, so the grader recomputes
digests for covering ranges: UTC day boundaries (with or without
milliseconds or seconds, date-only, end of day), local midnights at each
whole-hour offset, and the run's own `startedAt` and cost-event
instants. A pending run (no cost event) counts only through
`costs/summary`, `costs/data-quality` or its own run record, because the
other reports sum cost events. A link to the run does not count, because
the run id is in the task description. The run-log redaction is not
changed.
- Oracle versions bumped: `routine-audit.v7`,
`spend-breakdown.structural.v6`, `identity-profile.v4`.

## Verification

- Red/green: each new test fails on the old code and passes on the new
code.
- `internal-remaining.test.ts`: every candidate zone, reported by its
own name, its ICU name and known browser aliases, passes at every hour
in January and July. A different clock, a zone with daylight saving time
and an unknown name fail.
- `identity-personal.test.ts`: the settled read returns the saved name
after empty reads, and returns a wrong name when the saved name never
arrives.
- `ProfileSettings.test.tsx`: the first rendered frame with a cached
session shows "Alex Chen", not `""`.
- `identity-accounting.test.ts`: the cohort 16 week-range reads pass, as
do other covering ranges. Negative controls fail: no reads, run-list
reads with unchecked inputs, ranges that exclude the run, another
company, a denied read, a read by another run, and a read of a different
run record with a link to the target run.
- The retained cohort 16 spend-breakdown observation passes the new
grader, and fails with its reads removed.
- `pnpm test:e2e:runner:unit`: 686 files passed. `pnpm
test:e2e:runner:typecheck` passed. `ui` ProfileSettings tests passed.
`scripts/check-token-gates.mjs` is clean.
- One narrow paid run of the three cases on this branch (result in a
comment below).

## Risks

- Low risk. The changes are in eval code, plus a small Profile page
state change.
- A covering range in a form outside the recomputed set (for example
08:00 at -03:00) is not recognized. That is a false negative, not a
false pass.
- The Profile page now keeps an unedited field in step with the session.
A refetch still resets drafts, as before.

## Model Used

- Claude, model ID `claude-opus-5-5`, used through Claude Code with tool
use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 22:05:47 -06:00
DottaandClaude Opus 5.5 c5218f5ec4 test(restart-chaos): count distinct warm close failures and let R5 wait out environment acquire (#21852)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness (`tests/restart-chaos`) runs zero-downtime
gate waves and grades each wave and each task
> - Gate wave A1 of rerun 6 (w-20261116T001158-s7106) failed. Its "warm
sessions that did not close" count was 28, but only 2 sessions failed to
close. Task t010 also failed R5, but its run was healthy.
> - The harness's environment-delete retry loop re-ran one failing close
every 3 s, so one failed session was logged 27 times. The harness
counted log lines, not failures. Also, no sandbox capture existed from
the time of the failed idle close, so the cause was not recorded.
> - R5 saw 253 s of silence on t010's run aedc1f6e. The run spent 251 s
in `environment.acquire` (a Daytona restore of an archived sandbox, with
one 90 s plugin call timeout). The "run started" event is written only
after acquire, so the run had no event during that time. The run then
succeeded.
> - This pull request makes the warm-shutdown grade count distinct (run,
reason) failures, captures the sandbox at the first idle-close failure,
and lets R5 wait up to 10 minutes while a running run has no run event
yet.
> - The benefit is a wave grade that shows the real number of failed
sessions, records why a close failed, and does not report a slow
environment acquire as a dead run.

## Linked Issues or Issue Description

**Bug description**
Three findings from gate wave A1 of rerun 6 (w-20261116T001158-s7106):
1. `warm-shutdown.ts` reports one failure per log line. The wave showed
29 shutdown close failures (28 "did not close" lines and 1 "cleanup
failed before controller shutdown" line) for 2 sessions: run 300f0d61 (2
lines) and run 70d909cc (27 lines from the environment-delete retries).
No sandbox capture was taken when the idle close of 70d909cc failed, so
why the Runner did not suspend is not known.
2. R5 reported `silent_dead_run aedc1f6e… 181s` for task t010. The run
was in environment acquire for 251 s. It then succeeded.
3. Task t024 failed its content check `subtask_1_done`. The child saved
its ledger with Read-tool line-number prefixes (`1\tROUND 1 ...`). The
grader requires lines that start with `ROUND`. This is model behaviour:
a ledger that the agent corrupts is a real content failure. This pull
request does NOT change that check.

**Steps to reproduce**
1. Grade the A1 server log lines (fixture
`fixtures/a1-warm-close-failures.log`) with `gradeWarmShutdown`. The
result has 29 `shutdownCloseFailures` entries.
2. Grade the t010 samples (fixture
`fixtures/t010-slow-environment-acquire.json`) with `gradeR5`. The
result is `fail: silent_dead_run aedc1f6e…`.

**Expected behavior**
1. The grade shows 2 distinct shutdown close failures, each with its
line count, and the verdict stays "fail".
2. R5 passes a run that has no run event yet for up to 10 minutes. R5
still fails a run with no event past 10 minutes, and a started run that
is silent past 3 minutes.

## What Changed

- `warm-shutdown.ts`: `shutdownCloseFailures` and `idleCloseFailures`
are now lists of distinct `{ runId, reason, lines, firstLine }` entries,
keyed by run and failure reason. The reason drops the JSON detail after
the error text (for example timings), so retries of the same failure
merge. A line with no parsable run keeps its own message as the reason.
Each distinct shutdown failure still adds a problem, so the verdict is
unchanged in meaning: any distinct failure fails the wave.
- `warm-shutdown.ts`: new `watchFirstIdleCloseFailure` splits server
output into lines per stream and reports the first idle-close failure
once. New optional `sandboxCaptures` input and output, listed in the
markdown section.
- `supervisor.ts`: new `onServerOutput(chunk, source)` hook, called for
the primary and the second server's stdout and stderr. A hook error is
recorded in the timeline and never breaks logging.
- `main.ts`: on a Daytona warm wave, the first idle-close failure
triggers one `captureRunnerLogs("idle-close-failure-<run>")` (the class
R probe: processes, sockets, Runner runtime files including
`runner-state.json`, and logs). The harness waits for that capture
before the environment cleanup and before the warm-shutdown grade.
- `observer.ts` / `types.ts`: a running run whose events read back empty
(the read succeeded and there is no event yet) gets `preparing: true` in
its sample. A failed events read does not set it.
- `grader.ts`: new `PREPARATION_LIMIT_MS` (10 minutes). `deadRunningRun`
uses it when the run sample has `preparing`.
- Why 10 minutes: the observed acquire took 251 s, including one 90 s
plugin call timeout. The Daytona provider bounds each start or restore
at 5 minutes by default (`DEFAULT_DAYTONA_OPERATION_TIMEOUT_MS`), so 10
minutes covers one timed-out restore and its retry. This is the same
bound #21730 gave an active provider turn.
- New fixtures: `fixtures/a1-warm-close-failures.log` (the 34 real
close-failure lines of the A1 server log) and
`fixtures/t010-slow-environment-acquire.json` (the 87 real t010 samples
of run aedc1f6e, with the time of its first run event).
- `README.md`: document the distinct-failure count, the idle-close
capture and the preparation bound for R5.
- t024: no change. Decision: the ledger check stays strict, because a
ledger that the agent saved with line-number prefixes is a real content
failure.

## Verification

- Red on `origin/master` sources, green with this change (9 tests):
- `warm-shutdown.test.ts` › "reports each distinct (run, reason) once,
not each log line (w-20261116T001158-s7106 A1: 2 sessions, 29 lines)": 2
failures (300f0d61 with 2 lines, 70d909cc with 27 lines), 2 problems,
verdict fail.
- `warm-shutdown.test.ts` › "still fails on one distinct failure, and
keeps different reasons for the same run apart" (guards: the 27 lines of
one session still fail; two reasons for one run give two problems; an
unparsable line still fails).
- `warm-shutdown.test.ts` › "names the first idle-close failure once,
across chunk boundaries and interleaved streams" (a forced-stop line
does not trigger it).
- `warm-shutdown.test.ts` › "lists the sandbox captures taken at the
first idle-close failure", plus the two updated shape tests.
- `triggers.test.ts` › R5 live evidence › "does not flag a run that is
still acquiring its environment before its first run event
(w-20261116T001158-s7106/t010)": the real samples still fail without
`preparing`, and pass with `preparing` on the samples before the first
run event.
- `triggers.test.ts` › R5 live evidence › "still fails a run silent past
the preparation bound, and a started run silent past the stall bound"
(guards: 9.5 min preparing passes; 10.5 min preparing fails; 4 min quiet
without `preparing` fails).
- `triggers.test.ts` › R5 live evidence › "marks a running run as
preparing only while its events read back empty" (the flag clears after
the "run started" event; a failed events read never sets it).
- `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 71
files, 833 tests pass.
- `npx tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean.
- No wave was run. The live idle-close capture path in `main.ts` is not
exercised by a unit test.

## Risks

- Low risk. The change touches only the test harness.
- `warm-shutdown.json` changes shape: `shutdownCloseFailures` and
`idleCloseFailures` are now objects, not strings. Scripts that read
these fields as strings must read `firstLine`. Summary counts now show
sessions, not lines, so they are lower than before for the same log.
- A run that dies before its first run event now fails R5 after 10
minutes instead of 3. The fault is still caught, but later.
- The idle-close capture can add up to 45 s (the capture deadline)
before the environment cleanup on a wave that had an idle-close failure.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1116.0-canary.3
2026-11-15 20:55:32 -06:00
DottaandClaude Opus 5.5 d149e30323 fix(native): stop a finished warm owner's retained Runner when suspend is refused, and release a suspended Runner whose process left (#21853)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes an idle session after 5 minutes, and closes it
again for an environment delete, idle sleep, or a restart.
> - In a zero-downtime release-swap test, two warm closes failed and did
not recover.
> - In the first case, the Runner of a finished run refused
`runner.suspend`. The server kept the Runner alive and the session
registered. An environment delete returned 409 thirty-five times, and
only an outside sandbox destroy released the session. A live Runner and
a paid sandbox were held, and idle sleep was blocked.
> - In the second case, the Runner durably suspended and exited, but its
process check returned exit 44 (`process_identity_changed`) on every
attempt. Exit 44 also covers "a live process without `--runner-id`", so
no proof could release the owner.
> - Also, the bounded idle retries from #21782 did not run for the first
case.
> - This pull request stops the retained Runner of a finished owner with
the verified port-holder retirement, releases the owner only when a
fresh check reports `process_not_running`, splits exit 44, accepts a
durably suspended Runner with no live process that carries its runner
id, and keeps the idle retries alive after a failed sweep.
> - The benefit is that a finished warm session can no longer hold a
live Runner and a sandbox until someone destroys the sandbox.

## Linked Issues or Issue Description

Refs #21782
Refs #21835

**What happened?**
Release-swap gate wave, two warm-session close failures:

1. A finished run's idle close failed with
`native_session_close_unrecoverable`: `runner.suspend` failed,
`runnerRetained: true`, Runner lifecycle `ready`. Only close attempt 1
was logged. An environment delete with `forceStopUnsettled` could not
discard the session, because `closeWarmNativeSession` discards only with
a Launcher task owner or a suspended Runner. The delete got 409
thirty-five times. The session was released only after an outside
sandbox destroy.
2. A Runner's `runner-state.json` showed lifecycle `suspended`, with
drain and suspend `completed`. The process check exited 44 on every
attempt, and the Runner process was gone later while the sandbox stayed
started.

**Expected behavior**
A finished owner whose Runner refused suspend is stopped and released,
and a failed stop keeps it held. A durably suspended, settled Runner
with no live process that carries its runner id releases its owner. A
bare `process_identity_changed` never releases an owner.

**Steps to reproduce**
1. Keep a warm sandbox session idle after its run succeeds.
2. Make its close fail with `NativeSessionCloseUnrecoverableError` with
`runnerRetained: true`.
3. Call `closeWarmNativeSessionsForEnvironment({ forceStopUnsettled:
true })`. On master the result is `failed: 1`, every time.

**Why the retries did not run**
- The transport memoizes `close()`. A second close returns the same
rejected promise and does no sandbox work. The 27 environment-delete
failures have identical timings (`totalMs: 6055`), and the other
Runner's daemon log shows no sandbox commands during its three retries.
So a retry can only succeed through a fresh proof, not through the
close.
- `closeIdleWarmNativeSessions` (environment delete, idle sleep,
restart) clears the entry's idle timer, which is also the pending retry
timer, and never re-arms it after a failed close. The test "keeps
retrying a failed idle warm close after a failed environment sweep
cleared its retry timer" is red on master for this reason.
- The log shows one more `close()` request on the same transport in the
same second as attempt 1, and no attempt 2. Every path that closes an
idle owner from outside the timer clears that timer. The log does not
show which caller it was, so this part is not proven.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`:
- A warm entry keeps its sandbox runner and remote directory when the
run used a sandbox that the server reaches directly (no Launcher task).
  - `warmCloseFailureProvesOwnerGone` adds two proofs:
- The close failed with `NativeSessionCloseUnrecoverableError`,
`runnerRetained: true` and not suspended. The server stops the session's
own Runner with `retireSandboxRunnerPortHolders` (owner run must be
`succeeded` or `cancelled`, same agent, idle state; TERM, then KILL). It
releases the owner without a checkpoint only if a fresh process check
then exits 45 (`process_not_running`). A stop that cannot be confirmed
keeps the owner.
- The close failed with the new `process_runner_id_missing`. A fresh
sandbox inspection must show lifecycle `suspended`, settled state (empty
outbox, no pending terminal delivery, provider cleanup, or warm
transition), matching run, session and runner ids, and no process that
carries `--runner-id <id>`.
- Both the idle-timer failure path and `closeIdleWarmNativeSessions`
(environment delete with `forceStopUnsettled`, idle sleep, restart) use
these proofs.
- The first Runner-check or refused-suspend close failure for a sandbox
owner logs one evidence line: stage, reason, Runner lifecycle, suspend
command status, proof window, and an inspection of the sandbox (durable
lifecycle, settled, identity match, marker pid, and pid/state of each
`paperclip-runnerd` or runner-id process). It logs no nonce and no
command line.
- A sweep that fails to close an owner whose idle close already failed
re-arms the bounded idle retry.
-
`server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts`:
`REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT` exits 48
(`process_runner_id_missing`) for a live process whose command line
lacks the runner id. Exit 44 now means only an identity file mismatch.
- `server/src/services/native-runtime/sandbox-runner-port-holders.ts`:
`retireSandboxRunnerPortHolders` takes `only: { sessionDirectory, runId
}` to retire one session's Runner. New `sandboxSessionRunnerExited` and
`inspectSandboxSessionRunner` (a read-only `node` script).

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts
src/services/native-runtime/sandbox-runner-port-holders.test.ts
src/services/native-runtime/sandbox-runner-port-holders.integration.test.ts
src/services/runner-lost-mid-turn-continuation.test.ts
src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts`
- Red on master (master sources with these tests): "stops the retained
Runner of a finished warm owner that refused suspend ..." returns
`failed: 1`. "keeps retrying a failed idle warm close after a failed
environment sweep ..." sees 2 closes, not 3. "releases at controller
restart a warm owner whose Runner durably suspended ..." returns
`failed`.
- Negative tests: a Runner that survives TERM and KILL stays `failed: 1`
with no exit check. A run that is still `running` is never signalled. A
live process that still carries the runner id, and a bare
`process_identity_changed`, keep the owner.
- Script tests: the process check exits 48 for a live process without
`--runner-id`, 0 with it, and 44 for a changed marker. The inspection
script reports state, marker pid and runner processes, never the nonce
or a command line, and fails closed without `/proc` or through a
symlink. The port-holder integration test stops only the named session's
finished Runner (a real local process) and then proves it exited.
- `pnpm verify` exited 2 (opaque setup dependency), so focused checks
and required CI give the coverage. `npx tsc --noEmit -p server`: no
errors.

## Risks

- A finished owner's retained Runner is now stopped when its suspend was
refused. The session is released without a new checkpoint. The last
saved checkpoint stays the recovery source, which is the same as an
environment delete today.
- The stop uses the same rules as a successor's port-holder retirement:
finished owner run, same agent, idle Runner state, and a verified
process identity. Any doubt keeps the owner.
- The stop can take up to about 20 s (TERM and KILL waits) inside a
sweep, while the entry stays fenced.
- The first close failure of a sandbox owner runs one read-only
inspection command (5 s limit).
- Previous-release story: no persisted state or launch identity changes.
The scripts are sent with each command, so a server always runs its own
exit-code table. Exit 44 from the previous release maps to
`process_identity_changed`, which still never releases an owner.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1116.0-canary.2
2026-11-15 20:54:47 -06:00
DottaandClaude Opus 5.5 49a2b7b245 test(recovery): record a live process's OS-observed start time in owner-proof tests (#21847)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Recovery continues a cut-off task automatically. It first lets a
previous process that still runs on this host settle, for about ten
minutes at most.
> - A server test proves that wait. It failed once on master at
03309f1650 in the "Cloud readiness" run, and the failure blocked the
staging release.
> - The test stamps the run with `Date.now() - process.uptime()`. The
product compares the stamp with the start time that the OS reports, and
a gap of 1 s or more means "a different process with a reused PID".
> - The OS time is truncated to whole seconds (Linux `btime`, macOS `ps
-o lstart`), and Node starts some milliseconds after the fork. Thus the
gap is sometimes 1 s or more, and the product correctly continues the
task.
> - This pull request makes the test stamp the same OS-observed start
time that production records.
> - The benefit is a deterministic test that still proves the wait and
its ten-minute upper bound.

## Linked Issues or Issue Description

**What happened?**
`server/src/services/fresh-continuation-fallback.test.ts > lets a
previous process still running on this host settle first, for about ten
minutes at most` failed on master (Cloud readiness run 43776843790, job
"verify / General tests (server (2/10))"). Received `{ kind: "continued"
}`, expected `{ kind: "exact_recovery_pending", reason:
"previous_process_running" }`.

**Expected behavior**
The test passes on every host. It only fails when the product does not
wait for a live previous process.

**Steps to reproduce**
1. Run the test about 15–30 times on macOS under CPU load (for example
24 busy loops on 18 cores).
2. About 1 in 13 runs fails with the same assertion as CI.
3. On Linux, the gap is `frac(boot time)` plus Node startup. Thus a
runner whose boot time has a fractional second near 1 fails, and other
runners pass.

**Paperclip version or commit**
03309f1650 (master).

**Additional context**
This is not a product regression. Production writes `processStartedAt`
from `readProcessStartedAt` (`persistHeartbeatRunProcessMetadata`),
which is the same source that the owner proof reads. Thus production
compares two equal values. Neither #21831 nor #21835 changed this path.

## What Changed

- `fresh-continuation-fallback.test.ts`: `liveIdentity()` now records
`observedProcessStartedAt(process.pid)`, the OS-observed start time that
a run records. It throws if the host cannot report the time, so the test
cannot pass by accident.
- `completed-turn-cleanup-hold-settle.test.ts`: the live controller
stamp now uses `currentProcessStartedAtMs()`, the value that this server
stamps on its own claims. It had the same `Date.now() -
process.uptime()` pattern.
- No product code changed. No assertion changed or became weaker. The
ten-minute upper bound assertion stays.

## Verification

- Gap probe (fresh Node processes, `Date.now() - uptime` minus the OS
start time):
- macOS under load: 2 of 200 processes had a gap of 1 s or more (max
1017 ms).
- Linux (`node:24-bookworm-slim`): gap 340–396 ms, of which 337–345 ms
is `btime` truncation. This value is fixed for each host boot, so some
CI hosts are always near or past 1 s.
- `npx vitest run src/services/fresh-continuation-fallback.test.ts -t
"previous process still running|recycled PID"`, repeated under CPU load:
see the before/after counts in the comment below.
- `npx vitest run
src/__tests__/completed-turn-cleanup-hold-settle.test.ts`: 10 passed.

## Risks

- Low risk. Only two test files changed.
- If a CI host cannot read a process start time, the changed tests now
throw a clear error. Before this change the product would also return
"unprovable" there.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1116.0-canary.1
2026-11-15 19:22:57 -06:00
DottaandClaude Opus 5.5 03309f1650 fix(native): release an idle warm owner whose sandbox Runner port a later run took over (#21835)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes idle sessions, and closes them again before it
exits.
> - A sandbox has one Runner port. Before a new run launches its Runner
on a reused sandbox, it retires every other Runner on that port.
> - In a zero-downtime release-swap test, a run finished on the outgoing
server and stayed warm. 77 ms later, a follow-up run started on the
incoming server. It reused the same sandbox, started a fresh provider
session, retired the old Runner, and launched its own Runner on the same
port.
> - The outgoing server's idle close could not suspend the old Runner.
Its checkpoint was correctly refused with
`native_runner_checkpoint_owner_lost`. The idle timer retried 3 times
and then kept the owner registered. That phantom owner blocked idle
sleep for 4.8 minutes, and the shutdown close failed again.
> - The existing proofs that a warm owner is gone (#21782) do not count
"a later run launched its own Runner on this sandbox".
> - This pull request adds that proof. When it holds, the server forgets
the owner without a checkpoint. The checkpoint fence does not change.
> - The benefit is that a superseded warm owner no longer blocks idle
sleep or fails a clean shutdown.

## Linked Issues or Issue Description

Refs #21627
Refs #21782

**What happened?**
An idle warm owner finished on the outgoing server. A follow-up run on
the incoming server reused the same sandbox and launched its own Runner
on the sandbox's single Runner port. The owner's idle close failed with
`native_runner_checkpoint_owner_lost` 4 times. The owner stayed
registered, held the idle-work flag, and blocked idle sleep. At SIGTERM
the restart sweep logged "idle warm native sessions did not close" with
"runner did not durably suspend before checkpoint". No task was
affected.

**Expected behavior**
A failed idle close releases its owner when a later run proves that the
owner's Runner is gone. The checkpoint stays refused while the successor
is live.

**Steps to reproduce**
1. Finish a warm run on a sandbox and keep it idle.
2. Start a successor run for the same task on another server. Give it a
lease on the same sandbox, its own `native.process_identity_recorded`
event and its own `session.started` event.
3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result
is `closed: 0, failed: 1`, and `readNativeSessionIdleWork()` stays
`present`.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: new
`warmRunnerPortTakenBySuccessor`, used by
`warmCloseFailureProvesOwnerGone` after the existing sandbox-termination
proof. It holds only when another run has all of these:
- an `environment_leases` row with the same environment and
`providerLeaseId` as the owner;
- its own `native.process_identity_recorded` event (`source_event_id is
null`, so an inherited identity does not count);
- a `session.started` event from its own Runner instance
(`source_instance_id` equals the run's `runner_instance_id`);
  - both events created after the owner run's `finishedAt`.
- When the proof holds, the idle-timer path and the controller-restart
path log "warm native session's Runner or sandbox is gone; released its
owner" and forget the owner without a checkpoint. Both paths already
call `warmCloseFailureProvesOwnerGone`.
- `validateNativeRunnerCheckpointOwner` does not change. A checkpoint
still cannot land under a live successor.
- Cost: the proof runs only after a failed close. It does one
primary-key read of the owner run and one query on
`environment_leases.provider_lease_id` (indexed), joined to
`heartbeat_runs` by primary key, with `heartbeat_run_events` read
through `(company_id, run_id)`.
- New real-database test file
`server/src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts`.
- `native-session-executor.test.ts`: one environment-delete test counts
the extra proof read when the first proof does not hold.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts`:
6 passed.
- "releases an idle warm owner whose checkpoint is refused because a
successor on another server launched its own Runner on the same sandbox:
own_runner": the close is refused with
`native_runner_checkpoint_owner_lost`, the result is `closed: 1, failed:
0`, the idle-work flag is `none`, the owner has no checkpoint row, and
`assertNativeRunnerCheckpointOwner` still throws
`native_runner_checkpoint_owner_lost`.
- These stay failed and keep the owner registered: the successor is on
another sandbox; the successor has only
`native.process_start_requested`; the successor has only inherited
events (`source_event_id` set); the successor's events come before the
owner finished.
- "releases a superseded idle warm owner from its idle timer without
writing a checkpoint": the idle timer releases the owner after the first
refused close.
- Red on master (test file only, fix reverted): `own_runner` fails with
`closed: 0, failed: 1`; the idle-timer test fails with the idle-work
flag still `present`. The 4 negative cases pass on master and with the
fix.
- `cd server && npx vitest run
src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts`:
9 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: all
passed.
- `cd server && npx tsc --noEmit -p .`: no errors.
- `pnpm verify`: see the comment below for the result.
- Not run: a restart-chaos wave. The release-swap timing is reproduced
in the database test, not on a live sandbox.

## Risks

- A superseded owner is now released without a checkpoint. Its old
provider conversation cannot be resumed. The successor had already
started a different conversation and holds the task, so no one would
resume the old one.
- The proof needs the successor's own Runner identity and own session
start after the owner finished, on the same sandbox lease. A successor
that reused the owner's Runner, only requested a start, or recorded its
events earlier does not count. In those cases the close fails as before.
- The proof adds two small indexed reads after a failed close only. The
normal close path does not change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1116.0-canary.0
2026-11-15 17:49:01 -06:00
DottaandClaude Opus 5.5 da8cfda42e perf(recovery): stop counting a run start as a change to every task of its agent (#21831)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues. The `stranded-assignments`
queue checks assigned tasks that have no live run. One check costs about
22 database statements.
> - #21729 made the 5-minute backstop re-check only tasks whose own rows
changed. A change to the assignee agent's row counts as a change to
every task of that agent. #21765 left out the agent's heartbeat writes.
> - Execution start in `heartbeat.ts` writes `agents.status = 'running'`
and `updated_at`, but not `last_heartbeat_at`. So each run start still
makes every task of that agent "changed". The backstop window overlaps
the previous one by 5 minutes, so each run start re-checks those tasks
in two windows.
> - On an almost idle hosted instance with one live run and a new run
every few minutes, this gives bursts of about 1,450 checks for about 855
settled tasks, every few minutes.
> - This pull request stops counting an agent row whose current status
is `running` as a change to its tasks.
> - The benefit is no re-checks of settled tasks beside a live run: 540
backstop checks in 30 simulated minutes drop to 6 (only the live task
itself), and statements drop from 15,280 to 3,984.

## Linked Issues or Issue Description

Refs #21729, #21765, #21795

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The changed-tasks backstop of the `stranded-assignments` and
`assigned-work-invariant` recovery queues
(`server/src/services/recovery/changed-tasks.ts`).

**Subsystem affected**

Server: issue recovery background work.

**Current behavior**

On staging at near idle (one live native run, about six new runs in 30
minutes), the stranded-check statement bundle runs in bursts of about 10
checks per second for about 2.5 minutes, then stops, then starts again
about 4 minutes later. The tasks do not change. The cause: every run
start writes the agent row without a heartbeat time. The backstop then
loads every open task of that agent as "changed", in two consecutive
windows.

**Proposed behavior**

An agent row counts as a change to its tasks only when the agent is not
`running`. A pause, a resume, a hold or a configuration change of an
idle agent still counts. A run's own task still gets its run hints and
an immediate targeted check.

**Reason and benefit**

Recovery work becomes proportional to activity, not to the number of
tasks held by an agent that just started a run.

## What Changed

- `server/src/services/recovery/changed-tasks.ts`: the assignee-agent
clause of `changedTaskCondition` now requires `ag.status <> 'running'`.
A comment explains why run lifecycle writes are left out.
- `server/src/__tests__/issue-recovery-live-run-agent-census.test.ts`
(new): a staging-shaped census on the embedded test database with the
real recovery workers and a fake clock. It seeds 800 settled in_review
tasks with a succeeded latest run (3 workers, about 266 tasks each), 30
in_progress and 30 todo tasks without live runs, and one live native
run. The run's agent gets the execution-start write, and the run
publishes a run hint every 10 seconds for 30 simulated minutes. It also
has three correctness guards: a task of the live agent whose own row
changes without a hint is checked within the backstop bound; a resumed
agent's task is checked within the backstop bound when the resume sends
no hint; and the hourly full pass still checks every settled task while
the run stays live.
- `server/src/__tests__/issue-recovery-deadlines.test.ts`: a fast test
that a run-start write is not a change and a pause during the run is a
change.

## Verification

Red on `origin/master` (fix reverted), green with the fix:

| 30 simulated minutes beside one live run | before | after |
| --- | --- | --- |
| stranded checks, backstop wake | 540 | 6 |
| stranded checks, hint wake (the live task's own run hints) | 178 | 174
|
| checks of settled tasks | 534 | 0 |
| database statements | 15,280 | 3,984 |
| work signals beyond the run's own hints | 0 | 0 |

- `npx vitest run
src/__tests__/issue-recovery-live-run-agent-census.test.ts` (4 tests):
the census fails on master with `expected 534 to be less than or equal
to 50` and passes with the fix. The three guards pass.
- `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts -t "run
start"`: fails on master, passes with the fix.
- `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts
src/__tests__/issue-recovery-idle-backstop.test.ts
src/__tests__/issue-recovery-activity-hints.test.ts
src/__tests__/issue-recovery-signal-fanout.test.ts`: 31 passed, 1
skipped.
- `tsc --noEmit -p server/tsconfig.json`: clean.

## Risks

- Low risk. The change narrows one clause of the backstop query. No
schema change.
- If an agent is resumed and then starts a run in the same window, the
resume no longer counts through the agent row. A resume already sends
its own `pause_lifted` hint (`agent-lifecycle-driver.ts`), and the
hourly full pass still covers every task. Before this change, a run
finish already hid a resume the same way, because the finish writes a
heartbeat time.
- A configuration change of an agent while it runs is caught by its next
idle write or by the hourly full pass, not by the 5-minute backstop.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 16:49:44 -06:00
8299f7c389 perf(runner): check a listening Runner's backlog every 10 ms, not every reconnect delay (#21821)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote `paperclip_runner` agent runs `paperclip-runnerd` in a
sandbox or VM. A provider ingress proxies Core's WebSocket to a Runner
that listens (`--listen-address`).
> - In listen mode, the Runner checks for an incoming connection only
once per `--reconnect-delay-ms` (default 250 ms). It waits that delay
again after each disconnect, and it backs off after a failed handshake.
> - That delay exists to pace redials in dial mode. A listener never
dials. Core's connection waits in the kernel backlog until the next
`accept`.
> - Every listen-mode connect pays part of that wait. A warm run
attachment pays it in full: the Runner disconnects at the authority
rotation, and Core's new-authority connect then waits up to 250 ms or
more.
> - This pull request makes a listening Runner check for a connection
every 10 ms. Dial mode keeps the configured reconnect delay.
> - The benefit is about 0.4 s less on each warm run attachment and
about 0.1 s less on each cold start, for every listen-mode Runner.

## Linked Issues or Issue Description

**What happened?**
A warm follow-up turn on a listen-mode remote Runner reached its first
turn later than a cold turn on the same agent. In traces, a cold start
makes one Core→Runner connect of 134–178 ms. Each warm attachment makes
an old-authority reconnect (72–233 ms) and then a new-authority connect
of 411–504 ms. This was 6 of 6 warm runs.

**Expected behavior**
A connect to a listening Runner completes in about one network round
trip plus the WebSocket handshake. It does not wait for a reconnect
timer.

**Steps to reproduce**
1. Start `paperclip-runnerd` in listen mode with the default
`--reconnect-delay-ms`.
2. Let it reach the disconnected wait (startup, a park claim, or a
`run.attach` authority rotation).
3. Connect to the listener. The upgrade is not answered until the next
250 ms check.

**Root cause**
`durable/runner.rs` schedules the next `accept` at `now +
config.reconnect_delay` when no connection is pending, after a
disconnect, and as the base of the failure backoff. The Launcher-style
hosts do not pass `--reconnect-delay-ms`, so the 250 ms default applies.
The tests use a 1 ms delay, so they did not show the wait.

## What Changed

- `durable/transport.rs`: add `RunnerTransportEndpoint::retry_delay()`.
A listener returns `min(configured, 10 ms)`. A dial endpoint returns the
configured delay without change.
- `durable/runner.rs`: the four places that waited
`config.reconnect_delay` (no pending connection, failure backoff base,
two disconnect paths) now use
`endpoint.retry_delay(config.reconnect_delay)`.
- `DURABLE_TRANSPORT.md`: document the listen-mode connection check
interval.
- Add a unit test for the listener and dial delays.
- Add `tests/listener_accept.rs`. It starts `paperclip-runnerd` in
listen mode with the default reconnect delay. One connection drops
before authentication, as a superseded route's redial does. The test
then checks that the next upgrade is answered in under 120 ms. Before
this change, that upgrade waited 188–313 ms.

## Verification

- `cargo test -p paperclip-runner-core --lib` passes locally.
- `cargo clippy -p paperclip-runner-core --all-targets` reports no new
findings in the changed files.
- `cargo fmt --all -- --check` passes.
- New tests:
`durable::transport::tests::listener_checks_its_backlog_sooner_than_a_dial_reconnects`
and `tests/listener_accept.rs`.
- `tests/listener_accept.rs` fails on the previous `runner.rs` and
passes with this change (3 runs each).
- Existing listener and warm-attachment tests pass (`listener_*`,
`warm_run_attachment_*`, `warm_external_attachment_*`).
- I did not measure the change on a live host. The effect is visible in
the next deployment as a shorter `runner.transport.registration` on warm
attachments.

## Risks

- Low risk. Dial mode does not change.
- A disconnected listener now wakes every 10 ms instead of every 250 ms.
Each wake is one non-blocking `accept` and a non-blocking host-control
read. The loop already slept in 25 ms steps.
- Failed handshakes on a listener now back off from 10 ms (10, 20, 40 …
320 ms) instead of from 250 ms. Core's own reconnect loop still paces
its redials, and the ingress requires credentials, so this does not
create a hot loop.
- The change needs a Runner release before hosts pick it up.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, trace queries).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1115.0-canary.20
2026-11-15 22:17:52 +00:00
Devin FoleyandPaperclip 304e40fe9c fix(auth): report effective Cloud owner access to the board UI (#21817)
## Thinking Path

> - Paperclip manages AI agents and the environments where they run.
> - Cloud owners can receive instance-admin rights for their own
dedicated instance through the existing owner setting.
> - Authentication computes these rights for each request without
storing an instance-admin role.
> - The board UI reads `/api/cli-auth/me` to decide whether to show
administration controls such as Tailscale.
> - That response used only stored roles, hiding controls that the Cloud
backend already authorized.
> - This change reports the effective Cloud request rights so the UI
agrees with the backend.

## Linked Issues or Issue Description

**What happened?**

A Cloud owner with `enableOwnerInstanceAdmin` enabled can successfully
read Tailscale connections, while `/api/cli-auth/me` reports
`isInstanceAdmin: false`. The Connectors page therefore hides Tailscale.
The same stale summary can hide other instance administration controls.

**Expected behavior**

The access summary should reflect the Cloud request's effective
privileges. Turning off owner elevation must remove those privileges on
the next request. Session and board-key callers must continue to use the
fresh stored-role snapshot.

**Steps to reproduce**

Sign in as a Cloud stack owner with owner instance-admin elevation
enabled and no stored instance-admin role. Open Connectors and compare
the access summary with a Tailscale connections request: the summary
reports no admin access even though the backend permits the request.

Searched existing PRs for `"cli-auth/me" cloud` and `tailscale access`;
no duplicate fix was found. Related:
[#21526](https://github.com/paperclipai/paperclip/pull/21526), which
added device sign-in and exposed this hosted permission-summary gap.

## What Changed

- `/api/cli-auth/me` uses the trusted Cloud actor's computed
`isInstanceAdmin` value for `cloud_tenant` requests.
- Session and board-key requests retain fresh stored-role checks. No
roles are persisted and no authorization gates or owner settings change.
- Regression tests cover Cloud elevation and de-elevation, non-Cloud
stored access, and rejection of anonymous/agent callers. Tailscale
documentation explains Cloud owner access.

## Verification

- Reproduced the deployed mismatch in an authenticated Cloud owner
browser session: the access summary reported false while the Tailscale
list request succeeded. No feature connection or configuration was
changed.
- Both Cloud access regression cases failed before the fix.
- `pnpm exec vitest run server/src/__tests__/cli-auth-routes.test.ts
server/src/middleware/cloud-tenant-actor.test.ts
ui/src/pages/apps/TailscaleConnector.test.tsx` passes: 88 tests.
- `pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify --explain` exits 2 and selects broad CI. All 57 checks
pass on final head `6044d5097396de95f49bcbb7ce7bbb1e445a80fa`; no checks
remain pending. Greptile scored this head 5/5 with no unresolved
comments, and the branch is conflict-free.
- `git diff --check` and a local credential/PII scan pass.
- The fix has not been deployed or accepted in a hosted browser yet.

## Risks

The summary exposes administration controls to Cloud owners already
authorized by the backend. Non-owner access and disabled elevation
remain denied by the existing authentication policy. No migration,
provider configuration, or deployment change.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex in T3 Code, with xhigh
reasoning. Context window size was not exposed. No subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the bug template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket or instance details
- [x] I have run tests locally and they pass
- [x] I have added regression tests
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no unresolved comments
- [x] I will address actionable reviewer comments before requesting
merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
canary/v2026.1115.0-canary.19
2026-11-15 12:49:54 -08:00
DottaandClaude Opus 5.5 f71aa135a0 fix(server): release sandbox leases for runs stopped before runner attach (#21788)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a `sandbox` environment run each task in a paid remote
sandbox that an environment lease tracks
> - When a run stops, the control plane must release that lease, and
terminating an agent waits until every lease of the agent is released
> - In a load test, a run stopped after its sandbox was ready but before
its runner started never released its lease, so the sandbox kept running
and the agent stayed in "terminating" forever
> - In the same test, terminating an agent whose failed run had a
scheduled same-run retry waited about 18 minutes, until the stale retry
time
> - This pull request makes lease release finish for every stop window,
records any release that must wait with a retry time, adds a hint-driven
safety net, and makes termination cancel a pending native retry at once
> - The benefit is that a stopped run never leaves a paid sandbox
running, and agent termination finishes in seconds without manual repair

## Linked Issues or Issue Description

**What happened?**
A load test with ten OpenCode agents in a `sandbox` environment stopped
its agents mid-run (pause, then terminate). Two defects appeared.
1. A run cancelled while it selected its provider pack (after the
sandbox was acquired and the workspace was prepared, before the runner
started) kept its environment lease `active` for more than 40 minutes.
The agent stayed in lifecycle `terminating`. The lifecycle worker
retried every pass and wrote `updatedAt` each time with no recorded
reason. Delete returned 409 "Complete termination before deleting the
agent".
2. A failed native run with a scheduled bootstrap retry kept its lease
until the stale retry time. Agent termination took about 18 minutes. The
run still showed "retry_scheduled" after the agent was terminated.

**Expected behavior**
A stopped run releases its sandbox lease within seconds. If release must
wait, the wait has a recorded reason and retry time, and a background
pass finishes it. Terminating an agent cancels a pending retry, releases
its lease, and reaches `terminated` promptly.

**Steps to reproduce**
1. Start a native run in a `sandbox` environment.
2. Stop it after the sandbox is acquired and the workspace is prepared,
before the runner starts.
3. Pause and then terminate the agent.
4. See that the lease stays `active` and the agent stays `terminating`.

**Paperclip version or commit**
Reproduced on `master` at 62033b9994 (load test build 2026.1106.0+2564).

**Relevant logs or output**
Root causes found in the code:
- `releaseEnvironmentLeasesForRun` returned with no record when
workspace retention refused a cancelled run. The refusal was permanent
when the run's native finalization owner had expired but was not
cleared. The owner is cleared only after `agentExecutionsHaveStopped`
sees every lease released, so the two waited on each other.
- `releaseRunLeases` silently skipped a lease whose prepared native
workspace was still "pending export", even for a run that never started
a runner and so never changed those files.
- The destroy path deferred on any busy warm session in the same
environment, also with no record.
- `stopInvocationsForAgents` returned on the first run that was still
live, so it did not release the leases of the other runs of the agent.
- The agent stop cancelled only `queued`, `running` and
`scheduled_retry` runs. A failed native run with a pending same-run
retry kept its lease until that retry fired.
- The lifecycle worker deferred a pending host step by 2 seconds every
pass, with no reason and no backoff.

## What Changed

- `releaseEnvironmentLeasesForRun` (`run-control.ts`):
- A native run with no runner launch evidence (no
`native.process_start_requested`, no runner event, no process id) skips
workspace retention and warm-session closing. No runner process exists
to fence, and no work ran against the prepared files.
- When retention is refused only because the run's native finalization
owner lease has expired, the release clears that expired owner
(compare-and-set on the exact expired owner of a cancelled run with no
result) and retries retention once. A recorded recovery then stops and
retains the sandbox, so the files stay recoverable.
- Every release that cannot finish now records
`metadata.releaseDeferral` (`reason`, `attempts`, `retryAt`,
disposition) on the active lease and publishes an `orphan-active-leases`
hint. The retry time backs off from 30 seconds to 15 minutes, then
retries hourly without end. The record keeps `failingSince` and the last
reason. A new Stop or terminate operation, or a server restart, triggers
an immediate attempt.
- `nativeWorkspaceExportPendingForLease` treats a cancelled run with no
runner launch evidence as owing no export.
- Safety net: `sweepOrphanedActiveLeases` (the existing hint- and
deadline-driven orphan-lease work) now retries the run's own release for
a due deferral or for a cancelled native run, after the native owner
lease has expired. The orphan-lease loader uses the stored `retryAt` as
the due time. No new timer or poller.
- Agent stop:
- It cancels a failed native run that has a pending same-run retry (or
an interrupted cancellation of one), through the existing
retry-cancellation path. A pause stamps `pauseStop` on those runs too,
so a budget resume can continue them.
- It releases the leases of every terminal run, and it skips a lease
whose deferral is not yet due.
- `agentInvocationStopBlocker` and `agentExecutionStopBlocker` return
why the stop must still wait.
- Lifecycle worker: a pending host or plugin step records
`pendingReason` and `pendingPasses` on the operation, and its retry
backs off from 2 seconds to a 60 second cap. When a release leaves a run
with no unsettled lease, the stopping agent's saved retry time is
cleared and the lifecycle queue is notified, so Pause or Terminate
finishes on the next pass.
- Cancelled-workspace preservation: a stopped run that never started a
runner has its prepared reference moved to
`nativeWorkspaceSyncReleasedUnchanged` when its lease is released, so
the preservation worker no longer selects it. The worker records the
same settlement itself for older runs, and waits on a live owner's exact
lease expiry.

## Verification

- Red on `master`, green on this branch (embedded PostgreSQL, fake
plugin sandbox provider):
-
`server/src/__tests__/heartbeat-lease-release-before-runner-attach.test.ts`:
agent pause then terminate, and board Stop, for each window between
sandbox acquire and runner attach (artifact discovery, pack select, pack
upload, runner launch, transport connect, session open). Each test
checks that the provider gets a stop or destroy call, the lease is
released, the run is cancelled, and the agent reaches `terminated`
within one extra worker pass. It also checks the orphan-lease safety net
for a leaked lease and a recorded deferral that the background pass
finishes when due.
-
`server/src/__tests__/heartbeat-terminate-pending-native-retry.test.ts`:
terminating an agent whose failed run has a bootstrap retry 17 minutes
away cancels the retry, releases the lease and reaches `terminated` at
once. The test covers an open task and a cancelled task.
-
`server/src/modules/agent-lifecycle/__tests__/pending-backoff.test.ts`:
a host step that stays pending records its reason, and its retry delay
grows to a 60 second cap.
- Related suites pass: run-control, recovery, orphaned active lease
sweep, environment runtime, agent lifecycle, native workspace
preservation and recovery, held comment wake.
- `server` typecheck passes.

## Risks

- A cancelled native run with no launch evidence now gets its prepared
sandbox destroyed instead of kept. The proof is the same launch evidence
that restart recovery uses.
- A launched cancelled run whose retention owes nothing is stopped and
retained, as before. A retention that throws (lock or write error, or a
live owner) now keeps the lease active with a recorded reason and retry
time, instead of a silent skip.
- Release now clears an expired native finalization owner of a cancelled
run. An old controller whose lease already expired then fails its next
owner check and stops, which is the normal lease rule.
- On the first start after deploy, the orphan-lease work releases leases
already leaked by cancelled native runs.
- Task state after pause then terminate: the run-scoped stop keeps the
task status (an open task stays open, a cancelled task stays cancelled),
and a run that held prepared files gets the existing workspace recovery
record on its stopped sandbox. No new state is added.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code search, shell, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1115.0-canary.18
2026-11-15 12:54:23 -06:00
DottaandClaude Opus 5.5 b4f71fc276 perf(recovery): stop re-checking settled workspace flags and snapshot ledger rows every pass (#21795)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs all background recovery on one Node main thread, and
each database statement costs time on it
> - The ownership reconciliation pass settles in-progress records every
30 s while it has records to watch
> - Two of its kinds listed records that can never change again: dirty
remote workspaces whose writer reached runtime, and snapshot ledger rows
that are uploaded or still held by a stale writer's epoch
> - Each such record cost a transaction, a row lock and up to four reads
on every pass, forever, and kept the 30 s watch armed on an idle
instance
> - This pull request lists only the records that the settle step can
still act on, locks the writer run only when the settle will write, and
skips one stranded-task read that can never match a native run
> - The benefit is that recovery work follows activity, not history

## Linked Issues or Issue Description

**Describe the problem**
On a hosted instance with 7 native runs, the ownership pass ran about
7.6 + 3.2 + 3.5 + 2.8 statements per second over records that cannot
change, and took a row lock on `heartbeat_runs` for each stale dirty
workspace on every pass. At 1,000 agents this becomes thousands of
statements per second.

**Steps to reproduce**
Seed about 100 dirty remote workspaces whose writer run reached runtime
and ended, 50 pending ledger rows still held by those writers, and 100
uploaded ledger rows. Run the ownership reconciliation worker. The new
census test
`server/src/__tests__/recovery-settled-records-census.test.ts` does this
on the embedded database.

**Expected behavior**
A settled record costs zero statements on later passes. An idle instance
runs no SQL from this pass.

**Actual behavior**
On master, one pass costs 714 statements and 150 row locks. An idle half
hour costs 40,800 statements and 9,000 row locks (the settle watch
re-checks every 30 s).

Refs #21765 (the stranded-assignment hint scoping that this rebases on)

## What Changed

- `remote_workspace_unsaved_files` lists a dirty workspace only when its
writer run can still be a preparation-only writer: stage `preparing`, no
`runtime_mode_resolved_at`, no process, session or runner id, no
finalization row and no work-started event (or no writer at all). The
SQL lives with the settle step in
`remote-workspace-unsaved-files-settlement.ts`.
- The unsaved-files settle reads the writer run without a lock. It takes
`for update` only after every check passes, just before it writes, and
checks the locked row again.
- `remote_snapshot_ledger` lists only `pending` rows with no object key,
whose operation is a workspace save, and whose workspace epoch has moved
past the row's writer. Uploaded rows belong to snapshot retention, as
before.
- `terminalRunSettlingUntil` in `recovery/service.ts` skips its
`heartbeat_runs` read when the latest run already in memory is not a
legacy run of another server boot. `getLatestIssueRun` now also selects
`runtime_mode` and `controller_boot_id`. The live-execution check still
runs first.
- New census test with a staging-shaped fixture, a statement counter
grouped by statement shape and caller, and prompt-pickup checks. New
red/green test for the stranded-task lease read.

Census (embedded PostgreSQL, same fixture, base 531db3c10b vs this
branch):

| Window | Master statements | Master row locks | Branch statements |
Branch row locks |
| --- | --- | --- | --- | --- |
| One full ownership pass | 714 | 150 | 62 | 0 |
| One settle-watch re-check | 681 | 150 | 29 | 0 |
| Idle 30 min (real worker) | 40,800 | 9,000 | 0 | 0 |
| Active, 7 live runs hinting every 10 s | 47,257 per 30 min | 9,000 |
2,763 per 10 min (about 8,300 per 30 min, of which about 4,200 are the
test's own hint writers) | 0 |

Confirmed callers on master (per full pass): `select
execution_workspace_snapshots` full row at
`remote-snapshot-ledger-settlement.ts:20` (150), `select heartbeat_runs
... for update` at `remote-workspace-unsaved-files-settlement.ts:20`
(100), `select workspace_operations ... for update` at
`remote-snapshot-ledger-settlement.ts:25` (50), `select
execution_workspaces.metadata` at
`remote-snapshot-ledger-settlement.ts:36` (50). The
`execution_replacement` queue costs 2 statements per wake over the same
history (discovery and the deadline read), so it is not the source of
the snapshot or lease reads in this shape.

## Verification

- `cd server && npx vitest run
src/__tests__/recovery-settled-records-census.test.ts
src/__tests__/issue-recovery-terminal-run-settling.test.ts` (fails on
master: 714/40,800 statements and the native lease read; passes here)
- `cd server && npx vitest run
src/__tests__/remote-snapshot-ledger-settlement.test.ts
src/__tests__/ownership-reconciliation-report.test.ts` (51 passed,
unchanged)
- `cd server && npx vitest run
src/__tests__/recovery-deadline-acceptance.test.ts
src/__tests__/legacy-continuation-authority.test.ts
src/__tests__/heartbeat-process-recovery.test.ts` (502 passed)
- The census test also checks that a preparation-only workspace is still
settled in the first pass, an abandoned pending ledger row is still
retired, a ledger row held by a stale writer is retired in the next pass
once the workspace epoch moves, and an `active` run lease whose run
ended is still listed and released.

## Risks

- The report shows fewer in-progress records for these two kinds.
Records that the settle step can never change (a dirty flag whose writer
reached runtime, an uploaded ledger row) are no longer listed, so they
no longer appear as `I2_settled` violations in the activity log. On
master each process wrote one such violation per record.
- A pending ledger row held by its stale writer's epoch is no longer
re-checked every 30 s. It is found by the next full pass after the
workspace epoch moves. A new writer or an epoch abandonment both come
with run signals that wake the pass, but on an instance with no other
activity this can now wait until the next wake instead of at most 30 s.
- A dirty workspace whose writer run row is missing is no longer listed.
On master it was re-checked forever and the settle refused it every
time.
- No schema, API or UI change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:13:18 -06:00
c2e7df2dcc fix(workspaces): use project storage for a fresh retry after unowned task storage (#20756)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A project can use a storage provider plugin. That plugin prepares
each task's git worktree
> - A board Retry after a failed run can start a fresh attempt. The
fresh attempt starts from the project source
> - `preparePluginWorkspace` returns `null` when the task is still bound
to a workspace that no storage provider owns. It does this also for a
fresh attempt
> - On a plugin task environment, the host then reserves an empty
adapter-managed "Fresh task workspace" with no repository. The runner
fails in it. The next Retry finds that empty workspace bound to the
task, returns `null` again and reserves another empty workspace. The
task can never start again
> - This pull request lets a fresh attempt use the project storage
provider even when the previous workspace is not provider-owned
> - The benefit is that one Retry gives the task a normal provider
worktree again

## Linked Issues or Issue Description

**What happened?**
Tasks moved off an older execution environment kept a binding to a
workspace that no storage provider owned. A board Retry selected
`fresh_isolated_workspace`. `preparePluginWorkspace` returned `null`
because the bound workspace was not plugin-owned. The host fell back to
`reserveFreshPluginTaskWorkspace`, which made an empty `adapter_managed`
workspace (strategy `project_primary`, no repository, no recovery
snapshots). The run failed in about one second with "Plugin-provided
Runner execution operation unavailable; reconcile the same task before
retrying". Each later Retry made another empty workspace, chained to the
last one.

**Steps to reproduce**
1. Use a project with a storage provider plugin that supports fresh
workspaces.
2. Bind a task to a workspace that the provider does not own (for
example an adapter-managed workspace).
3. Call `preparePluginWorkspace` for a fresh attempt with that workspace
as the previous workspace.

**Expected behavior**
The storage provider prepares a new worktree for the fresh attempt.

**Actual behavior**
`preparePluginWorkspace` returns `null` and never calls the provider.

## What Changed

- `server/src/services/plugin-workspaces.ts`: the early `return null`
for a previous workspace that no provider owns no longer applies to a
fresh attempt. The fresh attempt then uses the project's storage
provider, as for a task with no bound workspace.
- `server/src/__tests__/plugin-fresh-workspaces.integration.test.ts`:
new test "prepares provider storage for a fresh retry that follows a
workspace the provider does not own".

## Verification

- The first commit adds only the test. CI on that commit (`d09ad235`,
[failed
job](https://github.com/paperclipai/paperclip/actions/runs/42976154239/job/130393285731))
shows only this test fail on the current code (`preparePluginWorkspace`
returns `null`).
- The second commit adds the fix. CI shows the test pass.
- The other tests in the file keep the fresh-attempt guarantees: an
older provider still fails closed, a changed binding revision still
stops the call, and a lost reply still reconciles to the same operation.
- I did not run the suite locally. The test needs embedded Postgres, and
this environment does not install packages. CI runs it.

## Risks

- Low to medium risk. A fresh attempt in a project with a storage
provider now uses that provider even when the previous workspace was
local or adapter-managed. Before, such an attempt used the fallback
path. This matches the project's storage policy, and a fresh attempt
does not reuse the previous workspace's files.
- Projects without a storage provider are not affected: the function
still returns `null` for them.
- Non-fresh preparation is not affected.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
(shell, file edits). Standard reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: lucktastic <13757243+lucktastic@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:10:25 -08:00
DottaandClaude Opus 5.5 c6cdd57f7a perf(native-runtime): reuse a same-content image provider pack across builds (#21784)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote agent run needs the provider pack (Node, OpenCode and the
Runner bridge) inside its sandbox before the provider can start
> - The server reuses a pack that is installed in the sandbox image only
when its full manifest equals the server's own manifest
> - The manifest includes `runnerSourceRevision`, which is the commit
that built the server, so an image pack fails this check on every later
server build, also when every file in the pack is the same
> - The run then uploads the whole pack again, which cost about 15 s per
task in a recent load test
> - This pull request accepts an image pack whose sealed content equals
the expected pack and that differs only in build provenance, and caches
verified archive hashes in the server process
> - The benefit is that a sandbox image stays usable across server
releases that do not change the pack, and the server does less hashing
on its main thread

## Linked Issues or Issue Description

Refs #21781 (merged), which names the failed check on the run spans.

**What existing behavior does this improve?**
Selection of the remote provider pack for a new sandbox
(`provider_pack.select`).

**Subsystem affected**
Server: native runtime, remote provider pack verification and upload.

**Current behavior**
`provider_pack.verify_preinstalled` compares the complete manifest of
the image pack with the server's manifest. Any difference fails,
including `runnerSourceRevision` and `runnerInputsDigest`, which only
record where the pack was built. The run then uploads the full pack. The
server also hashes the same archive or pack directory again for every
task.

**Proposed behavior**
When the run is not bound to a saved pack and the operator did not
select a custom pack, the remote verification accepts an image pack when
(1) its own digest seals its payload, and (2) its schema and payload
equal the expected pack after the two provenance fields are removed.
Every artifact, the `dist` tree, the candidate trees, the package
versions, the Node version, the target platform and the OpenCode version
are still checked in the sandbox, against the same values. The server
checks the reported manifest again and selects it through the pack
authority, as the compatibility path already does. An upload still
happens for different content, a corrupted or partial pack, or no pack.
The server also remembers a verified transport archive (by path, inode,
size and times), so it does not hash it again for the next task.

**Reason and benefit**
In the harness (fake sandbox, 150 ms per command, 300 ms + 10 MiB/s per
transfer, 24 MiB pack), the cross-build case goes from 4.1 s and 24 MiB
uploaded to 1.1 s and 0 bytes. Other cases do not change.

**Breaking changes**
None. A saved or reattached session, and an operator-selected custom
pack, still require the exact pack identity.

## What Changed

- `remote-provider-pack-verification.ts`: new `acceptProvenance` option.
The remote script accepts a sealed pack with the same content and prints
its manifest. The server checks that manifest (`sameContentManifest`)
and returns it. A forged or different report fails with
`installed_manifest_invalid`.
- `native-session-executor.ts` (`prepareRemoteProviderPackOnTarget`):
staged and preinstalled verification use the option when
`acceptProvenanceDifferences` is set. The function returns the selected
manifest. The run caller sets the option only through
`providerPackProvenanceReuseAllowed` (not retained, no explicit custom
pack, selectable authority), and selects a different manifest through
the pack authority.
- `provider-pack-layers.ts`: `verifyArchive` skips the hash for an
archive it already verified with the same file identity.
- `remote-provider-pack.ts`: `providerPackProvenanceReuseAllowed` gate,
with tests for each case.
- `provider-pack-selection-harness.test.ts`: new harness. Uploads go
through the workspace confinement check of the Daytona sync guard. It
runs the real remote verification in local processes, with modelled
sandbox latency and transfer speed, for five cases, before and after.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/provider-pack-selection-harness.test.ts
src/services/native-runtime/provider-pack-archive.test.ts
src/services/native-runtime/remote-provider-pack-verification.test.ts
src/services/native-runtime/remote-provider-pack.test.ts
src/services/native-runtime/retained-provider-pack-source.test.ts
src/__tests__/kimi-connection-probe.test.ts` passes.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-artifact-preparation.test.ts
src/services/native-runtime/remote-provider-pack-authority.test.ts`
passes (1541 passed, 7 skipped).
- Red/green: with the source changes reverted, the harness case
`same_content_other_build` and the archive cache test fail. With the
changes, they pass.
- Harness output (load average about 9 on a shared machine; modelled
time = wall time + modelled sandbox latency):

| Case | Before | After |
| --- | --- | --- |
| Exact image pack | preinstalled, 1.1 s, 0 MiB | preinstalled, 1.0 s, 0
MiB |
| Same content, other build | uploaded, 4.1 s, 24 MiB | preinstalled,
1.1 s, 0 MiB |
| Different content | uploaded, 4.1 s, 24 MiB | uploaded, 4.0 s, 24 MiB
|
| Corrupted image pack | uploaded, 4.0 s, 24 MiB | uploaded, 4.1 s, 24
MiB |
| No image pack | uploaded, 3.7 s, 24 MiB | uploaded, 3.8 s, 24 MiB |

- The pack step runs no database statements.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- This changes which provider pack the server accepts. It accepts a
different manifest digest only when all content fields are equal and
every file is hashed in the sandbox, and only for a run that is not
bound to a saved pack. The selected manifest goes through the existing
pack authority, so launch profiles and receipts bind the installed
pack's own identity.
- The archive hash cache trusts each archive file's identity (inode,
size, times) on the server. The sandbox still checks the uploaded bundle
digest and verifies every uploaded file, so a changed local file still
fails closed.
- This does not help when the image pack has other content (for example
a different contract digest). See the hand-off notes for the image
build.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1115.0-canary.17
2026-11-15 12:07:56 -06:00
DottaandClaude Opus 5.5 b5447432bd fix(ui): keep the active company on Cases and Status navigation links (#21794)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI builds company-prefixed links, such as `/PAP/cases`,
from short paths in the sidebar
> - The Cases and Status links use `/cases` and `/status`, but these
roots were not in the board route list
> - The router read `cases` as a company prefix and left the link
unprefixed, so the link only worked through the unprefixed redirect
> - When a person clicked Cases before the company list loaded, the
redirect picked an arbitrary company and showed its empty case list
> - The same race made `tests/e2e/cases-read-only.spec.ts` fail in "ci /
e2e shard (2/8)" on unrelated PRs (#21683, #21715, #21765)
> - This pull request adds both roots to the board route list and makes
the e2e test force the race
> - The benefit is that Cases and Status always open in the company the
person is in, and the flaky e2e test becomes deterministic

## Linked Issues or Issue Description

Refs #21683
Refs #21715
Refs #21765

**What happened?**
The read-only Cases e2e test failed about once per day with `Expected:
3, Received: 0` links in the case list (job 132875164249 on #21765, job
132680942843 on #21715). The failure screenshot shows the sidebar of a
different company ("Passive artifacts …", created by the previous spec)
and "No cases yet". The server log shows that the page asked for the
cases of the other company and never for the test company.

**Expected behavior**
A click on the Cases or Status link opens that page for the company in
the URL, also when the click happens before the company list has loaded.

**Steps to reproduce**
1. Have two or more companies.
2. Open `/<PREFIX>/issues` on a slow connection (the e2e test holds `GET
/api/companies`).
3. Before the company list loads, open More and click Cases.
4. The URL becomes `/cases`. When the list loads, the app redirects to
`/<OTHER>/cases` and shows another company's empty case list.

**Paperclip version or commit**
`master` at 46e5e4450f.

## What Changed

- `ui/src/lib/company-routes.ts`: add `cases` and `status` to
`BOARD_ROUTE_ROOTS`, so `applyCompanyPrefix("/cases", "PAP")` returns
`/PAP/cases`. Remove the part of the `caseHref` doc comment that
described the old behavior.
- `ui/src/lib/company-routes.test.ts`: unit test for both roots (red
before the fix, green after).
- `tests/e2e/cases-read-only.spec.ts`: hold `GET /api/companies` until
the Cases link is clicked, then require the URL `/<PREFIX>/cases`.
Without the fix the URL is `/cases`, so the test fails every time
instead of sometimes.

## Verification

- `cd ui && npx vitest run src/lib/company-routes.test.ts
src/App.cases-routing.test.tsx
src/lib/router.company-task-link.test.tsx`: 23 passed. The new test
failed on `master` (2 failed).
- E2E, before the fix (spec change only): `npx playwright test --config
tests/e2e/playwright.config.ts tests/e2e/cases-read-only.spec.ts
--repeat-each=5`: 5 of 5 failed with `Received string:
"http://127.0.0.1:3199/cases"`.
- E2E, after the fix: same command with `--repeat-each=50` on a loaded
machine (load average 6 to 12): 50 of 50 passed.
- The e2e config starts its own server on port 3199
(`reuseExistingServer: false`) with a fresh `mkdtemp` home and its own
embedded PostgreSQL data directory, so these runs did not touch any
other instance.

## Risks

- Low risk. A company whose issue prefix is exactly `CASES` or `STATUS`
now gets the same treatment as a company named after any other board
route (for example `TASKS`): a bare `/CASES` path is read as the Cases
page. The existing `/<PREFIX>/issues` exception still applies.
- No API, schema, or migration change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, local Playwright runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1115.0-canary.16
2026-11-15 11:15:17 -06:00
DottaandClaude Opus 5.5 df9914dbd9 fix(native): let the outgoing server checkpoint its idle warm session while the incoming follow-up waits for it (#21791)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap. A warm agent session stays
with the outgoing server that ran its last turn.
> - Since #21774, a follow-up that the incoming server claims during the
overlap waits (`native_controller_recovery_waiting_for_owner`) and
checks again every 5 s.
> - About 1 s after the traffic switch (#21627), the outgoing server
tries to save and close its idle warm session. The checkpoint fence
refuses that save with `native_runner_checkpoint_owner_lost`. The task's
current run is the parked follow-up. That run has its own finalization
row under another controller boot, and the fence does not accept that
kind of successor.
> - The close retry from #21782 fails the same way each time. So the
follow-up waits until the outgoing server exits, or until the
predecessor's finish time plus the idle window plus 2 min. In the CHA-5
case that was about 225 s, and the worst case is about 425 s. Speed is a
requirement.
> - This pull request records which owner a parked follow-up waits on,
and lets the fence accept the outgoing owner's final save when the
follow-up is parked on exactly that owner and has not started.
> - The benefit is that the outgoing server saves and closes the session
on its first attempt, and the follow-up continues at its next 5 s check.

## Linked Issues or Issue Description

Refs #21774, #21782, #21697, #21627, #21279

**What happened?**

During a release-swap overlap, a warm follow-up was claimed by the
incoming server while the outgoing server still owned the idle warm
session. The follow-up waited correctly. The outgoing server's idle
close failed with `native_runner_checkpoint_owner_lost` on every
attempt. The wait ended only when the outgoing server exited, about 225
s after the follow-up was created.

**Expected behavior**

The outgoing server saves and closes the session while the follow-up
waits. The follow-up sees the saved session and continues within one 5 s
check.

**Steps to reproduce**

1. Run a warm native session on a remote sandbox and let a run finish on
server A.
2. Start server B with an overlap and switch traffic to it.
3. Post a follow-up comment so that B claims the follow-up while A is
still alive.
4. A's idle close fails with `native_runner_checkpoint_owner_lost`, and
the follow-up waits until A exits.

**Paperclip version or commit**

master `d14a3b88b1`.

## What Changed

- `NativeControllerRecoveryWaitingError` carries the owner that the
follow-up waits on: the predecessor run and its controller boot.
`waitForLiveWarmSessionOwner` and the live-owner wait in
`inspectRemoteWarmSessionRecovery` set it.
- `deferNativeControllerRecovery` stores that owner in the finalization
`failureDetail.waitingForOwner`. The executor passes it through.
- `validateNativeRunnerCheckpointOwner` accepts one more successor kind:
a parked follow-up waiting on this exact owner. All of these must be
true, or the fence still refuses:
- The checkpoint holds an accepted result and has no new work (unchanged
rule).
- The follow-up finalization is `retryable_failure` with
`native_controller_recovery_waiting_for_owner`, `attempt = 0`, no
result, and no lease owner (no claim is running it).
  - `waitingForOwner` names this owner's run and controller boot.
- The follow-up run is a running native run for the same company and
task, on the same native session as the owner's run.
- The follow-up has no process fields, no own process-start events, and
no checkpoint rows.
- The fence locks the follow-up's finalization row (`for share`), so a
resumption claim and the save cannot interleave.
- Tests in `native-controller-recovery.integration.test.ts` (real
Postgres):
- The outgoing server saves and closes its idle warm session while the
incoming follow-up waits, and the follow-up resumes within one recheck.
- Negative cases: the save is still refused when the follow-up has its
own process, its own process-start event, its own checkpoint, or a live
resumption claim, or when it waits on a different owner boot or run.

## Verification

- Red on master: with the `native-runner-checkpoint.ts` change removed,
the scenario test fails with `native_runner_checkpoint_owner_lost` at
the outgoing server's first close attempt. The follow-up keeps waiting.
Green with the change.
- The negative cases pass with and without the change.
- Measured in the test scenario (warm idle window 600 s):
- Before: the follow-up waits until the outgoing server exits, or about
720 s (finish + 600 s idle window + 120 s margin).
- After: the outgoing save and close succeed on the first attempt. The
follow-up is claimable and its wait resolves at `retryAt`, 4998 ms after
it parked.
- `vitest run` of `native-controller-recovery.integration.test.ts`,
`native-runner-restart-recovery.integration.test.ts` and
`native-harness-backup-storage.integration.test.ts`: 349 passed, 14
skipped.
- `vitest run` of `native-session-executor.test.ts`,
`remote-warm-session-recovery.test.ts`,
`native-runner-checkpoint-save.test.ts`,
`native-runner-checkpoint-retirement.test.ts` and
`native-runner-checkpoint-authority.integration.test.ts`: 1566 passed, 7
skipped.
- `tsc --noEmit` for `server` passes.
- `pnpm verify` exits 2 (opaque setup dependency). Required CI supplies
the broad coverage.
- Not run: a release-swap wave. The fix is not yet measured on a live
overlap.

## Risks

- The fence is wider. It accepts only a final save with an accepted
result and no new work, and only while the current run is parked on this
exact owner and has not started. A follow-up that has a claim, a
process, process-start events or checkpoint rows still causes a refusal.
- A follow-up that a server parked before this change has no
`waitingForOwner`. The fence refuses its owner's save as before. At its
next recheck, the follow-up parks again with the owner recorded.
- While a resumption claim runs the follow-up's recheck (a short window
every 5 s), the save is refused. The close retry from #21782 then tries
again.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 11:02:31 -06:00
DottaandClaude Opus 5.5 cec8c2b3f4 perf(recovery): name the run or task in run writers' recovery hints instead of the company (#21765)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues on one Node main thread. The
`stranded-assignments` and `assigned-work-invariant` queues check
assigned tasks that have no live run. One stranded check costs about 19
database statements.
> - #21552 scoped run hints to the run's task graph. #21729 made the
5-minute backstop re-check only changed tasks. A hint with only a
company id still makes the queue re-check every open task of the
company, 25 per pass.
> - Under load on staging (7 native runs), the stranded check ran about
6.7 times per second, and its statement bundle was more than a third of
all statements. An activity fixture shows the cause: ordinary run
writers send a company-only hint once per run, and one company hint
costs about 850 checks. At 1,000 agents this work never stops.
> - This pull request makes those writers name the run (or the task)
they already know. A run that has no task now resolves to no tasks
instead of the whole company. The changed-tasks backstop no longer
counts an agent's own run heartbeat as a change to every task of that
agent.
> - The benefit: with 7 runs a minute plus one taskless run, stranded
checks drop from 18.4 to 1.0 per second and stranded-check statements
from 344 to 7 per second. Real company-wide changes (pause lifted,
budget, company status, tree holds) keep the full company pass.

## Linked Issues or Issue Description

Refs #21729, #21552, #21676, #21724

**What existing behavior does this improve?**

The issue recovery task queues (`stranded-assignments`,
`assigned-work-invariant`, `resolved-dependencies`, `stale-issue-locks`)
and the writers that wake them.

**Subsystem affected**

Server: issue recovery background work and the run lifecycle writers
that publish recovery hints.

**Current behavior**

Many run lifecycle writers know the run but publish a company-only hint,
for example `notifyExecutionRunWork(tx, { companyId })` in the native
finalizer and projection, `settleNativeTerminalWake`, the claimed-wake
settlement, and an environment lease release of a run without a task. A
queued run without a task publishes a company hint. A graph hint for a
run without a task resolves to "whole company". A comment sends a
company hint to `assigned-work-invariant` and `resolved-dependencies`.
Each of these re-checks all open tasks of the company. Also, the
changed-tasks backstop treats any change to the assignee agent's row as
a change to all of that agent's tasks, and every run start and finish
writes that row.

**Proposed behavior**

A writer that knows the run sends the run's graph hint. A writer that
knows the task sends the task hint. A run without a task affects no task
queue entry. An agent row change that is only the agent's own run
heartbeat does not mark its other tasks changed. Company-wide facts keep
the company hint.

**Reason and benefit**

The work is now proportional to activity, not to (activity x history).
See Verification.

## What Changed

- `server/src/services/issue-recovery-notifications.ts`:
- `notifyIssueRecovery` with a task sends the task hint to all four task
queues (before: only to `stranded-assignments` and `stale-issue-locks`;
`assigned-work-invariant` and `resolved-dependencies` got the company
hint). Only comments (`issue-comment-writes.ts`) call it with a task.
- `resolveIssueGraph` returns `[]` when a run-only graph has no task. A
missing task seed still returns `null` (whole company).
- `server/src/services/recovery/work.ts`: a load for an empty graph
returns no rows. The `assigned-work-invariant` queue emits the same
`check` debug event as `stranded-assignments` (with
`PAPERCLIP_DEBUG_WORK_SIGNALS=assigned-work-invariant`), with the load
reason.
- `server/src/services/recovery/changed-tasks.ts`: the assignee-agent
part of the changed-tasks condition skips an agent whose last write was
its run heartbeat (`last_heartbeat_at = updated_at`). A resume, pause or
configuration write (no heartbeat stamp) still counts.
- `server/src/services/heartbeat/queue.ts`: a queued run without a task
gets its id before the insert and publishes a run hint, not a company
hint. The daily-cap comment release publishes the run's graph hint.
- `server/src/services/environments.ts`, `work-intent-outbox.ts`,
`chat-inbound-wake-notice-notifications.ts`: a lease release passes its
run id. `promoteDeferredWorkIntents` scopes `lock_released` with only a
run to that run's graph. `notifyChatInboundWakeNotices` accepts an
optional `runId` for its settlement hint.
- Native and recovery run writers now pass the run id they already have
to `notifyExecutionRunWork`, `writeExecutionControl` or
`notifyExternalConversationSettlement`: `native-run-finalizer.ts` (3),
`native-run-projection.ts` (2), `native-session-executor.ts` (3),
`wake-request-claimed-settlement.ts`, `native-superseded-retry.ts`,
`native-safe-replacement.ts`, `native-restart-recovery.ts` (2),
`native-failed-workspace-recovery.ts`,
`native-recovery-preparation-failure.ts`,
`execution-control-reconciliation.ts`,
`execution-recovery-resolution.ts` (2). These are the native equivalents
of the legacy run paths in `heartbeat/queue.ts`, which already pass the
run id.
- Tests:
- `server/src/__tests__/issue-recovery-activity-hints.test.ts` (new):
staging-shaped company (780 in-review tasks with succeeded native runs,
40 in progress, 30 to-do, 50 agents) on embedded PostgreSQL, the real
recovery, dispatch and retry-cancellation workers, a fake clock, and
real process-adapter runs. Each run is woken by a user comment through
`heartbeat.wakeup`, starts, calls back to post an agent comment and move
the task to `in_review`, and finishes. One (or five) taskless runs a
minute run beside them. It counts checks by queue and reason, task-queue
signals by caller, and statements by owner. It asserts the workload
(runs started and succeeded, agent comments, run events, zero agent
write failures, at most 10% failed runs), then no company check and no
company-only task-queue signal. In CI it runs the 7-runs-a-minute
workload for 4 minutes (about 1 minute of real time). With
`RECOVERY_ACTIVITY_REPORT=<file>` it runs both workloads at full length,
attributes statements by stack, and writes the numbers below to that
file.
- `server/src/__tests__/issue-recovery-deadlines.test.ts`: pickup tests
for each narrowed writer (see Verification).

### Publishers of company-only (or `*`) hints to the recovery task
queues

"Per" is how often the writer fires on a busy instance. "Known" says
whether the run or task is known at the call site.

Narrowed in this pull request:

| Call site | Event | Per | Known | Now |
| --- | --- | --- | --- | --- |
| `heartbeat/queue.ts:5396` `enqueueWakeup` | queue a run without a task
(timer, manual, chat without a task) | each taskless run | run (id
generated) | run graph (no tasks) |
| `environments.ts:1780` `releaseLease` ->
`notifyChatInboundWakeNotices` | environment lease released | each run
without a task | run | run graph |
| `environments.ts:1792` `releaseLease` ->
`promoteDeferredWorkIntents(lock_released)` | lease released | each run
without a task | run | run graph |
| `environments.ts:1934` `releaseLeasesForRun` | leases of a run
released | each run without a task | run | run graph |
| `issue-comment-writes.ts:12` (`assigned-work-invariant`,
`resolved-dependencies` only) | comment | each comment | task | task |
| `native-runtime/native-run-projection.ts:54`
`settleNativeTerminalWake` | native run's wake claim settled | each
native run finish | run | run graph |
| `native-runtime/native-run-projection.ts:177` | native run status
projected | each native run finish | run | run graph |
| `native-runtime/native-run-finalizer.ts:408`, `:629`, `:2080` | native
finalization terminal / failure / projection | each native run finish |
run | run graph |
| `native-runtime/native-session-executor.ts:13018`, `:13085`, `:13284`
| native execution control deadline, cancelled retry, pre-result
settlement | each native run end | run | run graph |
| `recovery/wake-request-claimed-settlement.ts:67` | claimed wake
settled after the run | each run whose claim is reconciled | run | run
graph |
| `heartbeat/queue.ts:590` | daily-cap comment key release | each run
cancelled at the daily cap | run | run graph |
| `native-runtime/native-superseded-retry.ts:42`,
`native-safe-replacement.ts:613`, `native-restart-recovery.ts:1033`,
`:1935`, `native-failed-workspace-recovery.ts:164`,
`native-recovery-preparation-failure.ts:30`,
`execution-control-reconciliation.ts:270`,
`execution-recovery-resolution.ts:830`, `:965` | native recovery
terminal writes | each recovered run | run | run graph |
| `resolveIssueGraph` for any run-only hint of a run without a task (for
example `setWakeupStatus`, `run-lifecycle.ts:1570`) | run lifecycle |
each taskless run event | run | no tasks |

Left with the company hint (company-wide fact, or the graph is not
known):

| Call site | Event | Per | Why kept |
| --- | --- | --- | --- |
| `budgets.ts:371`, `:386`, `:429`, `:437`, `:454` | budget policy
changed, scope paused or resumed | operator or budget event |
eligibility of every task can change |
| `work-intent-outbox.ts:79` with `pause_lifted`, `tree_hold_released`,
`scheduling_resumed`, `reconciler_backstop`, and `accounting_settled`
with an active hard stop | hold release | operator event;
`accounting_settled` once per run only with a hard-stop policy | can
admit any task; guarded by "keeps the company pass for hold releases
that can admit any task" |
| `companies.ts:77`, `:350`, `:353`, `:361`, `:512`, `:522` | company
created, paused, resumed, archived | operator | company-wide |
| `agent-lifecycle.ts:65`, `:72`, `agent-lifecycle-driver.ts:29`,
`agent-configuration.ts:130`, `:159`, `agent-deletion.ts:61` | agent
paused, resumed, configured, deleted | operator | can change many tasks,
and other agents' review paths |
| `access.ts:570`, `:752`, `:912`, `:926`, `:960`, `:975`, `:1175`,
`board-claim.ts:129`, `:142`, `middleware/auth.ts:706` | membership and
authority changes | operator, first sign-in | company authority |
| `chat-channels/endpoints.ts:461`, `identity-links.ts:323`,
`resource-reconciliation.ts:290`, `slack/setup-services.ts:50`,
`chat-github-management.ts:182`, `:352`, `chat-work-notifications.ts:50`
| chat authority changes | operator | company authority |
| `instance-settings.ts:666`, `:669`, `:680` (`*`) | instance setting
changed | operator | instance-wide |
| `issue-tree-control.ts:809`, `:889`, `:1236`, `:1339` | tree hold
created or released | operator | many tasks |
| `company-deletion.ts:83`, `company-portability.ts:6560` | company
delete, import | rare | many tasks |
| `issues.ts:8015` (blockers replaced), `:11165` (task created),
`:12685`, `:12699` (task deleted), `:12991`, `:14618` | task graph
changes | per task create or blocker edit | graph moves; #21552 kept
these |
| `documents.ts:615`, `:673`, `:757`, `:808`, `:860` | document delete
and restore | rare | #21552 kept these |
| `task-watchdogs.ts:854`, `:1841` | watchdog changes | rare | #21552
kept these |
| `system-recovery.ts:17`, `:70`, `task-retry.ts:294`,
`workspace-recovery-capture-policy.ts:59`,
`workspace-recovery-stop-proof.ts:122` | system and workspace recovery |
per recovery | not shown to stay in one graph |
| `agent-conversations.ts:607`,
`chat-channels/telegram/attachment-failure.ts:83`,
`native-runtime/chat-attachment-reuse.ts:1754`,
`native-runner-file-handoff.ts:1317` | conversation and attachment paths
| per failure | not shown to stay in one graph |
| `wake-queue/adapters/postgres.ts:376`, `:410`, `:555`, `:1169`,
`routes/issues.ts:14552`, `issues.ts:12070`, `:12171`,
`heartbeat/run-control.ts:1128`, `:1770`, `projects.ts:100`, `:1051`,
`environments.ts:1617` (`notifyChatInboundWakeNotices` without a task) |
saved notice and wake changes | per manual wake or project change |
company admission state; not reached by run activity in the fixture |
| `heartbeat/run-state.ts:1746`,
`native-harness-backup-expiration.ts:356`, `:526`,
`explicit-native-continuation.ts:705` | session goal clear, backup
expiry, explicit continuation | rare | no run id at the call site |
| `fast-responses.ts:201`, `:559`, `:672`, `:1053`,
`chat-github-response-comments.ts:62`, `:229`, `:440`,
`chat-github-reviews.ts:1063` (`notifyChatResponseAvailability` with a
delivery only) | chat response availability | per chat response | no
task at the call site; not measured here (chat paths are not in the
fixture) |

Per-run-event path: appending run events, saving native checkpoints,
renewing leases and streaming deltas publish no recovery task-queue
hint. `heartbeat-run-events.ts` reaches only live-run recovery queues
(`native.process_*`, `turn.*` events), chat progress, response
presentation, and `notifyNativeStatusReassessment` for control-plane
evidence events only. Checkpoints and lease renewals reach
`notifyLiveRunRecovery`, which does not include the task queues. A run's
start and finish publish its graph hint from several writers (about 20
task-queue signals per legacy run in the fixture, for example
`notifyChatRunMilestones` in `run-identity.ts` activation,
`claimQueuedRun`, `run-completion.ts`, `setWakeupStatus`). They coalesce
into about 3 checks of the run's own task per run. That is not one hint,
but it is bounded by the run's graph. This pull request does not change
it.

## Verification

- Activity fixture (`issue-recovery-activity-hints.test.ts`), checks per
second by queue and reason, and statements per second. Each workload
minute is exactly 60 s of fake-clock time, and the rates use the elapsed
fake-clock time. "Before" is this branch's test against the
`origin/master` service files. Both versions did the same work (7
runs/min: 150 runs started, 143 to 145 succeeded, 133 to 135 agent
comments, about 1,280 run events; 50 runs/min: 629 to 630 started, 620
to 622 succeeded, about 590 agent comments, about 5,550 run events). The
test now asserts this workload and zero agent write failures before it
accepts the counts.

  | | master | this branch |
  | --- | --- | --- |
| 7 runs/min + 1 taskless run/min (600 s), stranded checks/s | 18.30
company + 0.07 hint | 0.91 hint + 0.06 backstop, 0 company |
  | 7 runs/min, statements/s in stranded checks | 343.9 | 7.2 |
  | 7 runs/min, all statements/s | 577.5 | 220.4 |
  | 7 runs/min, company-only task-queue signals | 710 | 0 |
| 50 runs/min + 5 taskless runs/min (360 s), stranded checks/s | 16.86
company + 0.31 hint | 1.05 hint, 0 company |
  | 50 runs/min, statements/s in stranded checks | 314.0 | 7.4 |
  | 50 runs/min, all statements/s | 1,717.6 | 1,387.5 |
  | 50 runs/min, company-only task-queue signals | 2,864 | 0 |

"Statements in stranded checks" counts statements whose stack includes
`reconcileStrandedAssignedIssues`. The other statements are the runs
themselves. Recovery load queries (graph lookups and hint loads) are
12.5 to 17.2/s at 7 runs and 69 to 72/s at 50 runs in both versions.
`stale-issue-locks` saved deadlines check about 1 to 1.6 tasks/s in both
versions.
- Without the taskless runs, master already showed the backstop degrade
under activity: with 7 runs a minute, 2.46 backstop checks/s, because
every run writes its agent's row and marks all of that agent's tasks
changed. With this branch: 0.27/s.
- New tests in `issue-recovery-deadlines.test.ts`, red on
`origin/master` service files and green here:
- "re-checks a run's own tasks for its settlement, lease and dispatch
writers, and nothing for a run without a task": for `notifyRunDispatch`,
`notifyExecutionRunWork`, `notifyExternalConversationSettlement`,
`notifyChatInboundWakeNotices` and
`promoteDeferredWorkIntents(lock_released)` with a run id, the run's
graph (task, parent, dependent) is checked within one pass and none of
300 backlog tasks is. For a run without a task, no task is checked. A
comment checks only its task in `stranded-assignments` and
`assigned-work-invariant`.
- "counts an agent's own run heartbeat as no change to its other tasks,
and a resume as a change".
- "resolves a run without a task to no tasks and a missing task to the
whole company".
- Unchanged and passing: the #21729 pickup tests in
`issue-recovery-idle-backstop.test.ts` (finished run, user comment,
raised budget, resumed agent within a minute; missed hint within the
backstop bound; outside change within the hourly pass), the #21552 tests
in `issue-recovery-deadlines.test.ts` (including "keeps the company pass
for hold releases that can admit any task" and "falls back to the
company pass when a run affects more tasks than a hint names").
- Regression, embedded PostgreSQL: `issue-recovery-deadlines`,
`issue-recovery-work-debug-attribution`, `issue-recovery-signal-fanout`,
`issue-recovery-work-lifecycle` (42 tests);
`native-run-projection-work`, `native-recovery-notice-work`,
`muse-terminal-settlement`, `heartbeat-held-comment-wake`,
`superseded-workspace-export-recovery`,
`native-recovery-preparation-failure`, `upgrade-readiness-post-result`,
`chat-wake-conversation-batch`, `environment-service`,
`retained-native-session-cleanup-work`, `slack-conversation-lifecycle`,
`workspace-recovery-dependency-finalize`,
`native-failed-run-owner-release`, `chat-run-milestone-work` (462
tests).
- `tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- **Recovery time bounds. The owner must see this.** No hint-driven
pickup becomes slower for a task in the run's graph. Two missed-hint
bounds change:
- An agent change that sends no hint and is followed by a run heartbeat
of the same agent inside the 5-minute window (for example a resume whose
hint was lost, after which the agent ran another task) is now found by
the hourly full pass, not the 5-minute changed-tasks pass. A normal
resume sends `pause_lifted` and is checked at once (tested).
- A task outside the run's graph whose recovery depended on the end of a
taskless run, or on the native finish writers above, is now found by its
own hint, the 5-minute changed-tasks pass (its own rows), or the hourly
full pass. Before, any run finish re-checked the whole company. I found
no recovery decision that waits on another run of the agent: concurrency
waits are queued runs that dispatch resumes.
- A run-only hint for a run row that does not exist resolves to no tasks
(it was the whole company). A run row is not deleted while its tasks are
open, except by company deletion, which sends the company hint.
- Native finish writers now send run-scoped hints to the live-run
recovery queues too (`notifyLiveRunRecovery`,
`notifyLiveResourceRecovery`, attention retention). The legacy run paths
already send these with the run id.
- The fixture uses process-adapter runs, not native runs. The native
writers are narrowed by code reading. Their effect on the hosted staging
instance is not verified.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:47:34 -06:00
DottaandClaude Opus 5.5 10ddb26dbb test(server): accept dash's exit code for a sandbox-denied shell write (#21792)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner workspace bridge runs agent commands in a `bwrap` sandbox
that binds only `/usr`, `/bin`, `/lib` and the task workspace
> - A server test checks that a sandboxed command cannot read or write a
file outside the workspace
> - The test only runs where `/usr/bin/bwrap` exists. CI jobs that the
gate sends to GitHub's `ubuntu-latest` now get the `ubuntu-26.04` image,
which has `bwrap`, so the test runs on CI for the first time
> - There it fails on every run, because it expects the write attempt to
exit with 1, but Ubuntu's `/bin/sh` (dash) exits with 2 when a
redirection fails
> - This pull request checks that the write fails and that the file is
unchanged, without depending on the shell's exit code
> - The benefit is that the "General tests (server (7/12))" job passes
again on the new image, and the test still proves the sandbox denial

## Linked Issues or Issue Description

**What happened?**
`server/src/services/native-runtime/runner-bridge-tools.test.ts` ›
`Runner workspace bridge > runs useful commands but denies files outside
the workspace and injected credentials` fails in "ci / General tests
(server (7/12))" on unrelated branches, on the first run and the retry:
- PR #21765, head `98197c87e9`
- branch `feat/run-loop-detector`, run 43732047843, head `b8c6e27429`,
job 132866487949, image `ubuntu-26.04` version 20261111.196.1

```
AssertionError: expected { exitCode: 2, signal: null, …(3) } to match object { exitCode: 1 }
 ❯ src/services/native-runtime/runner-bridge-tools.test.ts:180:19
```

On jobs that ran on the RunsOn `ubuntu24` image there is no
`/usr/bin/bwrap`, so the test is skipped (7 skipped instead of 6).

**Expected behavior**
The test passes wherever the command sandbox is available, and it fails
if the sandboxed command can read or change the outside file.

**Steps to reproduce**
1. In a Debian or Ubuntu container, run `/bin/sh -c 'printf stolen >
"$1"' sh /nonexistent/dir/private.txt`.
2. dash prints `cannot create /nonexistent/dir/private.txt: Directory
nonexistent` and exits with 2. bash exits with 1. `cat` of the same path
exits with 1.
3. In the sandbox, the outside temporary directory is not bound, so the
path does not exist. This is the same case.

**Root cause**
A stale test expectation, exposed by a CI image change. No commit on
`master` changed the bridge, the sandbox or this test today (the last
change to the test is from 2026-11-08). The test expected the shell
write to exit with 1. That is bash behavior. On Ubuntu, `/bin/sh` is
dash, which exits with 2 for a failed redirection. Until today no CI
image had `bwrap`, so the test never ran on CI.

**Production impact**
None. The sandbox denies the access as intended: the `cat` step on the
same run exits with 1 and its output does not contain the secret, and
the write fails with `Directory nonexistent` because the outside
directory is not visible in the sandbox. This is not a sandbox
regression.

## What Changed

- The write step now checks that the outside file still contains
`PRIVATE` first, then that the command was not stopped and exited with a
non-zero code. It no longer requires exit code 1 from the shell.
- The read step and the success step are unchanged.

## Verification

- Linux container (`node:24-bookworm-slim`, `/bin/sh -> dash`): the
write command exits with 2 under dash and 1 under bash, and both are
failures, as the test now accepts.
- `npx vitest run
src/services/native-runtime/runner-bridge-tools.test.ts` in `server/` on
macOS: 11 passed, 6 skipped (the sandbox tests need Linux with `bwrap`).
- I cannot run the sandbox test on this Mac. It runs in this pull
request's CI only when the job lands on an `ubuntu-26.04` runner.

## Risks

- Low risk. Only one test assertion changes. The test still fails if the
write succeeds, because the file content check runs first and the exit
code must be non-zero.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, GitHub CLI, Docker).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:47:19 -06:00
DottaandClaude Opus 5.5 3ee6049f71 perf(native-runtime): name why a preinstalled provider pack was not reused (#21781)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote agent run needs the provider pack (Node, OpenCode and the
Runner bridge) inside its sandbox before the provider can start
> - The server reuses a pack that is already installed in the sandbox
image, or it uploads its own pack
> - In a recent load test every task uploaded the pack, and this step
was about 43% of the time to first token
> - The compatibility check and the preinstalled-pack verification
failed, but the run records did not say which check failed
> - This pull request records the failed check on the run's performance
spans
> - The benefit is that the next run names the reason, so the image or
the check can be fixed without guesswork

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The run performance spans for provider pack selection. Today a failed
`provider_pack.compatibility` or `provider_pack.verify_preinstalled`
span only says `outcome: failed`. The code catches the error and drops
it.

**Subsystem affected**
Server: native runtime, remote provider pack selection
(`server/src/services/native-runtime`).

**Current behavior**
`provider_pack.select` records `reason: cached_packs_unverified` when it
uploads. The compatibility check throws one generic message for about
ten different conditions, and the caller swallows it. The remote
verification script throws a generic `manifest mismatch` without the
fields that differ.

**Proposed behavior**
Each failed check has a short, fixed name (for example
`installation_unprotected`, `contract_digest`, `artifact:nodeCommand`,
`manifest_mismatch:runnerSourceRevision`). The trace records this name
as the `check` attribute on the failed span. `provider_pack.select`
records the last preinstalled rejection as `check` when it uploads. The
compatibility span records `target: image` or `target: workspace`.

**Reason and benefit**
Upload of the provider pack costs about 15 s per task on the test
instance. To remove it, we must know which check rejects the image pack.
The attribute holds only fixed check names and manifest field names,
never paths or values.

**Breaking changes**
None. Error messages keep their existing text. The upload decision does
not change.

## What Changed

- `remote-provider-pack-verification.ts`: the compatibility check names
the failed condition (`failureCheck`). The remote verification script
prints a fixed `paperclip-pack-check=<name>` line, and for a manifest
mismatch it names the differing manifest fields. Missing files now give
a named check instead of a raw stack trace.
- `native-run-trace.ts`: a failed `measure` records the error's
`failureCheck` as the safe `check` attribute.
- `remote-provider-pack.ts`: `provider_pack.select` records the last
preinstalled rejection (or `no_preinstalled_pack`) when it uploads.
- `native-session-executor.ts`: the `provider_pack.compatibility` span
records `target`, the preinstalled-pack selection passes its rejection
callback, and a failed link has the `link_failed` check. No other lines
change.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-run-trace.test.ts
src/services/native-runtime/remote-provider-pack.test.ts
src/services/native-runtime/remote-provider-pack-verification.test.ts
src/__tests__/redaction.test.ts` passes (131 tests).
- Red/green: with the source changes reverted and the new tests kept, 12
tests fail. With the changes, all pass.
- The verification test runs the real remote script in a Node process: a
provenance-only difference gives
`manifest_mismatch:runnerSourceRevision`, a wrong artifact gives
`artifact_digest:nodeCommand`, and a missing manifest gives
`manifest_unreadable`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The change adds diagnostics only. The selection order, the
checks themselves and the upload path do not change.
- The `check` attribute is filtered by the existing safe-attribute list
and a strict character pattern, so it cannot carry paths or secrets.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:46:33 -06:00
DottaandClaude Opus 5.5 46e5e4450f fix(runner): leave an undispatched tool call to the next controller on restart detach (#21786)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner keeps an agent session alive while its
controller restarts, for example during a deploy, and the next
controller adopts it
> - A Paperclip tool call can be between its committed input and its
handler when the controller detaches
> - In that window the old controller journaled "The turn stopped before
this operation was dispatched" as the final, non-retryable result, but
the turn did not stop
> - The agent then got a false failure for a call that never ran, and
the next controller could not recover the call. A restart must never
fail work
> - This pull request leaves that call unsettled for the next
controller, the same as a call that was already in its handler
> - The benefit is that a controller restart no longer turns an agent's
pending tool call into a permanent error, and a CI test that failed on
four pull requests now passes deterministically

## Linked Issues or Issue Description

**What happened?**
`src/live/runnerd-codex-transport-orphaned-tool-at-attach.test.ts` ›
`settles an earlier generation's in-flight tool call at attach even when
that attach never finishes startup` failed in "ci / Verify Paperclip
Runner (vitest package (3/3))" on 2026-11-15. In each job the first try
and the CI retry both failed:
- PR #21757, head `d04825d812`, job 132824209377
- PR #21713, head `7758d56719`, job 132681892394
- PR #21729, head `c63a05a013`, job 132732839971

Every failure is the same, at line 74, right after
`detachControllerForRestart()`:

```
AssertionError: expected [ { …(8) } ] to deeply equal []
+   "type": "semantic_tool.result", "status": "pending",
+   "result": { "error": { "code": "semantic_tool_not_dispatched",
+     "message": "The turn stopped before this operation was dispatched.", "retryable": false } }
```

**Expected behavior**
When a controller detaches for restart, a tool call that it has not
dispatched stays unsettled. The next controller recovers it at attach,
the same as a call that is in its handler.

**Steps to reproduce**
1. Hold the semantic tool dispatch at a gate after the Runner commits
the tool input.
2. Call `detachControllerForRestart()`, then release the gate.
3. On `master`, the control-plane state has the
`semantic_tool_not_dispatched` result above. It is byte-identical to CI
(same command id and payload). The new test case `(still before dispatch
at detach)` does this.

**Root cause**
This is a Runner defect, not a test timing problem.
`#detachControllerOnce` sets `#closed`, resolves the closed signal and
resolves the turn-start admission with `false`. A call in
`#handleSemanticToolInput` that has not reached the handler then throws
`SemanticToolNotDispatchedError`. The durable control plane turns that
error into a journaled `semantic_tool_not_dispatched` result. That
result is correct when the turn stops. It is false when the controller
only detaches for restart, because the Runner and the turn continue. The
old test waited only for the committed input, so on a loaded CI runner
the detach sometimes came before the handler call.

**Production impact**
Yes. During a controller restart or deploy, an agent tool call in this
window got a non-retryable "turn stopped" error after the Runner was
adopted, and the next controller did not recover the call.

## What Changed

- `durable-prp-control-plane.ts`: new `SemanticToolHandedOffError`. When
the dispatcher rejects with it, the control plane journals no result.
The input stays unsettled for the next controller.
- `runnerd-codex-transport.ts`: the semantic tool callback goes through
`#dispatchSemanticToolInput`. If the call ends as not dispatched after
the transport detached for restart, it throws
`SemanticToolHandedOffError`. A normal close still journals
`semantic_tool_not_dispatched`.
- `runnerd-codex-transport-orphaned-tool-at-attach.test.ts`: the test
waits for the exact stage instead of the committed input. It runs two
cases: the call is in its handler at detach, and the call is held before
dispatch across the detach. Both then check the existing attach recovery
and the final `completed` result.

## Verification

- Red on `master`: the `(still before dispatch at detach)` case fails
every run with the CI assertion and payload.
- Green on this branch: `npx vitest run
src/live/runnerd-codex-transport-orphaned-tool-at-attach.test.ts` passes
2/2.
- Loop on macOS with 4 extra busy-loop processes (load average about 13
before them), 50 runs each: this branch 50/50 (100 test cases). The old
`master` test was 50/50 locally too, so the local loop does not show the
CI interleaving. The gate does.
- All 36 runner test files that use `detachControllerForRestart` or
`SemanticToolNotDispatchedError`: 1180/1182 pass with 3 workers under
load. The 2 failures (`runnerd-codex-transport-settlement.test.ts` and
`runnerd-acpx-source-reconciliation.test.ts`, `kill EPERM`) pass 36/36
when run alone.
- `npx tsc -p tsconfig.json --noEmit` and `npx tsc -p tsconfig.test.json
--noEmit` pass in `packages/paperclip-runner`.
- The other two static-check flakes from the same day are test-only and
are fixed in a separate pull request.

## Risks

- Medium-low risk. This changes Runner restart behavior for one window.
A call that was not dispatched now goes to the next controller's
recovery (`recoverSemanticToolInput`) instead of getting a final error.
That is the same path that already handles a call in its handler, so no
new recovery code is involved.
- If no successor ever attaches, the input stays unsettled. That is the
same state as for a call that was in its handler at detach.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:27:04 -06:00
DottaandClaude Opus 5.5 7a9177af0f test(runner): remove coreutils and pid-file timing assumptions from static checks (#21787)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner package has a "static checks" CI job that runs
`node --test` script tests
> - Two of these tests failed on unrelated pull requests on 2026-11-15
and passed on re-run
> - Each failure costs a re-run of the failed jobs and delays merges,
and it teaches people to re-run instead of read
> - One test assumed that a copied `/bin/sleep` acts as `sleep` under
any name. The other read a pid file before the child finished writing it
> - This pull request removes both assumptions, and one more instance of
the second pattern in the same test family
> - The benefit is that these tests no longer depend on the runner image
or on process scheduling

## Linked Issues or Issue Description

**What happened?**

1. `scripts/stage-runner-binary.test.mjs` failed in "ci / Verify
Paperclip Runner (static checks)" on PR #21765, head `98197c87e9`, job
132850068174:
- `atomically publishes while the previous executable is running`:
`AssertionError: 1 !== null` at line 124 (`old.child.exitCode`), after
77 ms.
- `a source rebuild selects fresh dev bytes ...`: `Expected "actual" not
to be strictly deep-equal` (line 192).
- That job ran on a GitHub-hosted `ubuntu-26.04` image (20261111.82.6).
The same tests pass on the RunsOn `ubuntu24` image, for example job
132822472729.
2. `scripts/materialize-kimi-distribution.test.mjs` › `setup
cancellation physically stops an in-flight owned child` failed in the
same job on PR #21758, head `40e38fc28a`, job 132822472729:
`AssertionError: The real child did not start` after 149 ms.

**Expected behavior**
The tests pass on every supported runner image and on a loaded runner.

**Steps to reproduce**
1. stage-runner-binary: in a Linux container, replace `/bin/sleep` and
`/bin/echo` with one multicall binary that picks the utility from
`argv[0]` and exits 1 for an unknown name (this is how uutils coreutils
works on Ubuntu 26.04). Run `node --test
scripts/stage-runner-binary.test.mjs`. On `master` both tests fail with
the CI messages, 20 of 20 runs.
2. kimi: make the child create the pid file, wait 300 ms, then write the
pid. On `master` the test fails with `The real child did not start` in
about 125 ms, the same as CI.

**Root cause**
1. On Ubuntu 26.04, `/bin/sleep` and `/bin/echo` are the same uutils
multicall binary. The test copies it to `paperclip-runnerd` and runs it.
The binary sees `argv[0] = paperclip-runnerd`, finds no utility of that
name, and exits 1. The fresh-build fixture (a copy of `echo`) also has
the same bytes as the old artifact (a copy of `sleep`), so the "fresh
bytes" assertion fails too.
2. The child wrote the pid with `writeFileSync`, which creates the file
and then writes it. The test loop broke on the first successful read. An
empty read gave `Number("") === 0`, so the loop stopped and the
assertion failed. The loop also had a fixed budget of 100 polls of 10 ms
for a Node child to start.

**Production impact**
None. Only tests change. The staging code works with any executable. The
Kimi setup cancellation code is correct: the deterministic repro shows
that the child is stopped once the test reads a real pid.

## What Changed

- `stage-runner-binary.test.mjs`: the test helper passes `argv0` to
`spawn`. The copied `sleep` runs with `argv0: "sleep"`, so a multicall
binary still acts as `sleep`. The fresh-build fixture is now a copy of
`/bin/sh` and runs `-c "exit 0"`, so its bytes always differ from the
old `sleep` artifact.
- `materialize-kimi-distribution.test.mjs`: the child writes `pid.tmp`
and renames it to `pid`, so a read never sees a partial file. The test
waits until the pid is valid or the command settles. The 3 s command
timeout and 5 s test timeout are unchanged.
- `provision-kimi-package.test.mjs`: the wait for the `download-ready`
marker continues until the child exits, instead of a fixed 1 s budget.
The existing 5 s kill timer still bounds it.
- No retries, skips or timeout changes.

## Verification

- stage-runner-binary, Linux container (`node:24-bookworm-slim`) with a
multicall `sleep`/`echo` built from a 15-line Go program that dispatches
on `argv[0]`:
  - `master`: 0/20 pass, same two assertions as CI.
  - this branch: 50/50 pass.
  - this branch with stock coreutils, and on macOS: all 9 tests pass.
- kimi setup cancellation:
  - delayed-write repro: `master` fails, this branch passes.
- loop on macOS, `master` test without extra load: 59/60 (one natural
failure). This branch: 60/60.
- loop with 4 extra busy-loop processes (load average about 13 before
them): `master` 100/100, this branch 100/100.
- `node --test scripts/stage-runner-binary.test.mjs
scripts/materialize-kimi-distribution.test.mjs
scripts/provision-kimi-package.test.mjs`: 19 pass, 1 skipped (public
download test, opt-in).
- The other CI flake from the same day,
`runnerd-codex-transport-orphaned-tool-at-attach.test.ts`, is a Runner
defect. It is fixed in a separate pull request.

## Risks

- Low risk. Only tests change.
- The rebuild test now uses `/bin/sh` as the fixture for the fresh
build. This file exists on every supported Linux and macOS runner.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, GitHub CLI, Docker).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1115.0-canary.15
2026-11-15 10:13:19 -06:00
DottaandClaude Opus 5.5 d14a3b88b1 fix(native): release an idle warm session whose Runner is proven gone, and retry a failed idle close (#21782)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes idle sessions, and closes them again before it
exits.
> - When an idle close failed, the idle-timer path kept the session
registered and never tried again, unless the sandbox lease was already
inactive.
> - The controller-restart path tried once more, but it accepted no
proof that the session was already gone. It checked a destroyed sandbox
only for a forced environment delete.
> - In a release-swap test, one idle close failed with
`runner_remote_process_identity_unavailable`. The session stayed
registered until shutdown, the shutdown close failed again, and the
sandbox was held until the host stopped it.
> - This pull request treats a failed close as complete when the Runner
is proven gone or the sandbox is stopped or destroyed, retries a failed
idle close a bounded number of times, and logs which Runner check
failed.
> - The benefit is that a dead warm session is released, and the next
failure names its cause.

## Linked Issues or Issue Description

Refs #21627

**What happened?**
In a zero-downtime release-swap test, a warm session's idle close failed
4 s after the server left new run starts. The failure was
`runner_remote_process_identity_unavailable`. The idle-timer path kept
the session registered with no retry. At SIGTERM,
`closeIdleWarmNativeSessionsForRestart` tried again and failed the same
way. The log showed "idle warm native sessions did not close" and the
restart-chaos harness counted a warm-shutdown failure. The log did not
say which Runner check failed, so the cause of the unreadable identity
is not known.

**Expected behavior**
A close whose Runner is proven gone, or whose sandbox is stopped or
destroyed, releases the session. A failed idle close is tried again a
bounded number of times. The failure text names the Runner check stage
and reason.

**Steps to reproduce**
1. Keep a warm remote session idle.
2. Make its close fail with a `RemoteRunnerRecoveryError` (for example,
`process_verification/process_not_running`).
3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result
is `failed: 1`.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`:
- New `warmCloseFailureProvesOwnerGone`. It uses existing proofs only:
the close error's Runner diagnostic is
`process_verification/process_not_running`, or a lease row for the
sandbox has a termination receipt (`hasRemoteTerminationReceipt`) with
state `destroyed`, or the warm run's own lease row has a receipt with
state `stopped`. `pending_cleanup` and a lease without a receipt prove
nothing. A lease read error counts as no proof.
- `closeIdleWarmNativeSessions` (controller restart, environment delete,
idle sleep) uses this proof for every failed close, not only for a
forced environment delete. A released Runner detach failure is
unchanged.
- The idle-timer path uses the same proof. Its earlier status-only rule
from #21504 (any lease status other than `active` or `retained`) is
removed, because `pending_cleanup` is written before the provider stop.
Without proof, it re-arms the timer and tries again, every 5 s, at most
3 times (`WARM_IDLE_CLOSE_RETRY_MS`, `WARM_IDLE_CLOSE_MAX_RETRIES`). The
log line records the attempt number.
- `describeWarmCloseFailure` appends the Runner check as
`[stage/reason/lifecycle]` when the error carries one.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
three tests.
- "an idle warm session whose close fails with
runner_remote_process_identity_unavailable and whose Runner is proven
gone is released at controller restart"
- "names the Runner check in a warm close failure and keeps an unproven
owner registered at controller restart"
  - "retries a failed idle warm close a bounded number of times"
- The #21504 idle-path test is now "forgets a warm owner whose idle
close failed only once its sandbox has a stop or destroy receipt". A
stop or destroy receipt releases the owner. `pending_cleanup`,
`released` without a receipt, and `retained` keep it.
- "reads the warm owner's own lease rows at controller restart" (a stop
receipt for this run releases the owner; `pending_cleanup`, no receipt,
or another run's receipt do not)

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: 1519
passed, 7 skipped.
- Red on master (with only the two exported constants added so the file
compiles): all three new tests fail. The first fails with `"failed": 1`.
- `npx tsc --noEmit -p server`: no errors.

## Risks

- A session is released without a checkpoint only when its Runner is
proven gone or its sandbox is stopped or destroyed. The last saved
checkpoint stays the recovery source, which is the same as the forced
environment delete path today.
- Environment delete and idle sleep now also accept the proof. Before,
they accepted only a destroyed receipt during a forced delete.
`pending_cleanup` is written before the provider stop, so it never
counts as proof here.
- The idle retry adds at most 3 more close attempts per failed session,
5 s apart.
- A failed idle close whose sandbox stop is still pending now keeps its
owner until a receipt lands or a successor retires it. Before, the
status alone released it.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:09:32 -06:00
DottaandClaude Opus 5.5 171e70c11d refactor: drop span and run-log event catalogs from the observability docs (#21779)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `doc/observability.md` and `doc/run-log-events.md` described every
OpenTelemetry span and every run-log event
> - `rules/data-paths.mdc` told authors to update
`doc/run-log-events.md` for each event, so both files only grew: 2,313
and 1,263 lines
> - Each span and event is already defined next to the code that writes
it, and its name is a literal string that a search finds in one step
> - This pull request removes the per-span and per-event catalogs and
keeps only what code does not say
> - The benefit is two short docs that do not drift from the code, and
no rule that makes them grow again

## Linked Issues or Issue Description

Refs #21670, #21700, and #21743 (the same cleanup for other docs).

**Issue type**

Outdated or confusing documentation.

**Where is the issue?**

`doc/observability.md`, `doc/run-log-events.md`, and
`rules/data-paths.mdc`

**What's wrong?**

The two docs repeated span names, span attributes, and event payloads
that the code defines. Operator setup started at line 138 of
`doc/observability.md`. The rule required a doc edit for every new
run-log event.

**Suggested fix**

Keep operator setup and the Sentry privacy contract. Point to the code
for span and event definitions. Remove the rule line that requires a doc
entry per event.

## What Changed

- `doc/observability.md` is now 619 lines. It keeps: how to enable
tracing, `service.version` resolution, scope, how to enable Sentry, DSN
delivery, environment attribution, privacy settings, server request
data, the core of "Run failure context", browser data, fail-open
behavior, the default capture set, and operator responsibilities. Kept
text is unchanged except the intro and two sentences noted below.
- Deleted from `doc/observability.md`: budget, Slack receipt, task
detail, and browser tracing notes; controller preparation phases; native
Runner spans; sandbox startup spans; sandbox duplex transport; shared
skill preparation; invocation attribution; and about 500 lines of
per-failure detail in "Run failure context".
- Replaced that per-failure detail with one paragraph. It names the
failures that `unexpected` mode keeps out of Sentry, and says the
remaining diagnostic contexts are defined in `server/src/sentry.ts`.
"Default capture set" now says the same.
- `doc/run-log-events.md` is now 23 lines. It says what the run log is,
where events are defined and appended, and keeps the payload storage
note.
- `rules/data-paths.mdc`: the run-log line now says to define an event
where it is written, not in a doc. The span-allowlist line now says the
allowlist is in the code that opens the span. The review levels and
globs do not change.
- Repointed links in two feature maps, two agent-access map files,
`tests/runner-e2e/SUITES.md`, `doc/runner-api-tools.md`, and the
Telemetry README. Updated the `honeycomb-tracing` skill and one code
comment in `packages/adapter-utils/src/http2-bridge-server.ts`.

## Verification

- A script built the new `doc/observability.md` from line ranges of the
old file and asserts the first line of each range.
- A link check on all changed Markdown files found 0 missing files and 0
missing anchors. The remaining inbound anchors (`#enabling-tracing`,
`#sentry-error-monitoring`) still exist.
- No test reads either doc.
- `node scripts/check-maps.mjs` passes.
- `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail.
- `pnpm verify` was not run. This worktree has no installed
dependencies. The only code change is a comment.

## Risks

- The privacy contract no longer lists each run-failure diagnostic
context field. `server/src/sentry.ts` is now the list. A reviewer of a
Sentry change must read that file.
- No browsable list of span names or run-log event names remains. A
search for the quoted name finds the producer.
- `.greptile/files.json` still scopes `rules/data-paths.mdc` to both
docs. Both files still exist, so it does not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code desktop, with tool
use. Earlier analysis in this session used Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:06:05 -06:00
DottaandClaude Opus 5.5 531db3c10b perf(db): keep dedicated connections for the next burst (#21769)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process runs every agent on one Node main thread, and
each new database connection costs a TLS handshake, about 0.36 ms of
SCRAM key derivation on that thread, and a type-fetch query
> - `withDedicatedDbConnection` gives lock holders a private session;
since #21693 it keeps at most 4 released sessions and closes the rest
> - When the same burst of more than 4 dedicated holders comes back,
every holder above 4 opens a new connection again
> - A harness on embedded PostgreSQL measured 504 new connections per
1,000 uses for bursts of 8 and 760 for bursts of 16
> - This pull request keeps released dedicated clients for reuse while
idle plus in-use clients stay within
`DATABASE_DEDICATED_CONNECTION_MAX`; the driver idle timeout still
closes their sessions when they are not used
> - The benefit is 8 and 16 new connections per 1,000 uses for the same
bursts, with no change to any pool maximum or to
`databaseConnectionBudget()`

## Linked Issues or Issue Description

**What is the problem?**
A staging measurement showed `SET
application_name='paperclip-runner-checkpoint'` at 3.17/s and `SET
application_name='paperclip'` at 3.86/s with 7 live runs. This looked
like connection churn. The investigation found two separate facts:

1. The `SET` statements are not new client connections. postgres.js
sends `application_name` in the startup packet, never as a `SET`
statement (`StartupMessage` in postgres.js `src/connection.js`).
PgBouncer in transaction mode sends `SET application_name` itself when
it gives a client transaction a server connection that a client with
another `application_name` used last. With one shared server pool, every
checkpoint transaction that lands after a main-pool transaction costs
one `SET`, and the reverse. The rate follows checkpoint transactions,
not connects. The checkpoint pool opens at most 3 connections per 1,000
saves (measured below).
2. Dedicated connections did churn by construction for recurring bursts
of more than 4 holders (measured below). This pull request fixes that.

**Steps to reproduce**
Run `packages/db/src/connection-reuse-census.test.ts`. It counts
`pg_stat_database.sessions` around 1,000 operations.

**Expected behavior**
A recurring burst reuses the sessions that the previous burst opened.

## What Changed

- `withDedicatedDbConnection` keeps a released client idle when fewer
than `DEDICATED_CONNECTION_IDLE_MAX` (4) are idle, as before, or when
idle plus in-use clients stay within the dedicated limit
(`DATABASE_DEDICATED_CONNECTION_MAX`, default 24). A client whose action
failed, an ended client and clients after shutdown are still closed.
- Each retained client keeps the main pool's `idle_timeout` (default 60
s), so its session closes after that idle time, as the main pool's do.
The next use reconnects.
- The worst case stays at `2 x DATABASE_DEDICATED_CONNECTION_MAX + 4`: a
new client opens only when no client is idle, and a client is kept above
4 only while idle plus in-use is within the limit.
`databaseConnectionBudget()` and every pool maximum are unchanged.
- New `connection-reuse-census.test.ts` measures connections opened per
1,000 checkpoint saves and per 1,000 dedicated uses.
- `dedicated-connection-pool.test.ts`: the burst test now checks that
the next burst reuses all sessions and that they close after the idle
timeout (1 s in the test). Before, it checked that at most 4 stayed
open.
- `doc/DATABASE.md` describes the new retention rule.

## Verification

Connections opened per 1,000 operations, embedded PostgreSQL,
`pg_stat_database.sessions` (load average about 10.5 on a shared
machine):

| Operation | Before | After |
| --- | --- | --- |
| Checkpoint saves, concurrency 1 / 7 / 24 | 1 / 3 / 3 | 1 / 3 / 3 |
| Dedicated uses, steady concurrency 1 / 4 / 8 / 16 | 1 / 4 / 8 / 16 | 1
/ 4 / 8 / 16 |
| Dedicated uses, recurring bursts of 8 | 504 | 8 |
| Dedicated uses, recurring bursts of 16 | 760 | 16 |

- `pnpm --filter @paperclipai/db exec vitest run
src/connection-reuse-census.test.ts
src/dedicated-connection-pool.test.ts
src/dedicated-connection-admission.test.ts src/client-options.test.ts`:
38 passed. The burst cases fail on `master` (504 and 760 against a bound
of 8 and 16).

## Risks

- After a burst, up to `DATABASE_DEDICATED_CONNECTION_MAX` dedicated
sessions stay open for up to `DATABASE_IDLE_TIMEOUT_SECONDS` (60 s)
instead of closing at once above 4. This stays inside the connection
budget. With `DATABASE_IDLE_TIMEOUT_SECONDS=0` they stay open until
shutdown, like the main pool.
- Retained sessions keep the existing rule: actions keep session state
transaction-scoped.
- The PgBouncer `SET application_name` statements stay. Removing them
needs one shared `application_name` for all pools, which removes the
per-pool names that operators and tests use. That is a separate
decision.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:51:55 -06:00
DottaandClaude Opus 5.5 8f1d39da79 perf(server): cache the instance settings row per process (#21768)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process runs every agent on one Node main thread, and
each database statement costs main-thread time
> - The `default` instance settings row is read on hot paths: live run
event delivery, run log redaction, the stranded-task recovery check,
runner tool calls, adapter routes and run preparation
> - A staging measurement with 7 live runs showed the full-row settings
read at 10.24 statements per second, and 3.5 to 5.4 per second when idle
> - The row changes only when an admin or a boot step writes it, so
almost every read returns the same value
> - This pull request adds a per-process read-through cache for that
row, with in-process invalidation on every write and a 2 second age
bound for changes made by another process
> - The benefit is at most one settings read per 2 seconds per server
process, whatever the load

## Linked Issues or Issue Description

**Is your feature request related to a problem?**
The full `instance_settings` row read (`select id, singleton_key,
default_environment_id, general, experimental ...`) runs at 10.24
statements per second on staging with 7 live runs, and 3.5 to 5.4 per
second at idle. It scales with run events and recovery checks, not with
settings changes.

**Describe the solution you would like**
Cache the row per process. Drop the cached row when a settings write
commits, in this process and, through the existing work-signal
notification, in the other process during a deploy overlap. Bound the
age of a cached row to 2 seconds, so any other change (a direct database
edit, a second process outside an overlap) takes effect within 2
seconds.

**Describe alternatives you have considered**
A database trigger with LISTEN/NOTIFY would also catch direct edits, but
it needs a migration, and work notifications cross processes only during
a deploy overlap today. The existing signal plus a 2 second bound keeps
the change small.

## What Changed

- New `server/src/services/instance-settings-cache.ts`: a read-through
cache keyed by the database client. Concurrent misses share one read. A
cached row is at most `INSTANCE_SETTINGS_CACHE_TTL_MS` (2000 ms) old,
measured from the start of the read. Every caller gets its own copy.
Reads through a transaction handle skip the cache.
- `instanceSettingsService` `get`, `getGeneral` and `getExperimental`
use the cache. Write paths (`update`, `updateGeneral`,
`updateExperimental`, and the feedback vote save) still read the row
fresh.
- Every settings write publishes an `instance-settings-changed` work
signal through the existing work-signal channel
(`@paperclipai/db/work-signals`). Inside a transaction it is the first
signal of the write, so its `pg_notify` payload goes out before the
other wakes of the same commit (for example `status-card-evaluation`).
This process drops the cached row when the write commits. During a
deploy overlap, when work notifications cross processes, the other
process drops its cached row on receipt, before a worker woken by the
same write reads the settings. A reconnect of the notification listener
also drops the row.
- `fillUnsetInstanceDefault` writes through `update({ db: tx })`, so its
signal is sent when its transaction commits.
- A read that starts before an invalidation never stores its result, so
an old value can not come back after a write.
- The test setup clears the cache before each test, because many tests
write the row directly.

Hot callers of the cached row:

- Live run event delivery: `live-run-event-redaction.ts`
(`censorUsernameInLogs`), once per delivery batch.
- Run output and run lifecycle redaction: `heartbeat.ts`
`getCurrentUserRedactionOptions`, used by `run-output.ts`,
`run-lifecycle.ts`, `run-completion.ts`.
- The stranded-task recovery check: `recovery/service.ts` reads
`getExperimental().enableAgentChat` once per checked conversation issue.
- Runner tool calls: `paperclip-runner-tool-authority.ts`
(`enableIsolatedWorkspaces`, `enableExternalObjects`).
- Adapter routes: `app.ts` (`enableNativeRunner`, `enableOpenAiDot`) per
request.
- Run preparation: `heartbeat.ts` (`instance_settings` preparation
step), `queued-run-line.ts`, `queued-run-dispatch-liveness.ts`,
`run-control.ts`.
- Issue and comment paths in `issues.ts` (`censorUsernameInLogs`,
`enableAgentChat`).

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/instance-settings-cache.test.ts` (embedded PostgreSQL):
- 150 hot reads (`getGeneral`, `getExperimental`, `get` from new service
objects) after one warm read: 0 statements. Without the cache the same
test measures 150 statements (red run).
  - 20 concurrent misses: 1 statement.
- `updateExperimental` and `updateGeneral` are visible at once to a new
read in the same process.
- `fillUnsetInstanceDefault` (transaction write of the default
environment) is visible at once after it returns.
- A change written by a second client (another process) becomes visible
within 2000 ms plus a 500 ms test margin.
- A caller that changes the returned object does not change the cached
row.
- A read through a transaction handle sees the transaction's uncommitted
write, two such reads issue 2 statements, and the cache does not keep
the write after rollback.
  - A read that started before an invalidation is not stored.
- A received `instance-settings-changed` notification from another
process drops the cached row (next read issues 1 statement).
- `updateExperimental({ enableStatusCards: true })` sends
`instance-settings-changed` as its first notification, before
`status-card-evaluation`; `updateGeneral` sends it alone.
- The server tests that use instance settings or write the row directly
(230 files): see the run notes in the first comment.

## Risks

- A direct database edit, or a change from a second process outside a
deploy overlap, takes up to 2 seconds to apply in this process. Before
this change it applied at the next read. This includes access flags such
as `enableOwnerInstanceAdmin` (read per cloud-tenant request) and the
default environment.
- A change made through the settings service applies at once in this
process, and during a deploy overlap at once in the other process.
- A new code path that writes the `default` row must call
`announceInstanceSettingsChange`, or its change applies within 2
seconds.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
user-facing setting or command changed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:50:19 -06:00
DottaandClaude Opus 5.5 3706f520fd test(scaling): keep ramping through recovered runs, read spend from cost records, and wait for termination before deleting agents (#21776)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The scaling effort uses a real-agent ramp harness
(`scripts/scale-agents-ramp.ts`, #21695 and #21757) to load a hosted
test instance through the product API
> - Its first run with agents that really ran (10 agents, 105 tasks, 106
runs) stopped on one failed run, recorded $0 model spend, and left
cleanup errors
> - Read-only inspection of the instance showed three harness faults.
The failed run had a same-run bootstrap retry scheduled 33 s later, but
the harness stopped and cleaned up first. Run rows carry no cost, but
cost records do (63.7 cents for the run's agents). Terminated agents
leave the company agent list, so the harness treated them as done and
sent deletes that Paperclip refused with 409
> - These faults make the ramp stop too early, make its spend cap inert,
and make cleanup report the wrong cause
> - This pull request keeps the ramp going through runs that Paperclip
recovers by itself, reads spend from cost records, records board latency
per route, and makes cleanup wait for termination and name what it left
behind
> - The benefit is that the next ramp grades the product, not the
harness, and its spend cap works

## Linked Issues or Issue Description

Refs #21695
Refs #21757

**What happened?**
The ramp stopped at step 1 with `failed_runs_zero failed (observed 1)`.
The run failed with `runner_ingress_bind_conflict` before its provider
session opened. Its run events show `same-run resume persisted` with
`recoverable: true` and `nextAttemptAt` 33 s later. The harness stopped
within 9 s and paused the agents, so the retry was blocked
(`native_retry_admission_ineligible`). The result showed `Recorded model
spend: $0`. The harness read `costUsd` from run rows, which is always
null; `GET /api/companies/<id>/costs/by-agent` holds 63.71 cents over
104 cost events for the 10 agents. Cleanup reported two `DELETE
/api/agents/:id returned 409: Complete termination before deleting the
agent`. Those agents were still `lifecycleState: terminating`; the
harness waited on the company agent list, which hides terminated agents,
so it treated them as done. A later `--cleanup-only` changed nothing
because it also found agents only through that list. The board API p90
failed (765 ms), but the result had no per-route timings to say which
request was slow.

**Expected behavior**
A failed run whose task finishes by itself is counted, not a stop. A
failed run whose task does not finish within a grace stops the ramp. The
share of failed runs is graded at the step end. Spend comes from
Paperclip's cost records. Cleanup deletes an agent only after
termination completes, and names the agents and sandbox leases still
pending.

**Steps to reproduce**
1. Run `node --test scripts/scale-ramp/scale-ramp.test.ts` with the new
fake and tests and the `origin/master` `ramp.ts`, `cleanup.ts`,
`criteria.ts` and `report.ts`.
2. 16 tests fail, among them every new test (recovered run, unrecovered
run, spend from cost records, spend cap, board per route, termination
wait, still-terminating agents and `--cleanup-only`, leases released
during the wait).
3. With this change, all 60 pass.

**Paperclip version or commit**
Instance on `2026.1106.0+2564.git.3244780bd`; harness from #21757.

## What Changed

- `criteria.ts`: replace `failed_runs_zero` with
`failed_runs_unrecovered_zero` (immediate) and `failed_run_rate` (at the
step end). Step metrics add `runsEnded`, `runsFailedRecovered`,
`runsFailedUnrecovered`, `runsFailedPending` and `board.byRoute`.
- `ramp.ts`: a run counts as failed once it was seen `failed` or
`timed_out`. It is recovered when its task reaches `done`. It is
unrecovered when the task rests not done, needs Retry, or has no live
run for `--recovery-grace-minutes` after the failure. A task whose
interruption has an automatic retry scheduled (`state: retry_scheduled`,
`nextRetryAt` or `retryWait`) does not count as needing Retry. Spend
comes from `costs/by-agent` for the run's agents on each poll, and each
step reports the spend in its window. Board probes keep a distribution
per route.
- `config.ts`: add `--max-failed-run-percent` (default 2) and
`--recovery-grace-minutes` (default 15).
- `cleanup.ts`: read agents by id as well as from the list, so
terminated agents in the state file are found. Wait for `lifecycleState`
`terminated` or `rejected` by id before a delete. Report
`agentsStillTerminating`, `activeSandboxLeaseIds` and
`sandboxLeasesReleasedDuringWait`, and name each lease still active with
its run and acquire time.
- `report.ts`: the summary shows recovered, unrecovered and pending
failed runs, board latency per route, and each cleanup error.
- `fake-paperclip.ts`: run rows carry no cost; `costs/by-agent` returns
cents per agent. Optional failure of the n-th run, a scheduled recovery
run, a termination delay (terminated agents leave the list and a delete
answers 409 until termination completes), and leases that stay active
after their run.
- `doc/plans/2026-11-13-thousand-concurrent-agents.md`: the runbook
describes the failed-run criteria, the spend source and the cleanup
behavior.

## Verification

- `node --test scripts/scale-ramp/scale-ramp.test.ts`: 60 of 60 pass.
- Red check: with the new fake and tests but the `origin/master`
`ramp.ts`, `cleanup.ts`, `criteria.ts` and `report.ts`, 16 tests fail,
including all 8 new ones.
- `tsc -p scripts/scale-ramp/tsconfig.json`: no errors.
- Evidence from read-only GETs on the test instance: run `f1892d43`
events (bind conflict, scheduled same-run retry, blocked after the
pause), run rows (`costUsd` null on all 106), `costs/by-agent` (63.71
cents for the 10 agents), agent rows by id (`terminating` while absent
from the list), lease rows.
- Not run: a new real ramp (needs the owner's go-ahead).

## Risks

- Low risk. The change touches only the scale-ramp harness, its fake,
its tests and one runbook section. No product code changes.
- A failed run no longer stops the ramp at once. An unrecovered one
stops it after the grace (15 min by default), so a broken step can run
up to that long. The step-end failed-run rate still fails a step with
more than 2% failed runs.
- Spend now needs cost read access for the board token. If
`costs/by-agent` cannot be read, the result reports spend as not
measured and the spend cap does not trigger.
- Cleanup can now take up to 5 more minutes while it waits for
termination by id.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only HTTP GETs to the test instance).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:39:28 -06:00
DottaandClaude Opus 5.5 2b23119ed2 fix(native): keep a warm follow-up that inherited its predecessor's process identity waiting on the same run (#21774)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap. A warm agent session stays
with the outgoing server that ran its last turn.
> - #21697 makes a follow-up wait while that outgoing server is still
alive, instead of starting a second Runner over its session.
> - Runtime selection copies the warm predecessor's process fields
(`processPid`, `processGroupId`, `processStartedAt`) onto the follow-up
and records `inheritedProcessIdentityRunId`.
> - The wait records itself with `deferNativeControllerRecovery`. That
function refused any run with process fields, so it refused every warm
follow-up that the wait stopped. The run failed with
`native_controller_recovery_wait_authority_changed`.
> - This pull request lets the deferral accept an inherited identity,
and clears those inherited fields in the same transaction so the usual
resumption claim can pick the run up.
> - The benefit is that a warm follow-up in a deploy overlap stays on
the same run and does not fail.

## Linked Issues or Issue Description

Refs #21697

**What happened?**
In a zero-downtime release-swap test, a follow-up comment arrived 0.25 s
after its predecessor finished on the outgoing server. The follow-up run
started on the incoming server 1.09 s later.
`waitForLiveWarmSessionOwner` found the outgoing server alive and asked
to wait. `deferNativeControllerRecovery` then refused with
`native_controller_recovery_wait_authority_changed`, because the run
carried the process fields that runtime selection copied from its warm
predecessor. The run failed after 2.7 s with
`native_continuation_requires_reconciliation`. Held recovery started a
fresh run. The task finished, but the run history shows one failed run.

**Expected behavior**
A warm follow-up whose only process identity is inherited from its
predecessor waits on the same run. The resumption owner claims it at the
deadline. A run whose own process has started is still refused.

**Steps to reproduce**
1. Run a warm remote agent turn on server A.
2. Switch new run starts to server B while A still serves.
3. Post a follow-up within about 1 s of the turn's finish.
4. The follow-up fails with
`native_controller_recovery_wait_authority_changed`.

## What Changed

- `server/src/services/native-runtime/native-controller-recovery.ts`
(`deferNativeControllerRecovery`):
- Process fields refuse the deferral only when they are the run's own.
Fields count as inherited when
`runnerProfileJson.inheritedProcessIdentityRunId` is set and the run has
no own `native.process_start_requested` or
`native.process_identity_recorded` event. `run-cost-accounting.ts` uses
the same proof.
- With an inherited identity, the same transaction clears `processPid`,
`processGroupId` and `processStartedAt`. It keeps
`inheritedProcessIdentityRunId`. `claimNativeSessionResumptions`
requires those fields to be null.
-
`server/src/services/native-runtime/native-controller-recovery.integration.test.ts`
(real Postgres):
- New: "keeps a warm follow-up that inherited its predecessor's process
identity on the same run until the outgoing owner's deadline".
- Two new cases in "does not publish an admission wait after authority
changes": `own-process` (process fields, no marker) and
`inherited-then-own-process` (marker plus an own process-start event).
Both stay refused and keep their fields.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts
-t "inherited its predecessor|does not publish an admission
wait|unstarted follow-up"`: 9 passed.
- Red on master: the new test fails with `Error:
native_controller_recovery_wait_authority_changed`. The own-process
cases pass on master and with the change.
- Green: the run is `retryable_failure` with `nextAttemptAt` at the
deadline, its process fields are null, no claim happens before the
deadline, and one claim happens at the deadline.

## Risks

- The follow-up now waits instead of failing. The wait ends when the
outgoing server saves a suspended checkpoint, when it exits, or at the
predecessor's finish plus its idle window plus 2 minutes. In the
observed case, the outgoing server's idle close could not save its
checkpoint (`native_runner_checkpoint_owner_lost`), so the wait would
end only when that server exits. See the follow-up note below.
- An own process is still refused. The proof uses the run's own
process-start events, the same proof as run cost accounting.

**Follow-up note (not in this PR):** The outgoing server's checkpoint
write checks the task's current run. A native successor that is parked
in this wait is not one of the accepted successor kinds, and the idle
close is not retried. So the wait does not end early from a checkpoint.
In the observed timeline, the wait is about 225 s (until the outgoing
server exited). The worst case with the default 5-minute idle window is
about 425 s.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:19:23 -06:00
DottaandClaude Opus 5.5 b9cf64f5bf test(server): check the agent row lock with FOR SHARE so lock checks cannot block each other (#21775)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each master commit must pass Cloud readiness before its canary and
image publish
> - `agent-file-restore-locking.test.ts`, added in #21694, failed master
Cloud readiness run 43712563988 on 2026-11-15, so that commit's canary
did not publish
> - The test reported the agent row as locked during the download, but
its own sampler measured 0 ms of product lock time
> - The cause is in the test: its two lock checks used `FOR UPDATE
NOWAIT` on one pool and could block each other
> - This pull request checks the lock with `FOR SHARE NOWAIT`, which
still detects the product's `FOR UPDATE` but never conflicts with
another check
> - The benefit is that the test fails only when the product really
holds the lock

## Linked Issues or Issue Description

**What happened?**
`downloads a stale projection without holding the agent row lock` failed
in master Cloud readiness run 43712563988, attempt 2:
`expected [{ objectKey, agentRowLocked: true }] to deeply equal [{
objectKey, agentRowLocked: false }]`.
The test's metric line showed `lockedMs: 0`.

**Steps to reproduce**
1. Make each sampler check hold its `FOR UPDATE NOWAIT` lock for 50 ms
(to make the overlap certain).
2. Run `pnpm exec vitest run
src/__tests__/agent-file-restore-locking.test.ts -t "stale projection"`
in `server/`.
3. The test fails 3 of 3 times with `agentRowLocked: true` and
`lockedMs: 0`, the same as CI.

**Expected behavior**
Only a lock held by the product makes the check report "locked".

Refs #21694

## What Changed

- `agentRowLocked()` uses `FOR SHARE NOWAIT` instead of `FOR UPDATE
NOWAIT`. The product takes `FOR UPDATE` on this row
(`agent-file-store.ts`, `agent-instruction-revisions.ts`), which
conflicts with `FOR SHARE`. Two `FOR SHARE` checks do not conflict.
- New test: three concurrent checks all report unlocked, a `FOR UPDATE`
held on another connection is reported locked, and the row reads
unlocked again after release.

## Verification

- Red: with the forced 50 ms overlap and the old `FOR UPDATE` probe, the
test failed 3 of 3 runs with the CI error.
- Green: with the same forced overlap and `FOR SHARE`, 4 of 4 tests
passed in 3 of 3 runs.
- Final file (no forced overlap): `pnpm exec vitest run
src/__tests__/agent-file-restore-locking.test.ts` in `server/`: 5 of 5
tests passed in 5 of 5 runs.

## Risks

- Low risk. Test-only change. The new test makes sure the check still
detects a real `FOR UPDATE` lock.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:15:45 -06:00
DottaandPaperclip 62033b9994 fix(qa): schedule recovered cleanup promptly and wait for task panels (#21660)
## Thinking Path

> - Paperclip lets people manage AI agents and their saved work.
> - QA journeys check browser controls and the cleanup of owned agent
resources.
> - A recovered workspace queues sandbox destruction after its files are
saved.
> - The cleanup queue treated that first attempt as a retry and waited
five minutes.
> - Agent termination correctly waited for destruction, so QA cleanup
failed first.
> - This PR runs that first cleanup attempt promptly and keeps the
existing safety gates.
> - It also waits for task-panel transitions before the browser acts
again.

## Linked Issues or Issue Description

**What happened?**

The full scheduled QA smoke completed with 23 passed and 1 failed.
Stop/redirect passed all eleven behavior checks but failed cleanup. The
cancelled run had a saved workspace and a bound stop receipt before
agent termination. Its sandbox was destroyed about five minutes later.
The shared cleanup schedule applied the retry delay before the first
destruction attempt.

The browser regressions also found that the current panel helper can
consume all four actions while an opener remains visible during panel
mounting.

**Expected behavior**

A recovered workspace's first cleanup attempt runs on the next eligible
worker pass. A failed attempt retains normal backoff. Agent termination
still waits for completed resource cleanup. Task-panel navigation waits
for the next controls before it acts again.

**Steps to reproduce**

1. Queue recovered-workspace destruction after the cleanup worker has
become idle.
2. Read the next cleanup deadline and advance one normal worker pass.
Before this fix, the deadline is five minutes late and no teardown runs.
3. Open task document controls with a delayed side-panel transition.
Before the panel fix, both collapsed-panel controls fail.

Refs #19224 and #19292 for retaining stop proof during workspace
recovery. This change keeps the full cleanup requirement for agent
termination. Current master supplies the shared task-panel helper; this
branch uses it and retains additional browser regressions.

## What Changed

- Use one shared first-attempt predicate for the cleanup deadline, sweep
selection and ownership report. It now includes recovered workspace
cleanup bound to the same run.
- Preserve worker spacing, in-flight claims, ownership and saved-file
checks, retry delays and attempt limits.
- Wait for panel controls after opening the panel. Wait for the selected
tab or menu action to finish.
- Add real PostgreSQL cleanup regressions, keep the Chromium controls,
and document the cleanup timing.

## Verification

Tested on Linux x86_64, Node v25.9.0. Local commit
`7c8e0f12aa4b19aaf53669141ab3299ce76db963` and published head
`892c5dd4b4638796a138400d8bbe30aee19c71dd` have the same tree,
`d135a98bdff58e3320cdc83976ffe1cf47f48b66`.

- Red: both new real-PostgreSQL cleanup tests failed on the old code.
The deadline was five minutes late and the first normal sweep made zero
teardown calls.
- Green: 87 cleanup, ownership and lifecycle tests pass. The
restored-work case stays unconfirmed until the real cleanup worker
destroys the lease; failure still backs off and the worker becomes idle
again.
- Red after master integration: both delayed document-panel browser
cases failed. Green: all 18 Chromium support checks pass.
- 77 journey contract tests pass.
- Product E2E typecheck passes with the documented low-memory path. The
initial ordinary attempt was killed by the local memory limit and is not
counted as a pass.
- The only follow-up after the passing checks removes new code comments
to follow the contribution guide. Runtime and test behavior are
unchanged. Final-head [CI run
43714742904](https://github.com/paperclipai/paperclip/actions/runs/43714742904)
completed successfully at this exact head. All 63 PR checks are
terminal: 58 success, 2 neutral and 3 conditional skips; no failures or
pending checks. Typecheck, build, tests and the final verify gate pass.
Greptile and Paperclip Review both score this head 5/5 with no
actionable findings; both historical inline threads are resolved.
- Hosted QA recovery is still unverified. No extra model run, merge,
deployment or QA setting change was made.

## Risks

Recovered workspace destruction starts sooner. Existing producer,
file-preservation, ownership, claim and provider checks still decide
whether teardown is allowed. Failed attempts retain the existing
cooldown. Agent termination and purge gates are unchanged. The panel
helper waits for visible controls and still fails when controls or saved
values are absent.

The PostgreSQL integration uses a controlled provider boundary. It does
not prove a new live Daytona or deployed QA pass. That requires the
fixed revision on QA and a later normal scheduled smoke.

## Model Used

OpenAI GPT-6-based Codex, with tool use and code execution. The host
does not expose a more specific model ID or context-window size for this
run.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-15 08:52:49 -06:00
DottaandPaperclip afca556728 fix(github): clarify configured review task completion (#21748)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub bots publish reviews and report the result of their assigned
task.
> - A GitHub check and a Paperclip task have separate completion rules.
> - The current prompt explains how to wait, but gives little completion
guidance.
> - A bot can keep a complete review in review while it waits for CI or
merge that its assignment does not require.
> - This pull request makes the configured completion rules clear in
GitHub prompts.
> - The benefit is a finite review task with an accurate status and
publication evidence.

## Linked Issues or Issue Description

Related change: Refs #17318 keeps advisory review waits separate from
human attention. This change adds completion guidance in the same GitHub
prompt owner.

**What happened?**

A bot can publish a complete current-head 5/5 review and still wait for
CI or merge. Manual request prompts also lack the task completion
guidance used by automatic reviews.

**Expected behavior**

When the configured assignment allows completion at 5/5, a complete
published review can finish Done. CI and merge are completion conditions
only when the assignment requires them. Explicit tracking rules still
apply.

**Steps to reproduce**

1. Configure an agent to finish after a complete published current-head
5/5 review.
2. Start a GitHub review with CI results unavailable.
3. Inspect the generated prompt. It explains a response wait but does
not explain this completion case or publication safeguards.
4. With this change, automatic, mention, and comment prompts carry the
same explicit guidance.

**Paperclip version or commit**

Base commit: `83541e89ad3b90740ffc61a32c396dddb5834526`.

**Deployment mode**

Built from source. The prompt change applies to GitHub bot tasks.

## What Changed

- Add shared guidance within the GitHub prompt module for configured
completion at 5/5 and finite discussion replies.
- Require a complete assessment, confirmed publication, and the current
head in the 5/5 guidance.
- Preserve explicit CI, merge, and below-5 tracking rules. Keep the
shared status arbiter and check thresholds unchanged.
- Add six prompt cases. State that CI is unavailable in the existing
lifecycle completion case.
- Document the configuration example and its limits.

## Verification

- Six new prompt cases failed before the change.
- `pnpm exec vitest run
server/src/services/chat-github-review-policy.test.ts`: 40 passed.
- `pnpm exec vitest run
server/src/services/native-runtime/external-chat-wait.integration.test.ts
-t 'settles a published'`: 9 passed.
- `pnpm exec vitest run
server/src/services/native-runtime/external-chat-wait.integration.test.ts
-t 'inherits.*review|published.*guard|unknown publication|pending
publication|changed head|inherited.*review'`: 5 passed.
- `git diff --check` passed.
- `pnpm verify` returned exit 2. The connector document requires broader
CI coverage.
- Local preflight installed dependencies and built shared and plugin SDK
packages. It stopped because Cargo is unavailable. Full local typecheck,
build, and test suites were not run.
- [Greptile reviewed this
commit](https://github.com/paperclipai/paperclip/pull/21748#issuecomment-6894002401)
at 5/5 with no actionable findings. Its check passed for
`36dd146cdc00b0f3d7f6fc2971f24166ef0f6bf5`.
-
[CI](https://github.com/paperclipai/paperclip/actions/runs/43711597984)
passed for `36dd146cdc00b0f3d7f6fc2971f24166ef0f6bf5`. Typecheck, build,
Rust/Runner, all server tests, and all eight E2E test groups passed. The
final [CI
gate](https://github.com/paperclipai/paperclip/actions/runs/43711597984/job/132808715297)
passed at 12:17:58 UTC on 2026-11-15.
- At 12:23 UTC, all 61 check records were complete: 57 passed, 1 was
neutral, and 3 were skipped. Snyk also passed. No checks failed or
remained pending.

## Risks

This change gives prompt guidance. It does not enforce a score-to-status
rule. Custom assignments can still require CI, merge, or later review.
The focused tests check prompt content and existing lifecycle behavior;
they do not prove every live agent or webhook journey.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. This session does not expose the exact model variant or
context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in this PR with the bug report fields
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My public branch name describes the change and contains no
internal ticket id
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-15 08:52:01 -06:00
DottaandClaude Opus 5.5 e6562755e1 fix(issues): cover a blocked issue whose own run is queued after its blockers finish (#21770)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task is blocked on its subtasks. When the last subtask is
done, Paperclip queues a run for the parent to continue.
> - The board marks a blocked task "needs attention" when no live work
covers it. The check looks at the blockers, then at each blocker's runs.
> - When every blocker is done, the check returned "needs attention" at
once. It did not look at the parent's own queued or running run.
> - In a release-swap test, a parent showed "needs attention" for about
12 minutes while its own continuation run waited in the queue. Nobody
had to do anything. The state cleared by itself.
> - This pull request makes the check return "covered" when the blocked
task has its own queued or running run.
> - The benefit is that people see no false "needs attention" while
Paperclip already continues the work.

## Linked Issues or Issue Description

Refs #21731

**What happened?**
In a zero-downtime release-swap test, a blocked parent task had two
subtasks. Both subtasks were done. Paperclip queued the parent's
"children completed" run, which waited about 12 minutes for a free agent
slot. During that time, `blockerAttention` was `needs_attention` /
`attention_required`, with 0 unresolved blockers and the parent itself
as the terminal blocker. The board showed the parent in the blocked
inbox. Then the run started, finished, and the parent became done.

**Expected behavior**
A blocked task whose own run is queued or running is covered by live
work. It must not ask a person for attention.

**Steps to reproduce**
1. Create a blocked parent task with one or more subtasks that block it.
2. Mark every subtask done.
3. Keep the parent's own run queued (for example, when the agent has no
free slot).
4. Read the parent from the issue list. `blockerAttention.state` is
`needs_attention`.

## What Changed

- `server/src/services/issues.ts`: in `readIssueBlockerAttentionBatch`,
when a blocked root has no unresolved blockers, the function now checks
the active-issue set that it already built. If the root has its own
queued or running run (or an active wake), it returns `covered` with
`blockingTreeLive: true`. Otherwise it returns `needs_attention` as
before.
- The change adds no query. The active-issue set already includes the
roots, through their `executionRunId` and their wake requests.
- `server/src/__tests__/issue-blocker-attention.test.ts`: two tests.
- "classifies a blocked parent whose children are all done as covered
while its own children-completed run is queued"
- "keeps a blocked issue with no unresolved blockers and no active run
as needs_attention"

## Verification

- `cd server && npx vitest run
src/__tests__/issue-blocker-attention.test.ts`: 29 passed.
- Red on master: the first new test fails with `"state":
"needs_attention"`. Green with the change.
- The second new test passes on master and with the change (negative
case).

## Risks

- Low risk. The change only affects a blocked task with zero unresolved
blockers. That task now reads as covered while its own run is queued or
running. When the run ends without a status change, the task reads as
`needs_attention` again.
- The covered result has no reason (`reason: null`), because no blocker
covers it. The UI label is then "Blocked", not "waiting on 0 sub-tasks".

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:51:38 -06:00
DottaandPaperclip 5b750fdf25 feat(ui): add foldable sidebar subtasks with bounded refreshes (#21649)
## Thinking Path

> - Paperclip helps people manage AI agents for work.
> - The sidebar helps people return to recent tasks.
> - Recent parents need foldable child tasks.
> - Busy companies must not refresh every child group for unrelated
work.
> - Exact unread counts must preserve personal reads and task access.
> - This pull request adds child groups and bounds their refresh work.
> - People can scan subtasks without repeated background requests.

## Linked Issues or Issue Description

**Subsystem affected**

Task navigation in the standard and Classic sidebars.

**Problem or motivation**

Recent parent tasks did not show their subtasks. The first
implementation refreshed every previously viewed group. One unrelated
update caused 52 child-page requests and one summary request after
viewing 50 groups. Exact child counts also repeated privacy and unread
checks per child.

**Proposed solution**

Show direct subtasks with independent folding and ten-row pages. Refresh
affected groups and summary batches. Stop offscreen reads. Bound
child-request concurrency. Aggregate viewer facts once and preserve the
existing inbox unread rule.

**Roadmap alignment**

This extends existing task navigation. The existing feature PR is reused
for its performance repair. No new PR duplicates it.

## What Changed

- Add shared parent groups to both sidebar layouts. Preserve Classic row
actions and active-child visibility.
- Keep unread state personal to each task. Folding and paging do not
mark tasks read. Counts include readable children beyond ten.
- Target live refreshes with authorized current and previous parent IDs.
Reuse parent metadata from the narrow run-binding read. Resolve fallback
lookups once per delivery batch. Keep the previous summary fallback for
older servers.
- Limit child requests to three at a time. Cancel active and queued
reads when groups leave the viewport.
- Discard extra cached pages on Show less and collapse.
- Compute exact navigation counts in one SQL statement. Materialize
readable candidate IDs once. Use indexed participation sources and
conservative unread candidates before the shared inbox rule.
- Skip recursive privacy walks for standalone open tasks without a
project. Keep the recursive rule for all restricted or inherited shapes
and all delegated principals.
- Add request-count, cancellation, reparenting, read-receipt,
inbox-parity and privacy regressions. Include a disposable PostgreSQL
benchmark.
- Update product and API documentation. Retain the production Storybook
states.

## Verification

- Published head: `4148861437c1b6b635479ad92509dbc71155a232`. Its tree
matches local `04be9d66d481d9bcd1c4d6158acf063a1e65ac7e`:
`a23533e93d3d155918f0254d296a7fd2f0c7c12f`.
- Rebased on master `ca23330c69b1f2b25bdadfcbda68317db97deaef`. Remote
publication preserves prior feature commits.
- The request regression failed before the fix and passes after it. An
unrelated event now sends zero child requests. Current servers also
avoid the unrelated summary request. Old-server events keep one summary
fallback.
- A relevant update after Show less reads one page. Reparenting
refreshes two affected groups. Offscreen reads stay at zero and resume
once on return. Active and queued cancellation tests pass.
- The same synthetic authenticated HTTP workload with 50,000 read
children improved from 767 ms to 92.84 ms median. Five warm samples per
case. The all-unread case improved from 463.85 to 351.65 ms. Untouched
children improved from 208.01 to 131.09 ms. The old-receipt case
improved from 799.70 to 245.80 ms. These local measurements are not a
production capacity guarantee.
- Focused PostgreSQL routes, inbox-parity, privacy and WebSocket tests
passed: 38 tests.
- All 325 affected UI tests passed. All 38 focused PostgreSQL, privacy
and WebSocket tests passed. UI typecheck and low-memory server
declaration checks passed.
- All 41 Chromium Storybook checks passed. They cover 20 states in both
themes and keyboard, folding, paging and unread behavior. They use
production components with HTTP fixtures.
- Both new review findings failed as regressions before their fixes and
pass afterward. Explicit child reads include waiting external
conversations. Deleting a child publishes a separate parent-authorized
update without deleted-child content. Both review threads are resolved.
- `pnpm verify --explain` returned exit 2 because product documentation
requires broader CI. [The prior CI
run](https://github.com/paperclipai/paperclip/actions/runs/43711489306)
found type errors in the new test fixtures. The same errors failed
locally. The fixture fix passes `node
server/scripts/typecheck-layers.mjs --layer test-tree-0` and all 21
issue summary route tests. Production code is unchanged. That CI run
then found a live-delivery read regression. The existing test failed
locally with eight statements against its six-statement budget. A new
PostgreSQL regression also found three parent lookups for two same-batch
calls. Run delivery now reuses parent metadata from its narrow
run-binding query. Fallback lookups resolve to one promise. Both
regressions pass without increasing the budgets. All 74 targeted route,
WebSocket and privacy tests pass. The affected below, above and
test-tree-0 typecheck layers pass. [CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/43715541480/attempts/2)
passes on the published head. All 62 check runs are terminal and clean:
57 succeeded, two were neutral and three were skipped. Typecheck, build,
policy, all general and serialized server suites, E2E and the [final
verify
gate](https://github.com/paperclipai/paperclip/actions/runs/43715541480/job/132829783231)
pass. The separate security status passes. Attempt 1's server shard 7/12
reached its 20-minute job limit without a test verdict. One retry of
that job passed without a source change. [Greptile completed
successfully on this
head](https://github.com/paperclipai/paperclip/runs/132816163906) with a
[5/5
score](https://github.com/paperclipai/paperclip/pull/21649#issuecomment-6883916606).
All five review threads are resolved. The final CI gate is complete.
- A full local Storybook build exceeded available memory while server
typechecking ran. Browser verification uses the current-source
development server. Rebuilding stale Runner and SDK declarations
resolved the initial server check errors. The first browser attempt
lacked Chromium; all checks passed after installation.
- Reproduce the database measurements with `pnpm --filter
@paperclipai/server exec tsx scripts/benchmark-sidebar-subtasks.ts`.
- Extra local shard diagnosis ran 164 suites. Under this shell's `0077`
creation mask, 157 suites passed and three had fixture permission or
timing failures. Rerunning those three with `0022` resolved the
permission failures. The remaining ten cases passed with one worker: two
workspace cases and eight archive cases. The eight archive cases also
passed on the pre-performance-fix revision, whose archive source is
identical. The broad local shard is not claimed as one passing
execution. The same-head CI retry supplies complete final CI coverage.
- Nothing is merged or deployed.

## Risks

- Exact counts still grow with the readable hierarchy and comment
history. The change reduces repeated work; it does not cap or
approximate counts.
- A relevant summary batch can contain several recent parents. Older
servers retain a broader summary fallback until upgraded.
- Live delivery adds bounded, permission-checked parent lookups. Short
delivery-window reuse limits repeated reads.
- The authorization fast path applies only to open tasks with no privacy
parent and no project. Restricted cases retain the shared predicate.
- No migration is required.

## Model Used

OpenAI Codex, based on GPT-6. Used code execution, native GitHub tools,
PostgreSQL and Chromium verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked a public issue or described the issue in this
PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal issue
id
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open findings
- [x] I will address all review comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-15 08:51:04 -06:00
DottaandClaude Opus 5.5 eb65fc0941 fix(server): turn agents' made-up-host Paperclip links into relative routes (#21762)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents post chat replies and task comments that link board pages and
attachments
> - Some models invent a hostname for these links, such as
`https://paperclip.local/COM/agents/<id>` or
`https://paperclip/api/attachments/<id>/content`
> - Those hosts do not exist, so the person clicks a broken link
> - The chat guidance already says never to invent a hostname, but a
model can still do it
> - This pull request rewrites such links to relative app routes before
an agent comment is saved, and adds the rule to the Paperclip skill
> - The benefit is that every board and attachment link an agent writes
opens, on whatever origin the reader uses

## Linked Issues or Issue Description

**What happened?**
Chat-eval cohort 16 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master 819f61f1fd). In
`agents.edit-instructions`, the reply linked Morgan as
`[Morgan](https://paperclip.local/COM/agents/<id>)`. In
`onboarding.first-work`, the worker's summary linked its attachments as
`https://paperclip/api/attachments/<id>/content`. Neither host exists.

**Expected behavior**
An agent's link to a Paperclip board or API route opens. Genuine
external links, the instance's own origin, and links inside code stay as
written.

**Steps to reproduce**
Post an agent comment that contains
`[Morgan](https://paperclip.local/<PREFIX>/agents/<id>)`. Before this
change it is stored as written. After it, the stored body links
`/<PREFIX>/agents/<id>`.

## What Changed

- New `server/src/services/agent-invented-host-links.ts`. It rewrites an
absolute link to a relative route only when all of these are true:
- The host is made up: `paperclip`, or `paperclip.<reserved TLD>`
(`local`, `localhost`, `internal`, `test`, `example`, `invalid`, `lan`),
or a loopback host.
- The path is a known board route (`/<PREFIX>/<board root>…`) or, on a
made-up Paperclip host only, `/api/…`.
- A loopback host (`localhost`, `127.x`, `[::1]`) qualifies only for a
board route under the comment's own company prefix. A local app that the
agent built keeps its link, including its `/api` paths.
  - The origin is not this instance's configured public origin.
  - The link is outside fenced and inline code.
- Markdown link targets keep their place. Bare links and autolinks
become Markdown links, so they stay clickable.
- `issueService.addComment` applies it to agent-authored comments,
before the redaction and dedupe steps, so a retried identical comment
still deduplicates. The run-presentation check still compares the text
that the agent wrote. People's comments are not changed.
- `skills/paperclip/SKILL.md` now tells agents to write board and
attachment links as relative paths and never add a hostname.

## Verification

- `pnpm vitest run
server/src/services/agent-invented-host-links.test.ts` (19 tests):
recorded cohort 16 links are rewritten. Negative controls stay
unchanged: external links, a production Paperclip host, the instance's
own origin, local app links, another prefix on localhost, a non-route
path, another single-label host, inline code, a fence, and relative
links.
- `server/src/__tests__/issues-service.test.ts` new case: an agent
comment is stored with the relative route, and a person's identical
comment is stored as written. It fails on master and passes with this
change.
- Per-message cost (local tsx micro-benchmark): the gate on a comment
with no candidate host takes 0.3 µs (500 chars), 1.2 µs (2 KB) and 4.7
µs (8 KB). Only when a candidate exists does it read the company prefix
(one indexed select) and the public origin (config read once per
process), and rewrite (4–40 µs). Nothing runs before a turn starts.
- `server` typecheck passes.

## Risks

- A link on a made-up host to a board route of another company prefix
becomes relative. It opens that route on the reader's instance, where
access checks still apply.
- External channels that cannot resolve relative links already get
relative links from agents today. This change does not alter their
delivery.

## Model Used

- Claude, `claude-opus-5-5` (Claude Code agent with tool use and code
execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:44:06 -06:00
DottaandClaude Opus 5.5 348ed0ae10 fix(runner): keep closing tool-call tags and model control tokens out of OpenCode chat (#21761)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner OpenCode driver streams model text into the
chat as progress and selects the final reply
> - DeepSeek v4.1 flash (through OpenRouter) wrote tool-call scaffolding
and tokenizer control tokens as ordinary text
> - #21361 hides tool-call markup, but its detector matched only opening
DSML tags, and nothing recognised control tokens
> - A person saw raw `</|DSML| invoke>` lines in a progress message, and
in another conversation got a degenerate reply instead of an answer
> - This pull request extends the same detector to closing DSML tags and
to model control tokens, on the streaming path and at completion
> - The benefit is that scaffolding never reaches the chat, and a
degenerate reply goes to the existing correction instead of being posted

## Linked Issues or Issue Description

Refs #21361

**What happened?**
Chat-eval cohort 16 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master 819f61f1fd)
showed two defects.
1. In `task-coordination.block-on-design`, after a real
`paperclip_finish` call, the model streamed a text part that held only
closing tags (`\r\n</|DSML| invoke>\n</|DSML|
calls>…</invoke>\r\n</parameter>`). The driver emitted it as an
`agentMessage` progress delta, so the chat showed raw markup. The cause:
`SENTINEL_FORMS` matched `<|DSML|` but not `</|DSML|`.
2. In `cases.case-history`, the reply to the second follow-up was
`<|begin▁of▁sentence|><?xml version="1.0" …?>\n<output>\nSearch output
truncated. Let me search more specifically…\n</output>`. The driver took
it as the final text part and posted it. The question was never
answered.

**Expected behavior**
Tool-call scaffolding and model control tokens never appear in progress
or in the reply. A reply made only of them goes to the empty-response
correction, which asks the model for the real answer. Real answers,
including answers about XML, chat templates or tool tags, stay as
written.

**Steps to reproduce**
Feed the recorded text parts above to the OpenCode driver (the new
`closing-markup` and `control-token-reply` cases in
`empty-response.test.ts` do this).

## What Changed

- `tool-call-markup.ts`: closing DSML tags (`</|DSML|`, `</|DSML|`)
count as tool-call markup.
- `tool-call-markup.ts`: new `controlTokenStart` finds DeepSeek, ChatML
and Llama control tokens (for example `<|begin▁of▁sentence|>`,
`<|im_start|>`, `<|eot_id|>`) outside code. A new
`visibleBeforeModelScaffolding` holds back both kinds while text
streams.
- Sentinels inside inline code spans are ignored, as fenced code already
was.
- A per-line fast path skips the sentinel scan when a line has no `<|`,
`<|`, `</|` or `</|`.
- `opencode-server-driver.ts`: prose before a control token stays the
answer, with the token removed (for example a trailing end-of-sentence
token). A part with no prose before the token is discarded with the
`opencode_control_token_reply_discarded` diagnostic. The existing
empty-response correction then asks for the answer, with an opening that
names the control tokens. Unlike tool-call markup, this does not make
earlier prose ineligible.

## Verification

- `pnpm vitest run packages/paperclip-runner/src/drivers/opencode/` (20
files, 296 tests pass).
- Red/green: with master's detector and driver, 9 new detector tests and
the `closing-markup` and `control-token-reply` driver cases fail. With
this change they pass.
- The tests use the recorded cohort 16 text. Negative controls: an XML
answer, an answer that names an `<output>` tag, control tokens in inline
code and in a fence, HTML, and comparison operators are left alone. A
driver case checks that `answer.<|end▁of▁sentence|>` keeps the answer
with no correction.
- Per-message cost (local tsx micro-benchmark, 20k calls, one prose line
with `<`): 500 chars 0.3 → 0.8 µs; 2 KB 2.2 → 2.6 µs; 8 KB about 8–16 µs
before and 7–9 µs after (noise range). The detector runs per text-part
update, as before. Nothing runs before the turn starts.
- Paid narrow eval run: results are added in a PR comment.

## Risks

- A real answer that writes a literal control token outside code is cut
at the token. Prose before it stays. That text is never meaningful
prose.
- Closing DSML tags in plain prose (outside code) now hide the rest of
the part. DSML tags are model-internal, so prose does not contain them.

## Model Used

- Claude, `claude-opus-5-5` (Claude Code agent with tool use and code
execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1115.0-canary.13
2026-11-15 08:44:00 -06:00
DottaandClaude Opus 5.5 56654adecd perf(server): reuse sandbox provider schemas for five seconds on the remote-run path (#21758)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs can use remote sandboxes that a sandbox provider plugin
supplies (for example Daytona)
> - Each sandbox operation resolves the environment config, and that
step reads the provider config schema to find the secret-ref fields
> - On a hosted instance with 7 remote runs, this schema read was the
most frequent statement: 17.95 per second, about 2.5 per second for each
run
> - The server runs all agents on one Node main thread, so each
statement costs main-thread time; at 1,000 agents this read alone is
about 2,500 statements per second
> - This pull request reuses the retained schema for a short, bounded
time and drops it at once when the plugin registry changes a plugin
manifest
> - The benefit is that a live remote run reads the schema about once
per 5 seconds for each provider, not once per operation

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Every plugin sandbox operation (exec, file sync in/out, runner ingress
attach, capability snapshot, workspace realize, lease resume, lease
release) calls `resolveEnvironmentDriverConfigForRuntime`. That call
runs the `with selected_provider as materialized ...` statement in
`resolvePluginSandboxProviderConfigSchema`, every time.

**Subsystem affected**
Server: plugin environment driver
(`server/src/services/plugin-environment-driver.ts`) and the plugin
registry manifest writes.

**Current behavior**
The schema read runs once per operation. A census test that drives the
real runtime calls of one remote run counts 19 schema reads for 19
operations (see the table below). The hosted measurement was 17.95
executions per second with 7 runs.

**Proposed behavior**
The read runs at most once per 5 seconds for each provider and database
client. A plugin install, upgrade or uninstall through the plugin
registry in the same process takes effect on the next operation.

**Reason and benefit**
The schema is instance-wide plugin manifest data. It does not change
during normal runs. Re-reading it per operation spends main-thread and
database time for no new information.

**Breaking changes**
None. No API, schema or saved-state change.

## What Changed

- `resolvePluginSandboxProviderConfigSchema` now returns the retained
schema without a database read when the entry was checked less than
`PROVIDER_SCHEMA_REUSE_MS` (5,000 ms) ago and the plugin manifest
revision did not change. After that, it uses the existing cheap revision
check, as before.
- A lookup sequence number stops an older, slower lookup from replacing
a newer retained entry.
- A missing provider is still never retained, so a provider that becomes
available is used on the next operation.
- New module `plugin-manifest-revision.ts` holds an in-process manifest
revision counter. It is a separate module so that the registry and the
driver do not import each other.
- `plugin-registry.ts`: `install` (both the new-row and the reinstall
paths), `update` and `uninstall` announce a manifest change after the
write completes. This is 5 changed lines plus 1 import line. Status-only
writes do not announce, because the schema read does not use plugin
status.
- `plugin-loader.ts`: `installPlugin` runs `registry.install` inside an
outer transaction. It now announces the manifest change again after that
transaction settles, so a schema read made before the commit cannot be
reused after it. This is 1 changed line plus 1 import line.
- Statement census per operation, before and after. One remote run, one
secret-ref field in the provider config, embedded test database, fake
in-process plugin worker:

| Operation | Count | Schema reads before | Schema reads after | All
statements before | All statements after |
| --- | --- | --- | --- | --- | --- |
| Lease acquire | 1 | 1 | 1 | 26 | 26 |
| Runner attach (ingress endpoint) | 1 | 2 | 0 | 11 | 9 |
| Capability snapshot | 1 | 1 | 0 | 6 | 5 |
| Exec | 10 | 10 | 0 | 50 | 40 |
| File write (sync in) | 1 | 1 | 0 | 5 | 4 |
| File read (sync out) | 1 | 1 | 0 | 5 | 4 |
| Workspace realize | 1 | 1 | 0 | 5 | 4 |
| Lease resume (reattach) | 1 | 1 | 0 | 6 | 5 |
| Lease release | 1 | 1 | 0 | 18 | 17 |
| **Total** | 18 | **19** | **1** | 132 | 114 |

- Other reads that stay on each operation (reported, not changed in this
pull request): each exec resolves each secret-ref field with 3 reads
(secret, binding, version) and 1 `secret_access_events` insert. Runner
attach and the capability snapshot also read the plugin row by id. These
are part of the secret authorization and audit path, so they keep their
own checks.

## Verification

- `cd server && npx vitest run
src/__tests__/plugin-sandbox-provider-schema-census.test.ts` (new). On
master the census test fails with 19 schema reads. With this change it
passes with 1.
- The new tests also prove these points:
- A registry upgrade that changes the secret-ref fields takes effect on
the next call in the same process.
- A direct manifest change (as from another server process) takes effect
after the 5-second bound, and no read occurs inside the bound.
  - A registry uninstall takes effect at once.
- A plugin reinstall through `pluginLoader.installPlugin` takes effect
after its install transaction commits, even when a schema read ran
inside the uncommitted transaction. This test failed before the
post-commit announcement and passes with it.
- A missing provider is not retained. The next call after install reads
and uses it.
- Two companies share the provider schema but each gets only its own
secret values. A cross-company secret id still fails with "Secret must
belong to same company".
  - A new database client (a process restart) starts cold, as before.
- `cd server && npx vitest run src/__tests__/plugin-database.test.ts`:
passed (plugin install transaction paths).
- `cd server && npx vitest run
src/__tests__/sandbox-provider-schema-reads.test.ts`: the existing tests
now step the clock past the bound where they change the database
directly.
- `cd server && npx vitest run
src/__tests__/environment-runtime.test.ts`: 210 passed.
- `cd server && npx vitest run
src/__tests__/bundled-plugin-registration.test.ts
src/__tests__/external-object-refresh-work.test.ts
src/__tests__/plugin-lifecycle-restart.test.ts
src/__tests__/plugin-loader-error-retry.test.ts
src/__tests__/plugin-environment-driver-sandbox-capabilities.test.ts
src/__tests__/plugin-sandbox-provider-config-validation.test.ts
src/__tests__/environment-probe.test.ts`: 82 passed, 1 skipped.
- `cd server && pnpm typecheck`: passed.

## Risks

- Staleness bound: a plugin manifest change made by a different server
process (for example during an overlapping deploy) can take up to 5
seconds to apply here. During that time, a secret-ref field that an
upgrade adds is not resolved yet, and a field that an upgrade removes is
still resolved. The bound is short and it applies only to which config
paths are secret refs.
- The retained value is instance-wide plugin manifest data, keyed by
database client and driver key. It contains no company data and no
secret values. It does not take part in an authorization decision.
Secret resolution, the environment binding check and the access audit
still run on each operation for the calling company.
- Memory stays bounded by the existing limits: 32 entries per database
client, 64 KiB per schema.
- `plugin-registry.ts` has a small edit in `install`, `update` and
`uninstall`. Other open work in that file can cause a simple merge
conflict.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #17581

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:09:04 -06:00
DottaandClaude Opus 5.5 15afc32137 test(scaling): wait for agent setup and gate the ramp on a first started run (#21757)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The scaling effort uses a real-agent ramp harness
(`scripts/scale-agents-ramp.ts`, added in #21695) to load a hosted test
instance through the product API
> - Its first real run held a 20-minute step with 10 tasks, 10 queued
runs and 0 started runs, and reported only "concurrency_reached failed
(observed 0)"
> - The cause was in the harness: it bound the agents to a plugin
Runner-task environment, which runs only agents with a managed AI
connection. Each agent stayed in setup (status `paused`, setup state
`verifying`) with that reason, so each run stayed queued. The product
did what it is designed to do
> - The harness did not wait for agent setup, did not read the setup
error, and held the full step with nothing running. Its fake API created
agents ready at once, so its tests could not catch this
> - This pull request makes the harness wait for agent setup, gate the
ramp on one started run, and stop within minutes with Paperclip's own
reason
> - The benefit is that a wrong environment or any other start blocker
costs about one minute and prints the cause, not a 20-minute step with
no explanation

## Linked Issues or Issue Description

Refs #21695

**What happened?**
A ramp of 10 then 50 agents on a hosted test stack created 10 agents, 10
tasks and 10 runs. In 20 minutes no run started. The result said only
"concurrency_reached failed (observed 0)". Cleanup found every agent
already paused and cancelled 10 queued runs. The tasks never left `todo`
(status version 1, no checkout run). The agents were bound to the
"Paperclip Sandbox (Daytona)" plugin environment, whose `daytona-tasks`
driver declares `supportsTasks: true`. For such an environment,
lifecycle verification refuses a `paperclip_runner` agent that has no
`runtimeConfig.aiConnection` (`ai_connection_required`, see
`server/src/services/agent-harness-verification.ts` and its test
"refuses a plugin Runner-task agent without a managed AI connection
before allocating a target"). The failure pauses automatic verification,
so the agents never become ready.

**Expected behavior**
The harness must not create a full step of work before agents can run.
When agents cannot finish setup, or when no run starts, it must stop
soon, print the task, run and agent state with Paperclip's reason, and
clean up.

**Steps to reproduce**
1. Run `node --test scripts/scale-ramp/scale-ramp.test.ts` with the new
fake (agents start in setup; `agentSetupError` keeps them there) and the
old `ramp.ts`, `setup.ts` and `config.ts`.
2. The new test "stops within minutes with Paperclip's reason when the
agents never finish setup" fails with `concurrency_reached`, the same
result as the real run.
3. With this change, it stops with `agents_ready` and the setup reason
in under 5 virtual minutes.

**Paperclip version or commit**
Instance on `2026.1106.0+2555.git.819f61f1f` (master at 11:34Z on
2026-11-15); harness from #21695.

## What Changed

- `setup.ts`: add `waitForAgentsReady`, `agentReady` and
`describeAgent`. The harness reads the company agent list until each
agent has setup state `ready` and is not paused. It stops when an agent
reports a setup error for 60 s, or after `--agent-ready-minutes`
(default 10), and prints the agent's status, setup state, holds and
Paperclip's reason.
- `ramp.ts`: before the first step, a preflight creates one agent and
one task and creates the rest only after that task's run starts. A new
check stops the ramp when no run has started within
`--start-timeout-minutes` (default 5) of the first task. The stop reason
names the task status, any execution blocker, the run status, error
code, error and liveness, and the agent's setup state. New stop
criteria: `agents_ready` and `run_started`. Run records keep `error`,
`errorCode` and `livenessReason`.
- `config.ts`: add `--start-timeout-minutes` and
`--agent-ready-minutes`.
- `fake-paperclip.ts`: agents start as the real product creates them
(status `paused`, setup state `preparing`) and become `ready` after
`agentReadyDelayMs`, or stay in `verifying` with `agentSetupError`.
Issue creation queues a run even for an agent in setup, as the real
product does. Runs start only for a ready agent.
- `scale-ramp.test.ts`: three tests. Agents that never finish setup stop
the run with `agents_ready` and the reason, with one agent and no task
created. A first task that never starts stops the run at the preflight
with `run_started`, the task, run and agent state, and one cancelled
run. The rest of a step is created only after the preflight task starts.
- `doc/plans/2026-11-13-thousand-concurrent-agents.md`: the Phase 3
runbook describes the start checks, the first run's cause, and which
environments the harness can use.

## Verification

- `node --test scripts/scale-ramp/scale-ramp.test.ts`: 43 of 43 pass.
- Red check: with the new fake and tests but the `origin/master`
`ramp.ts`, `setup.ts` and `config.ts`, 3 tests fail. The setup test
fails with `concurrency_reached` (the real run's result) where
`agents_ready` is expected.
- `tsc -p scripts/scale-ramp/tsconfig.json` (same options): no errors.
- Read-only inspection of the hosted instance (GET only): task rows
(`todo`, status version 1, no checkout run), activity (agent created, no
pause, 10 `heartbeat.cancelled` at cleanup), the plugin manifest
(`daytona-tasks` `supportsTasks: true`), and the company AI connections
(no OpenRouter connection). The run and agent rows were deleted with the
agents at cleanup, so their setup error was inferred from the code path
and the cleanup counts, not read.
- Not run: a new real ramp (needs the owner's go-ahead).

## Risks

- Low risk. The change touches only the scale-ramp harness, its fake and
its tests, and one runbook section. No product code changes.
- A real ramp now spends up to one preflight start (seconds to minutes)
before the first step. The preflight task is part of step 1 and the
preflight running samples are dropped, so step metrics keep their
meaning.
- An agent whose setup error clears within 60 s does not stop the run.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only HTTP GETs to the test instance).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:07:03 -06:00
DottaandClaude Opus 5.5 3244780bd0 fix(queue): keep a message a starting run's first turn took out of the queue (#21747)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A message posted while a run is starting goes into that run's first
turn, and the task page shows it as sent
> - The page shows it as sent only while the live queue lists it as a
first-turn message
> - When that queue closes, for example after a later Steer, the page
has no lasting record of the delivery
> - It then infers that the message is queued and shows it again with a
Steer button
> - This pull request reports each run's own first-turn receipt in the
run list, and the page places those messages in that run
> - The benefit is that a sent message stays sent

## Linked Issues or Issue Description

Follow-up to #21661, found while I verified that change on staging.

**What happened?**
With a Codex runner, two messages were posted while the run was
starting. The first turn took them and the agent acted on both. Then a
third message was steered into the running turn. After the Steer, the
first two messages came back into the queue with Steer buttons. The
server queue was empty.

**Expected behavior**
Messages that a run's first turn took stay in the conversation as
delivered.

**Steps to reproduce**
1. Start a task with a Paperclip runner agent.
2. While the runner is starting, post one or two messages.
3. After the turn starts, post another message and click Steer.

## What Changed

- `activityService.runsForIssue` returns `firstTurnCommentIds` from the
run's `firstTurnQueuedInput` receipt. The field is set only when the
receipt names this run, because a retry copies its predecessor's
context.
- `IssueDetail` adds those ids to the run's start inputs, so the
messages are placed in that run and are not inferred as queued.
- `RunForIssue` type gets the new optional field.

## Verification

Red/green tests. Each failed before the fix and passes after it:
- `server/src/__tests__/activity-service.test.ts`: "reports the queued
messages a run's own first turn took, not a predecessor's". 33/33 pass.
- `ui/src/pages/IssueDetail.test.tsx`: "keeps a message the running turn
already took out of the queue after the live queue closes". The
IssueDetail, queue helper, and task-chat suites pass (69 files, 1298
tests). UI typecheck passes.

Staging, after #21661 deployed, Codex runner: two messages posted while
the run started were taken into the first turn and acted on. No
Interrupt, error banner, or composer wait was shown. A mid-turn Steer
was accepted in seconds. This PR fixes the one defect that the check
found.

## Risks

- Low. One new read-only field on the run list. A run that took messages
and then failed before its provider accepted the turn now shows those
messages in that run. The next run still delivers them, and the queue
still lists them while they wait.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1115.0-canary.12
2026-11-15 13:05:34 +00:00
DottaandClaude Opus 5.5 e7f31705da perf(release): stage the verified host release daemon instead of compiling it in publish_canary (#21555)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every master push runs a canary, and only one canary runs at a time,
so its duration sets how often merged work ships
> - `publish_canary` takes about 30 min. 8 min of it is `pnpm build`,
and 4 min 40 s of that is an uncached `cargo build --release` of
`paperclip-runnerd`
> - The release then replaces that binary with the exact-source daemons
that `release_runner_binaries` already built and verified for all four
targets
> - This pull request stages the verified daemon for the publisher's own
target during a release, instead of compiling it
> - The benefit is about 4.5 min less per canary, with the same verified
bytes published

## Linked Issues or Issue Description

**What happened?**
In canary run 43487884481 (2026-11-13), the `Publish canary` step took
28.6 min. Its `pnpm build` ran 19:42-19:50. The
`packages/paperclip-runner build` took 5.3 min, and its `cargo build`
reported `Finished release profile ... in 4m 40s`. Every other package
waited for it. `release.sh` then runs `assemble-daemons`, which
overwrites `dist/bin` with the four exact-source release daemons.

**Expected behavior**
A release does not compile a binary that it discards. The Runner build
stamp stays valid, so the server build still reuses the Runner build.

Refs #21554 (the other canary speed-up)

## What Changed

- `packages/paperclip-runner/scripts/build-runner-binary.mjs` (new):
when `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR` is set, it checks the
release daemons with the existing `releaseDaemonManifest` (source
revision equal to `HEAD`, digest, architecture, build metadata) and
stages the host target's daemon at `dist/bin` with the existing
`stageRunnerBinary`. Otherwise it runs `pnpm run build:binary` as
before.
- The Runner `build` chain calls this script in place of `pnpm run
build:binary`. `build:binary` itself and the image build are unchanged.
- `stage-release-runner-binaries.mjs` exports `metadataContract`.
- Tests: `stage-runner-binary.test.mjs` (staging per target, wrong
source, bad digest, missing target, non-release fallback);
`runner-image-identity.test.ts` step list.
- `doc/RELEASING.md` describes the behavior.

## Verification

- `node --test
packages/paperclip-runner/scripts/stage-runner-binary.test.mjs`: 9 of 9
pass.
- `server`: `vitest run src/__tests__/runner-image-identity.test.ts`: 42
of 42 pass.
- With the real daemon artifacts of canary run 43487884481 (receipts
pointed at this commit): `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR=...
pnpm run build` in `packages/paperclip-runner` finished in 15 s with no
`cargo` line. The build stamp was recorded, and
`server/scripts/prepare-runner-vendor.mjs` then reported
`action:reuse,reason:inputs_unchanged`. The staged daemon runs.
- After merge: the next canary's `packages/paperclip-runner build` log
shows "Staged the verified exact-source linux-x64 release daemon" and no
`Compiling` lines.

## Risks

- Low. A release still fails closed: a daemon with the wrong source,
digest, architecture or metadata stops the build, and `assemble-daemons`
checks all four again afterwards.
- Local, Docker and CI builds without the variable are unchanged.
- On macOS, staging re-signs the binary, as it does for a compiled one.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1115.0-canary.11
2026-11-15 06:40:21 -06:00