Commit Graph
2830 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 f2e449f8f6 fix(retry): continue an accepted Retry past chat input held because it predates the stop (#21129)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person's Retry on a connected task (for example a GitHub pull
request review) must start work. #21080 made a Retry whose chat replay
is refused after the click continue from the recovered saved work
> - After #21080 was deployed, accepted Retries on a busy GitHub review
agent still did not start
> - Those tasks had a saved chat request that arrived while the failed
run was still running. Admission holds such a request as
`message_predates_stop`, so it can never start alone
> - #21080 refused to continue while any saved chat input existed, so
this held input blocked the continuation. For a second group, the latest
run was an admission that the recovery hold cancelled before dispatch,
and the recovered saved work belongs to the earlier failed run
> - This pull request lets the continuation adopt input that was held
because it predates the stop, and lets the recovered run that held a
cancelled admission count as the saved work
> - The benefit is that an accepted Retry starts work on these tasks too

## Linked Issues or Issue Description

Follow-up to #21080. No public issue exists. This is the bug
description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

After #21080 was deployed, `POST /api/issues/:id/retry` returned 202 for
blocked tasks of a GitHub review agent (`paperclip_runner`), but no run
started. Each Retry receipt changed to `failed` about 5 seconds later
(on the 14:48 Retries, 5 seconds for one task and about 75 seconds for
two others), with no run and no `issue.retry_continued_from_saved_work`
activity. Every affected task had a saved chat request (a newer GitHub
review request) with an `issue.execution_waiting` record of reason
`message_predates_stop`. The request arrived while the failed run was
running, so admission holds it permanently. The interruption projection
kept pointing at that held request. In a second group of tasks, the
latest run was cancelled before dispatch with
`execution_reconciliation_required`, and the recovered saved work
belonged to the earlier failed run that held it.

Root cause: in `continueRefusedTaskChatRetry`
(`server/src/services/task-retry.ts`), any saved chat input from
`selectTaskRetryInput` stopped the continuation, so the receipt settled
`failed`. The held input can never start by itself, so nothing ran. The
recovered-work check also looked only at the latest run, so it missed
the recovered run that held a cancelled admission.

**Expected behavior**

An accepted Retry starts work. Saved input that admission holds because
it predates the stop joins the continuation, and the task continues from
the recovered saved work.

**Steps to reproduce**

1. Bind a task to a GitHub chat conversation. Let a native run fail with
`native_continuation_requires_reconciliation`, and recover its saved
work.
2. Save a chat request while that run is still running, so admission
holds it with `message_predates_stop`.
3. Click Retry while the chat identity lock is busy, so the source check
is deferred and then refused.
4. The route returns 202. The Retry receipt changes to `failed`, and no
run starts.

**Paperclip version or commit**

a65644d1e1 (the deployed #21080). Reproduced on ef2b53b675.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

`paperclip_runner` (native runs, and legacy admissions cancelled before
dispatch). The fix is in core Retry code and is not adapter-specific.

## What Changed

- `server/src/services/task-retry.ts` (`continueRefusedTaskChatRetry`):
- Saved chat input whose source comment was written no later than the
failed run's stop boundary (`userStopBoundaryAt`: the recorded user Stop
request, otherwise `finishedAt`) no longer stops the continuation. This
is the same boundary and timestamp that admission uses. Admission holds
that input as `message_predates_stop`. The continuation coalesces its
receipts into the Retry receipt (`coalescedIntoWakeupRequestId`), so the
task stops waiting on it. The receipt ids are recorded in the activity
details.
- Saved input created after the stop, and unadmitted chat input, still
settle the Retry as failed, as before.
- When the latest run is an admission cancelled before dispatch
(`execution_reconciliation_required`, never started), the recovered-work
check also accepts the run named by that run's
`executionWait.recoveryActionId`.
- `doc/execution-semantics.md`: both rules.
- Tests in `server/src/__tests__/chat-channels.integration.test.ts`: two
new variants of the #21080 regression test.
`person_with_input_before_stop` saves a chat request before the failed
run stops. `person_after_held_admission` adds a legacy admission that
the recovery hold cancelled. Both start one run that resumes the
recovered workspace (`resume_preserved_workspace`). The held-admission
fixture binds the task to the recovered workspace, as on the hosted
tasks. Refusal edges `input_after_stop` and `input_inside_user_stop`
settle the Retry failed with no run and leave the input deferred.

## Verification

- Red before the fix: both new variants fail. The Retry receipt is
`failed` (or stays held), and no run starts.
- Green after the fix: both variants pass, and the held input receipt is
coalesced into the Retry receipt.
- `npx vitest run src/__tests__/chat-channels.integration.test.ts -t
"[Rr]etry"` with `src/services/task-retry.test.ts` passes 369/369 on the
final commit.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Staging was read with GET requests only.

## Risks

- Behavior change: input that arrived during a failed run no longer
starts its own turn through the chat retry. It is adopted into the
saved-work continuation, and the comment stays in the task thread.
Before this change, that input could never start at all.
- Input that arrived after the stop keeps its own admission path.
- No migration. Receipts that already failed stay failed. The next click
takes the new path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:34:36 -06:00
DottaandClaude Fable 5.2 4e932fd178 perf(attention): index the agents and review paths once for every review of a request (#21132)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task in review needs a path to its next action: a reviewer, a
pending interaction or approval, a monitor, a run, a wake, an external
reply or a recovery. A review without a path is stalled, and the
attention feed tells a person about it.
> - The server decides this for each task in review
(`listIssueReviewAttentionMap` in `server/src/services/issues.ts`, with
the classification in
`server/src/services/recovery/issue-graph-liveness.ts`). The task list,
the blocked inbox and the attention feed call it. `GET
/attention/summary`, which the board sidebar reads on most pages, runs
it for every task in review of the company.
> - The classification did the company-wide work again for each task.
Each check of an agent copied the agent list, indexed it again and
walked the reporting chain. Each task scanned every run, wake,
interaction, approval, external wait and recovery of the request. Each
stalled review also filtered and sorted every agent again, with one such
check for each agent.
> - Thus the work grew with reviews × agents for covered reviews and
with stalled reviews × agents² for stalled reviews. It also allocated a
new list and a new index for each check, and the garbage collector of
the server paid for that.
> - On a hosted instance, an allocation sample of the server put 4.7% of
all allocated bytes under this function, and garbage collection used 25%
of the main thread. The two slowest board requests there are the
attention summary and the task list, which both call it.
> - This pull request builds one index for one classification: the
agents by id, the eligibility of each agent, the invokable agents of
each company in id order, and the paths of each task. Each task then
costs its own paths.
> - The benefit is that the review classification of a summary request
falls from 62 ms to 5 ms for 900 reviews and 60 agents, and from 123 ms
to 8 ms for 300 reviews and 180 agents. The request allocates 57% to 91%
fewer bytes. Every finding, every review path and every count is the
same.

## Linked Issues or Issue Description

Refs #20851 and #20858 (they batched the review-wait provenance check
and the task read decision of the same requests; this change is the next
cost on that path).
Refs #19194 and #13729 (open; they change the failed-run lookup of the
attention feed, which is a different statement and a different
mechanism; this pull request does not touch it).

**What existing behavior does this improve?**

The time and the memory that the server uses to decide the review
attention of the tasks in review, for `GET
/api/companies/:companyId/attention/summary`, `GET
/api/companies/:companyId/attention`, `GET
/api/companies/:companyId/issues` and the blocked inbox.

**Subsystem affected**

Server: task-graph liveness classification
(`server/src/services/recovery/issue-graph-liveness.ts`) and the
review-attention map in `server/src/services/issues.ts`.

**Current behavior**

- `classifyIssueReviewPaths(input, issue)` built `new
Map(input.agents…)` for each task and scanned six path lists of the
request for each task.
- `isInvokableAgent` copied the agent list for each check.
`getAgentOrgChainHealth` then indexed that copy again.
- `ownerCandidatesForRecoveryIssue` ran for each stalled review. It
filtered and sorted all agents, and it checked each agent again in two
more loops. Each check did the copy and the index above.
- `hasActiveExecutionPath` built a joined list of the runs and the wakes
for each blocker it looked at.
- `listIssueReviewAttentionMap` searched the review list for each open
recovery task (`reviewIds.includes`) and for each review
(`issues.find`).

Evidence from a hosted instance (sanitized):

- Server main thread at 93% of one core. Garbage collection: 25% of the
main thread (387 pauses in 32 s, the longest 224 ms).
- Allocation sample, 30 s: 77.7 MB/s in total.
`listIssueReviewAttentionMap` and its callees: 4.7% of all bytes
(`classifyIssueReviewPaths` 4.4%, `classifyIssueGraphLiveness` 2.3%,
`reviewFinding` 2.3%).
- `GET /attention/summary` took 16.3 s of server time and `GET /issues`
up to 15.3 s. Most of that time is waiting behind other work on the same
thread (see "Not verified").

**Proposed behavior**

One classification indexes its input one time. The cost of one task is
the number of its own paths. The cost of the agents is one eligibility
decision for each agent. The results do not change.

**Reason and benefit**

The summary request runs on most board pages. Its CPU time and its
allocation also delay every other request, because the server has one
main thread and one garbage collector.

**Breaking changes**

None. `classifyIssueGraphLiveness` and `classifyIssueReviewPaths` keep
their signatures and results. `packages/shared` does not change.

## What Changed

- `server/src/services/recovery/issue-graph-liveness.ts`
- New internal index for one input (`createIssueGraphLivenessIndex`):
agents by id, a memo of `getAgentWorkEligibility` for each agent record,
the invokable agents of each company in id order, and the paths of each
task for the six path lists. The path lists are indexed on first use and
keep the order of the input. The company check of each path stays.
- `classifyIssueGraphLiveness` builds the index one time and uses it for
the review paths, the waiting paths of blockers, the owner candidates
and the assignee eligibility of blockers.
- New export `createIssueReviewPathClassifier(input)`. It returns the
review paths of many tasks from one index.
`classifyIssueReviewPaths(input, issue)` stays for single tasks.
- The memo uses the agent record as its key, not the agent id. Thus an
input that lists one id two times gets the same answers as before.
- The eligibility decision itself is still `getAgentWorkEligibility` of
`@paperclipai/shared`, with the same agent list. The index calls it one
time for each agent, not one time for each check.
- `server/src/services/issues.ts` (`listIssueReviewAttentionMap` only):
one classifier for all reviews of the request, a set for the review ids,
and the review by position instead of a search. No query changes.
- Tests and the reproduction (see Verification):
- `server/src/__tests__/helpers/issue-graph-liveness-reference.ts`: the
classification as it was before this change. It is the oracle of the
equivalence tests.
- `server/src/__tests__/issue-graph-liveness-equivalence.test.ts`,
`server/src/__tests__/review-attention-classification-equivalence.test.ts`,
`server/src/__tests__/helpers/review-attention-fixture.ts`.
- `server/scripts/benchmark-review-attention.ts`: the reproduction
through the real routes.

## Verification

Reproduction and before/after measurement. The script starts a local
PostgreSQL, seeds a synthetic company, mounts the real `attentionRoutes`
and `issueRoutes` with a board user, and sends the two requests over
HTTP. With `--baseline-ref` it runs the pre-fix modules in a second
process against the same database. It fails unless both versions return
the same summary, the same full attention feed, the same task list and
the same review attention for every task in review.

```sh
pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-attention.ts --baseline-ref=ef2b53b675feebb1bfbd6701627f6c302f8c6916 --tasks=3000,9000 --agents=60
pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-attention.ts --baseline-ref=ef2b53b675feebb1bfbd6701627f6c302f8c6916 --tasks=3000 --agents=180
```

Fixture: one company with tasks of every status (10% in review, 10%
blocked), four of five tasks are children, a reporting tree of agents
with paused, terminated and pending agents. The reviews cover every kind
of path and three ways to stall (31% stalled). The blocked tasks wait
behind chains, cycles, cancelled, done, backlog and unassigned blockers.
Open recovery tasks (5% of the tasks) name reviews, blocked tasks and
done tasks. 6% of the reviews wait for a GitHub reply. A second company
with the same shapes is in the same database.

Embedded PostgreSQL 18.1, Node v26.4.0, `VACUUM ANALYZE` after seeding.
Each version ran in a new process, three rounds in alternating order, 2
warm-up requests and 5 measured requests in each round. A CPU profile of
3 more requests gives the last two columns.

| Tasks / agents / reviews | Request | Version | Statements | Server ms,
median (min) | CPU ms, median (min) | Main thread busy ms, median (min)
| Allocated MB | Review classification ms | GC ms |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 3000 / 60 / 300 (94 stalled) | attention/summary | before | 69 | 121.4
(112.7) | 131.5 (114.4) | 80.2 (73.2) | 170.4 | 16.8 | 5.0 |
| 3000 / 60 / 300 (94 stalled) | attention/summary | after | 69 | 105.9
(94.7) | 113.3 (96.3) | 67.3 (58.6) | 73.3 | 2.4 | 3.4 |
| 9000 / 60 / 900 (283 stalled) | attention/summary | before | 85 |
280.9 (263.9) | 281.4 (253.4) | 203.7 (186.8) | 509.8 | 62.1 | 30.0 |
| 9000 / 60 / 900 (283 stalled) | attention/summary | after | 85 | 224.8
(213.4) | 220.1 (205.8) | 149.9 (142.3) | 219.1 | 4.8 | 6.6 |
| 3000 / 180 / 300 (94 stalled) | attention/summary | before | 69 |
241.2 (203.9) | 248.7 (227.4) | 197.1 (171.1) | 950.2 | 123.2 | 21.2 |
| 3000 / 180 / 300 (94 stalled) | attention/summary | after | 69 | 127.8
(108.5) | 128.5 (102.0) | 76.7 (64.8) | 87.3 | 7.8 | 5.1 |
| 3000 / 60 / 300 | task list (500 rows, 48 reviews) | before | 59 |
102.7 (96.7) | 92.4 (83.6) | 54.7 (48.9) | 73.2 | 2.6 | 2.4 |
| 3000 / 60 / 300 | task list (500 rows, 48 reviews) | after | 59 |
106.2 (91.2) | 89.9 (78.8) | 55.2 (44.8) | 63.4 | 0.9 | 1.6 |
| 9000 / 60 / 900 | task list (500 rows, 50 reviews) | before | 42 |
194.3 (186.0) | 77.5 (69.1) | 43.3 (38.4) | 57.3 | 1.0 | 1.8 |
| 9000 / 60 / 900 | task list (500 rows, 50 reviews) | after | 42 |
189.6 (182.3) | 72.1 (69.5) | 41.9 (39.4) | 57.8 | 1.2 | 2.5 |
| 3000 / 180 / 300 | task list (500 rows, 48 reviews) | before | 59 |
124.6 (95.7) | 110.2 (99.4) | 67.8 (58.1) | 165.4 | 15.4 | 3.8 |
| 3000 / 180 / 300 | task list (500 rows, 48 reviews) | after | 59 |
104.1 (91.9) | 90.7 (79.9) | 56.2 (47.1) | 78.2 | 2.9 | 5.3 |

How to read the table:

- "Review classification ms" is the time in
`listIssueReviewAttentionMap` and everything it calls on the main
thread, for one request, from the CPU profile. It is the direct measure
of this change. Before the fix it was the largest block of application
code in the summary handler: 22% of the busy time at 3000 tasks, 20% at
9000 tasks, and 61% with 180 agents. `getAgentOrgChainHealth` under
`isInvokableAgent` under `ownerCandidatesForRecoveryIssue` had most of
it. After the fix it is 4% to 9%.
- The scaling: three times the tasks gave 3.7 times the classification
time before and 2.0 times after. Three times the agents gave 7.3 times
before and 3.3 times after.
- "Allocated MB" is all bytes that the server process allocated for one
request (`total_allocated_bytes`), including the HTTP client of the
script.
- Other agents used the same machine during the measurement (load
average 8 to 16 on 18 CPUs), so the medians of server time have noise.
The profile columns and the allocation column are the stable ones.
- The task list changes little with 60 agents, because one page has
about 50 reviews. With 180 agents its classification falls from 15 ms to
3 ms and its allocation by half.
- Statement counts do not change: this change removes no statement.

Red/green regression test for the mechanism
(`server/src/__tests__/issue-graph-liveness-equivalence.test.ts`):

- `reads each agent and each path a number of times that does not grow
with the reviews`. The test counts the reads of the agent status and of
the path task id for 20 and for 400 stalled reviews with 40 agents and
360 paths. On the code before this change: `expected { agentStatus:
200800, pathIssueId: 144000 } to deeply equal { agentStatus: 10040,
pathIssueId: 7200 }`. After: 316 agent reads and 360 path reads for both
sizes (one read of each path). The test also runs the oracle and asserts
that its reads grow, so it shows what it catches.
- `classifies the review paths of a page from one index`. The same count
for `createIssueReviewPathClassifier`.

Equivalence tests (old code and new code on the same input):

- `returns the findings and review paths of the pre-fix classification
for random task graphs`. 400 seeded random graphs of two companies. Each
graph has reviews with and without paths, blocked chains and cycles,
terminal blockers, relations and paths that name the other company or a
missing task, reporting chains with cycles, missing managers and
managers of the other company, and ids that the input lists two times.
The test compares the findings of the graph and the review paths of
every task, for both `classifyIssueReviewPaths` and the new classifier.
It asserts that the graphs reach all six finding states and all nine
kinds of path, each at least 20 times.
- `returns the pre-fix findings for blocked chains, cycles and terminal
blockers`. One named graph with the expected state of each finding, so
the comparison cannot pass on two wrong answers.
-
`server/src/__tests__/review-attention-classification-equivalence.test.ts`,
on real PostgreSQL through the real routes. The test wraps the
classification module: every classification that a request runs is also
run by the oracle on the same input, and a difference fails the test.
- `gives every review the paths the fixture was built with, and the
pre-fix classification of every request`. `GET
/issues?status=in_review`, `GET /issues` and `GET
/issues?attention=blocked` (the blocked inbox, whose graph has the
chains, cycles and terminal blockers). Each of the 64 reviews has the
path kinds and the stalled reason that the fixture states for it,
including reviews that wait for an external reply and reviews that open
recovery tasks cover.
- `counts the reviews that need a person, leaves out the ones a person
dismissed, and reads one company`. `GET /attention/summary` and `GET
/attention`: the review count and the review cards are the reviews that
the fixture built to need a person. The second company has its own
counts and none of the first company's tasks. Three dismissed cards
leave the count and stay stalled in the task list.
- The benchmark above compares the complete responses of both versions
through the routes at each size (summary, full feed, task list, review
attention of every review).

Commands run locally:

- `pnpm --filter @paperclipai/server exec vitest run` for
`issue-graph-liveness-equivalence`,
`review-attention-classification-equivalence`, `issue-liveness`,
`recovery-classifiers`, `issue-review-attention`,
`issue-blocker-attention`, `blocked-inbox-review-wait-scope`,
`heartbeat-issue-liveness-escalation`, `attention-retention-work` and
`attention-service`: 210 tests passed. In that run the `afterAll` hook
of `attention-retention-work` and of `attention-service` (the stop of
the test database) timed out at 30 s while other agents loaded the
machine. No assertion failed. CI runs both files.
- `tsc --noEmit` for the server package: no errors. The new test files
and the script are outside the server `tsconfig`; a focused `tsc`
program for them reports no error in these files.
- `pnpm check:module-boundaries`, `node scripts/check-no-git-push.mjs`
and `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`:
pass.
- The first CI run of this pull request failed `ci / Runner support
suite`. An earlier revision also let the eligibility functions of
`packages/shared` accept a prebuilt index, and the source of
`packages/shared` is pinned for the runner SDK source proof
(`tests/runner-e2e/source-sdk-public-inputs.json`). The fix does not
need that change, so the current revision leaves `packages/shared` as it
is on `master`: no pin moves, and the UI, the CLI and the runner use the
same eligibility code as before. `source-sdk-public-inputs.test.mjs`,
`source-sdk-evidence.test.ts` and `source-sdk-public-type.test.ts` pass
locally on this revision (58 tests), and the drift guard reports no
stale pin.
- `pnpm verify` returns exit 2 (broader verification required) because
the change adds a file under `server/scripts/`. CI supplies the broad
coverage. The repo-wide typecheck, build and test suite were not run
locally (shared machine).

Not verified:

- The effect on the hosted instance. This pull request does not deploy.
The local requests take 0.1 s to 0.4 s, and the hosted requests took 15
s. Thus most of the hosted time is not this code: the hosted main thread
is saturated by other work, and each of the 69 to 85 statements of a
summary request waits for its turn on that thread. This change removes
CPU time and allocation from that thread. It does not remove a
statement.
- The number of agents and of stalled reviews of the hosted company. The
hosted allocation sample has most of the bytes in
`classifyIssueReviewPaths`, which is the cost for each review (reviews ×
agents). The local fixture has more stalled reviews, where the owner
candidates cost more.
- A remaining cost inside the classification: each eligibility decision
of `@paperclipai/shared` indexes the agent list again. One
classification now does this one time for each agent (agents² map
entries for one request, 3 ms with 180 agents), not one time for each
check. The helper can accept a prebuilt index in a later change; that
change moves the SDK source pins, so it needs its own review.
- A second, independent cost that this pull request does not change:
`GET /attention/summary` returns counts only, but it builds the full
feed. After this change, the largest application blocks in its profile
at 9000 tasks are the full task list of the 900 blocked tasks (15 ms on
the main thread, of which 9 ms decode description previews that the
summary never returns), the blocker attention of those tasks (9 ms), and
the text of its 85 statements. A later change can give the summary a
count-only path.

## Risks

- The attention feed decides what a person is told needs them. The risk
is a classification that differs from the previous one. Mitigation: the
rules did not move. Only the lookups changed (a map lookup instead of a
scan, a stored eligibility instead of a new one). The oracle tests
compare the findings and the paths for random graphs, for a named graph
and for every classification of the route tests. The benchmark compares
the full responses.
- The index is a snapshot of the input.
`createIssueReviewPathClassifier` documents that the input must not
change while the classifier is in use. The two callers build the input
and use it in the same synchronous block.
- When the input has no `now`, one classification now reads the clock
one time. Before, `classifyIssueGraphLiveness` read it one time for the
blocker checks and one more time for each review. All callers in the
server that classify many tasks pass `now` or run in one synchronous
call, so the values differ by less than the run time of the call.
- The oracle file is a copy of the previous classification (about 640
lines, test-only). A later change of a classification rule must change
the oracle in the same way. The header of the file says so.
- The repository rule in `CONTRIBUTING.md` says that a pull request has
no code comments. This pull request has comments where the code has a
constraint that the names do not show (the memo key, the snapshot rule,
the purpose of the oracle, the usage of the script), as the merged
changes around this code do.
- No migration. No query change. No change to company scope or to actor
permissions: the change is after the reads, on rows that the same
queries return.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (code reading, editing, shell, tests, CPU profiles).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 10:26:24 -06:00
Nicky LeachandPaperclip 782d6bead4 Configure managed plugin Runner lifecycle (#21065)
## Thinking Path

> - Paperclip runs agents in managed environments.
> - An agent can use an explicit environment or the instance default.
> - A lifecycle plugin must receive the driver that provisions the
agent.
> - A managed plugin environment also needs to select the Runner process
lifecycle.
> - This change gives lifecycle selection and provisioning the same
environment lookup.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Managed plugin environment provisioning and lifecycle configuration.

**Current behavior**
Lifecycle calls lack the selected plugin driver. Managed plugin
configuration cannot select warm Runner execution.

**Proposed behavior**
Resolve the environment once per lifecycle operation. Pass the driver
key for the plugin that serves it. Parse and persist the managed Runner
lifecycle mode.

The portable ownership and workspace placement changes are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066) and
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Share agent environment resolution between required plugin selection
and the lifecycle RPC.
- Pass `environmentDriverKey` to the plugin that serves the selected
environment.
- Allow managed plugin environments to set `warm` or `per_turn` and
persist the selected mode.
- Advertise `pluginEnvironmentRunnerLifecycleMode` in the release
capability manifest so senders can check support before including the
field.

## Verification

- Built shared types, the plugin SDK, and Runner. Server typecheck and
SDK source-pin check passed for the initial layer.
- The capability contract test failed before the manifest fix and passed
after it. Independent verification passed all 160 tests across managed
configuration, plugin environment persistence, lifecycle routing, and
environment reconciliation.
- Routing coverage includes the instance default, an explicit second
driver from the same plugin, and an explicit sandbox.

- [Hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/43193311862)
passed on `f07b192c21`. The first attempt timed out in an unchanged
workspace-recovery fixture and then collided with a background recovery
action. The exact test and all 44 tests in its suite passed locally; the
isolated shard retry passed without code changes.

## Risks

- Existing managed configuration can omit the lifecycle mode. Senders
must check the target release for `pluginEnvironmentRunnerLifecycleMode`
before sending the new field. Publishing this signal does not update
external deployment tools.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 08:18:04 -08:00
DottaandClaude Fable 5.2 1af987690d perf(storage): hash and encrypt workspace snapshots off the server thread (#21138)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process serves every request and agent turn on one
thread. The same process saves each remote workspace as one encrypted
archive in object storage, and restores it before a turn
> - `server/src/services/remote-workspace-snapshot-storage.ts` hashed,
encrypted and decrypted every byte of those archives on that thread.
`server/src/storage/s3-multipart.ts` computed the Content-MD5 of each
uploaded part on that thread too
> - Hashes and ciphers are synchronous in Node. A CPU profile of a busy
hosted server showed this work on the main thread, which was at 93% of
one core
> - Each megabyte of a snapshot delays every request and turn that waits
behind it
> - This pull request moves those passes to a small, bounded set of
worker threads. The checks, their order, their failures and the stored
format do not change
> - The benefit is that a 1 GiB save uses about 0.7 s of the server's
thread instead of about 2.8 s, and an unrelated request during the save
is answered in 1.3 ms (p99) instead of 16.7 ms

## Linked Issues or Issue Description

No public issue exists. I searched open and closed pull requests for
`workspace snapshot hash encrypt`, `event loop` and `archive io`.
Related merged work: #19900 gave these stream passes 1 MiB buffers,
#20554 removed a read-back from migration, #20722 took runner artifact
hashes off the event loop, #20709 records long event-loop blocks. No
open pull request covers this change.

**What happened?**

A 30 second CPU profile of the main thread of a hosted server (commit
1d153f62b6, Node 24, main thread at 93% of one core) showed:

- the two stream transforms in the snapshot save at 3.9% and 1.6%
inclusive. The first hashes the archive (SHA-256). The second hashes the
ciphertext (SHA-256);
- native hash `update` at 5.4% self time for the whole process. About
1.0 s of it was under those two transforms and 0.25 s was under
`fileRangeMd5`;
- the AES-256-GCM cipher of the same pipeline, and Buffer allocation
under the file read streams. Garbage collection was 25% of the thread.

What the code did with each byte, on the server's thread:

| Path | Pass | Digest or cipher | Purpose |
| --- | --- | --- | --- |
| Save | read archive | SHA-256 of plaintext | bound into the wrapped
key, checked on each restore |
| Save | same read | AES-256-GCM | confidentiality and authenticity |
| Save | same read | SHA-256 of stored bytes | recorded on the snapshot
row, checked on each restore and read-back |
| Save | read each part (S3) | MD5 | `Content-MD5`: the object store
rejects a part it did not receive intact |
| Save, read-back callers | read the kept file | SHA-256 of plaintext |
proof that the published file is the saved archive |
| Restore | download | SHA-256 of stored bytes | integrity of the object
against the snapshot row |
| Restore | read ciphertext | AES-256-GCM | authenticity |
| Restore | read plaintext | SHA-256 of plaintext | integrity against
the digest in the wrapped key |
| Restore | read plaintext again | none (copy) | publication beside the
destination |

**Expected behavior**

Digest and cipher passes over a snapshot do not run on the thread that
serves requests. Every check stays, with the same failure.

**Steps to reproduce**

1. On the parent commit, run `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-workspace-snapshot-crypto.ts --mib=1024` (copy the
script from this branch).
2. Read `upload.mainThreadCpuMs`, `upload.mainThreadBusyMs` and
`upload.unrelatedRequest`. The benchmark uses the real save and restore
functions, the real S3 client, and a local object store that checks each
part's `Content-MD5`. A separate process requests a trivial route from
the measured server and times each reply with its own clock.

**Paperclip version or commit**

`master` at 9e910ff9ae. The profile is from 1d153f62b6.

**Deployment mode**

All modes that save remote workspaces. The profile is from a hosted
server with S3 storage.

**Node.js version**

Profile: Node 24. Local benchmark and tests: Node 26.4.0 on darwin
arm64.

## What Changed

- `server/src/storage/file-crypto-pool.ts` (new): runs file digests and
AES-256-GCM passes on worker threads.
- Bounds: at most 2 jobs run at once. Each worker has two reused 1 MiB
read buffers and one write in flight. A waiting job holds file names and
byte ranges, not file contents. Waiting jobs are served in order.
- Cancellation: a waiting job is cancelled at once. A running job stops
at its next chunk, closes its files, and then the caller sees the same
`AbortError` as before.
- Shutdown: an idle worker does not keep the process alive. A busy
worker does, until its job ends or its caller's signal aborts it.
- Key handling: the pool copies the data key for the worker, moves that
copy (it is not cloned), and the worker erases it. A copy that never
reaches a worker is erased on this thread. The caller keeps and erases
its own key as before.
- The worker program is sent as source text, as
`remote-workspace-snapshot-archive.ts` already does. It needs no extra
file in the build or the runtime bundle.
- Save (`uploadWorkspaceSnapshot`): the one read that digests the
archive, encrypts it and digests the stored bytes runs in a worker. The
identity check around that read (`archive_changed`) stays on this thread
and is unchanged.
- Part upload (`fileRangeMd5`): the `Content-MD5` of each part is
computed in a worker. The first round of parts is digested before any
part is sent, so the parts still leave together and a rate limit still
answers one burst (the existing throttle test covers this). Every digest
of that round has ended, also after a failure or a cancellation, before
the upload settles.
- Restore (`loadWorkspaceSnapshotArchive`): the download no longer
hashes on this thread. One worker digests the downloaded object while
another decrypts it into private scratch. The results are then checked
in the previous order: object size and digest, header, key, GCM tag.
- Restore, one pass less: the decrypted archive was read twice (digest,
then copy). It is now digested while it is copied beside the
destination, and linked to the destination only when the digest matches.
The digest covers exactly the published bytes.
- Read-back callers (`verifiedArchivePath`): the digest of the kept file
runs in a worker, through the same open descriptor. The file identity
checks are unchanged.
- `server/scripts/benchmark-workspace-snapshot-crypto.ts` (new): the
reproduction and benchmark.
- Docs: `doc/run-log-events.md` (what each `storage.snapshot.*` phase
now does) and `doc/DEVELOPING.md` (the worker threads and the
benchmark).

Module boundaries (`rules/module-boundaries.mdc`): the pool is in
`server/src/storage/` because both consumers are parts of the same
upload, and storage must not import from services. Its contract is file
paths, byte ranges, an algorithm, and AES-GCM parameters. The snapshot
format (magic bytes, additional data, key wrapping, the order of checks)
stays in the snapshot service. The part layout and `Content-MD5` stay in
the S3 module.

Not changed, and why:

- The stored format, the ledger, attempt journaling, atomic publication,
and `recoverWorkspaceSnapshotUpload` /
`recoverWorkspaceSnapshotArchive`.
- `verifyObject` (full read-back for a store that did not verify the
upload, and for a retried save) still hashes on this thread. Its bytes
arrive from the network on this thread. To move it, the object must
first be written to disk.
- Socket writes of part bodies stay on this thread. I measured 1 MiB
part-body read chunks in place of 64 KiB: main-thread time did not
change and the longest event-loop delay increased, so I did not keep it.
- `remote-workspace-transport.ts` also hashes an archive on this thread.
It is outside the snapshot storage path.

## Verification

Regression tests, red then green (`cd server && pnpm exec vitest run
src/services/remote-workspace-snapshot-storage.test.ts
src/storage/file-crypto-pool.test.ts`):

- Parent implementation with the tests of this branch: 3 tests fail.
- `keeps encrypting a save while the server thread is occupied by other
work`: `expected 4194323 to be 25166144`. The test blocks the thread
until the encrypted file is complete. On the parent the file stops at 4
MiB.
- `answers unrelated requests while a save is being encrypted, and
digests or encrypts none of it on the server thread`: `expected 8388627
to be less than 65536` (bytes hashed on the thread).
- `digests and decrypts none of a save or a restore on the server thread
when the store verified the upload`: `expected 33556772 to be less than
65536`.
- This branch: 58 of 58 and 6 of 6 pass.
- The tests use synchronization, not time. A named pipe keeps a save in
the encrypt step for as long as the test feeds it. Deadlines are 30 s
and matter only for a failure.
- Integrity tests that pass on both the parent implementation and this
branch, which shows that the failures are the same: a truncated object
(`object_hash_mismatch`), an oversized object (`object_size_mismatch`),
an archive that grows, shrinks or is rewritten in place after its
identity was read (`archive_changed`, before any upload), a ciphertext
that was changed with the recorded digest changed to match (GCM
failure), a plaintext that differs from the digest in the key
(`plaintext_hash_mismatch`, staged copy removed), and the order object,
header, key.
- Cancellation: during encryption the save rejects with `AbortError`,
and leaves no scratch file, no object and no recorded attempt. During
the store write it leaves no object.
- Part digests (`src/__tests__/storage-s3-multipart.test.ts`): `leaves
no part digest running or waiting when the source is cancelled|removed
during the first round`. Before the review fix in a354242865 it failed
with `{ running: 2, waiting: 0 }` at multipart cleanup.
- Pool tests: digests equal single-threaded digests for a file, a range
and a descriptor; encrypt and decrypt round trip; each damaged
ciphertext is reported; at most 2 jobs run and the others wait in order;
a waiting job and a running job are both cancelled.

Benchmark, same machine, 5 alternating pairs (parent implementation,
then this branch) for each size. Values are medians, with the range in
parentheses.

Command: `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-workspace-snapshot-crypto.ts --mib=1024` (and
`--mib=200`).

1 GiB archive:

| Measure | Save, before | Save, after | Restore, before | Restore,
after |
| --- | --- | --- | --- | --- |
| Main-thread CPU time | 2,834 ms (2,717 to 3,007) | 742 ms (658 to 881)
| 1,602 ms (1,378 to 1,837) | 510 ms (470 to 589) |
| Main-thread busy time (event-loop utilization) | 2,938 ms | 858 ms |
1,673 ms | 597 ms |
| Total process CPU time | 6,735 ms | 4,762 ms | 5,564 ms | 3,306 ms |
| Wall time | 3,914 ms (3,656 to 4,253) | 2,580 ms (2,507 to 3,200) |
2,502 ms (1,756 to 2,922) | 1,799 ms (1,508 to 2,492) |
| Longest event-loop delay | 12.6 ms | 6.0 ms | 10.4 ms | 5.4 ms |
| Event-loop delay p99 | 9.2 ms | 2.0 ms | 2.1 ms | 1.8 ms |
| Unrelated request p99 (p50 is 0.2 to 0.3 ms in all runs) | 16.7 ms |
1.3 ms | 4.7 ms | 1.3 ms |
| Unrelated request, slowest | 18.5 ms | 4.5 ms | 10.0 ms | 3.5 ms |
| Peak resident memory | 495 MiB | 408 MiB | 524 MiB | 476 MiB |

200 MiB archive:

| Measure | Save, before | Save, after | Restore, before | Restore,
after |
| --- | --- | --- | --- | --- |
| Main-thread CPU time | 571 ms | 181 ms | 378 ms | 141 ms |
| Main-thread busy time | 602 ms | 236 ms | 395 ms | 167 ms |
| Total process CPU time | 1,235 ms | 935 ms | 1,385 ms | 840 ms |
| Wall time | 842 ms | 679 ms | 538 ms | 382 ms |
| Longest event-loop delay | 12.4 ms | 4.0 ms | 7.3 ms | 2.9 ms |
| Unrelated request p99 | 16.5 ms | 2.0 ms | 1.9 ms | 1.9 ms |
| Peak resident memory | 437 MiB | 398 MiB | 510 MiB | 424 MiB |

How to read these numbers:

- The parent did not block the thread for seconds at one time. It worked
in 1 MiB slices, so the longest single delay was about 12 ms. The cost
was the sum: the thread was busy for about 75% of a 1 GiB save. That sum
is what the profile showed, and it is what decreases.
- A worker thread moves CPU time; it does not remove it. Total CPU time
also decreased here (about 29% for a save, 41% for a restore) because
the workers reuse two buffers in place of one new 1 MiB Buffer for each
chunk, and a restore reads the decrypted archive once in place of twice.
- Wall time decreased because the part digests no longer wait for the
thread that sends the parts, and a restore digests and decrypts at the
same time.
- What remains on the server's thread is network and file streaming:
socket writes of the parts, the download, and the database calls.
- Limits: the machine was shared with other work (1-minute load average
8 to 14 on 18 cores) and timings varied between runs, which is why the
pairs alternate. Earlier runs on a busier machine (load 15 to 28) showed
the same CPU-time ratios with wall times up to 4 times longer for both
implementations. The object store was on loopback without TLS.

Other checks:

- `cd server && pnpm exec vitest run src/storage
src/services/remote-workspace-snapshot-storage.test.ts
src/services/remote-workspace-snapshot-reuse.test.ts
src/__tests__/remote-workspace-file-resources.test.ts` and the 8
`src/__tests__/storage-s3-*.test.ts` files: 249 tests in 15 files pass.
- `tsc --noEmit` over the changed TypeScript files: clean. `pnpm
check:module-boundaries`: pass.
- The pool, bundled with the options of
`server/scripts/bundle-runtime.mjs` and run with plain `node`: encrypt,
decrypt and digest round trip, and the process exits.
- `pnpm verify` exits 2 (broader verification required, because of the
new file in `server/scripts`). Required CI supplies that coverage.
- Not verified: the hosted server after deploy, Linux x86 with Node 24
outside CI, a real object store with TLS, and archives larger than 1
GiB.

## Risks

- This is integrity and confidentiality code. The mitigation is that
each check and its failure is pinned by a test that passes on the parent
implementation and on this branch, and that the format is unchanged, so
objects saved before and after this change restore with either version.
- Order of work in a restore: the ciphertext is now decrypted into
private scratch while the object digest is computed, not after it. No
output is used before the digest, header, key and tag checks pass, and
the scratch directory is removed on every failure. A damaged object is
reported with the same error, but after the decrypt work.
- The staged copy beside the destination (`.workspace-snapshot-*.tmp`)
is now written before the plaintext digest is known, and after the GCM
tag is verified. It is removed on a mismatch. The destination itself is
linked only after the digest matches, as before.
- Resource trade-off: up to 2 more threads, each with about 2 MiB of
buffers and a small heap (64 MB limit), that stay for the life of the
process once used. In exchange the server's thread does about 70% less
work for each snapshot. With more than 2 snapshot passes at one time,
the others wait. A part digest can wait behind a long encryption.
- Run-log timing shifts: `storage.snapshot.digest` is now about 0 ms,
`storage.snapshot.verify_plaintext` includes the copy, and
`storage.snapshot.publish` is only the link. `doc/run-log-events.md`
describes this.
- A worker that stops unexpectedly fails its job with
`file_crypto_worker_exited`. The save or restore then fails as for any
other I/O error, and the next job starts a new worker.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code with
tool use (shell, file edits, local test and benchmark runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 10:11:37 -06:00
DottaandClaude Fable 5.2 5b698b68cf perf(run-events): append a native event in one transaction of two statements (#21131)
One appended native run event cost eight statements on the database
connection (BEGIN, SAVEPOINT, run lock, replay probe, sequence update,
insert, count/max read, COMMIT). The query builder built five of them
again for every event, and each event created two transaction objects.

A native source event now takes the run lock and then runs one statement
that finds a replayed event or reserves the sequence and inserts, and
reads the retained count and maximum for the acknowledgement. The lock
stays a statement of its own so the append reads a snapshot taken after
the lock was granted. The control-plane port passes its controller check
to the append instead of opening an outer transaction, so there is no
SAVEPOINT. The hot statements are fixed SQL text.

Events that wake a queue still register the wake before the write and
only when the event is new. Order, sequence numbers and durability do
not change: every event still commits before its append returns.

Local benchmark (server/scripts/benchmark-run-event-append.ts), one
native event: 8.3 -> 4.6 statements, 589 -> 222 KB allocated,
1.09 -> 0.53 ms CPU, 28 -> 11.5 garbage collections per 1,000 events.
One ordinary event: 105 -> 27 KB allocated.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 09:58:31 -06:00
DottaandClaude Opus 5.5 9e910ff9ae fix(chat): keep raw IDs and internal codes out of chat replies (#21081)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and the agent replies in plain
text
> - In a full chat-eval cohort, 47 of 158 passing replies showed raw
UUIDs, error codes, field keys, tool names or server paths
> - Users must not see internal jargon or raw IDs, so these replies are
a product defect, even when the facts are correct
> - Agents copy these terms from API error messages and saved fields;
the chat directive did not tell them to translate
> - This pull request fixes the sources: the chat directive, and API
denial messages that put IDs and policy codes in the human message
> - The benefit is plain replies for every model, while machine codes
stay available in separate fields

## Linked Issues or Issue Description

**What happened?**

Chat replies quote internal names and raw IDs. Examples: run, handoff,
revision, interaction and trigger UUIDs in prose;
`native_session_retry_exhausted`, `board_access_required`,
`review_policy_denied`, `local-board`, `enableCases`; a raw user ID
inside the inbox denial message.

**Expected behavior**

Replies name records the way the person sees them (task identifier,
title or name, with a link) and explain codes in plain words. IDs stay
only in link targets, in commands the person must run, or when the
person asks for an ID.

**Steps to reproduce**

Run the chat-features OpenCode cohort (for example `cases.close-case`,
`routines.webhook-trigger`, `inbox-search.archive-without-grant`) and
read the agent replies.

**Paperclip version or commit**

`master` at f30005440b.

**Agent adapter(s) involved**

Not adapter-specific (core bug). Seen with the Paperclip runner and
OpenCode.

**Deployment mode**

Local dev server (embedded PGlite) and the CI chat-eval runner.

**Installation method**

Source checkout (`pnpm dev`).

## What Changed

- `AGENT_CHAT_DIRECTIVE`: add one rule. Name records by their
identifier, title or name and link them. Do not write raw IDs, internal
codes, field keys, tool names, process names or server paths, even in
inline code. The rule has exceptions for link targets, commands the
person runs, and IDs the person asks for. The rejection rule now asks
for the platform's reason "in plain words".
- Authorization denials (inbox management, membership, responsible user)
no longer put user or agent IDs in the explanation. The `reason` and
`code` values do not change.
- Review-policy denials explain the rule in plain words. The
`review_policy_denied` code and the `policy` and `allowedActor` fields
do not change.

## Verification

- Red/green: `server/src/__tests__/agent-conversations.test.ts`
(directive rule, and it reaches the chat task prompt),
`server/src/__tests__/authorization-service.test.ts` (inbox denials
carry no IDs), `server/src/__tests__/issue-review-policy.test.ts` and
`server/src/services/issue-thread-interaction-resolution.test.ts` (plain
review messages). Each new assertion failed before the fix.
- `pnpm verify` and the server typecheck pass locally.
- A paid OpenCode run on 8 affected chat cases is in progress; results
will be added here.

## Risks

- Low risk. Only human-readable messages and prompt text change. No
client matches the old message text; codes and machine fields stay the
same.
- The directive grows by one paragraph. A model can still copy an ID;
the rule reduces this but cannot prevent it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:37:07 -06:00
DottaandClaude Fable 5.2 8bdd53c11f perf(server): start no git process for server info outside a checkout (#21122)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reports which commit it runs. `GET /api/health` and `GET
/api/liveness` return it, and the board UI and the hosting control plane
poll these routes
> - The commit comes from `getServerInfoSnapshot()` in
`server/src/server-info.ts`. The health route calls it on each request.
The snapshot keeps its Git metadata for 3 seconds, then reads it again
with `execFileSync("git", ["show", ...])`, which is a synchronous child
process
> - A container or packaged install has the `git` command but no
checkout. There `git show` can only fail, and the server then uses the
commit that the build supplied. The failed start still stops the event
loop for the length of a process start
> - A CPU profile of a busy hosted server showed this cost on the main
thread
> - This pull request looks for a checkout before it starts `git`.
Without a checkout the server takes the build-commit path directly
> - The benefit is that health polls start no child process on a server
that does not run from a checkout, and the responses stay the same

## Linked Issues or Issue Description

No public issue exists. I searched open and closed pull requests for
`server-info git health`, `readGitInfo` and `getServerInfoSnapshot`.
Related merged work: #8753 added the 3 second refresh so that the debug
view follows a checkout, #10563 and #10566 added the build commit,
#18649 took the process start time off the Git probes. No open pull
request covers this change.

**What happened?**
`server/src/app.ts` creates the health routes without a `serverInfo`
option, so each `GET /api/health` request calls
`getServerInfoSnapshot()`. When the 3 second cache is expired, the call
starts `git show -s HEAD`. On an install without a checkout the command
fails each time, so the failure is paid again on each refresh. The rate
is one `git` start for each request when requests are 3 seconds or more
apart, and one start for each 3 seconds when requests are more frequent.
It is not one start for each request in a burst.

A 30 second CPU profile of the main thread of a hosted server (commit
1d153f62b6, Node 24, main thread at 93% of one core, process about 4 GB
resident) showed `getServerInfoSnapshot` → `readGitInfo` → the `git
show` command at 0.6% inclusive. That is about 190 ms in 32 seconds,
which agrees with one failed start for each 3 second refresh. The
profile showed only the `git show` command under `readGitInfo`, not the
status or branch commands that run after a successful `git show`. The
native child-process spawn was 0.9% self time in the full profile; other
callers also start processes.

**Expected behavior**
A server that does not run from a Git checkout starts no `git` process
for health or liveness requests. It reports the build commit, as before.
A server that runs from a checkout continues to read Git after each 3
second refresh, because the commit and the local changes of a checkout
can change while the process runs (#8753).

**Steps to reproduce**
1. Run `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-health-server-info.ts --requests=200` on the parent
commit. The benchmark puts a counting `git` wrapper first on `PATH`. The
wrapper records the start and then runs the real `git`.
2. Read the row `packaged install: no checkout, build supplies the
commit`, pattern `polled every 5000 ms`. It shows 200 `git` execs for
200 health requests, and 1 exec when the routes are created.

**Paperclip version or commit**
`master` at ef2b53b675. The profile is from 1d153f62b6.

**Deployment mode**
All modes. The profile is from an `authenticated` hosted server that
runs the container image. The image installs `git`, and the build
context excludes `.git` (`.dockerignore`).

**Node.js version**
Profile: Node 24. Local benchmark and tests: Node 26.4.0 on darwin
arm64.

## What Changed

- `server/src/server-info.ts`: the default `git show` command first
looks for a checkout. A checkout is a `.git` entry in the working
directory or in one of its parents, or a `GIT_DIR` value. Without one,
the command throws before it starts a process, and `readGitInfo` takes
its existing build-commit path. The same check applies when
`/api/liveness` reads the snapshot at route creation.
- The 3 second refresh, the checkout path, the injected-command path for
tests and `processStartedAt` are not changed.
- `server/src/__tests__/health-server-info-git-probe.test.ts` (new):
regression tests that count the `git` processes that the server starts.
The tests use real Git and real directories.
- `server/scripts/benchmark-health-server-info.ts` (new): the
reproduction and benchmark.
- No route file is changed. `server/src/routes/health.ts` continues to
read the snapshot on each request.

## Verification

Regression test, red then green:

- `cd server && pnpm exec vitest run
src/__tests__/health-server-info-git-probe.test.ts`
- Parent commit (with the test file of this branch): 1 of 4 tests fails.
`starts no git process when the server does not run from a checkout`
fails with `expected 6 to be +0`: one start when the routes are created
and one for each of 5 polls. The response assertions in that test pass
on the parent commit.
- This branch: 4 of 4 tests pass.
- The tests assert the number of `git` starts and the response content.
They do not assert time. The first test compares the full response text
of anonymous health, board health and liveness with the responses of a
server that has the snapshot injected. The other 3 tests pass on both
commits. They keep the checkout behavior: a new commit is visible after
the refresh, also from a subdirectory; a linked worktree (`.git` is a
file) is read; `GIT_DIR` is obeyed.

Benchmark, same machine, 200 health requests per row, 3 runs each
(ranges shown). Each row alternates anonymous and board requests against
routes that are created as `server/src/app.ts` creates them.

| Scenario | Request pattern | Before: `git` execs (route creation +
requests) | Before: ms per request | After: `git` execs | After: ms per
request |
| --- | --- | --- | --- | --- | --- |
| No checkout, build supplies the commit | polled each 5 s | 1 + 200 |
12.5 to 13.8 | 0 + 0 | 1.05 to 1.53 |
| No checkout, build supplies the commit | burst | 1 + 0 | 1.47 to 1.63
| 0 + 0 | 1.19 to 1.57 |
| Source checkout | polled each 5 s | 3 + 600 | 36.3 to 50.2 | 3 + 600 |
36.8 to 40.7 |
| Source checkout | burst | 3 + 0 | 1.38 to 1.40 | 3 + 0 | 1.03 to 1.45
|

- Command: `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-health-server-info.ts --requests=200`
- The `polled` pattern moves the clock that the server reads forward by
5 seconds before each request. It does not wait for real time.
- The benchmark prints a SHA-256 prefix of the anonymous health body, of
`serverInfo.git` in the board health body and of the liveness body. The
values are the same before and after in all rows.
- The `git` call is synchronous, so the difference per request (about 11
to 12 ms locally, which includes the shell wrapper) is time that the
event loop is blocked.
- The source checkout rows do not change. This is the intended result.

Assumption and evidence (`rules/runtime-dependency-boundaries.mdc`): the
change assumes that real Git fails where the check finds no checkout,
and that the check finds each checkout that Git finds. The first
regression test runs real `git show` in the same directory and asserts
that it fails. The other tests run real Git from a repository
subdirectory, from a linked worktree and with `GIT_DIR`. The new check
itself is a small number of `existsSync` calls (one for each directory
level) on each 3 second refresh.

Other checks:

- `cd server && pnpm exec vitest run src/__tests__/server-info.test.ts
src/__tests__/health.test.ts src/__tests__/liveness.test.ts
src/__tests__/health-dev-server-token.test.ts
src/__tests__/health-workspace-readiness.test.ts
src/__tests__/cloud-warm-standby.test.ts
src/services/native-runtime/native-controller-identity.test.ts` and the
new test file: 58 tests in 8 files pass.
- `tsc --noEmit` over the three changed TypeScript files: clean.
- `pnpm verify` exits 2 (broader verification required, because of the
new file in `server/scripts`). Required CI supplies that coverage.
- Not verified: the benchmark on Linux with a multi-gigabyte process,
and the hosted server after deploy. The local numbers come from a
process of about 700 MB, where a process start is cheaper than on the
profiled server. That the profiled server has no checkout is a
conclusion from the image build and from the profile; I did not read its
`serverInfo` response.

## Risks

- Low risk. The response bodies of `/api/health` and `/api/liveness` are
the same as before for an install without a checkout and for a checkout.
- Behavior shift: a server whose working directory is inside a bare
repository, with no `.git` entry and no `GIT_DIR`, reported the commit
of that repository before. It now reports the build commit, or no
commit. This layout is not a supported way to run the server.
- A server that runs from a source checkout continues to start up to 3
`git` processes on each 3 second refresh while health is polled. This
pull request does not change that path, because #8753 made it read the
checkout live on purpose.
- If a checkout is created around the working directory while the server
runs, the next refresh finds it, because the check runs on each refresh.

## Model Used

- Claude Fable 5.2 (Anthropic, model ID `claude-fable-5-2`), in Claude
Code with tool use (shell, file edits, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 09:28:07 -06:00
DottaandClaude Opus 5.5 ef2b53b675 fix(environments): keep resuming an idle sandbox whose resume timed out (#21113)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on the next wake
> - When the resume call times out, the run failed after 90 seconds with
`reusable_sandbox_resume_failed (cause=transport_timeout)`
> - The provider keeps starting the sandbox after the host call times
out, so the sandbox is usually ready a little later
> - The run gave up after one call, and the task then waited behind a
Retry button
> - This pull request keeps the same run retrying an idle sandbox whose
resume only timed out, up to the provider's start deadline
> - The benefit is that recovery works automatically and a sandbox with
unsaved work is never destroyed

## Linked Issues or Issue Description

Refs #20626, #20828, #21020

**What happened?**
Three Codex runs on Daytona failed before provider dispatch with:
`Reusable sandbox lease "<id>" could not be resumed
(cause=transport_timeout); the lease was preserved and no replacement
was created.` Each run had `executionRecovery: { kind: "bootstrap",
providerWorkStarted: false }`. Each run took 100 to 122 seconds: one
resume call used the whole 90-second budget, so no in-run retry was
possible. The preserved leases were idle. Their last runs had ended
about two hours before, with a confirmed stop-and-retain. The workspaces
were not safe to replace, so the run failed strictly. The automatic
bootstrap retry was then refused ("The automatic retry could not
revalidate the original user continuation."), and all three tasks ended
`blocked` behind a Retry.

**Expected behavior**
A resume that only timed out, before any provider work, on a sandbox
that no live run holds, continues in the same run. The run keeps the
same sandbox and does not create a replacement or destroy the sandbox.

**Steps to reproduce**
1. Let a task's reusable Daytona sandbox stop and stay idle for some
hours (it auto-archives after 60 minutes).
2. Retry the task. The workspace has unsaved work, so replacement is not
safe.
3. The `environmentResumeLease` call takes longer than 90 seconds. The
host call times out (`RPC call "environmentResumeLease" timed out after
90000ms`), the 90-second budget is spent, and the run fails with
`reusable_sandbox_resume_failed`.

**Paperclip version or commit**
Staging at `849f109f` (current `master` has the same code path).

**Deployment mode**
Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases
on.

**Agent adapter(s) involved**
`codex_local` (Codex) on a Daytona sandbox environment. The change is
adapter-independent.

**Root cause**
`server/src/services/environment-runtime.ts`, reusable resume loop: the
first `environmentResumeLease` call got the whole 90-second budget as
its RPC timeout. When that call timed out, the loop saw no time left and
threw. The 90-second budget assumes that the bounded bootstrap retry
resumes the same sandbox later. That retry is not always admitted. A
timeout is different from a refusal: the provider is still starting the
sandbox.

## What Changed

- `reusable-sandbox-resume-replacement.ts`: add
`reusableSandboxLeaseIsIdle` (lease is `released` or `retained` and no
other live run holds it). `reusableSandboxReplacementIsSafe` now uses
it, with the same rules as before. Add
`REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS` (330 seconds, the Daytona start
deadline plus the call margin).
- `environment-runtime.ts`: when the 90-second budget ends on a
`transport_timeout` and the preserved lease is idle, extend the deadline
one time, up to the provider start deadline (at most 330 seconds from
the first attempt). The run keeps retrying the same sandbox with
backoff. Each call still has at most 90 seconds. Refusals, a live owner,
and other failures keep the current behavior. The run never replaces or
destroys the sandbox because of this extension.
- `doc/execution-semantics.md`: document the idle-timeout extension.

## Verification

- Red before the fix, with the new tests in
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`
(staging shape: released lease, finished owner run, workspace with
unsaved work, 300-second provider deadline, host RPC timeout): 2 tests
failed with the staging error `could not be resumed
(cause=transport_timeout); the lease was preserved and no replacement
was created.`
- Green after the fix: 16/16 pass. The timed-out idle sandbox resumes in
the same run with no create or destroy call. If every call times out,
the run stops at or before 330 seconds, makes more than one call, and
keeps the lease `released`. A sandbox that a live run still holds gets
one call and fails at 90 seconds, as before.
- `pnpm exec vitest run src/__tests__/environment-runtime.test.ts
src/services/environment-runtime-reusable-lease-probe.test.ts
src/services/remote-workspace-transport.test.ts`: 378 pass.
- `tsc -p server/tsconfig.json --noEmit` passes.
- `pnpm verify` exits 2 because a doc is a global CI input. Required CI
supplies the broad coverage.
- Not run: a live Daytona resume on staging. No later run attempted the
three preserved leases, so it is inferred, not observed, that the
sandboxes were ready after the host call timed out.

## Risks

- A run can now wait up to 330 seconds in preparation before its first
turn, but only when the resume timed out on an idle sandbox. Before this
change, that run failed and the task waited for a person.
- A provider that is down (not slow) now takes up to 330 seconds to fail
instead of 90 seconds. The failure is the same
`reusable_sandbox_resume_failed`, and the lease is still preserved.
- No persisted fields change. Leases saved by older releases follow the
same rule.
- The refused bootstrap retry admission for task Retry continuations is
a separate gap. This pull request does not change it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:28:32 -06:00
DottaandClaude Opus 5.5 704b3f4d63 fix(server): a card accepted while its run settles keeps one continuation (#21094)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run ends while its task is in review, Paperclip checks that
the task still has a maintained review path. If it has none, Paperclip
queues one review-path recovery run
> - An approval card for a tool action sends its continuation through a
delivery receipt. The receipt waits until the source run has settled and
the action has finished
> - A person accepted a card while the source run was still settling.
The interaction was no longer pending, and the continuation wake did not
exist yet, so the review-path check found no path and queued a recovery
run
> - The real continuation then ran too, so the task got two follow-up
runs for one decision
> - This pull request counts an undelivered tool review continuation as
a maintained review path
> - The benefit is that a card accepted during settle produces exactly
one continuation run

## Linked Issues or Issue Description

**What happened?**
An agent posted a tool approval card and its run finished. The person
accepted the card 1.3 s later, while that run was still settling. The
review-path check saw no pending interaction and no queued wake, and it
queued an `issue_review_path_lost` recovery run. The accept continuation
was deferred behind that run and then started as a second follow-up.

**Expected behavior**
The accepted card's pending continuation counts as the task's review
path. Exactly one continuation runs.

**Steps to reproduce**
Have an agent in a task that is in review post a tool approval card.
Accept the card while the creating run settles and before the tool
action delivers its result. Before this change, a review-path recovery
run was queued in addition to the continuation.

## What Changed

- `server/src/services/issues.ts` (`listIssueReviewAttentionMap`): reads
tool review continuations for the review tasks in one statement. A
continuation counts as a queued review path when (a) its receipt is
undelivered, its card is resolved, and the requesting assignee can still
receive it (not paused, terminated or pending approval), or (b) its
delivered continuation wake, or that wake's run, is still active. The
path list leaves out a continuation whose wake or run is already listed,
so each continuation shows once. The path shows as "Queued tool action
review wake".
- `heartbeat-issue-liveness-escalation.test.ts`: a new embedded-Postgres
test runs a heartbeat for a task in review with an accepted card whose
tool action is still executing. It checks that no review-path recovery
wake is queued for that run. It then completes the action, delivers it,
and checks that exactly one continuation wake exists.

## Verification

- The new heartbeat test failed before the change (one
`issue_review_path_lost` wake) and passes after it.
- A new parameterized attention test covers an undelivered receipt, a
delivered continuation (listed and not listed in the wake list), a
pending card, a terminated assignee and a consumed continuation. The
terminated, pending-card and delivered cases were red against the
earlier revisions of this change.
- `pnpm vitest run
src/__tests__/heartbeat-issue-liveness-escalation.test.ts
src/__tests__/issue-review-attention.test.ts
src/services/recovery/review-path-recovery.test.ts
src/__tests__/attention-service.test.ts
src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts
src/__tests__/tool-gateway-service.test.ts`: 332 passed.
- Product evidence: a paid chat-eval run of
`questions-and-approvals.approve-tool-once` on a branch that combines
this change with three related fixes. The result is in a PR comment.

## Risks

- Every caller of review attention (review-path recovery, its
escalation, the attention feed and the approvals view) now sees a
pending tool review delivery as a covered path. That is the intended
state: the delivery service always sends one continuation for it, and
its periodic sweep retries a failed delivery.
- One more indexed query for each review-attention read. It reads only
undelivered receipts for the given tasks.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:16:32 -06:00
DottaandClaude Opus 5.5 81d2e4d8d9 fix(native): end a run whose finalizer a newer run superseded, so it stops holding its task (#21106)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result, then a finalizer exports the
workspace and applies the result to the task.
> - When a newer run already committed the task's decision, the
finalizer retires the older result as `native_finalization_superseded`.
> - That retirement set the finalizer terminal, but it did not end the
run. The run stayed `running` and held the task and every run queued
behind it, with no way out except a person.
> - A stuck run must never hold a task and its queue forever.
> - This pull request ends the run when its finalizer is retired as
superseded, and it ends runs that an older release left in that state.
> - The benefit is that the task continues with its queued runs by
itself.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. The workspace export then failed
(`remote_workspace_archive_fs_eacces`). The task's last decision came
from another run that was created 42 seconds later but ran earlier. The
finalizer judged this run superseded: it set the finalizer to
`terminal_failure` with `native_finalization_superseded`. The run stayed
`running` with "Recovery needed". Over one hour later it still held the
task, and five queued runs waited behind it. A server upgrade at 12:02Z
did not change it.

**Expected behavior**
A run whose finalizer is retired ends. The task keeps the newer run's
decision, the retired result is never applied, and the queued runs
proceed.

**Steps to reproduce**
1. Accept a native result for a run.
2. Commit a status decision on the same task from a run that was created
later.
3. Let the older run's finalization fail once, so the finalizer records
the failure.
4. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is still `running`.

**Root cause**
`recordRetryableFailure`
(`server/src/services/native-runtime/native-run-finalizer.ts`) projects
a terminal run status only when `exhausted && !supersededByNewerRun`. A
superseded retirement therefore never ended the run. No other path ends
a run whose finalizer is already `terminal_failure`: the reconciler
skips `terminal_failure` rows.

**Paperclip version or commit**
Staging Core `3eca7a6112`. The behavior is the same on master
`482063aeca`.

## What Changed

- `native-run-finalizer.ts`: a superseded retirement now ends the run
with the same status mapping as an exhausted finalization (workspace
failure: `failed`; otherwise the provider's terminal state). It does not
change the task, does not apply the result, and does not keep the
workspace export lease. The run's error says in plain words that a newer
run already decided the task.
- `native-run-finalizer.ts`: add `endSupersededNativeFinalizationRuns`.
It ends a native run that is still `running` while its finalizer is
`terminal_failure` / `native_finalization_superseded`. This covers rows
that an older release wrote. It locks in the finalizer's order
(coordinator, task, run), and re-checks the row under the lock.
- `native-run-projection-work.ts`: the registered native run projection
worker (it loads at startup, pages through all candidates, and wakes on
finalization changes) also selects such runs and ends them with
`endSupersededNativeFinalizationRuns`.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/__tests__/native-finalization-recovery.test.ts`, red on
master and green on this branch:
- "retires an older failed finalizer when a newer run already committed
the issue": the retired run now ends (`succeeded`, `finishedAt` set).
Master: the run stays `running`.
- "ends a run an earlier release left running after retiring its
finalizer as superseded": the shape of the staging row (workspace export
failed, superseded finalizer, run `running`). The repair ends it as
`failed` with the plain message. The task, its decision and the
finalizer stay unchanged, and no status decision is written for the run.
A second pass changes nothing.
- `server/src/__tests__/native-run-projection-work.test.ts`, "ends, at
startup, a run an earlier release left running after retiring its
finalizer as superseded": the registered worker ends the staging-shaped
row at startup and settles its claimed wake. Red without the worker
change.
- Related suites pass: `native-finalization-recovery`,
`native-run-projection-work`, `native-finalization-reconciler-timing`,
`native-reconciliation-timing`, `native-workspace-finalizer`,
`native-workspace-finalizer-failure`, `native-run-finalizer-telemetry`.
- `tsc --noEmit` for `server`.

## Risks

- Low risk. The change only ends runs whose finalizer is already
terminal and superseded. It never applies a result and never changes the
task.
- A superseded run whose workspace export failed now releases its
sandbox lease through the normal settlement instead of waiting forever.
Its result was not applied, because a newer run decided the task.
- Not changed here: the superseded check compares run creation times. A
run created earlier but executed later is still judged superseded. This
PR makes that verdict end the run; it does not change the verdict.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:07:18 -06:00
DottaandClaude Opus 5.5 0a2f618cae fix(native): leave a run another live server is preparing alone, and re-adopt a run a shutdown handed off (#21107)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade replaces the server. The old and the new server serve
together for a few minutes, then the old server gets SIGTERM and hands
its native runs to the next controller.
> - #21046 taught the legacy reaper to leave a run with its owner while
the owner's server process is proven alive. Native runs have a similar
gap.
> - Runtime selection writes a native run's coordinator with no
controller and no lease, and stops the run lease. The executor claims
the coordinator about 20 seconds later. In that window the other
server's sweep sees no owner and marks the run "ownership could not be
verified".
> - When the old server then hands the run off before any Runner
checkpoint exists, no sweep looks at it again. Only the next server boot
does, 15 to 40 minutes later. The task and its queued runs wait.
> - This pull request leaves such a run with its live preparing server,
and lets the serving server re-adopt a handed-off run through the
ordinary restart claim.
> - The benefit is that a run started during an upgrade continues within
one sweep, with no false "blocked" mark and no wait for another upgrade.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, an upgrade started a new server at
12:19:57Z. The old server started a native run and selected its runtime
at 12:20:32Z. At 12:20:35Z the run logged "Native execution ownership
could not be verified; automatic recovery is blocked"
(`observed_owner_unverified`). The old server's executor claimed the run
at 12:20:51Z. The old server got SIGTERM and suspended the run at
12:20:54Z (`awaiting_runner_reattach`). After that the run showed
"Confirming execution" for 29 minutes, and two queued runs waited behind
it. At 12:49:45Z the next upgrade's startup recovery restarted its
bootstrap, and the run succeeded at 12:58Z.

**Expected behavior**
- A server does not mark a run unverified, or recover it, while another
live server is preparing it.
- A run that a graceful shutdown hands to the next controller is
re-adopted by the serving server at once, not at the next boot.

**Steps to reproduce**
1. Seed a native run whose coordinator is `observed` with no controller
and no lease, and whose `controllerProcess` names another server with a
live controller instance lease.
2. Run the recovery sweep on this server. See that the run gets
`native_execution_ownership_unverified`.
3. Claim the coordinator for the other server, then write what its
graceful shutdown writes (`restart_suspended`,
`awaiting_runner_reattach`, lapsed lease) and stop its instance lease.
Give the run a predecessor's Runner PID.
4. Run the sweep again. See that nothing claims the run.

**Root cause**
- `recoverActiveExecutions`
(`server/src/services/heartbeat/recovery.ts`) treats an `observed`
coordinator without a live controller lease as an owner it cannot
verify. Between runtime selection and the claim, the only owner on
record is the run's `controllerProcess`, and the sweep did not check it.
- `listExpiredNativeRunnerAuthorities` (`native-runner-checkpoint.ts`)
selects runs for the restart claim only when they have a controller
checkpoint, or no process identity at all. A run handed off just after
its claim has no checkpoint, and a follow-up carries its predecessor's
Runner PID. The sweep then skips `awaiting_runner_reattach` runs, so
only a server boot looked at them again.

**Paperclip version or commit**
Staging Core `814fe5d20f` (old server) and `3eca7a6112` (new server).
The behavior is the same on master `849f109f62`.

## What Changed

- `recovery.ts`: before it lists expired native runs, the sweep checks
runs in the runtime-selection window (coordinator `observed`, no
controller, no lease, not tracked here). When the run's
`controllerProcess` names another server process that the shared owner
prover proves alive, the sweep leaves the run with that server: it does
not recover it and does not mark it unverified. It re-checks within 30
seconds. After 5 minutes without a claim
(`LEGACY_LIVE_OWNER_SETTLE_GRACE_MS`), or once that server is gone, the
existing rules apply.
- `native-owner-stamps.ts`: add `preparingControllerOwnerStamp` for that
proof.
- `native-runner-checkpoint.ts`: `listExpiredNativeRunnerAuthorities`
also selects a run that a graceful shutdown handed off and whose lease
lapsed: recovery state `awaiting_runner_reattach` or
`awaiting_evidence`, with a `restart_suspended` history entry that no
restart claim followed. It stays selected through ownership waits (the
departing server can still be alive at the first look). `recovery.ts`
schedules a 30-second re-check for an ownership wait that has no retry
of its own. The ordinary restart claim then proves the takeover and
picks the disposition, exactly as the startup pass of a new server does.
A row that an older release already marked unverified is re-adopted too:
the claim clears the mark.
- `doc/execution-semantics.md`: document both rules.
- No schema change. Nothing changes on the path that starts a run or a
turn: the new proof runs only in the recovery sweep, and only for runs
in the window.

## Verification

- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`,
block "a run the old server started during an upgrade overlap". Red on
master, green on this branch:
- "leaves the run with its live preparing server, then re-adopts it
after that server hands it off": two concurrent sweeps leave the run
`running` with no error and do not touch the coordinator. After the old
server claims, hands the run off and stops, one sweep of the new server
re-adopts it and reaches provider dispatch. Master: the run is marked
`native_execution_ownership_unverified`, and after the handoff nothing
claims it.
- "keeps a handed-off run eligible while the departing server is still
alive, and re-adopts it once it exits": the first sweep waits
(`awaiting_evidence`), the sweep after the departing process exits
re-adopts the run. Red with a filter that matches only the last history
entry.
- "re-adopts a handed-off run that an older release already marked
ownership-unverified": the stuck staging shape. Master: nothing claims
it.
- Related suites pass: `heartbeat-native-restart-suspension` (all 109),
`services/heartbeat/recovery.test.ts` (all 103).
- `tsc --noEmit` for `server`.

## Risks

- A run that another live server prepares, but never claims, now stays
`running` for up to 5 minutes before the existing rules apply. That is
the same bound #21046 uses.
- The guard trusts the shared owner prover. It reads "alive" only from a
live controller instance lease or a live PID in the same PID space.
- More handed-off runs reach the restart claim earlier. The claim keeps
all its fencing and evidence checks; this change only makes the serving
server ask the same question a new boot asks.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:06:57 -06:00
DottaandClaude Opus 5.5 a65644d1e1 fix(retry): continue an accepted connected Retry from recovered saved work when its chat replay is refused later (#21080)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks from a connected chat (for example GitHub pull request
reviews) show Retry when a run fails
> - A person's Retry on such a task first asks the chat retry owner to
repeat the exact failed chat request. #21038 made a refusal at the click
continue the task from saved work instead
> - The chat owner can also defer its source check (a busy identity,
endpoint, or agent row lock) and refuse it a few seconds later. Then the
route has already answered 202, and the task Retry receipt was marked
failed with no run
> - The task stayed blocked, and its execution blocker told the person
to send a message, while the interruption projection still offered Retry
> - This pull request makes that late refusal take the same board
continuation as a refusal at the click, and makes the blocker offer
Retry for recovered saved work
> - The benefit is that an accepted Retry always starts work, and the
two projections agree

## Linked Issues or Issue Description

No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a hosted deploy, about 20 blocked tasks of one busy GitHub review
agent (`paperclip_runner`, native remote runs) did not restart after
Retry. `POST /api/issues/:id/retry` returned 202 with `accepted: true`,
a `chatActionId`, and `presentation.state: retry_scheduled`. No run was
created. The task Retry receipt changed to `failed` 5 to 10 seconds
later, and the task stayed blocked. The activity record of each click
(`chat.failed_run_retry_requested`) has `sourceAuthorizationPending:
true`, so the chat owner deferred its source check at the click. No chat
wake row was written, so the chat replay ended before admission. The
last run failed with `native_continuation_requires_reconciliation` /
`runner_remote_process_identity_unavailable`. Its saved work was
recovered (`workspaceRecovery.status: recovered`, remote lease destroyed
with a termination receipt). The projections disagreed: `interruption`
had `canRetry: true`, but `executionBlocker` had `canRetry: false` and
`nextAction: "Saved work recovered. Send a message to continue."`.

Root cause: when the chat retry action settles `failed`,
`releaseUnadmittedTaskRetryInput` in
`server/src/services/chat-channels/task-retry-input.ts` marked the
linked task Retry receipt `failed`. #21038 added a board-continuation
fallback only for a refusal inside the click transaction. A refusal
after a deferred source check (for example 8 lock-busy attempts on the
agent row, about 7 seconds) had no fallback. Separately,
`getExecutionBlocker` hid Retry for every chat-bound task (`canRetry &&
!chatBinding`), and `workspaceRecoveryDisplay` always said "Send a
message to continue" for recovered work.

**Expected behavior**

A Retry that is accepted starts work. If the exact chat replay is
refused after the click, the person's Retry continues the task from its
recovered saved work. The execution blocker and the interruption both
offer Retry, and neither tells the person to send a message to unblock a
failed run.

**Steps to reproduce**

1. Bind a task to a GitHub chat conversation. Make its latest native run
fail with `native_continuation_requires_reconciliation` and recover its
saved work.
2. Hold the chat identity advisory lock while you click Retry, so the
source check is deferred (`sourceAuthorizationPending: true`).
3. Hold a row lock on the agent while the chat retry processes, so the
deferred check fails 8 times.
4. The route returned 202. The chat action settles `failed`, the task
Retry receipt changes to `failed`, and no run starts.

**Paperclip version or commit**

`master` at a8580160 (the deployed revision). Reproduced on f30005440b.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

`paperclip_runner` (native remote runs). The fix is in core Retry code
and is not adapter-specific.

## What Changed

- `server/src/services/task-retry.ts`: new `settleRefusedTaskChatRetry`.
When a chat retry action settles `failed`, the linked task Retry receipt
becomes an ordinary board continuation if all of these are true: it is a
person's Retry (or an automatic Retry that the same person's click has
joined), its source run is still the latest run and its saved work is
recovered (system recovery `complete`, workspace recovery `recovered`),
the clicking person still has task access, the chat wake has no run, and
no saved or unadmitted chat input remains. Any other refusal still
settles the receipt as failed, so the existing source-edited, revoked,
and identity-expiry outcomes do not change. A paused agent does not fail
the receipt: it stays deferred and starts when dispatch allows. The
receipt drops `chatActionId`, stays `deferred_issue_execution`, notifies
the deferred-receipt worker, and records
`issue.retry_continued_from_saved_work`. Otherwise the receipt is failed
as before. `reconcileTaskChatRetryReceipts` uses the same check, so a
crash before settlement is also covered. Its failed write matches the
chat binding it read, so a stale pass cannot fail a receipt that
settlement already converted.
- The task Retry intent now records `initiatedBy` for an automatic
Retry, so an automatic Retry keeps the exact-replay requirement after
the click too.
- `server/src/services/chat-channels/task-retry-input.ts`:
`releaseUnadmittedTaskRetryInput` calls the task owner's settlement
instead of failing the receipt directly.
- `server/src/services/execution-blocker.ts` and
`workspace-recovery-receipt.ts`: for a failed or timed-out run whose
saved work is recovered, the blocker offers Retry also on a chat-bound
task. The text follows actual Retry eligibility: "Saved work recovered.
Retry continues from it." when Retry is offered, "Saved work recovered."
when it is not (for example cleanup quarantine), and "Send a message to
continue" only for a stopped run. Only the task blocker decides Retry.
The run-level projection (`execution-projection.ts`) has no task
context, so it says "Saved work recovered." and never claims Retry.
`execution-projection.ts` uses the same text.
- `ui/src/lib/activity-format.ts`: labels for the new activity.
- `doc/execution-semantics.md`: the late-refusal rule.
- Tests: a new real-service case in
`server/src/__tests__/chat-channels.integration.test.ts` that reproduces
the hosted state (remote saved workspace, destroyed lease receipt,
recovered recovery action, deferred source check, refused replay), for a
person's Retry and for an automatic Retry. A gate case per continuation
condition (unrecovered workspace and revoked access settle failed; a
paused agent stays deferred and runs after resume), a stale-reconcile
race case, and a joined-click case. Two existing text expectations are
updated.

## Verification

- Red before the fix: with only the source changes removed, the new test
fails on the projection (`canRetry: false`, "Send a message to
continue."). With only the Retry settlement removed, it fails with the
receipt `status: "failed"` and no run.
- Green after the fix: `npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "continues an
accepted GitHub task Retry from recovered saved work"` passes 2/2. The
person's Retry starts one queued run that resumes the preserved
workspace, and the automatic Retry still settles as failed.
- Focused suites pass: `task-retry`, `execution-projection`,
`execution-projection-recovery`, `workspace-recovery-retry-copyback`,
`chat-failed-run-retry-boundary`, `workspace-recovery-admission`,
`remote-workspace-recovery-receipt-fallback` (372/372), and other
recovery, blocker and continuation suites.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Staging was read with GET requests only. No POST was sent.

## Risks

- Behavior change: a person's Retry on a connected task that the chat
owner refuses after the click now starts a board continuation. It is the
same continuation #21038 already uses for a refusal at the click, with
the clicking person's authority, and dispatch keeps every normal gate.
- Receipts that already failed on staging stay failed. The next click on
those tasks takes the fixed path. No migration.
- The blocker now reports `canRetry: true` for chat-bound tasks only
when the failed run's saved work is verifiably recovered.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:57:42 -06:00
DottaandClaude Opus 5.5 849f109f62 fix(server): agent inbox reads now apply private-task visibility (#21092)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents read a person's Mine list with `GET
/api/agents/me/inbox/mine`, and their own queue with `GET
/api/agents/me/inbox-lite`
> - The company task list routes apply private-task visibility for the
reading actor
> - The two agent inbox routes should use the same visibility rules, so
that each route returns the same tasks for the same actor
> - This pull request makes agent inbox reads apply private-task
visibility
> - The benefit is one consistent visibility rule across task list
routes

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Agent inbox reads now apply private-task visibility.

**Subsystem affected**
Server API: agent inbox routes.

**Current behavior**
The agent inbox routes build their lists without the issue read
condition that the company task list routes use.

**Proposed behavior**
The agent inbox routes use the same issue read condition and per-row
read decisions as the company task list routes.

**Reason and benefit**
All task list routes apply one visibility rule.

## What Changed

- `server/src/routes/agents.ts`: `GET /api/agents/me/inbox/mine` adds
`issueReadSqlCondition` to its list query and keeps only the rows that
`access.decideReads` allows.
- `GET /api/agents/me/inbox-lite` adds the same read condition to its
list query.
- `inbox-mine-routes.test.ts`: a new embedded-Postgres test for the
agent Mine read with private and open tasks.
- `agent-permissions-routes.test.ts`: the mocked route test expects the
read condition and the read decisions.

## Verification

- `pnpm vitest run src/__tests__/inbox-mine-routes.test.ts`: 7 passed.
The new test was red before the route change and is green after it.
- `pnpm vitest run src/__tests__/agent-permissions-routes.test.ts
src/__tests__/openapi-routes.test.ts`: all passed.
- A chat eval run on a branch that combines this change with three other
fixes. The result is in a PR comment.

## Risks

- Low risk. The routes return only tasks that the reading agent can open
through the task routes.
- The Mine route adds one batched read decision for each request.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:52:07 -06:00
DottaandClaude Opus 5.5 67403baaa4 fix(native): an agent termination Stop with no turn result stays a cancellation (#21093)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person terminates an agent, Paperclip stops the agent's
running native turn
> - Two paths can record that Stop: the terminate route ("Cancelled
because the agent was terminated ...") and the lifecycle worker ("Agent
lifecycle stop requested"). Both are system-initiated and expected
> - When the stopped turn ends without a semantic result, the native
executor keeps the run cancelled only for a person's Stop, a
reassignment, or a governance pause. A termination Stop fell through, so
the run ended `failed` with `native_session_interrupted`
> - Whether the run ended cancelled or failed depended on which path won
the race
> - This pull request treats an acknowledged termination Stop like a
person's Stop in the executor and in the lease-release terminalization
> - The benefit is that a terminated agent's interrupted run ends
cancelled, whichever path records the Stop first

## Linked Issues or Issue Description

Refs #20186 (the same treatment for a governance pause Stop)

**What happened?**
A person chose Terminate for an agent while its native turn ran. The
lifecycle Stop recorded an acknowledged native cancellation (initiator
system). The turn ended without a semantic result. The executor then
wrote the run as `failed` with `native_finalization_missing: session
returned no semantic result`.

**Expected behavior**
The run ends `cancelled`. A termination is an intentional Stop, not a
failure to retry.

**Steps to reproduce**
Terminate an agent while its native turn runs, so that the Stop
interrupts the turn before it records a result. Before this change, the
run became `failed` when the lifecycle worker recorded the Stop first.

## What Changed

- `governance-pause-stop.ts`: adds `AGENT_TERMINATED_STOP_REASON`,
`COMPANY_ARCHIVED_STOP_REASON` and `isAgentLifecycleStopRecord`. The
predicate accepts a system-initiated, expected control-plane
cancellation whose reason is a lifecycle, termination or company-archive
Stop.
- `acknowledged-native-stop.ts`: adds
`hasAcknowledgedNativeLifecycleStopIntent`, which requires that record
and an acknowledged, dispatched native Stop receipt.
- `native-session-executor.ts`: the no-semantic-result branch accepts
that intent, so it settles the coordinator as `native_retry_cancelled`
and raises `NativeCancellationPendingRecoveryError`, as for a person's
Stop.
- `heartbeat/run-lifecycle.ts`: lease-release terminalization keeps such
a run `cancelled`.
- The terminate route and the lifecycle driver use the shared reason
constants. The reason text does not change.

## Verification

- `native-session-executor.test.ts`: a new case for each reason
(lifecycle Stop, terminate route, company archive). Each was red before
the change and is green now. The earlier control case now uses a system
Stop that is neither a pause nor a termination, and it still takes the
retry path. Full file: 1414 passed.
- `heartbeat-run-lease-release-terminalization.test.ts`: a new case for
a lifecycle Stop when teardown wins the finalizer race. It was red
before the change (`interrupted`) and is green now.
- Product evidence: a paid chat-eval run of `agents.retire-agent` on a
branch that combines this change with three related fixes. The result is
in a PR comment.

## Risks

- A termination or company-archive Stop that interrupts a turn no longer
schedules an exact resume of that turn. The agent is leaving service, so
no resume can run.
- Other system Stops keep their current behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:19:21 -06:00
DottaandClaude Opus 5.5 6d8f59d494 fix(api): name the secret binding proposal as the way to give an agent a company secret (#21090)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents find product API routes through the runner API search,
which ranks operations by their OpenAPI summary and description
> - In the chat eval `secrets.configure-company-secret`, a person asked
the CEO to let Morgan use an existing deployment API key
> - The agent searched for agent configuration and environment variable
routes, tried `PATCH /api/agents/{id}` (403) and spent 3.5 minutes
before it used `POST /api/agents/me/secret-proposals`
> - The summary "Propose a company secret or agent secret binding" did
not match how agents describe the task
> - This pull request rewrites the summary and adds a description that
names the binding fields
> - The benefit is that the first search finds the supported route, so
the turn is short

## Linked Issues or Issue Description

**What happened?**
Runner API search did not rank `POST /api/agents/me/secret-proposals`
for the queries an agent uses to give another agent a company secret
("agent configuration update env secret", "agent environment variables
set secret ref"). The agent probed other routes for 3.5 minutes.

**Expected behavior**
Those searches return the secret binding proposal route in the first
results, and the result explains `secretId`, `configPath` and
`targetAgentId`.

**Steps to reproduce**
Call `searchRunnerApi({ query: "agent configuration update env secret",
limit: 3 })` on master. The proposal route is not in the results.

## What Changed

- `server/src/routes/openapi.ts`: new summary and description for `POST
/api/agents/me/secret-proposals`. The description states the
reporting-line target rule, the `env.<KEY>` config path, and that a
board member approves the binding.
- `server/src/__tests__/secret-binding-discovery.test.ts`: the four
queries return the route in their top three results. The test fails on
master (red) and passes with this change (green).

## Verification

- `npx vitest run` on 24 server guidance, discovery, OpenAPI and runner
API test files: 2305 passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
`changed: false`.
- Paid chat eval run: see the PR comment.

## Risks

- Low. Only OpenAPI text changes. Other search results can move down by
one place for queries that share these words.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:46:26 -06:00
DottaandClaude Opus 5.5 b853cc28ae perf(server): reach readiness without waiting for backlog-sized recovery passes (#21014)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server process starts, recovers saved work, and then reports
ready on `/api/ready`. The deploy health gate waits for that signal.
> - On a busy staging workspace, one boot took 85 s to bind the listener
and 4.4 minutes to report ready. Small, idle stacks take 20–90 s.
> - Two parts of that time do not need to block readiness. The server
published several hundred UI assets one at a time before it bound the
listener. After the listener bound, it waited for the first pass of each
periodic recovery queue, and those passes grow with the backlog.
> - The volume-free data gate, which can refuse a start, ran last. By
then the server had already dispatched queued runs that a refusal would
abandon.
> - This pull request moves the refusal gate to the start, publishes the
assets 8 at a time with replayable bodies, and runs the backlog-sized
first passes after readiness, in their original order.
> - The benefit is a faster time from container start to ready, and no
work admitted before a gate that can refuse to start.

## Linked Issues or Issue Description

No public issue exists. This is the problem description (enhancement
template).

**Is your enhancement related to a problem? Please describe.**
Server boot time grows with the saved backlog. These timestamps come
from the log of one staging boot (release 87fe4d9793). The container
started at 01:16:56. The listener bound at 01:18:21, 85 s after start.
The fresh queued-run dispatch completed at 01:18:25. The full queued-run
dispatch pass completed at 01:20:14. The volume-free data gate ran at
01:21:20. Readiness came about 4.4 minutes after start. During startup
the log showed hundreds of `An error was encountered in a non-retryable
streaming request.` lines.

**Describe the solution you'd like**
Readiness waits only for the correctness gates and for the adoption of
old runs. Work that the periodic queues already retry runs after
readiness. A gate that can refuse to start runs before the server admits
any work.

**Describe alternatives you've considered**
- Move all post-listen recovery to the background. We rejected this.
Native recovery can fail closed and must still fail the start. It also
adopts old runs.
- Skip asset re-publication when the image is unchanged. We rejected
this. The collision check (never overwrite different bytes) needs a read
per asset.

**Additional context**
Startup map (line numbers are from `origin/master` 82e0393598):
- Blocks the listener: migrations and the embedded database
(`server/src/index.ts:369-741`); config and identity (`:753-774`);
backfills (`:821-833`, where
`sweepSupersededPendingRequestConfirmations` scans every pending
confirmation); `qualifyCloudConfigurationForCurrentImage` (`:956`,
synchronous package hashing); `createApp` (`:964`), which awaited serial
UI asset publication (`server/src/app.ts:1110`,
`server/src/retained-ui-assets.ts:69-105`).
- Blocks readiness after the listener: first passes of tool reviews,
gateway sessions and delivery workers (`index.ts:1529-1558`); the
terminal workspace reaper (`:1652`); native recovery and hot-restart
adoption (`:1707-1779`); the first passes of 17 recovery queues
(`:1782-1890`), including the full queued-run dispatch (`:1805-1811`);
backup registration (`:1979-1999`); the data gate and package
qualification (`:2020-2026`).
- Grows with the backlog: the queue first passes (queued runs, pending
deliveries, watchdogs, native sessions and results), the terminal
workspace sweep, the native recovery scan, and the confirmation sweep.
UI asset publication grows with the build (hundreds of assets), not the
backlog.

## What Changed

- **Volume-free data gate runs first.** `assertEphemeralServerData` now
runs right after `claimVolumeFreeControllerState` and the runtime config
load. That is before `createApp`, before the listener binds, and before
any queued-run dispatch. It used to run at the end of startup. The
package qualification gate stays at the end, because bundled plugin
installs can change its inputs.
- **UI asset publication is faster but still a gate before work.**
`createApp` starts the publication while it builds its routes. It awaits
the result before its first job, queue or run starts, which is the same
point as on master. A colliding build still refuses to start before it
admits any work (Greptile review).
- **Publication sends Buffer bodies, 8 at a time.** Each asset is read
once into a Buffer. The S3 client can replay a Buffer body; it cannot
replay a stream. The new code sends no stream bodies, so it no longer
reaches the SDK path that logs "non-retryable streaming request". After
the first failure, no new assets start. A collision found on one asset
outranks a storage outage on another (Greptile review).
- **Backlog-sized first passes run after readiness.** The 17 queue
registrations after native recovery (residual recovery through
stopped-instruction collection) now run in a tracked background
sequence, in the same order. Each pass is isolated. A pass that fails is
logged with its name, and the later passes, the housekeeping and the
scheduler still run. A queue whose first pass failed stays registered,
and the coordinator retries it. Native recovery and hot-restart adoption
still complete before ready. Idle sleep stays blocked while the sequence
runs, and shutdown waits for it like other scheduler work.
- **Queued turns keep the right status.** A new flag,
`startupQueuedRunPassOutstanding()`, stays true until the first full
queued-run pass ends. Until then the execution projection still shows
"Waiting for server restart". The flag clears even when that pass fails,
because the dispatch queue then owns those turns.
- **Startup timeline log.** At readiness the server logs the duration of
each startup step (`startup timeline at readiness`). It logs again when
the background passes finish. No OpenTelemetry collector is needed.
- **Benchmark.** A new script,
`scripts/benchmark-retained-ui-assets.ts`, measures publication against
storage with a fixed round-trip latency. It can fail when time exceeds a
budget (`RETAINED_UI_BENCH_MAX_MS`).

Why each moved step is safe:
- **Data gate (earlier).** Its inputs are the migrated database and the
owner marker. `claimVolumeFreeControllerState` still runs first.
Interaction with #20952 and #20990, which are both in this branch: those
changes made the gate tolerate
`native_harness_backups_require_migration` and
`local_instruction_copies_require_settlement` when the filesystem
carries the volume-free owner marker. That covered the case where this
boot's own dispatch and recovery wrote such run state before the late
gate ran. With the gate first, no work of this boot can write run state
before the gate: the app, its queues, run admission, native recovery and
dispatch all start after it. A test checks this order. So the gate sees
only state that a previous process left, and the owner-marker tolerance
still decides that state. This is strictly safer. Nothing relied on the
gate running after recovery: the only work that could clear a blocker
was background work that the old position never awaited. Effect: a
refused start no longer leaves dispatched runs behind.
- **UI asset publication.** The await point is unchanged, so the
gate-before-work order matches master. Only the time it takes changes.
- **Recovery queue first passes (after ready).** Each queue owns its
work and retries it on deadlines, commit wakes, and intervals. A first
pass that ends after ready behaves like a later pass. Interrupted runs
are still classified and adopted by native recovery before ready, and
they resume without a Retry. No wait is added before a turn starts:
turns can start sooner. The HTTP API already served requests during
recovery, because readiness never gated requests. Observable changes:
- The deploy gate passes earlier. A queue that throws while it registers
is now logged instead of failing the start. Each queue's `ready` already
absorbed run errors.
- `startup queued-run dispatch pass complete` can now appear after
`Server startup recovery complete`.

Left alone (safety unclear or not measured):
- The first passes of tool review recovery, gateway sessions, the
delivery workers, the GitHub workers, and the terminal workspace reaper.
These run before native recovery, and I did not prove the ordering safe.
- `qualifyCloudConfigurationForCurrentImage` before `createApp`. It
hashes plugin packages synchronously; its result is only a warning
there.
- The confirmation sweep and principal access backfill before the
listener. They scan whole tables.

## Verification

- Red/green: with `server/src/index.ts`, `app.ts` and
`retained-ui-assets.ts` reverted to master, these new tests fail. With
the change they pass:
  - `server-startup-feedback-export.test.ts`:
- refuses ineligible volume-free data before the app, the listener or
any queued-run dispatch
- reports ready while the first queued-run dispatch pass still runs,
then finishes it and the passes after it
- runs every deferred recovery pass after readiness when one throws,
logs it, and keeps the failed queue registered (fails on the first
revision of this PR, passes now)
  - `retained-ui-assets.test.ts`:
    - re-publishes with bounded parallel, replayable writes
- refuses a colliding build even when another asset's storage outage
fails first (fails with `failure ??=`, passes now)
- Regression guards that also pass: "checks volume-free data once,
before any of this boot's own startup work can write run state" and
"stops starting new assets after one collides".
- Focused suites pass after merging `origin/master`:
`server-startup-feedback-export` (131 tests), `retained-ui-assets`,
`execution-projection`, `liveness`, `heartbeat-startup-fresh-dispatch`,
`ephemeral-server-startup-run-state`, `ephemeral-server-data`,
`ephemeral-migration`, `ephemeral-server-contract` and
`static-index-html`. `tsc --noEmit -p server/tsconfig.json` is clean.
- Benchmark (measured locally, 400 synthetic assets, 25 ms simulated
round trip): `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-retained-ui-assets.ts`

  | | serial (before) | 8 in flight (after) |
  | --- | --- | --- |
  | first publish | 10.5 s | 1.3 s |
  | restart re-publish | 21.3 s | 2.7 s |

This time is still before the listener, as on master, but it is about 8
times shorter and overlaps route construction.
- Local boot (measured, empty instance, embedded PostgreSQL, 3 boots
each, a shared and noisy machine):
  - listener to ready: before 385–427 ms; after 212–303 ms.
- Spawn to listener was 5.5–12.8 s in both cases. Module load and the
embedded database dominate it, and it does not differ.
- Not measured: the hosted saving. From the staging log, the inferred
saving is the queued-run pass and the passes after it (about 175 s:
01:18:25 to 01:21:20), plus most of the serial asset publication before
the listener. Confirm with the new `startup timeline at readiness` log
line after deploy.

## Risks

- **Medium: readiness semantics change.** The deploy gate now passes
before the first recovery passes finish. The old server can leave
earlier while the new server still dispatches its backlog. Queued turns
show "Waiting for server restart" until the dispatch pass ends, and the
queues retry on their own.
- **Low: earlier data gate.** It can refuse a start that the old order
would have admitted only if background work cleared a blocker first.
That was a race before this change.
- **Low: parallel publication against throttling storage.** Eight
requests at a time, with the existing per-asset backoff (0.5–8 s). A
persistent outage still lets the server start on local files.
- No migrations. No API contract changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, subagents for code search). Extended
reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21010 (retry a throttled asset upload).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:35:47 -06:00
DottaandClaude Opus 5.5 f8e1fb0278 perf(docker): take the orphan-reaping pull and pack chmod off the image path (#21013)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `docker.yml` builds and signs the standard Core image for every
master push. Paperclip Cloud deploys a Core commit only after this image
is published and attested.
> - A master commit takes 29 min (p50) and 56 min (p90) to reach a
published image. Every minute here delays a staging fix.
> - The run logs show where the time goes. The exact-source gate (about
14 min) is the critical path today. The build legs finish about 3 min
before it. After the gate, `merge-and-push` spends about 95 s to pull
the 4.9 GB amd64 image for one probe.
> - This pull request moves that probe into the build legs, which run
before the gate. It also removes a duplicate 187 MB layer from the amd64
image.
> - The benefit is about 95–100 s less from gate to signed image on
every master commit, with the same proof. arm64 is now also probed.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The time from a commit landing on master to the signed standard image
(`ghcr.io/paperclipai/paperclip:sha-<full>` plus its attestation).

**Subsystem affected**
`.github/workflows/docker.yml`, `Dockerfile`,
`scripts/standard-image-contract.mjs`.

**Current behavior**
Measured on run 43133035209 (sha 42fd790e, total 999 s):

- Push 01:13:47. The slowest exact-source job, `General tests (server
(7/10))`, ends 01:27:08. `Cloud source verified v1` waits 20 s for a
runner and ends 01:27:44. The watcher sees it at its next 30 s poll,
01:27:58.
- `build-and-push (amd64)` ends 01:24:52 and arm64 ends 01:21:59. Both
have slack before the gate. Run 43134250912 shows the same shape: the
gate opened 01:44:37 and amd64 ended 01:41:53.
- `merge-and-push` starts 01:28:03 and pushes the manifest at 01:28:34.
Then "Verify PID 1 reaps orphaned processes" pulls the full amd64 image
(01:28:34 to 01:30:09, 95 s). "Verify the immutable standard image" runs
the same probe again on the same bytes (5 s). The attestation ends
01:30:22.

Last 60 master pushes (2026-11-11 20:17Z to 2026-11-12 03:21Z): 43
cancelled or superseded, 14 succeeded, 2 failed, 1 in progress. Push to
first published image that contains the commit: p50 29.4 min, p90 55.5
min, max 85.6 min. A successful run takes 22.5 min (p50) from push. The
median gap between pushes is 4.5 min.

Where the amd64 "Build and push by digest" step spends its 582 s:

| Time | Phase | What happens |
| --- | --- | --- |
| 0–8 s | Load | Dockerfile frontend, cache manifest import |
| 8–54 s | Pull cached layers (46 s) | `deps`, `rust-chef`,
`runner-deps`, `runner-build` and `base` are **all cache hits**. The
`build` stage starts `FROM runner-build` and copies `deps`, so BuildKit
downloads and extracts them: a 932 MB node_modules layer (13 s download,
24.5 s extract) and about 650 MB of Rust `target/` |
| 54–108 s | Per-commit build (54 s) | `COPY . .` 10 s, mtime
normalization 11 s, UI 5 s, plugin SDK 3 s, server tsc 18.5 s, runtime
bundle 5.6 s |
| 108–206 s | `runner-provider-pack` (98 s) | Rebuilt every commit: it
is `FROM build` and stamps the commit. pnpm install about 25 s, pack 9
s, **Kimi smoke 37 s (amd64 only)**, archive 28 s |
| parallel | `production` | Tool layer is a cache hit but pulls 1.11 GB
(11 s). `COPY --from=build /app` 24 s |
| 206–294 s | Pack copy and chmod | `COPY` pack 7.5 s. `chmod -R a+rX`
plus UID check **71 s** (arm64: 20.5 s) |
| 294–420 s | Image export (126 s) | 115 s compressing new layers:
`/app` 1.40 GB, pack 1.01 GB, pack archives 1.06 GB, chmod copy 0.19 GB
(compressed). Push 10 s |
| 420–582 s | Cache export (172 s) | `mode=max`: 152 s "preparing"
(compressing the per-commit layers of `build` and
`runner-provider-pack`, which no later commit can reuse), 20 s upload |

What rebuilds every commit, and why: only the `build` stage and
everything after it. `PAPERCLIP_BUILD_COMMIT` is already declared after
the dependency layers, `CLI_TOOLS_CACHE_EPOCH` is weekly, and dependency
install already copies manifests first. The commit ARG is used by the UI
build, the server stamp, and the provider pack, so those must rebuild. A
pnpm `--mount=type=cache` gives nothing here: the runners are ephemeral,
and cache mounts are not exported to the registry cache.

Why amd64 is slower than arm64: both legs run on runs-on AWS 32-core
instances (m6a.8xlarge and m6g.8xlarge), not on `ubuntu-latest`. The
difference is content, not CPU. amd64 has a larger provider pack with
the Kimi smoke test (37 s), a 187 MB chmod copy layer (arm64: 0 MB), a
4.9 GB image (arm64: 2.9 GB), and a 172 s cache export (arm64: 100 s).

**Proposed behavior**
Run the PID 1 probe in each build leg, on its own architecture, against
the digest that leg pushed. Before signing, require that every Linux
image in the standard index is one of those probed digests. Set the
provider pack modes in the stage that writes the pack.

**Reason and benefit**
The probe pull leaves the post-gate path. The leg has about 3 min of
slack before the gate. The chmod change removes about 50 s and 187 MB
from the amd64 leg, which offsets most of the probe pull that the leg
now does.

**Breaking changes**
None. Tags, digests, attestation subject, and signer identity are
unchanged.

**Additional context**
The concurrency proposal and the build-in-PR plan are below, in "Risks".
They are not implemented.

## What Changed

- `docker.yml` `build-and-push`: new step "Verify PID 1 reaps orphaned
processes". It runs `scripts/assert-orphan-reaping.sh` in
`ghcr.io/...@<pushed digest>` with `--platform` set to the leg platform.
It runs before "Export digest", so `merge-and-push` only receives probed
digests.
- `docker.yml` `merge-and-push`: removed the two `docker run` probes
(tag and digest). "Resolve standard image digest" now runs
`standard-image-contract.mjs --resolve "$GITHUB_SHA" --verified-digests
/tmp/digests`.
- `standard-image-contract.mjs`: `--verified-digests <dir>` reads each
probed leg digest from the registry, checks that its bytes hash to that
digest, and collects its Linux image digests. It fails unless every
Linux image in the standard index is in that set. Registry content is
addressed by digest, so this proves that the signed index names exactly
the probed bytes, without a second pull.
- `Dockerfile`: `chmod -R a+rX /provider-pack` now runs in the same
`RUN` that writes the pack. `COPY` keeps the modes. The production stage
keeps the readability check as UID 65534.
- Tests: new contract tests (membership, bare manifest legs, digest
mismatch, empty set, lookup by digest). New workflow test: the probe
runs between the build and "Export digest", and the publisher does not
`docker run`. Updated the provider-pack Dockerfile test.
- `doc/standard-image-contract.md`: describes the per-leg probe and the
membership check.

Expected savings, per master commit:

| Change | Saving | Confidence |
| --- | --- | --- |
| Probe moved out of `merge-and-push` | 95–100 s from gate to
attestation (measured: 95 s pull + 5 s second probe) | High |
| Pack chmod in its own stage | About 50 s and 187 MB in the amd64 leg
(71 s vs 20.5 s on arm64, where the chmod layer is empty) | Medium-high
on size, medium on seconds |
| Net amd64 leg | About +10 to +45 s (probe pull estimated at 60–95 s,
minus about 50 s). Still before the gate | Medium |

## Verification

- `node --test scripts/standard-image-contract.test.mjs`: 10 pass (2
new).
- `node --test` on
`.github/scripts/tests/{cloud-readiness,docker-canary-promotion,post-merge-runner-routing,lockfile-refresh-workflows}.test.mjs`,
`scripts/__tests__/release-verify-workflow.test.mjs`,
`scripts/reuse-canary-image.test.mjs`,
`scripts/standard-image-contract.test.mjs`,
`scripts/cloud-source-verification.test.mjs`: 484 pass.
- `vitest run server/src/__tests__/docker-build-stamp.test.ts
server/src/__tests__/container-init-reaping.test.ts`: 22 pass. Red
check: with the `origin/master` `Dockerfile` and `docker.yml`, the 2 new
or changed assertions fail.
- Real registry, read only: `node scripts/standard-image-contract.mjs
--resolve 42fd790e80 --verified-digests
<dir>` with the two leg digests from run 43133035209 returns the
published index digest. With only the arm64 leg digest, it fails:
"linux/amd64 image sha256:0f0642e6… did not pass the build-leg
orphan-reaping probe".
- `actionlint .github/workflows/docker.yml`: the same 6 shellcheck
findings as `origin/master`, none new.
- Shell syntax check (`sh -n`) of the changed pack `RUN`.

Not exercised: the workflow itself (it runs only on master pushes, and I
did not dispatch it). No local `docker build`: the Docker daemon is not
running on this machine, and a full build does not fit a few minutes.
The leg probe pull time is an estimate from the publisher runner.

## Risks

- The leg now pulls its own image (4.9 GB amd64). AWS legs have about
175 GB free. Hosted tag builds run "Free runner disk" first, and the old
publisher pulled the same image on a hosted runner for tag builds.
- If the registry returns a leg index in an unexpected form, the resolve
step fails closed and no master image is signed. The real-registry check
above used today's exact shapes (OCI index with one platform image and
one attestation manifest per leg).
- Tag builds (`v*`, `nightly/v*`, `beta/v*`) keep the probe, now per
leg. They have no standard-image signing step, and the manifest list is
made only from probed digests.

### Deliberately not changed

- **The exact-source gate.** It is a deliberate "never publish an
unverified image" rule (`watch-release-source.mjs --require-proof` and
`--check-proof`, `doc/cloud-build-readiness.md`). I kept it. The owner
levers are: the slowest `cloud-readiness` shard (`server (7/10)`, 12
min, sets the gate), the 20 s runner wait for `Cloud source verified
v1`, and the watcher's 30 s poll (15 s average delay). I did not shorten
the poll because the org API rate limit was exhausted today.
- **Cache export (172 s amd64, 100 s arm64).** `mode=max` compresses
about 2 GB of per-commit pack layers every build. A fix: a separate
cache-only build of a small aggregate stage (`deps`, `runner-build`, the
production tool layer) with `mode=max`, and the main build with
`cache-from` only. Not done here: the legs are not on the master
critical path today, and I cannot measure it without a master run. It is
worth doing for tag builds or if the gate gets faster.
- **Cached layer pull (46 s).** The `build` stage starts `FROM
runner-build`, so it downloads about 650 MB of Rust `target/` that it
deletes at the end. The server build's `prepare:runner-vendor` uses that
target, so this needs a separate change.
- **Provider pack per commit (98 s amd64).** It rebuilds because it is
`FROM build` and stamps the commit. Its 1.06 GB transport archives also
duplicate the 1.01 GB pack inside the image. Keying the pack on its own
inputs is a product-contract change.

### Concurrency proposal (not implemented)

Today one run holds `docker-${{ github.ref }}` for 17–23 min. Most of
that is waiting for the gate, not building. A newer commit waits for
that run before its build starts (run 43134250912 waited 6.4 min).
Proposal:

1. Remove the workflow-level group. Add job-level groups:
`build-and-push` uses `docker-build-${{ github.ref }}-${{ matrix.arch
}}` with `cancel-in-progress: false`. At most one build per architecture
runs and the newest commit replaces the pending one. So not every commit
is built.
2. `merge-and-push` uses `docker-publish-${{ github.ref }}` with
`cancel-in-progress: false`. It skips publication when a newer master
commit already has a signed image, so the image order stays monotonic.
3. A superseded run must not keep its source watcher alive, or the API
load grows with each push. Fold the early watcher into the gate job, or
end it when its build job is cancelled.

Effect: the newest commit starts building at push, so its image
publishes at about gate time plus 45 s, instead of after the previous
run ends. That removes the 0–17 min queue wait, which is most of the gap
between the p50 of 29 min and the gate time of about 14 min.

### Plan: build in PR CI and promote on merge (not in this PR)

What must change:
1. Turn on the GitHub merge queue for `master`. The queue tests a
`gh-readonly-queue/...` commit and then fast-forwards `master` to that
exact commit, so the SHA stamped in the image (`PAPERCLIP_BUILD_COMMIT`,
`build-info.json`, the provider pack `runnerSourceRevision`, the UI
bundle) is the final master SHA.
2. On `merge_group`: build both architectures with **no secrets and no
registry credentials**, because the code under test is not yet on
master. Write each image as an OCI layout artifact. A second job that
runs no PR code pushes those artifacts by digest to a staging
repository, keyed by the merge-group SHA.
3. Run `cloud-readiness` on `merge_group` for the same SHA, so
exact-source proof exists before merge.
4. On the master push, `docker.yml` looks up the staged digests for
`github.sha`, checks that the tree and SHA match, retags them as
`sha-<full>`, and signs with the same identity
(`docker.yml@refs/heads/master`). If no staged image exists, it falls
back to the current build.

Guarantees to keep: never publish or sign an image without exact-source
proof for that SHA. The signer identity and the attested digest stay the
same. No secrets in any job that runs non-CODEOWNER code. The PID 1
probe runs on both architectures before signing. Superseded queue
entries do not publish.

Expected saving: from master push to signed image, about 1–2 min
(lookup, retag, attest) instead of 29 min p50 and 56 min p90 today. The
work moves before the merge, so the merge itself waits for the queue
(about the current gate time of 14 min). A staging fix then reaches
Cloud about 25 min sooner at the median.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used via Claude Code
with tool use (shell, file edits, read-only GitHub API and registry
reads). Default reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:29:11 -06:00
DottaandClaude Opus 5.5 db8848f559 fix(heartbeat): bound the I7 queued-run check with a resumable cursor (#21022)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #20986 added a reconciler check that reports a queued run whose
dispatch signal was lost (I7 `queued_run_not_dispatched`), and a worker
that re-drives such runs.
> - The Ponytail review of #20986 (4/5 on `fbab23cce6`) found three
issues. #20986 merged at that head before the fix commit was pushed.
> - The new I7 check read every queued run with no cap, while the report
still said that each I7 check reads at most 2,000 records.
> - This pull request adds that fix commit to master. The check now
reads a bounded slice per pass and continues from a cursor. The page
cursor no longer depends on the session date style. An agent with a
scheduled retry now counts as busy.
> - The benefit is that the reconciler pass stays bounded when many runs
are held on purpose, and the coverage statement is true again.

## Linked Issues or Issue Description

Refs #20986

Follow-up to the Ponytail review of #20986:
https://github.com/paperclipai/paperclip/pull/20986#pullrequestreview-5868879754
(review of `fbab23cce6`, score 4/5).

**What is wrong?**
1. `checkQueuedRunsDispatched`
(`server/src/services/recovery/ownership-invariant-report.ts`) paged
through every undispatched queued run with no cap.
`COVERAGE_GAPS.record_cap` still said each I7 check reads at most 2,000
records per pass.
2. The page cursor cast `created_at::text` back with `::timestamptz`.
That works only with the default `DateStyle`.
3. A queued run on an agent with a `scheduled_retry` run was reported
and re-driven, but `checkVisibleNextAction` already treats
`scheduled_retry` as an active run.

**Expected behavior**
The I7 queued-run check reads a bounded slice per pass and reaches every
run over later passes. The cursor format does not depend on session
settings. A scheduled retry counts as the agent's next action.

## What Changed

- `server/src/services/queued-run-dispatch-liveness.ts`: new
`readUndispatchedQueuedRuns`. It reads from an optional cursor until the
set ends or `maxRuns` rows were read, and returns the next cursor.
`listUndispatchedQueuedRuns` uses it with no limit. The cursor text is
`to_char(created_at at time zone 'UTC',
'YYYY-MM-DD"T"HH24:MI:SS.US"Z"')`, which keeps microsecond precision and
does not depend on `DateStyle`. The live-run exclusion now also covers
`status = 'scheduled_retry'`.
- `server/src/services/recovery/ownership-invariant-report.ts`: the I7
queued-run check reads at most `MAX_RECORDS_PER_KIND` (2,000) per pass.
It keeps a keyset cursor in `OwnershipReportState.queuedRunCursor`, and
the next pass continues from that cursor. A pass that stops at the cap
keeps a follow-up pass due (`retryNeeded`), like the I3 and I4 backlogs.
`COVERAGE_GAPS.record_cap` now describes this check.
- `server/src/__tests__/ownership-reconciliation-report.test.ts`: new
test. Bounded slices resume from the cursor and end with `next: null`. A
`scheduled_retry` run on the agent removes its queued runs from the set.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/ownership-reconciliation-report.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/services/heartbeat/dispatch-work.test.ts
src/__tests__/heartbeat-held-comment-wake.test.ts`: 75 tests pass on
this branch (fresh master + this commit).
- Server `tsc --noEmit` passes.
- The restart integration test from #20986 ("starts a continuation that
queued behind its predecessor's sandbox release after a graceful
restart") passed with this change on the #20986 branch, in both cases
(signalled and unsignalled).
- Restart-chaos CR1 360 s, Codex W2, `--daytona-warm
--ephemeral-server`, seed 9512 (`w-20261112T040314-s9512`, on the #20986
change set before this commit): the fault crossed the task and every
rule passed (R1–R6, T, U1–U5).

## Risks

- Low risk. The report reads less per pass. A very large held queue is
now covered over several passes instead of one.
- A queued run behind a `scheduled_retry` is no longer re-driven by the
backstop. The scheduled retry's own promotion signals dispatch.
- No schema change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:19:37 -06:00
DottaandClaude Opus 5.5 ad2d123dad fix(recovery): count a current response-wake wait as a visible next action in I7 (#20994)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The ownership reconciler checks invariant I7: an agent-assigned
`in_progress` issue must have a visible next action. A live run, a
pending wake, an open recovery action or a saved monitor deadline
counts.
> - #20935 added a parked wait. When a native agent yields
`response_wake` and no response source is admitted, the arbiter commits
`response_wake_waiting_for_input`. The task stays `in_progress`, and the
run shows "Waiting for a reply".
> - I7 did not know this wait. A task that really waits for the board
was reported as `no_owner_no_wake`.
> - This pull request makes I7 count that wait as a visible next action
while it is still current. I7 still reports the issue when the wait is
stale.
> - The benefit is that a genuinely parked task is not reported as a
silent stall, and a stale wait is still reported.

## Linked Issues or Issue Description

**What happened?**
The ownership reconciler reported `I7_visible_next_action` /
`no_owner_no_wake` for an issue whose latest applied decision was
`response_wake_waiting_for_input`. That is the visible "Waiting for a
reply" state from #20935.

**Expected behavior**
A current, committed response-wake wait is a visible next action. The
issue is reported again when a user comment arrives after the waiting
run started, or when that decision is no longer the issue's latest
decision.

**Steps to reproduce**
1. Commit a native run whose arbiter decision is
`response_wake_waiting_for_input`, and make it the issue's latest
decision.
2. Wait past the 5-minute next-action grace window.
3. Run an ownership reconciliation pass. Before this change, I7 reports
the issue.

**Additional context**
Follow-up to #20989 and #20935. The conditions match
`hasCommittedNativeResponseWakeWait`, which recovery already uses for
the same wait.

Refs #20935
Refs #20989

## What Changed

- `server/src/services/recovery/ownership-invariant-report.ts`:
`checkVisibleNextAction` excludes an issue with a parked response-wake
wait. The wait must meet all of these conditions:
- The issue's `lastStatusDecisionId` is an applied
`response_wake_waiting_for_input` decision.
  - Its native finalization is `committed`.
  - Its run succeeded and belongs to the current agent assignee.
  - The issue has no user assignee and is not hidden.
  - No user comment arrived after that run started.
- New test in
`server/src/__tests__/ownership-reconciliation-report.test.ts`:
  - A current wait is not reported.
  - A later user comment makes I7 report the issue.
  - A decision that is no longer current also makes I7 report it.

## Verification

- Red on master: `cd server && npx vitest run
src/__tests__/ownership-reconciliation-report.test.ts -t "response-wake
wait"`. I7 reported the waiting issue.
- Green: the full `ownership-reconciliation-report.test.ts` and
`environment-pending-cleanup-queue-owner.test.ts` pass (43 tests). `npx
tsc --noEmit -p server` passes.
- Restart-chaos harness, for reviewers: the harness U1/R1 grading should
treat a sample as waiting, not ownerless, when the issue's latest run
shows the execution projection phase `waiting_for_answer` ("Waiting for
a reply"). It should still fail when a user comment is newer than that
run. The harness reads I7 through `tests/restart-chaos/invariants.ts`,
so I7 changes with this PR. The U1 sampler does not read the projection
yet, and this PR does not change it.

## Risks

- Low risk. The change is in a report-only check. It writes nothing and
changes no admission, status or recovery path.
- If the conditions are too wide, a stalled task is hidden. The
conditions are the same as recovery's durable-wait check, and the test
proves that a stale wait is still reported.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:07:38 -06:00
DottaandClaude Opus 5.5 f30005440b fix(recovery): reconnect a live remote Runner with its admitted skills instead of re-downloading them (#21064)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner sandbox. A server upgrade
detaches each running turn, and the new server must reconnect to its
Runner (restart recovery).
> - A reconnect runs the full turn preparation before it contacts the
Runner. On a new server, skill preparation downloads every saved skill
version again, under one lock for the whole company.
> - After an upgrade, many reconnects wait behind each other's
downloads. The Runner stops answering after about 2 minutes, so a late
reconnect loses the turn.
> - This pull request lets a reconnect to the same run's live remote
Runner use the exact saved skill versions, without downloading them and
without the lock.
> - The benefit is that skill preparation no longer adds to the
reconnect delay, and it does not grow with the number of reconnects.
Normal turn starts do not change.

## Linked Issues or Issue Description

**What happened?**
After an automatic staging upgrade, runs that the new server reached
late failed. The measured time from detach to reconnect was 114–152 s
for the failed runs. Turn preparation took 30–130 s, and skill
preparation alone took 46 s on one run.

**Expected behavior**
A reconnect to a live Runner reaches it quickly. Its speed must not
depend on how many other runs prepare skills at the same time.

**Steps to reproduce**
1. Run a remote Runner turn for each of many tasks in one company.
2. Upgrade the server while those turns run, so the new server starts
with an empty skill cache.
3. See that each reconnect waits in
`heartbeat.prepare.configuration_skills` for the company skill cache
lock and downloads the skills again.

**Paperclip version or commit**
master at 4228bcf89c

**Additional context**
Root cause: `withRuntimeSkillEntries` holds the company-wide skill cache
lock (a Postgres advisory lock) while it materializes each saved skill
version. A new server has an empty skill cache, so each reconnect
downloads its skills again, one run at a time. The Runner already has
these files, and the reconnect path does not read them.

## What Changed

- `company-skills.ts`: new `admittedVersionsOnly` option. It still
checks that each saved version exists and that the agent can read the
skill. It does not materialize the files or take the company skill cache
lock.
- `heartbeat/live-runner-adoption.ts`: `adoptsLiveRemoteRunner` is true
only for a `reattach_remote_runner` claim for this run, with the claim's
own lease owner, on a native run.
- `heartbeat/execution-configuration.ts`: uses the option only when all
of these are true: the reconnect gate is true, the run has its saved
native input with a runtime context, and the run has its saved skill
versions. In that case it times the phase as
`configuration_skills_adopted` and adds one lifecycle run event
(`skillPreparation: "reused_admitted"`).
- `heartbeat.ts`: passes the gate to execution configuration.
- `native-runtime/runtime-context.ts` and `native-session-executor.ts`:
on the same reconnect to a remote Runner filesystem, the Runner
controller describes the Runner's skills by their admitted digests. It
rehydrates the pinned skill files (which takes the same lock) only if it
must launch a Runner process and stage them. All other starts rehydrate
as before.
- `doc/execution-semantics.md`: describes the new behavior.

Not in this PR (by design): a separate "dispatch-only" reconnect path,
and a new startup scheduling lane. The native executor needs about 40
values from preparation (runner environment, MCP token, managed AI
credentials, the reattached execution target, run output). Connector and
instruction preparation also feed the session fingerprint. That cut is
not mechanical, so this PR removes the largest measured single cost
only.

Other `heartbeat.prepare.*` costs that stay on the reconnect path (use
post-upgrade timings to choose the next cut):
- `configuration_secrets`, `configuration_connector_assignments`,
`configuration_connection_instructions`,
`configuration_connector_skills`/`_delivery`,
`configuration_agent_identity`, `configuration_ai_runtime`
- `environment_selection`, `workspace` (retained lease resume and
realization), `session`, `task_context`, `run_identity`
- `github_environment`, `github_launchers`, `run_output`,
`runtime_services`, `instructions`, `runtime_selection` (including
`runtime_selection.skill_assets` and `runtime_selection.workspace_sync`)
- The wait from detach to "run started" (queue and controller-preparing
waits) is outside `heartbeat.prepare.*`.

## Verification

- New test `reconnects to a live remote Runner with its admitted skills
without re-downloading them or waiting on the cache`
(`server/src/__tests__/heartbeat-execution-configuration.test.ts`):
- It admits a run, deletes its skill cache (like a new server), and
holds the company cache lock.
- The reconnect preparation finishes while the lock is held. It returns
the same key, runtime name and version. The session fingerprint is
identical. The cache is not created again.
- Normal preparation of the same saved selection still waits for the
lock and materializes the files ("normal start unchanged").
- Red on master source: `A live Runner reattach must not queue behind
skill downloads: expected null not to be null`. Green with this change.
- New test in `managed-skill-content.test.ts`: with the cache lock held
and an empty cache, `prepareRuntimeContextSource` for a reconnect
returns at once with the admitted context. Its staging step and a normal
start still wait for the lock and then rehydrate the same digests.
- New `server/src/services/heartbeat/live-runner-adoption.test.ts`: the
gate is false for a normal start, a scheduled resume, a dead-Runner
resume, an incomplete bootstrap, another run's claim, a different lease
owner, and a legacy run.
- `npx vitest run
src/__tests__/heartbeat-execution-configuration.test.ts
src/__tests__/skill-version-cache.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/heartbeat/live-runner-adoption.test.ts`: 289 passed, 14
skipped. After the controller change: restart-recovery, diskless
restart, runtime skill preservation, managed skill content, session
executor and execution configuration suites: 1686 passed, 14 skipped.
- `tsc --noEmit -p server/tsconfig.json`: pass.
- `pnpm verify` asks for broader coverage (exit 2). Required CI supplies
it.

After deploy, a reconnect run shows
`heartbeat.prepare.configuration_skills_adopted` and the
`reused_admitted` event, and no `skills.cache.acquire` or
`skills.version.materialize` phases.

## Risks

- Low. The option applies only to a same-run reconnect of a retained
remote Runner that has a saved runtime context and saved skill versions.
Every other start keeps the current path.
- Access revocation and deleted versions still fail the reconnect, as
before. A failure to download the files on this server no longer fails a
live reconnect, because the Runner already holds them.
- Nothing new is saved, so rows from older releases, two live servers,
and a deploy during a reconnect behave as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code search, editing, shell, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:58:01 -06:00
DottaandClaude Opus 5.5 424e2c6488 fix(recovery): keep live owners' runs through an upgrade and continue replaced preparations in the restart lane (#21046)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade replaces the server. The old and the new server serve
together for several minutes. A legacy run proves that its owner is
alive by renewing a 60-second lease on its `heartbeat_runs` row.
> - #21019 added an uncharged `server_restart` retry lane for runs that
a restart or replacement ended. On the first upgrade that shipped it, no
run used that lane. The new server also failed many runs as
`process_lost` while the old server that owned them was still serving.
> - Each of these runs then spent one failure retry. A run that was
already the last automatic retry was held for manual reconciliation, so
the task needed a person.
> - An upgrade is routine. It must not cost a task its work or a retry,
and it must not leave a task waiting for a manual Retry.
> - This pull request sends a run that a replacement stopped during
preparation to the `server_restart` lane. It also stops the reaper from
failing a run while the server process that owns it is proven alive.
> - The benefit is that a task continues by itself through an upgrade,
also when the database is slow during the swap.

## Linked Issues or Issue Description

Refs #21019, #21042

**What happened?**
A hosted staging server was replaced by an automatic upgrade. The new
server started at 06:18:46Z, became ready at 06:26:17Z, and the old
server received SIGTERM at 06:27:34Z. Between 06:23Z and 06:26Z the new
server failed 44 runs as `process_lost` ("Process lost -- server may
have restarted; retrying once",
`processLossDiagnostic.controllerReplaced: true`). The old server still
owned these runs and was still serving. Every reaped run was a Runner
run in `executionStage: preparing` with no process recorded. Each
successor was scheduled with `scheduledRetryReason: transient_failure`
and `executionRetryAccounting.failureRetries: 1`. No run on the server
has `retryReason: server_restart` or
`executionRetryAccounting.serverRestarts`.

The run evidence also shows why the leases lapsed. The old server's
lease renewals waited 19 to 50 seconds on the database
(`controllerLeaseLoss.renewalPendingMs`) while its timers fired on time
(`deadlineLateMs` 0 to 17 ms). #21042 fixes that renewal stall. This
pull request fixes how the reaper and the retry scheduler treat these
runs.

**Expected behavior**
- A server never fails a run as lost while the server process that owns
it is alive and serving.
- A run that a replacement really ended continues once, automatically,
without spending the failure budget.

**Steps to reproduce**
1. Seed a legacy `paperclip_runner` run in `executionStage: preparing`,
owned by another server boot, with a lapsed run lease.
2. Run `reapOrphanedRuns()` on the new server.
3. See that the successor is `transient_failure` with `failureRetries:
1`. When the chain already spent its failure budget, see that no
successor is created and the run is held for reconciliation.
4. Give the owner a live controller instance lease (the old server is
still serving) and run `reapOrphanedRuns()` again: the run is failed as
`process_lost` anyway.

**Root cause**
- `usesServerRestartLane` (`server/src/services/heartbeat/retries.ts`)
left every run with the bootstrap evidence `providerWorkStarted: false`
to the `interrupted_preparation` lane. But `scheduleBoundedRetryForRun`
gives that lane only to an `interrupted` run (or a native preparation).
The reaper writes the same evidence on a `failed`/`process_lost` run
that it stops during preparation. That run fell through to the charged
`transient_failure` lane. The #21019 tests used runs in `dispatching`,
so they did not catch it.
- `legacyExecutionNeedsReconciliation` held every run whose failure
budget was spent before it checked the bootstrap evidence. A run that a
replacement stopped before dispatch did no provider work, but it was
held for reconciliation, and the task waited for a person.
- The orphan reaper (`recoverActiveExecutions`,
`server/src/services/heartbeat/recovery.ts`) treated a lapsed run lease
as proof that the owner was gone. It checked the shared owner prover
only for "gone", never for "alive". During the swap the old server's
renewals were slow, so its leases lapsed while it was alive.

**Paperclip version or commit**
Staging Core `51d77648bc` (first build with #21019). The behavior is the
same on master `148d2b629c`.

## What Changed

- `retries.ts`: `usesServerRestartLane(run, preparationLaneTakesRun?)`
keeps the `interrupted_preparation` exception only when that lane really
takes the run. `scheduleBoundedRetryForRun` passes its own verdict.
Without one, only an `interrupted` run qualifies. A `process_lost` run
that a replacement stopped during preparation now continues in the
`server_restart` lane.
- `legacy-execution-recovery.ts`: a replaced preparation
(`isReplacedPreparation`) does not need reconciliation, also when its
failure budget is spent. The `server_restart` bound limits it.
- `recovery.ts` (orphan reaper):
- When a run's lease lapsed but the shared owner prover proves its owner
process alive (for example, the owner's controller instance lease is
live), the reaper leaves the run with that owner. The owner stops and
settles the run itself (`controller_lease_lost`, uncharged
`interrupted_preparation` lane). The reaper recovers the run once the
owner is gone or stopped. It also recovers the run when the owner has
not settled it within `LEGACY_LIVE_OWNER_SETTLE_GRACE_MS` (5 minutes)
after the lease lapsed.
- A setup that a replacement stopped before dispatch uses the
`server_restart` bound for its retry decision, whatever the adapter.
- `legacy-controller-lease.ts`: add `proveLegacyController(run)`, which
returns the shared prover's verdict. `legacyControllerProvenGone` uses
it, and the reaper computes the proof once per run.
- `server-replacement-interruption.ts`: add
`isReplacedPreparation(run)`: bootstrap evidence `providerWorkStarted:
false`, a recorded controller process stamp, and a server-replacement
interruption. A row saved by an older release without the stamp keeps
the charged lane (the existing documented rule).
- `legacy-execution-recovery.ts` (`settleInterruptedNativeBootstrap`): a
reconciliation hold that an older release saved on a replaced
preparation is settled once its owner is proven gone, even when the
failure budget is spent. The run continues in the `server_restart` lane.
- `legacy-controller-lease.ts`, `recovery/ownership-registry.ts`: the
`legacy_interrupted_preparation_hold` kind judges a finished run's hold
by its recorded controller process
(`finishedLegacyControllerOwnerStamp`). Revocation by the orphan reaper
swaps in a random boot id, so before this change a reaped preparation's
hold had owner "none" and was never settled.
- `recovery/service.ts`, `heartbeat.ts`: the stranded-task sweep reports
exhaustion from the scheduler's own `retry_exhausted` result, so a spent
`interrupted_preparation` lane blocks the task even when the synchronous
budget check reads an unspent restart bound.
- `doc/execution-semantics.md`: document these rules.
- No schema change. Nothing changes on the path that starts a run or a
turn (no new waits and no file work).

## Verification

- Red on `origin/master`, green on this branch
(`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs
ended by a server restart or replacement"):
- "continues a first attempt / a run that was already the last automatic
retry that a replaced server left preparing, in the restart lane". This
is the production shape: a `paperclip_runner` run in `preparing`, two
servers sweep at the same time. Master: `transient_failure` with
`failureRetries: 1`, and no successor for the spent chain. Now: one
`server_restart` continuation, due at once, with the failure count
unchanged.
- "leaves a preparing/dispatching run whose lease lapsed with its owner,
then recovers it once the owner stops". The owner's controller instance
lease is live and two servers sweep. Master: `process_lost` at once.
Now: the run stays `running` with its owner. After the owner stops, it
becomes `process_lost` and a `server_restart` continuation is scheduled.
- "recovers the run when its live owner has not settled it within the
grace".
- `interrupted-preparation-hold-settle.test.ts`: "releases an older
release's hold on a preparation a server replacement stopped with its
failure budget spent" for an unrevoked and a reaper-revoked lease (red
without the settlement and owner-stamp changes) and "blocks the task
when a replaced preparation's lane is spent even though the restart
bound is not" (red without the sweep change).
- Related suites: heartbeat process recovery, retry scheduling, orphaned
active lease sweep, recovery deadline (bootstrap and ordinary
bootstrap), legacy continuation authority, undispatched execution wait,
agent status recovery, legacy workspace restore recovery, legacy
execution recovery, legacy controller lease, server replacement
interruption, heartbeat recovery, provider failure classification,
reconciliation run read, and wake-queue policy.
- `tsc --noEmit` for `server`.

## Risks

- A run whose owner is alive but no longer drives it now stays `running`
for up to 5 minutes after its lease lapsed, and then the reaper recovers
it. The owner's own sweep still recovers a run that the owner itself
lost the handle for (this boot is never deferred).
- The guard trusts the shared owner prover. That prover reads "alive"
only from a live or grace controller instance lease, or from a live PID
in the same PID space. A server whose instance lease cannot renew for 60
seconds stops itself, so a stuck owner cannot hold runs forever.
- More runs continue in the uncharged lanes. The `server_restart` bound
(6 in 60 minutes) still stops a loop, and the stop is shown on the task.
- Rows saved by older releases: a `process_lost` row with the bootstrap
evidence and an old-boot owner now continues in the `server_restart`
lane instead of being held. These runs did no provider work.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only log and API inspection) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 02:18:39 -06:00
DottaandClaude Opus 5.5 4e519bd428 fix(chat): send writes with an empty responseText and point card declines to resolve-from-comment (#21050)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents call the Paperclip API through `call_api` and find
routes with `search_api`
> - #21035 added a pre-send refusal for writes that carry
`responseText`, and new summaries for the question cancel and withdraw
routes
> - Full chat cohort 5 (OpenCode with DeepSeek v4.1 flash) showed that
both changes went too far
> - The refusal blocked a valid write that had an empty `responseText`,
and the summaries hid the route that records a person's decline
> - This pull request narrows the refusal and corrects the guidance text
> - The benefit is that valid writes go through on the first try, and
agents settle a declined card instead of leaving it pending

## Linked Issues or Issue Description

Refs #21035, #21002

**What happened?**
- inbox-search.restore-archive: DeepSeek sent `responseText: {}` with a
valid body on `POST /api/issues/{id}/inbox-archive`. The refusal from
#21035 rejected 24 POSTs in 3 minutes. The model did not drop the key,
and the run was cancelled at the 240s window. In cohort 4 the same POSTs
went through on the first try.
- questions-and-approvals.reject-and-revise: the person said "Do not
publish it; just make a draft I can review." The CEO quoted the cancel
summary ("an agent withdraws its own pending card with POST
.../withdraw"), saw that the person created the publish card, and left
the card pending. The plan decision did not settle. In cohorts 3 and 4
the CEO declined the card with `resolve-from-comment` (resolver policy
`anyone`).

**Expected behavior**
- A write with an empty or default `responseText` is sent once. A write
whose `responseText` sets a window is still refused before it is sent.
- The guidance says that withdraw is only for the agent's own obsolete
card, and that a person's explicit decline or answer goes through
`resolve-from-comment` when the card's resolver policy allows.

**Steps to reproduce**
Run the two chat cases above with OpenCode and DeepSeek v4.1 flash on
master.

## What Changed

- `call_api` (`assertRunnerApiDispatchable`): the check reads the raw
`responseText` value. `{}` or `{ offsetBytes: 0 }` on a write is dropped
and the write is sent. A `responseText` with `limitBytes` or a later
offset is still refused. `saveResponse: false` on a write was already
accepted; a test now covers it.
- OpenAPI: the cancel and withdraw summaries say that withdraw is only
for the agent's own obsolete card, and that a decline or answer on the
person's explicit instruction uses `POST .../resolve-from-comment` when
the resolver policy allows. The `resolve-from-comment` summary says it
also records a decline of a card the person created.
- `AGENT_CHAT_DIRECTIVE`: a person's explicit decline in chat answers a
pending confirmation with reject, whoever created the card. Withdraw
never records the person's decision.
- `doc/runner-api-tools.md`: describes the narrowed refusal.

## Verification

- `server/src/services/native-runtime/runner-api.test.ts`: new test
sends `POST /api/issues/{id}/inbox-archive` with a valid body and
`responseText: {}`, `{ offsetBytes: 0 }`, `saveResponse: false`, or
both. The request is sent exactly once. Red before the fix (3 of 4 cases
refused), green after. The refusal test for a `responseText` with
`limitBytes` stays.
- `server/src/__tests__/openapi-routes.test.ts`: new test for the
cancel, withdraw and resolve-from-comment summaries. Red before, green
after.
- `server/src/__tests__/agent-conversations.test.ts`: new directive test
for the decline rule. Red before, green after.
- Also passed: `runner-api.integration.test.ts`,
`native-case-tools.integration.test.ts`, server `tsc --noEmit`. SDK
public input pins: `--check` reports no change.
- Paid run (OpenCode + DeepSeek v4.1 flash) on this branch for
inbox-search.restore-archive, questions-and-approvals.reject-and-revise,
gateways-profiles.rotate-token and budgets-costs.repair-cost-recording:
4 of 4 cells passed on 3f20fb2bcc (run 43177772183); details in a PR
comment.

## Risks

- A write with a populated `responseText` is still refused, as in
#21035. Only the empty and default shapes change. A dropped empty
`responseText` changes nothing, because the server already removed it
from writes before sending.
- The directive grows by two sentences, which adds a small number of
prompt tokens per chat turn.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 02:09:14 -06:00
DottaandClaude Opus 5.5 3c058c37e6 fix(retry): continue a connected task from saved work when its exact chat request cannot be repeated (#21038)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks that come from a connected chat (for example GitHub pull
request reviews) show a Retry button when a run fails
> - The task projection offers Retry, but the Retry route sends a
connected task to the chat retry owner, which only replays the exact
failed chat request
> - When that exact request cannot be replayed (the attempt already
saved a result, its output was published, the latest run did not start
from a chat message, or the requester's access changed), the route
returns 409 and tells the person to check the connection by hand
> - A Retry button that is shown must work, and recovery must not ask
the person to inspect anything
> - This pull request makes a person's Retry continue the task from its
saved work when the exact chat replay is refused, the same way a new
message on the task would
> - The benefit is that the projection and the route agree, and blocked
connected tasks become retryable after deploy with no migration

## Linked Issues or Issue Description

No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a hosted deploy, five blocked tasks that came from a connected GitHub
conversation showed Retry (`interruption.canRetry: true`). Each click on
Retry returned 409 `chat_failed_run_retry_not_authorized`. The message
was "The exact failed chat request can no longer be retried safely.
Check its current connection, access, and recovery state." or "This
attempt already recorded a result. Send a new message on the task to
continue from the saved work." The latest runs were legacy sandbox setup
failures, native runs that needed reconciliation, a run that started
from an issue monitor instead of a chat message, and a native run that
already saved its result. Root cause: `admitTaskRetry` in
`server/src/services/task-retry.ts` called the chat retry owner for
every connected task and passed its exact-replay refusal straight to the
person. The projection cannot predict that refusal.

**Expected behavior**

A shown Retry works. If the exact chat request cannot be repeated, Retry
continues the task from its saved work, the same way a new message on
the task would. Users do not get operator instructions.

**Steps to reproduce**

1. Bind a task to a restricted chat conversation.
2. Make its latest run fail so that the chat retry owner refuses the
exact replay. For example, use a run whose source is not a chat message,
or a native run with an accepted result.
3. Click Retry (`POST /api/issues/:id/retry`).
4. The route returns 409 `chat_failed_run_retry_not_authorized`.

**Paperclip version or commit**

`master` at 5a7378af5 (the deployed revision), reproduced on 51d77648bc.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

Not adapter-specific (core bug). Seen with `paperclip_runner` legacy and
native runs.

## What Changed

- `server/src/services/task-retry.ts`: a person's Retry runs the exact
chat replay inside a savepoint. On a
`chat_failed_run_retry_not_authorized` refusal, the savepoint rolls back
and the same Retry receipt stays an ordinary board continuation. The
clicking person is the responsible user, the run starts a fresh session
(`forceFreshSession`) from saved work, and no chat action, chat
authority, or output replay is created. Retry that Paperclip requests by
itself (`initiatedBy: restart_recovery | provider_limit_reset`) keeps
the exact-replay requirement. Other errors (403, lock contention,
database errors) still propagate.
- `server/src/services/chat-channels/failed-run-retry.ts`: plain refusal
wording. Runs saved by an earlier release with the old diagnostic are
still recognized by the pre-provider check.
- Tests: new red/green cases in `server/src/services/task-retry.test.ts`
(four refusal reasons on a chat task whose latest run came from an issue
monitor, plus automatic Retry still refused). Updated two real-service
route cases in `server/src/__tests__/chat-channels.integration.test.ts`
(accepted native result on an inherited GitHub task, and an unknown chat
source) from expecting 409 to expecting a board continuation without a
chat replay.
- A person's Retry on that person's own saved board turn (`source:
issue.comment.reopen`) that cannot be certified for replay also
continues from saved work, without replaying the saved message.
Automatic Retry still gets 409 `saved_board_message_not_replayable`.
- A person's Retry when no chat retry owner is wired also continues from
saved work. Automatic Retry still gets 409.
- The fallback is not used while the conversation holds saved or
unadmitted chat input for the task. That input keeps its own admission
path and the existing refusal.
- Docs: `doc/execution-semantics.md` and `doc/connections/GITHUB.md`.

### 409 paths in the task Retry route after this change

| Path | Who sees it | Result |
| --- | --- | --- |
| `chat_failed_run_retry_not_authorized` from the chat retry owner |
Person | Continues from saved work (board continuation) |
| `saved_board_message_not_replayable` | Person | Continues from saved
work |
| No chat retry owner wired | Person | Continues from saved work |
| Any of the three above while saved or unadmitted chat input is pending
| Person | 409. The pending input continues through its own admission
and delivery. |
| Any of the three above | Automatic Retry (restart recovery, provider
limit reset) | 409, recorded by the hook. No button. |
| `task_retry_busy` (chat authorization lock held) | Person | 409 with
`retryable: true` after 10 internal attempts. A new click works. |
| Idempotency key reused for a different task or user | Client error |
409. The UI makes a new key for each click. |

## Verification

- Red before the fix: `npx vitest run src/services/task-retry.test.ts -t
"connected task"` failed 4/4 with `Error: refused` (409). The board-turn
and missing-owner cases failed with "This saved message cannot be
retried safely..." and "Retry is unavailable for this connected
conversation." `src/__tests__/chat-channels.integration.test.ts -t
"continues an inherited GitHub task"` failed with `expected 202
"Accepted", got 409 "Conflict"`.
- Green after the fix: `src/services/task-retry.test.ts`,
`src/__tests__/durable-chat-wakeup.test.ts`,
`src/services/budget-resume-continuation.test.ts`: 160/160 passed.
- Full `src/__tests__/chat-channels.integration.test.ts`: 1949/1951
passed. The 2 local failures (`binds Slack tools...`, `materializes
direct external-chat finals...`) read `.claude/skills/slack/SKILL.md`
from the parent checkout of a nested local worktree (ENOENT). They do
not touch Retry and depend on the local checkout layout.
- `npx tsc --noEmit -p server/tsconfig.json`: passed.
- `pnpm verify` exited 2 (a doc file is a global CI input), so required
CI supplies broad coverage.
- Staging was inspected with GET requests only. No POST was sent to
staging.

## Risks

- Behavior change: a person's Retry on a connected task can now start a
board continuation where it was refused before. It uses the same
authority as a new message from that person, and dispatch keeps all
normal gates (task access, executor containment, recovery holds,
unadmitted chat input).
- The continuation does not publish the earlier result again and does
not reuse the chat requester's authority.
- No migration. Existing blocked tasks become retryable at the next
click.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 01:27:54 -06:00
DottaandClaude Opus 5.5 0dfbbdeb31 fix(chat): route company creation, decline retries and question cancellation correctly (#21035)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents follow the chat directive and discover API routes with
search_api and call_api
> - Full chat cohort 4 (OpenCode with DeepSeek v4.1 flash) showed four
guidance gaps
> - The missing-name rule overrode the board-only company-creation
boundary; a repeated request after a decline did not post a new approval
card; search_api showed the board-only question cancel route as open to
agents (and withdraw as board-only); write calls carried `responseText`
beside or instead of `body`
> - This pull request fixes the guidance, the route authorization
metadata and the pre-send refusal
> - The benefit is that agents take the supported path the first time
instead of asking for the wrong input, refusing a valid retry, or
calling a route that returns 403

## Linked Issues or Issue Description

**What happened?**
- onboarding.new-company: asked to "set up a new company for my website
business", the CEO cited the missing-name rule (#21007) and asked for a
business name with a question card. Creating a company is a board
action, so the agent should send the person to the creation flow.
- app-permissions.ask-before-action: the person declined a write card
and then repeated the same request. The CEO asked the person to tell it
to resubmit instead of posting a new approval card. The gateway already
allows a fresh request after a new human message
(`isNewHumanIntentAfterToolRejection`), but the decline text said "Do
not retry the same call".
- questions-and-approvals.multi-part-question: search_api listed `POST
/api/issues/{id}/interactions/{interactionId}/cancel` as
`board_or_agent`. The route rejects agents. The agent route,
`/withdraw`, was listed as board-only although its route admits the
card's creator agent and the task's assignee.
- Several cases: write calls through call_api carried `responseText`
with no body (for example "Received arguments: operationId, pathParams,
responseText").

**Expected behavior**
The agent links the company-creation flow; it posts a fresh approval
card for a repeated request; search_api shows the true authorization of
cancel and withdraw; a write with `responseText` is refused before it is
sent, with the same guidance #21002 gave `saveResponse`.

**Steps to reproduce**
Run the four chat cases above with OpenCode and DeepSeek v4.1 flash on
master.

Refs #21002, #21007

## What Changed

- `AGENT_CHAT_DIRECTIVE`: the missing-name rule applies to work the
agent creates itself and never comes before a board-only boundary. For a
new company the agent links [Create organization](/onboarding) (no
company prefix) and repeats the business details. A repeated request
after a declined approval card is an explicit retry: make the same call
so a fresh card is posted. A clarification question card puts candidate
links in its description, because labels show as plain text.
- Decline instructions in `tool-action-delivery.ts` and
`tool-gateway.ts` say that a later request from the person for the same
action is an explicit retry.
- OpenAPI: `POST /api/issues/{id}/interactions/{interactionId}/cancel`
is board-only and its summary names the `/withdraw` route for agents.
`/withdraw` is no longer marked board-only, which matches its route.
- `call_api`: a new write that carries `responseText` is refused before
it is sent; the message says `responseText` never carries request fields
and names the allowed body fields. Older saved calls still validate for
recovery. `doc/runner-api-tools.md` is updated.

## Verification

- `server/src/__tests__/agent-conversations.test.ts`: new directive
tests for the board-only boundary order, the decline retry and card
links.
- `server/src/__tests__/openapi-routes.test.ts`: cancel is `board`,
withdraw is `board_or_agent`, the cancel summary names withdraw.
- `server/src/services/native-runtime/runner-api.test.ts`: a write with
`responseText` is refused before sending (three window shapes), the same
write without it is sent once, and saved calls still validate.
- `tool-gateway.test.ts`, `tool-gateway-service.test.ts`,
`issue-thread-interaction-routes.test.ts`,
`tool-action-replay-intent.test.ts`: passed. `server` `tsc --noEmit`:
passed.
- Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) together with the
eval PR: see the PR comment for results.

## Risks

- A model that used to send a write with `responseText` and a valid body
now gets one refusal and must resend without it. Nothing is sent before
the refusal, so there is no duplicate side effect.
- The directive grows by a few sentences, which adds a small number of
prompt tokens per chat turn.
- Agents that read `/withdraw` as board-only before now see that they
can withdraw their own card; the route already enforced who may.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:27:35 -06:00
DottaandClaude Opus 5.5 940d681974 fix(heartbeat): hold a queued native run while its task's previous session finishes cleanup (#21031)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner runs keep a provider session per task. When a run
stops, its session close continues in the server process
> - The runner refuses every new session of the task until that close
settles. A close that cannot prove the old runner suspended needs a
stopped-process proof before it is released
> - A reassignment stops the old assignee's run mid-turn. The new
assignee's run was dispatched one second later into that gate, and it
failed with `native_session_cleanup_quarantined`
> - Only a Retry continued the task. Recovery must work without a
person, and the needed-retry rate must be 0%
> - This pull request keeps such a run queued until the close settles or
its old owner is proven stopped
> - The benefit is that the new run starts by itself after cleanup,
without a failed run or a Retry

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `tasks.edit-properties` (and
`tasks.assignment-conflict` in an earlier cohort), the CEO reassigned a
task while the old assignee's native run was mid-turn, with a Paperclip
tool call in flight. The reassignment cancelled that run. Its session
close then failed with `native_session_close_unrecoverable: runner did
not durably suspend before checkpoint`, and the runner kept an
operator-required cleanup gate for the task. The new assignee's run had
started 1 second after the cancel. The runner's admission waited for the
in-flight close. When the close escalated, admission refused the run
with `native_session_cleanup_quarantined`. No run was queued again.

**Steps to reproduce**
1. Run a native task and stop its run while its session close cannot
settle at once.
2. Queue a run of another agent for the same task.
3. The run is dispatched into the cleanup gate and fails before it
launches.

**Expected behavior**
The run waits for the cleanup, and starts once the close settles or the
old owner is proven stopped. It never fails, and no Retry is needed.

## What Changed

- `server/src/services/native-runtime/native-cleanup-admission-hold.ts`:
`nativeCleanupHoldsAdmission` reads the task's cleanup gate. For an
operator-required gate it first tries
`retireStoppedSessionCleanupGates`, which retires the gate when the old
owner's runner and provider processes are proven stopped. This is the
same proof that Retry uses.
- `server/src/services/heartbeat/queue.ts`: `claimQueuedRun` keeps an
unstarted `paperclip_runner` run queued while its task has a settling or
unproven close. This matches the existing
`predecessor_settling_in_process` hold, which covers only the same
agent. A reassignment changes the agent.
- `server/src/services/native-runtime/native-cleanup-release-wake.ts`: a
released close now also wakes queued-run dispatch. A close that
escalates to operator-required also wakes it, so admission tries the
stopped-process proof at once. The queued-run liveness redrive covers
any later proof.
-
`server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts`:
an assembled heartbeat test. It is red on master (the run is claimed
into a pending close and into an operator-required close). It is green
here (the run stays queued, then is admitted after the close settles). A
control case shows that a run for another task is not held.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts` (3
passed; 2 red on master)
- `cd server && npx vitest run
src/services/quarantined-task-retry.test.ts
src/services/heartbeat/queue.test.ts
src/__tests__/heartbeat-workspace-busy.test.ts
src/services/queued-run-dispatch-liveness.test.ts` (180 passed)
- `cd server && npx tsc --noEmit` passes.
- Paid E2E run on a combined branch:
https://github.com/paperclipai/paperclip/actions/runs/43161593494
(results in a comment below).

## Risks

- A run whose task has an operator-required gate that is never proven
stopped now stays queued instead of failing. Before, that run failed,
and Retry was blocked by the same proof. The liveness redrive re-checks
it with back-off.
- The hold applies only to unstarted `paperclip_runner` runs of the
gated task. Other tasks and agents are not affected.
- Previous-release saved state: no persisted shape changes. The gate
lives in process memory, and runs saved by the previous release are
claimed by the same rule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:53:54 -06:00
DottaandClaude Opus 5.5 ebaffddb01 fix(issues): release the outgoing assignee's never-started run on reassignment (#21030)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A lead agent creates tasks and assigns them to other agents
> - A task created without an assignee goes to its creator, with a
queued run that has not started yet
> - When the lead then reassigns the task, the route stopped only a
running run. The queued run kept the task's execution lock
> - The new assignee's wake waited behind that lock, so nothing ran
until the queue reached and withheld the old run
> - This pull request releases the outgoing assignee's never-started run
on reassignment and wakes the new assignee at once
> - The benefit is that reassignment starts the new owner's work
immediately, without a stale lock or a duplicate task

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `tasks.attachment-brief`, the CEO created a task
with no assignee, so it went to the CEO with a queued run. The CEO then
reassigned it to Morgan. The CEO's queued run kept the execution lock.
Morgan's assignment wake was deferred behind it. After 86 seconds the
queue cancelled the stale run as `issue_assignee_changed`. Before that,
the CEO got a 403 when it tried to cancel the run, and it created a
second copy of the task.

**Steps to reproduce**
1. Assign a task to an agent that is busy, so the task's assignment run
stays queued.
2. Reassign the task to another agent.
3. The queued run keeps the execution lock, and the new assignee's wake
is deferred.

**Expected behavior**
Reassignment cancels the outgoing assignee's never-started run for that
task and wakes the new assignee at once.

## What Changed

- `server/src/routes/issues.ts`: on reassignment, when no running run
holds the task, also stop the outgoing assignee's queued run that holds
the task's execution lock. The stop uses the existing `issue_reassigned`
code and the existing stop settlement.
- Reassignment stops pass `suppressImmediateRecovery: true`. Without it,
releasing the lock queued an `issue_assignment_recovery` run for the
outgoing assignee, which took the lock back before the reassignment was
saved. The new assignee's wake is the successor.
- `server/src/__tests__/issue-reassignment-queued-run.test.ts`: a
database-backed route test. It is red on master (the queued run stays
queued and holds the lock, and the new assignee's wake is
`deferred_issue_execution`). It is green here (the run is cancelled as
`issue_reassigned`, and the new assignee has an admitted run for the
task). A control case shows that a queued lock holder that is not the
outgoing assignee's run is not touched.

## Verification

- `cd server && npx vitest run
src/__tests__/issue-reassignment-queued-run.test.ts` (2 passed; red on
master)
- `cd server && npx vitest run
src/__tests__/issue-agent-mutation-ownership-routes.test.ts
src/__tests__/issue-queued-comments-routes.test.ts
src/__tests__/human-directed-work.test.ts` (378 passed)
- `cd server && npx tsc --noEmit` passes.
- Paid E2E run on a combined branch:
https://github.com/paperclipai/paperclip/actions/runs/43161593494
(results in a comment below).

## Risks

- Behavior change: a reassignment now cancels the outgoing assignee's
queued run that holds the task. Before, that run was cancelled later at
claim time with `issue_assignee_changed`. The eval and the run log
already recognize both codes.
- Reassignment stops no longer queue an immediate recovery run for the
outgoing assignee. If the reassignment write then fails, the periodic
recovery sweep still recovers the task.
- No schema or saved-state change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:53:49 -06:00
DottaandClaude Opus 5.5 a372271a40 fix(wake-queue): keep a task done for its delegator's comment posted during the closing run (#21032)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A lead agent delegates a task, and the assignee works on it in a run
> - When someone comments while that run is active, the comment waits as
a deferred wake for the next run
> - If the run marks the task done, an agent comment with resume intent
reopened the task. This started another assignee run and more lead wakes
> - When the comment came from the lead that delegated the task, during
the run that closed it, the lead gets the completed result anyway
> - This pull request keeps the task done in that case. Other agents'
feedback and any person's message still reopen it
> - The benefit is no repeated work after a task is finished, and no
lost person messages

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `secrets.least-scope-delivery`, the CEO created
Morgan's task. While Morgan's run was active, the CEO posted a redundant
instruction comment on it. Morgan's run completed the task. The deferred
agent comment then reopened the done task. This caused a second Morgan
run and extra CEO wakes, and the eval's run bound was exceeded. #20996
covers a person's owed answers. This change covers agent comments with
resume intent.

**Steps to reproduce**
1. Let agent A create a task assigned to agent B.
2. While B's run is active, let A comment on the task with resume
intent.
3. Let B's run mark the task done.
4. The deferred comment reopens the task, and B runs again.

**Expected behavior**
The task stays done. A receives the completed result and can follow up.
A person's message still reopens the task.

## What Changed

- `server/src/modules/wake-queue/application/use-cases.ts`: the "agent
feedback after completion" reopen rule now also requires that the
comments are not watcher comments posted during the finishing run.
Person messages, `issue_reopened_via_comment`, and other agents'
feedback do not change.
- `server/src/modules/wake-queue/application/ports.ts` and
`adapters/postgres.ts`: `isWatcherCommentDuringFinishingRun` is true
only when every comment exists, has an agent author and no person
author, comes from the task's creator agent or its parent's assignee
(not the finishing agent), and was created after the finishing run
started.
- When this is true, the deferred wake is cancelled with the existing
reason "Deferred execution wake no longer applies to a terminal task".
The comment stays on the task.
- `server/src/__tests__/heartbeat-comment-wake-batching.test.ts`: a new
heartbeat case. It is red on master (the task reopens and the assignee
runs again) and green here (the task stays done, one run). The existing
case, where a non-delegating agent's feedback reopens the task, still
passes.
- `use-cases.test.ts`: a unit scenario for the new rule.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-comment-wake-batching.test.ts
src/modules/wake-queue` (369 passed; the new case is red on master)
- `cd server && npx tsc --noEmit` passes.
- Paid E2E run on a combined branch:
https://github.com/paperclipai/paperclip/actions/runs/43161593494
(results in a comment below).

## Risks

- Behavior change: if a delegating agent asks for new work in a comment
during the closing run, the assignee does not get it automatically. The
delegator receives the completed task and can comment again or create
follow-up work. A person's message is never dropped by this rule.
- Comments with no author type (older rows) count as agent comments only
when they have an agent author and no person author.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:53:43 -06:00
Nicky Leach d68088465c Reuse retained Runner descriptions during startup (#21004) 2026-11-11 21:20:08 -08:00
DottaandClaude Opus 5.5 156453fe7c fix(recovery): continue runs that a server restart or replacement ended, without spending the failure budget (#21019)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A deploy replaces the server. The old server stops its running
legacy runs at SIGTERM (`server_shutdown_interrupted`). The new server
finds runs whose owner died without that drain (`process_lost`, "server
may have restarted").
> - Each of these runs got its follow-up from the bounded transient
retry. That retry spends the failure budget (2 attempts) and waits 30 to
60 seconds.
> - So the first deploy spent one failure retry. A run that was already
the last automatic retry got no follow-up at all. The task then rested
on a manual Retry.
> - A deploy is routine. An interruption by the server is not a failed
attempt, and it must never leave a task waiting for a person.
> - This pull request moves these runs to their own `server_restart`
retry lane. The lane never spends the failure budget, starts the first
continuation at once, and has its own bound.
> - The benefit is that a task continues by itself through several
deploys a day, and a real loop still stops and shows why.

## Linked Issues or Issue Description

**What happened?**
A zero-downtime deploy replaced a hosted server. In the 11 minutes
around the swap, 11 legacy runs ended with `server_shutdown_interrupted`
and 7 ended with `process_lost`. At least 7 tasks were left with no
follow-up run and no scheduled retry, so each needed a manual Retry. Two
of the stranded runs were already automatic retries.

**Expected behavior**
Each interrupted run continues once, automatically, on the surviving
server, with its original wake.

**Paperclip version or commit**
Core commit 7d9973405c. The behavior is
the same on master bf18b348c5.

**Steps to reproduce**
1. Start a legacy run with a conversation adapter (for example
`codex_local`) as the second automatic retry of its chain (failure
budget spent).
2. Send SIGTERM to the server (`drainRunningRunsForShutdown`), or let a
new server boot find the run without its owner (`reapOrphanedRuns`).
3. See that no successor run is created. The scheduler logs "Bounded
retry exhausted".

**Root cause**
- `enqueueProcessLossRetry`
(`server/src/services/heartbeat/recovery.ts`) calls
`scheduleBoundedRetryForRun`. That function put these runs in the
`transient_failure` lane, with `maxAttempts = 2` counted from
`executionRetryAccounting.failureRetries`
(`server/src/services/heartbeat/retries.ts`). A first attempt got a
charged retry 30 to 60 seconds later. A run with 2 spent failure retries
got `retry_exhausted`.
- The orphan reaper (`recoverActiveExecutions`) skipped the retry when
`failedRunRetryBudgetSpent(run)` was true. It also skipped a
conversation run that had no local child process recorded (for example a
run that was dispatching), because `shouldRetry` needed a local child, a
preparing stage, or a monitor wake.
- The stranded-task sweeper then saw the spent budget and moved the task
to `blocked`.

## What Changed

- `execution-recovery-attempt.ts`: add the `server_restart` retry lane.
It is an uncharged lane. Its counter (`serverRestarts`,
`serverRestartWindowStartedAt`) is saved in `executionRetryAccounting`,
so it is visible on each run. The bound is 6 continuations in one
60-minute window. A later interruption after the window starts a new
window.
- New `server-replacement-interruption.ts`:
`isServerReplacementInterruption(run)` names a legacy
`server_shutdown_interrupted` run, or a `process_lost` run whose owner
was another server boot. It reads the new
`processLossDiagnostic.controllerReplaced` verdict. For rows saved by an
older release, it reads the owner boot id or `runPredatesObserver`.
- `retries.ts`: `scheduleBoundedRetryForRun` uses the `server_restart`
lane for these runs (`usesServerRestartLane`). The first continuation is
due at once; later ones in the window back off like the
interrupted-preparation lane. `failedRunRetryBudgetSpent` reports the
restart bound for these runs, so the sweeper escalates only when that
bound trips. An interrupted preparation keeps its own lane while its
chain has failure budget left.
- `recovery.ts` (orphan reaper): record `controllerReplaced`. A
conversation run that a restart ended is eligible for continuation even
without a local child process, and it checks the restart bound, not the
failure budget.
- `wake-queue/adapters/postgres.ts`: carry the restart counter to
successors that the wake queue creates.
- No schema change. No change to how runs start (no new waits before a
turn).

Behavior that did not change:
- At most one successor per run: every path still uses the predecessor
row lock and the `retry_of_run_id` check in
`scheduleBoundedRetryForRun`. Two servers alive at once reuse the same
successor.
- A non-conversation adapter that reached dispatch stays held for
reconciliation, because a replay can repeat its actions.
- Native runtime runs keep their own recovery.
- A server that began shutdown already starts no queued run
(`closeProcessRunAdmission` in `server/src/index.ts`, checked in
`heartbeat/queue.ts`). Its queued and newly scheduled runs stay in the
database for the new server.

## Verification

- Red on `origin/master`, green on this branch
(`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs
ended by a server restart or replacement"):
- Graceful shutdown, first attempt: master scheduled a charged
`transient_failure` retry. Now one immediate `server_restart`
continuation.
- Graceful shutdown, already the last retry: master scheduled nothing.
Now one continuation; the failure count stays 2.
- Owner lost at boot (`process_lost`), first attempt and
already-retried: same results, with two servers running the boot sweep
at the same time. Only one successor.
- Old server exited after it committed the interruption and before it
scheduled the continuation: the new server's sweep continues the run.
- Run saved by an older release (no `controllerReplaced`, spent failure
budget): the sweep continues it instead of moving the task to `blocked`.
- The restart bound trips after 6 in one window, records `retryReason:
server_restart` on the run, and the sweeper escalates. The same count in
a closed window continues.
- A run that failed by itself (`adapter_failed`) with a spent failure
budget still gets no retry.
- Each continuation's execution continuation reports the original wake
(`trigger.retriedReason = issue_monitor_due`).
- Unit tests:
`server/src/services/server-replacement-interruption.test.ts`.
- Six earlier tests asserted escalation after "three deploy restarts"
spent the failure budget. They now spend the restart bound instead, so
they still prove escalation when the bound trips.
- Existing test `server/src/services/heartbeat/queue.test.ts` covers
"starts no run once shutdown closed run admission".
- `pnpm verify`: see the PR checks.

## Risks

- A run that a restart ends now continues more often (up to 6 times per
hour per chain, not 2 in total). A run that itself crashes the server is
still stopped by the bound, and the stop is shown on the task.
- Tasks that an older release already moved to `blocked` stay blocked.
Tasks that are still `in_progress` or `todo` with a stranded run
continue on the next sweep.
- A `process_lost` row from an older release whose owner was the same
boot that later restarted is read as a restart. That case is a lost
in-memory handle, which is also not a failure of the agent's work.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:56:49 -06:00
DottaandClaude Opus 5.5 8c761abae3 fix(environments): wait for a re-activating sandbox provider before refusing a lease resume (#21020)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a controller restart, a run whose remote Runner kept running
reattaches. First it resumes its exact sandbox lease through the sandbox
provider plugin (`resumeRunLease`).
> - #20982 made a plugin whose worker `initialize` timed out retry its
activation, so the plugin comes back seconds later instead of staying
`error`.
> - Chaos cell CR6 (#20983) shows the gap that remains. The reattaching
run asks for its lease during those seconds and gets
`PluginWorkerUnavailableError`. The setup retry covers that error only
for legacy runs, so the native restart-recovery run fails and needs
Retry.
> - This pull request makes the exact-lease resume wait, for a bounded
time, for the worker of a plugin that is `ready` (re-activating) or
recovering from activation timeouts.
> - The benefit is that a restarted run rides through a short plugin
re-activation instead of failing into Retry.

## Linked Issues or Issue Description

Refs #20982, #20983

**What happened?**
CR6 after #20982 (`w-20261112T041333-s9531`) passed in part. The plugin
re-activated 5 s after its `initialize` timeout, but both tasks finished
with `R3:retry_needed`. Run `a6b3f0f6` resumed its lease at 22:15:35,
inside that window. It failed with `setup_failed` /
`plugin_worker_unavailable` and then `agent_stop_unconfirmed`, because
`run-completion.ts` retries `PluginWorkerUnavailableError` only for
legacy runs (`runtimeMode === "legacy" && nativeSessionId === null &&
runnerInstanceId === null`).

**Expected behavior**
The resume waits for the plugin's worker while the plugin re-activates.
It resumes the same lease when the worker comes back. It still refuses,
as `PluginWorkerUnavailableError`, after the bound.

**Steps to reproduce**
1. Run chaos cell CR6: a K1 restart, then a stall of the replacement's
plugin worker during activation.
2. The reattaching run fails and needs Retry, although the plugin
recovers within seconds.

**Paperclip version or commit**
master at f1aef0cb58.

## What Changed

- `environment-runtime.ts`, `resumeRunLease`: the worker is not running,
but the plugin is `ready` or its activation-timeout recovery is in
flight. In that case the resume polls `isRunning`, every
`pluginWorkerReadyPollMs`, for up to `pluginWorkerResumeWaitMs` (default
60 s). That bound covers the first two activation retries (5 s, then 15
s). When the worker runs, the normal exact resume goes on. After the
bound it refuses with `PluginWorkerUnavailableError`, as before.
- Nothing changes when the worker is running: no wait, no added latency
on the normal path. A plugin that is `disabled`, `uninstalled`, or
`error` for a reason other than a timeout is not waited on.
- `doc/plugins/PLUGIN_SPEC.md` §12 describes the wait.

## Verification

- Red on master: `npx vitest run
src/__tests__/environment-runtime.test.ts -t "re-activating plugin's
worker comes back"` (in `server/`) fails with
`PluginWorkerUnavailableError`.
- Green: the test passes. The worker comes up on the fourth `isRunning`
check, and the lease is resumed with one `environmentResumeLease` call.
- The existing refusal test, now with a 50 ms bound, still refuses with
no provider call.
- `environment-runtime.test.ts` passes (205 tests), and `tsc --noEmit`
passes for `server`.

## Risks

- A restart-recovery run whose provider plugin never comes back now
waits up to 60 s more before it fails. It fails the same way as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:49:23 -06:00
DottaandClaude Opus 5.5 b033cc124d fix(evals): repair private human steps for webhook, device and key entry; name failed private steps (#21012)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat evaluation catalog checks agent replies and then the human
step that follows, such as a person who reveals a key or approves a
device
> - Some human steps run in a separate private browser process, because
they handle credentials. CI keeps the output of that process private on
purpose
> - Four cases failed only in that private step, so CI showed one fixed
message and no cause
> - I ran each private step locally against a dev server, where its
output is visible, and found the cause of three failures
> - This pull request fixes the harness and two product defects, and
adds a closed failure record so that the next CI failure names its step
> - The benefit is that these human steps work, and a private-step
failure is diagnosable without any credential in logs

## Linked Issues or Issue Description

Bug report (following `.github/ISSUE_TEMPLATE/bug_report.yml`).

**What happened?**
- `routines.webhook-trigger`: the private person opens the routine
overview. The trigger cards are only in the Triggers section, so the
step waits for a missing button. An unhandled response wait then stops
the process.
- `connection-setup.secure-api-key`: a key-protected MCP server answers
401 without a challenge header. Paperclip shows "Remote app returned
HTTP 401" and does not ask for a key. The person cannot enter the key in
the normal flow.
- `assistant-connections.device-login`: an expired device code page
shows "This item doesn't exist or was moved." The server sends "This
code is expired, already decided, or invalid." in the OAuth
`error_description` field, and the UI drops it.
- `onboarding.new-company`: the private step reads back all saved state
and then fails one proof check. CI does not say which check.

**Expected behavior**
- The person reveals the webhook key in the Triggers section and sends
authenticated events.
- A pasted MCP link that answers 401 shows the key field.
- An expired device code says that the code is expired.
- A private-step failure names its fixed step and error class, with no
value.

**Steps to reproduce**
1. Start a local dev server.
2. Create a routine with a bearer webhook, and run the private webhook
worker. It times out on the "Edit webhook" button.
3. Paste the address of an MCP server that answers a plain 401 into
Connectors > Add by link. No key field appears.
4. Open `/mcp-device?user_code=<expired code>`. The page shows generic
not-found text.

**Paperclip version or commit**
`master` (reproduced on origin/master before this change)

**Deployment mode**
Local dev server (embedded PGlite) and the CI chat-eval runner

**Installation method**
Source checkout (`pnpm dev`)

## What Changed

- `integrator-webhook-private-worker.ts`: open `<routine>/triggers`
(also for the setup-resume link), and attach a handler to the early
response wait so a failed click reports its real step.
- `server/.../catalog/discovery.ts`: a 401 with no `WWW-Authenticate`
header, to a connection that sends no credential, is handled like a
challenge without discoverable sign-in (`oauth_challenge`). The existing
UI guidance then asks for a key. A 401 to a connection that already
sends a credential keeps the old error.
- `ui/src/api/errors.ts`: read the OAuth `error_description` field as a
readable server message.
- New `private-child-failure.ts`: a private child records only a fixed
step name and an error class from a fixed list. The parent echoes a step
only when it is in the caller's closed list. Error text, stdout and
stderr stay private.
- The webhook, private browser and identity children use it. Each has a
closed step list file.
- `identity-new-company-native-proof.ts`: each proof check has a fixed
reason name. The private child reports `new-company-proof-<reason>` when
a check fails. The accepted conditions do not change.
- Tests: server red/green test for the bare 401, UI red/green test for
`error_description`, unit tests for the failure record (no text leaks,
closed list only, extra fields dropped).

## Verification

- Local dev server (embedded PostgreSQL), private workers run directly
with the real child process:
- Webhook, ready and setup-pending triggers: before, timeout on "Edit
webhook". After, receipt with invalid status 401, valid status 202 and a
linked task.
- Secure key entry with the synthetic notes provider: before, timeout on
the key field. After, an active connection with health `ok` and 2
authorized `tools/list` calls.
- Expired device page: before, `at step device-expired-page
(TimeoutError)`. After, `expiredPageVisible: true`.
- `server`: `vitest run src/__tests__/tool-access-service.test.ts -t
"pasted link answers"` (fails before the discovery change with 502,
passes after).
- `ui`: `vitest run src/api/errors.test.ts` (fails before, passes
after).
- `pnpm test:e2e:runner:typecheck`, `pnpm test:e2e:runner:unit`, server
`tsc --noEmit`.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD sha>
--check`: unchanged.
- One paid run of the four cases on this branch. Results are in a PR
comment.

## Risks

- A plain 401 from an MCP link now returns 422 `oauth_challenge` instead
of 502. The draft connection is removed, as for other challenges. Agents
that call the connect API see the "needs sign-in" answer instead of a
raw status. This only applies when the connection sends no credential.
- Some API errors that carry `error_description` now show that text
instead of generic copy. That text is written for people.
- The failure record is opt-in per child and contains only constants.
The parent ignores a record with any other field.
- `onboarding.new-company` is not fixed here. Its native hello probe
cannot start locally on macOS, so the failing check is not yet known.
This change makes the next run name it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits, local browser automation through Playwright). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:06:29 -06:00
DottaandClaude Opus 5.5 1bb4dcbe4b fix(routines): agents can only create authenticated webhook triggers (#21006)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Routines can have webhook triggers that outside services call
> - Agents can create and change routine triggers through the API
> - The choice of webhook authentication belongs to a person
> - This pull request lets agents create authenticated webhook triggers
only, and keeps every choice open to people on the board
> - The benefit is that agents can only create authenticated webhook
triggers

## Linked Issues or Issue Description

**What happened**

In a chat eval, an agent set up a routine webhook trigger with a weaker
authentication choice than the default.

**Expected behavior**

Agents can only create authenticated webhook triggers. When a sender
cannot authenticate, the agent asks a person, who decides on the board.

**Steps to reproduce**

1. Ask an agent in chat to give a routine a webhook for a test service.
2. Read the trigger that it creates.

## What Changed

- `server/src/services/routines.ts`: trigger creation, trigger updates
and revision restores refuse an agent that does not keep the webhook
trigger authenticated. The error tells the agent to use the default or a
signed mode, or to ask a person. Board users keep every option. An agent
can still edit other fields of a trigger that a person set up.
- `skills/paperclip/references/routines.md`: says agents can only create
authenticated webhook triggers.
- Test: new case in `routines-service.test.ts`.

## Verification

- `cd server && npx vitest run src/__tests__/routines-service.test.ts`:
104 passed. The new test fails without the service change (red) and
passes with it (green).
- `npx vitest run src/__tests__/routines-routes.test.ts
src/__tests__/company-portability.test.ts`: passed.
- The chat eval `routines.webhook-trigger` already requires one
authenticated trigger. It runs again on a combined throwaway branch;
results are in a PR comment.

## Risks

- An agent-run company import that would add such a trigger now fails
with the same error and rolls back. A person can run that import from
the board.
- Agents must use the default or a signed mode.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:05:16 -06:00
DottaandClaude Opus 5.5 5a7378af56 fix(heartbeat): re-drive a queued run whose dispatch signal was lost after a restart (#20986)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts queued runs. The run-dispatch worker
re-runs the queue only when a write sends a "queued run dispatch"
signal.
> - A follow-up run can queue while its predecessor still holds its
sandbox lease. The claim then declines the run without a log line, and
the run stays `queued`.
> - After a controller restart, nothing sent a new signal once that
lease was settled. The run stayed `queued` forever. The issue had no
Retry, and the reconciler counted the queued run as a valid next action.
> - This pull request sends a dispatch signal from the native
finalization writes that can clear a predecessor's cleanup gate, logs
why a claim was declined, and bounds the wait. The dispatch worker
re-drives an undispatched queued run after 60 s as a backstop, and the
ownership reconciler reports and re-drives one still queued after 5
minutes.
> - The benefit is that a follow-up starts within seconds of its
predecessor settling, a lost signal delays it by about a minute instead
of forever, and a run that still cannot start is visible.

## Linked Issues or Issue Description

Refs #20909, #20935, #20953, #20970

**What happened?**

Restart-chaos wave `w-20261111T223714-s9512` (master `6154b9485e` +
#20907 + #20909, Daytona warm, Codex, W2, CR1 with a 360 s hold,
`--ephemeral-server`, seed 9512), task t02. Run `0b95b1cd` crossed the
outage with an accepted result. The next controller finalized it at
22:47:59Z. Its continuation run `1ad79418` was queued at 22:48:00Z
(`retryOfRunId` = `0b95b1cd`, wake `d967700c`, source
`native_status_decision`). The run stayed `queued` with `claimed_at =
null` until the wave ended 19 minutes later. The agent stayed `running`
with no running run. The task had no Retry. The grade was
`R1:no_rest_within_budget` and U1 `no_owner_no_action ... queued 485s`.

**Root cause**

`claimQueuedRun` (`server/src/services/heartbeat/queue.ts`,
`lockIssueExecutionClaim`) declines a queued run while
`hasPendingRunCleanup` (`server/src/services/execution-blocker.ts`) is
true for its predecessor. After the restart, the predecessor's warm
`reuse_by_environment` lease was still `active` when the continuation
queued, so the first claim was declined. The decline is correct, but
nothing owned the retry. `registerRunDispatchWork`
(`server/src/services/heartbeat/dispatch-work.ts`) has `next: async ()
=> null`: the queue is only re-run by a later dispatch signal. At wave
end the lease was `retained` with `cleanupStatus: success`, which does
not block a claim. A dispatch pass in that state starts the run
(verified locally). So no pass ran after the gate cleared. The ownership
reconciler listed the run as `queued_run` with owner `run_dispatch`
(always "alive"), and I7 treats any queued run as a next action, so
nothing reported it.

This is not the #20935 throttle. The wave base predates #20935, and a
throttled intent keeps its wake intent `queued` with a `nextAllowedAt`
deadline. Here the run row existed and its claim was never retried. On
current master #20935 parks this specific `response_wake` self-wake, but
any continuation, comment wake or Retry that queues behind a settling
predecessor after a restart has the same gap. The red test uses a
`same_agent` continuation, which master still enqueues.

**Steps to reproduce**

`server/src/__tests__/heartbeat-native-restart-suspension.test.ts`,
"starts a continuation that queued behind its predecessor's sandbox
release after a graceful restart".

## What Changed

- Event-driven dispatch. A lease release already signals queued-run
dispatch (`environmentsSvc.releaseLease`, `writeEnvironmentCleanup`,
stop-and-retain settlement). I enumerated every `environment_leases`
status writer: each one that moves a lease out of a blocking state
already signals. The writers that do not signal only acquire a lease or
set `pendingCleanupInFlight`, so they add a block and do not clear one.
The native coordinator side had no signal, so:
- `notifyNativeFinalizationWork` (the shared notifier of coordinator
commits and terminal transitions) now also signals queued-run dispatch
for the company.
- The coordinator writes that clear `leaseOwner` or end the phase
outside that notifier now signal in the same transaction:
`releaseLapsedCoordinatorOwner`, native startup cleanup,
observed-workspace preservation, `record_finalization_error` in the
interaction bridge, the wake-queue terminal recovery, and the
reconciler's ambiguous-state terminal failure.
- Not changed: three non-transactional pre-dispatch closes in
`run-completion.ts` and `agent-execution-stop.ts`. Their callers queue
or settle the successor after the write, so the successor's own enqueue
starts the queue.
- `claimQueuedRun` now logs the reason when it declines a queued run at
the issue execution claim (`predecessor_cleanup_pending`,
`predecessor_still_executing`, `issue_execution_owned_by_another_run`,
case handoff, task chat cleanup), when a predecessor is still settling
in this process, and when a temporary budget hold keeps it queued.
- New `server/src/services/queued-run-dispatch-liveness.ts`:
`listUndispatchedQueuedRuns` reads queued runs in active companies whose
ready, non-paused agent has no running run. It reads every page with a
keyset cursor until the set is exhausted, so older held runs cannot hide
a newer stranded run. `createQueuedRunRedriveSchedule` re-drives each
such run 60 s after it was created, then every 2, 4, 8 minutes, capped
at 15 minutes. Run ids that leave the set are dropped.
- `server/src/services/heartbeat/dispatch-work.ts`: the queued-run
dispatch worker's `next` now plans those re-drives. It re-reads only the
companies the pass just dispatched, keeps the deadlines of every other
company, and runs no query when the pass dispatched nothing. Only an
instance-wide pass (startup or explicit wake) reads every company. It
adds each due company to the dispatch scopes and returns the next due
time.
- `packages/db`: new partial index `heartbeat_runs_queued_created_idx
(company_id, created_at, id) WHERE status = 'queued'` (migration
`0470_heartbeat_runs_queued_created_index.sql`, `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`, registered as a top-level migration in
`client.ts`). It returns `null` (no deadline, no SQL until a signal)
when no run is undispatched or scheduling is suppressed. Every re-drive
goes through the normal `startNextQueuedRunForAgent` and
`claimQueuedRun` gates, so it never bypasses a hold, budget, pause or
cleanup gate.
- `server/src/services/heartbeat.ts`: wires `listUndispatchedQueuedRuns`
into the worker (two lines).
- `server/src/services/recovery/ownership-invariant-report.ts`: new
check. A queued run still undispatched 5 minutes after creation is an I7
violation (`kind: queued_run`, `reason: queued_run_not_dispatched`),
written to the activity log once. A younger one sets the report's next
due time.
-
`server/src/services/native-runtime/native-finalization-reconciler.ts`:
`applyOwnershipReportBackstops` sends a queued-run dispatch signal once,
for each company with a newly found violation. Repeat passes do not
signal again, so a held run does not wake the report every 30 s.
- `tests/restart-chaos/README.md`: records the wave and the fix in the
CR results table. The existing CR1 cell (#20909) and the enforced U1
check already cover this class in the harness.

## Verification

- RED on master (`23909077ad`):
- `heartbeat-native-restart-suspension.test.ts` "starts a continuation
that queued behind its predecessor's sandbox release after a graceful
restart": the continuation is `queued` behind the predecessor's `active`
warm lease. The run-dispatch worker is registered, the lease then
settles to `retained`/`success` with no dispatch signal, and the run
stays queued: `AssertionError: expected null not to be null` after 75 s.
- `ownership-reconciliation-report.test.ts` "reports and re-drives a
queued run that no dispatch started while its agent had nothing
running": `expected [] to deeply equal [ ObjectContaining{…} ]` (no
violation for a run queued 19 minutes).
- `ownership-reconciliation-report.test.ts` "signals queued-run dispatch
when finalization work settles or a finished coordinator's lapsed owner
is released": `expected [] to include 'settled'`.
- GREEN on this branch: all pass. The restart test has two cases. When
the lease is released through `releaseLease`, the continuation starts
within 5 s. When the lease is settled by a write with no signal, the
backstop starts it within 75 s. A new report test puts three older held
runs ahead of a stranded run with a page size of 2, finds the stranded
run, and checks that the backstop signals once.
- `queued-run-dispatch-liveness.test.ts`: schedule bound, doubling, cap,
and going quiet.
- Focused suites pass (22 files, 495 tests):
recovery-deadline-acceptance, live-recovery-work,
heartbeat-run-dispatch-lifecycle, heartbeat-run-dispatch-work,
deferred-execution-receipt-work, heartbeat-native-cleanup-admission, the
ownership kinds suites, and others.
`heartbeat-native-restart-suspension` and
`ownership-reconciliation-report` pass in full. `tsc --noEmit` for the
server passes.

## Risks

- A queued run that is held on purpose (for example a temporary budget
hold or a case handoff waiting for release) is now re-checked at most
every 15 minutes after the first minute. Each re-check is one indexed
query and the normal claim gates. Such a run is also reported once as an
I7 violation.
- An instance with an undispatched queued run keeps a dispatch deadline,
so it does not idle while that run waits. An instance with no such run
is unchanged.
- Schema: one additive partial index, built concurrently. No data
change. State saved by an older version is read as is. Nothing runs
before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:04:58 -06:00
DottaandClaude Fable 5.2 82e0393598 fix(ui-assets): retry a throttled asset upload and never fail startup on a storage outage (#21010)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A hosted Paperclip server can run volume-free. Before it serves, it
publishes its UI build assets to shared object storage, so an older
browser tab can still load its chunks after the old container is gone
> - A build has several hundred assets. The server uploads them one
after another at startup
> - Object storage throttles a burst with `SlowDown` (HTTP 503). The
upload body is a stream, so the storage client cannot replay the
request, and the error leaves `startServer`
> - The process exits, the host restarts it, and the next start uploads
the same assets again. A workspace stayed down while four starts in a
row failed this way
> - This pull request retries a throttled asset and lets the server
start when storage stays unavailable
> - The benefit is that a storage slowdown cannot keep a server from
starting

## Linked Issues or Issue Description

Refs #20990, #20952 (other startup failures found on the same hosted
workspace).

**What happened?**

A hosted volume-free workspace received a new release. Each start logged
many `An error was encountered in a non-retryable streaming request.`
lines and then `Paperclip server failed to start` with `SlowDown: Please
reduce your request rate.` for a key under `ui-assets/v1/`. The four
starts failed after 44, 42, 79 and 90 seconds, each on a later asset
name.

**Expected behavior**

The server waits and repeats a throttled upload. If shared storage stays
unavailable, the server starts and serves its build from local files.

**Steps to reproduce**

1. Run a server with `PAPERCLIP_SHARED_UI_ASSETS=true` and S3-compatible
storage.
2. Make the storage answer `SlowDown` (HTTP 503) to one `PutObject`
during startup.
3. The server exits with `Paperclip server failed to start`.

**Paperclip version or commit**

`52cdf76bf3e0cdf29037b63bf0f5e99aa464eab7`

## What Changed

- `retainedUiAssets.publish`: each asset is published in a retry loop. A
throttled or server-side storage failure waits (0.5 s to 8 s, seven
waits at most) and repeats the asset with a fresh stream.
- `isTransientStorageError`: recognizes `SlowDown`, throttling, request
timeout and HTTP 429/500/502/503/504 answers.
- `app.ts`: when publication still fails with a transient storage error,
the server logs an error and continues. Other failures, such as a name
collision with different bytes, are still fatal.
- Tests for the retry, for the give-up path, and for the error
classification.

## Verification

- `npx vitest run src/__tests__/retained-ui-assets.test.ts` in
`server/`: 4 tests passed. The two new tests fail on `master` because
`publish` rejects on the first `SlowDown` and takes no `sleep` option.
- Not run locally: the full suite. CI supplies that coverage.
- Not verified: a live start against throttling storage.

## Risks

- A start can take up to about 31 seconds longer for each throttled
asset before it gives up.
- If storage stays unavailable, the server starts without its assets in
shared storage. An older browser tab can then fail to load a chunk after
the old container is gone. The server logs this at error level.
- Collision detection is unchanged.

## Model Used

Claude Fable 5.2 (`claude-fable-5-2`), through Claude Code with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 21:21:27 -06:00
DottaandClaude Opus 5.5 a9a08d8169 fix(plugins): retry a plugin activation whose worker initialize timed out (#20982)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins, sandbox providers such as Daytona among them, run as worker
processes that the server activates at boot. Activation sends the worker
an `initialize` RPC with a 15-second timeout.
> - When `initialize` timed out, the loader marked the plugin `error`
for the life of the server process. Only the next boot or an operator
retried it.
> - For a sandbox provider, that means every Daytona run on that server
fails. New runs fail with `lease_acquire_failed … plugin is currently
error`. Reattachments fail with "The exact sandbox lease cannot be
resumed by its verified provider". Retry hits the same dead plugin.
> - A worker that is slow or stalled for a moment at boot is transient,
not a broken plugin.
> - This pull request retries an activation whose `initialize` timed
out, with a bounded backoff. Meanwhile the server keeps serving, and
runs that need the plugin see a restarting worker, which setup retries.
Deterministic failures stay a permanent error.
> - The benefit is that a short plugin stall at boot no longer disables
a provider until the next deploy.

## Linked Issues or Issue Description

Refs #20966, #20857

**What happened?**
Chaos wave `w-20261111T232850-s9521` (cell CR4, first version) stopped
the replacement server's Daytona plugin worker while the server was
activating it. The worker's `initialize` RPC timed out after 15 s
(`INITIALIZE_TIMEOUT_MS`). `plugin-loader.ts` then called
`lifecycleManager.markError`, and the plugin stayed `error`. Task t03's
reattachment failed in `resumeRunLease`, and its Retry did not start a
new run. Task t04 failed 18 s after launch with `lease_acquire_failed`
("…plugin is currently error"), before its own fault.

**Expected behavior**
An activation timeout is retried, and the plugin ends `ready` when the
worker answers. Runs that need the plugin meanwhile are retried within
their bounded setup budget. A bad manifest, a worker that exits during
`initialize`, or a version mismatch is still a permanent error.

**Steps to reproduce**
1. Start the server.
2. Stop (SIGSTOP) a plugin worker for more than 15 s while the server
activates it.
3. Resume the worker. The plugin stays `error` until the next boot.

**Paperclip version or commit**
master at dfd2a6efca.

## What Changed

- `plugin-worker-manager.ts`: the "Worker initialize failed" error keeps
its cause, the `JsonRpcCallError` with `PLUGIN_RPC_ERROR_CODES.TIMEOUT`.
- `plugin-loader.ts`, `activatePlugin`:
- An activation whose error chain has the TIMEOUT code is retried after
5, 15, 45 and 120 s (`activationTimeoutRetryDelaysMs`).
- Before each retry, the loader tears down the partial runtime and keeps
the plugin `ready`. It sets `lastError` to "Activation timed out;
retrying in Ns (retry n of 4): …" so operators can see the state.
- A retry stops when the plugin is no longer `ready` or its worker
already runs. Shutdown clears pending retries.
- After the last retry, the loader calls `markError` with "Activation
timed out after 4 retries: …". Any other failure calls `markError` with
"Activation failed: …" at once, as before.
- `pluginLoader.requestActivationTimeoutRecovery(plugin)`: re-activates
a plugin in `error` whose `lastError` starts with "Activation timed
out", at most once per minute.
- `app.ts` exposes it through
`workerManager.requestActivationTimeoutRecovery`.
- `environment-runtime.ts`:
- Lease acquisition, for both pinned and key-resolved providers, calls
`requestActivationTimeoutRecovery` for such a plugin and reports
`worker_unavailable` (`PluginWorkerUnavailableError`, which setup
retries) instead of the terminal "plugin is currently error".
- `resumeRunLease` reports a stopped worker as
`PluginWorkerUnavailableError` when the plugin is `ready`, or is
re-activating from a timeout, instead of the generic "cannot be resumed
by its verified provider".
- This covers every plugin kind: activation retry is in the shared
loader. The on-demand path is in the sandbox-provider resolution, which
is the path a run waits on.
- `doc/plugins/PLUGIN_SPEC.md` §12 describes the rule.

## Verification

- Red on master:
`server/src/__tests__/plugin-loader-activation-timeout.test.ts` uses a
real PostgreSQL plugin registry and a bundled image fixture, and all 3
of its tests fail on master. On master, "initialize times out once, then
succeeds" stays `error` and the worker starts only once.
- Green, all 3 pass:
- A timeout followed by success ends `ready`, with a running worker and
no `markError`.
- Timeouts past the bound end `error` with the "Activation timed out
after 2 retries" message, and an on-demand request re-activates the
plugin to `ready`.
- A worker crash during `initialize` is a permanent error, with no retry
and no on-demand re-activation.
- `environment-runtime.test.ts` has new tests:
- A sandbox plugin errored by activation timeouts gives
`PluginWorkerUnavailableError` and requests re-activation. When no
re-activation is in flight, the error stays terminal.
- `resumeRunLease` on a stopped worker of a `ready` or re-activating
plugin gives `PluginWorkerUnavailableError` and sends no provider call.
- These files pass: `environment-runtime`,
`environment-run-orchestrator`, `plugin-loader-error-retry`,
`plugin-environment-driver-ready-recovery`,
`bundled-plugin-registration`, `plugin-lifecycle-restart`,
`plugin-lifecycle-delivery` and `plugin-dev-watcher`.
- `tsc --noEmit` passes for `server`.
- Chaos: cell CR6 (a plugin stalled during activation at boot) runs
before and after this change. The results are in a comment.

## Risks

- A plugin whose `initialize` keeps timing out now takes about 3 minutes
to reach `error`, instead of 15 s. During that time runs that need it
see a restarting worker and are retried within their setup budget.
- On-demand re-activation is limited to one attempt a minute per plugin,
so a stuck plugin cannot crash-loop.
- Nothing changes for deterministic activation failures.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 21:18:47 -06:00
DottaandClaude Opus 5.5 7d9973405c fix(chat): ask for a missing name only when the request and saved brief give none (#21007)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and the chat guidance tells the
agent how Paperclip works
> - #20888 added a rule: ask for a missing name before building
something that must carry it
> - In a chat eval, "Start a Launch project and have Morgan build its
landing page" made the agent create the project and then ask for the
brand name. It did not create Morgan's task
> - The person had named the project, so a question was not necessary
> - This pull request narrows the rule: ask only when the request, the
conversation and the work's saved brief give no name
> - The benefit is one necessary question when a name is truly missing,
and no extra question when the person already named the work

## Linked Issues or Issue Description

Refs #20888

**What happened**

For "Start a Launch project and have Morgan build its landing page", the
agent created the Launch project and asked for the brand name. It did
not create the landing-page task for Morgan.

**Expected behavior**

The agent uses the name the person gave (Launch), creates the project
and the task for Morgan, and returns both links. For "Build a landing
page for my new product." with no name anywhere, the agent still asks
one free-text question for the product name.

**Steps to reproduce**

1. Run the chat eval `tasks.website-without-project` on master.
2. Read the reply: the agent asks for a brand name and creates no task.

## What Changed

- Chat guidance (`AGENT_CHAT_DIRECTIVE`): use the name that the request,
the conversation or the work's saved brief gives. A project the person
names in the request counts. Ask only when none of these gives a name.
Do not take the name from unrelated records such as other existing
projects or the company name. The rule against invented or placeholder
names stays.
- Tests: updated and new text checks in `agent-conversations.test.ts`.

## Verification

- `cd server && npx vitest run
src/__tests__/agent-conversations.test.ts`: 64 passed. The new test
fails on master (red) and passes with this change (green).
- `npx vitest run` for `board-access-denial` and `codex-local-execute`:
passed.
- Paid chat evals `tasks.website-without-project` and
`questions-and-approvals.necessary-question` run on a combined throwaway
branch; results are in a PR comment. The necessary-question fixture has
an unrelated existing "Launch" project, so the guidance says not to take
the name from other projects.

## Risks

- Guidance text only. The model can still ask or not ask in edge cases.
The paid evals check both directions.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 21:00:33 -06:00
DottaandClaude Opus 5.5 601babe322 fix(chat): let agents read saved trace export status and give chat real company page paths (#20998)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and agents answer from the
company's saved settings
> - A board member can save a company trace export (an OTLP collector or
Braintrust) in Company settings
> - The settings route is for board connection managers only, so an
agent cannot read it
> - In a chat eval, the board saved and tested a collector. The agent
checked only its own config and secrets, and then told the person that
nothing was configured
> - Chat guidance also names company pages as `/<prefix>/...`. In
another eval, the agent copied that placeholder into a link, and the
link opened nothing
> - This pull request gives agents a read-only status of the trace
export, and gives chat the real company prefix in page paths
> - The benefit is that agents answer from saved settings, and the links
they write open real pages

## Linked Issues or Issue Description

**What happened?**
1. A board member saved and tested a trace export destination. The chat
agent was asked to acknowledge the saved configuration. The agent said
that no collector endpoint or credentials existed. Agents get 403 from
`GET /api/companies/{companyId}/observability`, and no other route
reports the saved state.
2. An agent reply linked `/<prefix>/company/settings/instance`. The chat
rendered it as `/%3Cprefix%3E/...`, which is not a real page. The
placeholder came from the chat directive, which names pages such as
`/<prefix>/company/settings/instance/profile`.

**Expected behavior**
1. An agent can read whether the company's trace export is configured,
enabled and credentialed, and the result of its latest test. The agent
cannot read the destination or any credential.
2. The chat directive and API search results that an agent reads give
real company paths, such as `/COM/company/settings/instance/profile`.

**Steps to reproduce**
1. As a board member, save and test an agent observability destination
in Company settings.
2. In chat, ask the CEO agent to acknowledge that the test collector
configuration is saved.
3. See that the agent cannot read the saved settings and denies them.

**Paperclip version or commit**
master at 9dfdeaad86.

## What Changed

- Added `GET /api/companies/{companyId}/observability/status`. Agents of
the company and all members can read it. It returns `configured`,
`enabled`, `provider`, `credentialsConfigured`, `updatedAt` and the
latest test status. It never returns the destination or a credential.
- Added the shared `AgentObservabilityStatus` type and
`agentObservabilityStatus()` mapper, and registered the route in the
OpenAPI catalog so that `search_api` finds it.
- The chat task markdown now fills the conversation's own company prefix
(from its identifier, for example `COM-1` gives `COM`) into the chat
directives in place of `/<prefix>/`.
- `search_api` results now get the same substitution for the caller's
company. The shared catalog is not changed.
- Pinned the changed shared source in
`tests/runner-e2e/source-sdk-public-inputs.json`.

## Verification

- `cd server && npx vitest run src/__tests__/agent-observability.test.ts
src/__tests__/agent-observability-status-guidance.test.ts
src/services/heartbeat/task-markdown.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts`
- Red/green: with the product change removed, 10 of the new or updated
tests fail (the agent status read gets 403, search does not find the
route, and the chat markdown still contains `/<prefix>/`). With the
change, all pass.
- `npx tsc -p server/tsconfig.json --noEmit` passes.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
reports no change.

## Risks

- Low risk. The new route is read-only and returns no destination and no
credential. The existing settings route keeps its board-only rule.
- The chat prompt text changes for chats with an identifier. The text is
constant for each company, so prompt caching is not affected inside a
conversation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:55:21 -06:00
DottaandClaude Opus 5.5 2a1620f143 fix(heartbeat): deliver a person's answer that waited behind a turn that closed the task (#20996)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can ask the person a question on its task, and the person
answers it in the task chat
> - The person can answer while the agent's turn is still running. The
answer then waits for that turn to finish
> - If that turn completes the task, the release step cancels every
waiting wake for the assignee on the done task. The person's own queued
messages reopen the task, but the answer was dropped and no turn
received it
> - A person who answers an agent's question expects the agent to get
the answer. Losing it without a trace is a silent failure
> - This pull request treats an answer saved before the task was
completed as owed input. It reopens the task and delivers the answer
once
> - The benefit is that a person's answer is never dropped because the
agent finished its turn first

## Linked Issues or Issue Description

**What happened?**
The paid chat eval `steering.approval-delivery` showed it. The person
answered the agent's naming question while the agent's turn was running.
The turn completed the task. The answer's own wake was then cancelled
with "Deferred execution wake no longer applies to a terminal task". No
turn received the answer.

**Steps to reproduce**
1. Start a turn for the assignee on a task.
2. While the turn runs, answer the agent's question on that task.
3. Let the turn mark the task done.
4. The answer's wake is cancelled, and no run receives it.

**Expected behavior**
The answer reaches the assignee exactly once. An answer saved after the
task was completed is still dropped, as before.

Refs #20981, #19331

## What Changed

- `wake-queue/application/use-cases.ts`: before cancelling a deferred
assignee wake on a done task, check whether it carries a person's owed
answer. If it does, reopen the task to `in_progress` and promote the
wake. An answer continues the assignee's own execution, so the existing
continuation status gate applies unchanged.
- `wake-queue/adapters/postgres.ts`: add `owedInteractionAnswer`. It
applies only to an undelivered response that a person resolved, that is
not a tool action, and that was saved at or before the task's
`completedAt`. It returns `blocked` when unresolved blockers forbid
`in_progress`. `reopenIssue` takes an optional target status.
- The blocked case cancels the answer wake with an explicit reason and
does not reopen the task, so the release step does not fail. A
regression test reproduces the earlier throw (fix for the Greptile P1).
- `heartbeat-answer-behind-settling-run.test.ts`: an end-to-end
heartbeat test. It is red on master (the answer wake is cancelled and no
run is bound to it) and green here (one run is bound and succeeds). A
control test shows that an answer saved after completion is still
dropped.
- `use-cases.test.ts`: unit cases for an owed answer, a not-owed answer,
a cancelled task, and an agent-requested wake.
- Eval `steering.approval-delivery` oracle v7: also accept the Acorn
document when the worker's settling turn saved it after the answer was
saved. The eval still requires exactly one later turn bound to the
answer. Calibration tests cover the retained shape, reworded text, a
pre-answer save, the wrong name, another agent, a stale revision, and a
missing bound turn.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-answer-behind-settling-run.test.ts
src/modules/wake-queue
src/__tests__/heartbeat-blocked-question-answer.test.ts` (351 passed)
- Red check: with the `server/src/modules` changes reverted, the new
delivery test fails.
- `cd server && npx vitest run
src/__tests__/heartbeat-comment-wake-batching.test.ts
src/__tests__/question-response-delivery.test.ts` passed. In
`issue-queued-comments-routes.test.ts` all 138 tests passed, but a
cleanup hook timed out locally. CI covers this file.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts` (58 passed)
- `tsc --noEmit` passes for `server` and `tests/runner-e2e`.
- Paid single-case run 43136466619 on 8a4eaab2c0 passed: one turn was
bound to the answer, there were 5 runs within the bound, and the task
was done. The later blocker commit adds only a blocker read on that
path.

## Risks

- Behavior change: a done task reopens when a person's answer was saved
before completion and no run received it. In that case the agent gets
one more turn. Answers saved after completion, cancelled tasks,
tool-action outcomes, and agent-requested wakes do not change.
- If the agent already read the answer by itself, the extra turn sees
the finished work and completes again. That is one extra turn, and the
answer is not repeated.
- Known gap: the answer is still dropped when the task has unresolved
blockers. This happens when a blocker reopens during the turn. Such a
task cannot move to `in_progress`, and an answer turn cannot run on any
other status. In this case the release step cancels the answer wake with
the reason "Queued answer cannot continue a task with unresolved
blockers", keeps the task done, and continues to the next wake. This is
not a regression, because master dropped the answer in every case. But
it does not meet the goal that a person's answer is never lost. It is a
separate follow-up.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:33:48 -06:00
DottaandClaude Opus 5.5 faa78dd818 fix(agents): keep a managed instructions bundle managed when a restarted volume-free server saves another setting (#20997)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent settings are saved through `PATCH /api/agents/:id`. On a
volume-free server, a save is refused when the agent's instructions are
external, because files outside the managed root do not survive a
restart
> - A managed bundle saves its instruction paths as absolute paths under
the home of the server that wrote them. A replacement volume-free server
can start with a new home
> - After such a restart, any `adapterConfig` change re-derived the
bundle mode from the saved `instructionsFilePath`. That path is outside
the new managed root, so the bundle was read as `external` and the save
failed with 422 "Volume-free servers require managed agent instructions"
> - This pull request keeps a managed bundle managed when the save keeps
its saved `instructionsFilePath`, and rewrites its paths under the
current managed root
> - The benefit is that a board user can change one setting (for example
`idleTimeoutMs`) after a restart, and a real switch to external
instructions is still refused

## Linked Issues or Issue Description

**What happened?**
In restart-chaos wave `w-20261112T012841-s7000` (volume-free mode,
Daytona, Codex, restarts K1 and K15 with a fresh server home), the board
sent `PATCH /api/agents/:id` with
`{"adapterConfig":{"idleTimeoutMs":301001}}`. All 3 tasks of the W2C
workload got 422 "Volume-free servers require managed agent
instructions".

**Expected behavior**
A partial `adapterConfig` change of an agent with managed instructions
succeeds on a restarted volume-free server.

**Steps to reproduce**
1. Create an agent with a managed instructions bundle on a volume-free
server.
2. Restart the server with a new `PAPERCLIP_HOME`.
3. Send `PATCH /api/agents/:id` with
`{"adapterConfig":{"idleTimeoutMs":301001}}`.
4. The server returns 422.

**Why Codex looked different from Claude**
It is not the adapter. The Claude wave that passed
(`w-20261112T010036-s6801`) had no faults, so its PATCH ran on the
server that wrote the bundle. The Codex wave sent its PATCH after K1 and
K15 restarts with a new home. The new test fails the same way for a
Claude agent after a fresh-home restart, and passes for both adapters on
the original server.

**The PATCH contract**
`adapterConfig` merges into the saved config unless the request sets
`replaceAdapterConfig: true` (validator in
`packages/shared/src/validators/agent.ts`, route in
`server/src/routes/agents.ts`). The UI full-config save sends
`replaceAdapterConfig: true` with the whole config. The merge was
already correct. The fault was in
`syncInstructionsBundleConfigFromFilePath`, which the route applies to
the effective (merged) config: it classified the bundle only from the
absolute file path and the current managed root. The contract does not
change.

## What Changed

- `server/src/services/agent-instructions.ts`
(`syncInstructionsBundleConfigFromFilePath`): when the saved bundle is
managed and the effective config keeps the saved `instructionsFilePath`,
keep it managed with the root that `deriveBundleState` already resolves
(the current server's managed root) and the saved entry file. A
different `instructionsFilePath` is classified as before.
- `server/src/__tests__/agent-volume-free-config-patch.test.ts`: route
tests with embedded Postgres for a Codex agent and a Claude agent: a
one-setting change after a fresh-home restart, the same change on the
original server, and a real switch to an external path, which is still
refused.
- `doc/DEVELOPING.md`: describe how a volume-free server keeps a managed
bundle across a new home.

## Verification

- Red on `origin/master` (`9dfdeaad86`), `cd server && npx vitest run
src/__tests__/agent-volume-free-config-patch.test.ts`:
- `× changes one codex setting after a fresh-home restart and keeps
managed instructions`
- `× changes one claude setting after a fresh-home restart and keeps
managed instructions`
- `AssertionError: {"error":"Volume-free servers require managed agent
instructions"}: expected 422 to be 200`
- The 4 other cases passed (the original server, and the refused
external switch).
- Green with the fix: the same file passes 6/6, together with
`agent-instructions-service`, `agent-instructions-routes`,
`ephemeral-server-contract` and `agent-configuration-save` (65 tests).
- The restart-chaos W2C workload (`tests/restart-chaos/rewake.ts`)
already sends this PATCH after K1/K15 restarts and caught the fault as
`T:config_changed_before_wake`. No new chaos cell is necessary.

## Risks

- Low risk. The change applies only when the saved bundle is managed and
the save keeps the saved `instructionsFilePath`. Then the paths are
rewritten under the current managed root, which is the root that bundle
reads already use.
- The volume-free guard is unchanged. A save that sets an
`instructionsFilePath` outside the managed root is still refused (tested
for both adapters).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:06:51 -06:00
DottaandClaude Opus 5.5 1949ac6788 test(chaos): fail the directory-lock crash point with the lock's own busy error (#20988)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness can fail one recovery step on purpose
(`--fail-recovery-step`), through a test-only crash point in the server
> - The `directory-lock` step fails
`agent_directory.operation_lock.before_acquire`, the per-agent Postgres
lock that a volume-free server takes around remote agent-directory work
> - The real lock path throws `AgentDirectoryOperationBusyError` when
another run holds the lock. Preparation turns that error into a deferral
and retries later
> - The crash point threw a plain 409 `conflict()` with the same
message, so the deferral never matched and the injected fault failed the
run. A volume-free wave on 2026-11-12 showed this: the run failed with
`native_continuation_requires_reconciliation` and needed Retry, where
real contention would have been deferred
> - This pull request makes the crash point throw the lock's own error
class
> - The benefit is that the `directory-lock` chaos cell tests the
product's real handling of a busy agent directory

## Linked Issues or Issue Description

**What is the problem?**
The test-only crash point
`agent_directory.operation_lock.before_acquire` throws a different error
class from the lock it stands in for.

**What is the expected behavior?**
A run that meets the injected failure takes the same path as a run that
meets real lock contention.

**Steps to reproduce**
Run a volume-free Daytona wave with `--fail-recovery-step
directory-lock:1`, workload W2 and a K2 restart. The recovering run
fails with `native_continuation_requires_reconciliation`; the stack
shows the crash point inside `prepareCopy`, under
`prepareWithAgentDirectoryDeferral`, which only defers
`AgentDirectoryOperationBusyError`.

## What Changed

- New test
`server/src/__tests__/agent-directory-lock-crash-point.test.ts`.
- `server/src/services/agent-directory-working-copies.ts`: the crash
point's `fail` returns `new AgentDirectoryOperationBusyError()` (same
409 status and message) instead of a plain `conflict()`.

## Verification

- `server/src/__tests__/agent-directory-lock-crash-point.test.ts` arms
the point in fail mode through a control file (as restart chaos does)
and checks that the injected failure is an
`AgentDirectoryOperationBusyError` that
`prepareWithAgentDirectoryDeferral` turns into a deferral. It fails on
master (the injected error is a plain `HttpError`) and passes with this
change.
- Server typecheck passes.
- The crash point is disarmed unless a chaos wave arms it, so production
behavior does not change.
- Live check: the next volume-free `directory-lock:1` chaos wave (W2,
K2, Codex) shows whether the recovering run is deferred instead of
failing. I will add the result to this PR.

## Risks

- Low risk. The change is in a test-only crash point that is disarmed in
production. The README step table already documents the error as the 409
"Agent directory operation is already active".

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:06:36 -06:00
DottaandClaude Fable 5.2 52cdf76bf3 fix(ephemeral): start a volume-free server over harness backups a volume-free server left unsaved (#20990)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A hosted Paperclip server can run volume-free: its durable state is
in the database and object storage, and its local disk is replaced with
every deployment
> - At startup, a volume-free server runs a readiness gate that refuses
to start when it finds data that still needs migration off a volume
> - One of those findings is a harness backup that never reached durable
storage. A volume-free server that stops in the middle of a run always
leaves some
> - The gate then refuses every later start, the process exits, and the
host restarts it into the same refusal. The workspace stays down until a
person repairs it
> - This pull request lets a server start over such backups when its
filesystem carries the volume-free owner marker
> - The benefit is that a volume-free server recovers from an
interrupted run by itself

## Linked Issues or Issue Description

Refs #20952 (the same crash loop for Runner controller files; this is
the next check in the same gate).

**What happened?**

A hosted volume-free workspace stopped while runs were in progress.
Every later start logged `Paperclip server failed to start` with
`Volume-free server data is not eligible:
native_harness_backups_require_migration`, and exited. The host
restarted the container and the server exited again. The workspace did
not serve any request for more than two hours.

**Expected behavior**

A volume-free server starts after an interrupted run. Restart recovery
handles the runs whose local harness backup was lost with the old
container.

**Steps to reproduce**

1. Run a volume-free server (`PAPERCLIP_EPHEMERAL_SERVER=true`) with
native Runner tasks in progress.
2. Stop the server process before the harness backup of an active
session reaches durable storage.
3. Start the server again. It refuses to start with
`native_harness_backups_require_migration`.

**Paperclip version or commit**

`87fe4d979355232570428deba480b47e1812a26c` (contains #20952).

## What Changed

- `assertEphemeralServerData`: when the only reason is
`native_harness_backups_require_migration` and the filesystem carries
the volume-free owner marker (`volumeFreeControllerStateOwned`), the
gate removes that blocker, logs a warning, and lets the server start.
- A filesystem without the marker is refused as before. All other
blockers are unchanged.
- `readEphemeralServerDataEligibility` is unchanged, so the migration
and enrollment callers still see the blocker.
- Added `ephemeral-server-startup-run-state.test.ts`.

## Verification

- Red: with the change to `ephemeral-server-contract.ts` reverted,
`starts a volume-free server over harness backups a volume-free server
left unsaved` fails (1 failed, 2 passed).
- Green: `npx vitest run
src/__tests__/ephemeral-server-startup-run-state.test.ts
src/__tests__/ephemeral-server-data.test.ts` in `server/`: 2 files, 8
tests passed.
- Not run locally: the full suite. CI supplies that coverage.
- Not verified: a live volume-free start. The hosted workspace that
showed the failure will receive this change through the normal release.

## Risks

- A server on a marked volume-free filesystem now starts with unsaved
harness backups. Those backups were on a disk that no longer exists, so
the refusal protected nothing. The affected runs depend on restart
recovery.
- The marker is the only evidence that there is no volume. A filesystem
that still is a volume and does not carry the marker keeps the refusal.
- The gate does not repair or delete the stale lease stamps. The warning
repeats at each start until restart recovery settles those runs.

## Model Used

Claude Fable 5.2 (`claude-fable-5-2`), through Claude Code with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 20:02:27 -06:00
DottaandClaude Opus 5.5 5d1bb5192f fix(retry): carry the original wake to every re-queued run (#20989)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A run serves one wake: a comment, an answer, an assignment, or a
one-shot task monitor check (`issue_monitor_due`).
> - When a run fails, Paperclip often starts a successor run by itself:
the bounded transient retry, the uncharged `interrupted_preparation`
re-queue, workspace or connection waits, missing-comment repair,
capacity retries and recovery successors. Each successor links to the
failed run through `retry_of_run_id`.
> - Most of these re-queues replace the wake reason with their own (for
example `transient_failure_retry`). The agent then does not know which
wake it must handle.
> - #20970 fixed this for the board Retry only. A no-fault chaos wave
showed the same loss on an automatic retry: the monitor run failed
before provider work, the automatic retry succeeded, and the agent did
not treat it as the monitor check. The monitor was already consumed, so
the task rested `in_progress` with nothing pending.
> - This pull request covers all re-queue paths at one shared point: the
continuation builder that every issue dispatch uses
(`buildExecutionContinuation`, called from `heartbeat/task-context.ts`).
It walks the re-queue chain to the first run that was not a re-queue and
records that run's wake reason as `trigger.retriedReason`. The wake
prompt already shows that field (#20970).
> - The benefit is that every successor of a failed run handles the same
wake as the run it replaces.

## Linked Issues or Issue Description

**What happened?**
No-fault restart-chaos wave on master `87fe4d9793`, volume-free mode
(`w-20261112T010036-s6801`, Claude, task t01, workload W2). The monitor
wake started run `78e27ef9`. It failed before provider work
(`native_predecessor_retirement_unproven`). The
`interrupted_preparation` re-queue (attempt 1/12) started run `0cd387a0`
on a fresh sandbox, and that run succeeded. The agent reported that no
board comment arrived and that the monitor wake was not visible. It
yielded `response_wake` again. The monitor was `triggered` with no
`nextCheckAt`. The issue rested `in_progress` with `canRetry=false`, no
live run, no wake and no monitor. The harness graded `R1
rested_not_done` and `R3 missing_retry`.

**Expected behavior**
A successor of a failed run tells the agent which wake it re-delivers. A
re-queued monitor check is handled as the monitor check.

**Steps to reproduce**
1. Wake a native agent with `issue_monitor_due` on a task.
2. Make the run fail before provider work, for example when the previous
run's sandbox session cannot be retired.
3. Let the `interrupted_preparation` re-queue run.
4. Before this change, the successor's continuation trigger has the
re-queue reason only, and its prompt does not name the monitor check.

**Additional context**
Follow-up to #20970.

Refs #20970
Refs #20935
Refs #20863

## What Changed

- `server/src/services/execution-continuation.ts`: new
`requeuedWakeReason`.
- It starts at the current run's `retry_of_run_id` column. If that is
empty, it uses the context `retryOfRunId`, or the source run of a board
Retry.
- It follows the chain within the same company and issue, at most 32
links, until it finds a run with no re-queue link. It returns that run's
wake reason.
- A `trigger.retriedReason` already saved on a run in the chain is
reused.
- `trigger.retriedReason` is set only when the original reason differs
from the current reason and is not itself a Retry reason. A re-queue
that keeps the original wake reason (`preserveWakeContext`) is
unchanged.
- This one point covers every successor that links with
`retry_of_run_id`: bounded transient retries, `interrupted_preparation`
(#20863), workspace and AI-connection waits, missing-comment repair
(`run-lifecycle.ts`), native capacity retries, safe replacement, and the
board Retry.
- Tests:
- `server/src/__tests__/heartbeat-native-predispatch-retry.test.ts`: the
assembled heartbeat service runs a monitor-due wake, a
predecessor-retirement failure before provider work, and the automatic
`interrupted_preparation` re-queue. It asserts the successor's saved
continuation and wake prompt.
- `server/src/services/execution-continuation.test.ts`: a two-link chain
of automatic re-queues, and a re-queue that keeps the original reason.

## Verification

- Red on master: `cd server && npx vitest run
src/__tests__/heartbeat-native-predispatch-retry.test.ts -t "delivers
the monitor check"`. The successor's trigger had `sourceRunId` = the
failed monitor run, but no `retriedReason`.
- Green: `heartbeat-native-predispatch-retry.test.ts`,
`execution-continuation.test.ts` and `heartbeat/retries.test.ts` pass
(78 tests).
- `npx tsc --noEmit -p server` passes.
- Chaos: the existing no-fault W2 cell in volume-free mode reproduces
this when the monitor run fails before provider work. I did not rerun a
wave.

## Risks

- Low risk. The change adds one optional trigger field to more wakes. It
changes no admission, status or authority path.
- The walk costs at most one indexed primary-key read per chain link,
only for runs that are re-queues.
- The walk follows only same-company, same-issue runs. Old saved runs
work, because they already store `wakeReason` and the `retry_of_run_id`
column.
- The resting state "agent yielded `response_wake` after the monitor was
consumed" is still possible when an agent really waits. #20935 shows it
on the run projection as "Waiting for a reply". The ownership
reconciler's I7 check still counts it as `no_owner_no_wake`. This PR
does not change I7.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:52:21 -06:00
8818402e7a feat(inbox): route requests to people and record personal outcomes (#20959)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - The inbox collects requests that need a human decision.
> - Teams need to know who owns each request and who may act on it.
> - A shared close operation must differ from hiding a card for one
person.
> - This pull request adds personal routing and durable outcomes to the
existing attention service.
> - People can claim or transfer work and inspect what another person
did for them.
> - Domain permissions still decide who may resolve each request.

## Linked Issues or Issue Description

**Subsystem affected**

Server, shared contracts, and database. This is the routing foundation
for a stacked inbox UI change.

**Problem or motivation**

A company-wide attention feed does not give teams a clear personal
queue. Archive, withdrawal, transfer, and decisions by another person
need distinct scopes and records.

**Proposed solution**

Resolve a request audience from its addressee, task participants, and
configured owners. Provide personal, unassigned, company, all-assignee,
and archived views. Add claim, transfer, personal archive and snooze,
shared withdrawal, receipts, and personal history.

**Alternatives considered**

A single shared inbox leaves ownership and personal hiding ambiguous.
The service keeps one canonical request and projects its audience
instead of copying requests for each person.

**Roadmap alignment**

This extends the existing Multiple Human Users and Agent Reviews and
Approvals capabilities in ROADMAP.md.

**Additional context**

Refs #19363 and #19520 for the current inbox navigation and confirmation
replacement rules. Refs #20268 for related actionable-request work. That
PR filters actionability; this stack adds audience routing and personal
outcomes.

This is part 1 of a two-PR stack. The inbox UI PR #20961 targets this
branch.

## What Changed

- Add nullable company and project decision owners and a company budget
owner.
- Add audience views and personal badge counts to the canonical
attention projections.
- Add claim, transfer, withdrawal, personal archive, snooze, and restore
operations with permission checks and activity records. Lock addressed
transfers to the current addressee or routing administrators, and
validate the recipient against canonical resolution policies.
- Add receipts and scoped history for work handled by another person.
- Keep closed-task requests and replaced confirmations out of current
cards.
- Add an idempotent migration and a concurrent receipt-history index.
- Update typed fixtures so the existing UI remains compatible with the
new contracts.

## Verification

- Server layered typechecking passes, including all three test-tree
partitions.
- Seven routing and interaction-resolution suites pass all 249 tests
against a dedicated external PostgreSQL test database on `4caecc21d8`.
Six new regressions failed before the repair; they cover takeover
followed by approval, legitimate delegation, excluded creators/review
requesters, and unauthorized claims.
- Prior focused verification covers private approval history and
receipts, transfer visibility, audience escalation, withdrawal
governance, hidden-card counts, per-recipient confirmation replacement,
and SDK source contracts.
- A synthetic three-person company passed 37 routing and lifecycle
acceptance checks. The UI stack adds browser acceptance for the shared
controls.
- The complete [GitHub Actions CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/43134997331)
passes on `4caecc21d891677323a7335f23b1a015cce24c13` (Linux/Node 24:
typecheck, build, general and serialized tests, Runner support, browser
shards, and aggregate gates). Greptile is 5/5 on that exact head with no
open findings. The fresh security scan passes after explicitly
rechecking the addressee-transfer repair.
- `pnpm verify` reports exit 2 because a CI-global script changed; full
CI supplies that broader coverage. Embedded PostgreSQL could not start
on this Mac because shared-memory slots were exhausted, so local
database checks use a separate external test database. Linux CI covers
the embedded setup.

## Risks

- Personal badge counts change to the viewer's assigned requests.
Accessible feed totals and task membership keep their existing scope.
- The migration adds nullable owner fields and a dismissal fingerprint.
Its index runs outside a transaction. The migration is classified and
rerunnable.
- Routing changes the addressee used by resolution authorization. The
current addressee or a routing administrator must authorize an addressed
transfer; the recipient must satisfy writer access, canonical resolver
exclusions, and task review policy under row locks.
- The shared validator export barrel crosses 1,000 lines because it
exports the new attention schemas. Schema implementation stays in the
dedicated attention validator module.

## Model Used

OpenAI GPT-6 through Codex. Used repository tools, code execution, and
browser checks. This session does not expose a more specific runtime
model ID or its context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
(OpenAPI contracts and docs/api/attention.md)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 17:49:39 -08:00
TonioandPaperclip 9dfdeaad86 Refine agent workspaces and organization charts (#20973)
Refine agent details, identity editing, responsive controls, and organization charts with three card tiers, clearer save feedback, and stable filtering.

Guard instruction-file renames and root configuration updates in one transaction through the agent lifecycle integration, preserving conflict checks, rollback, and configuration history.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-12 01:23:59 +00:00
Nicky Leach 45170b0cac Reuse Runner plugin capability checks (#20987) 2026-11-11 17:21:07 -08:00
DottaandClaude Opus 5.5 5cd5adfccc fix(native): link a person-owned agent blocker to the agent's open question whatever owner kind the model reports (#20969)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #20931 lets a person's answer continue a task that its agent blocked
while it waited for that answer
> - That link is recorded only when the blocker's unblock descriptor is
bound to the board
> - Codex reports such a wait as `paperclip_block` with `owner: {"kind":
"user", "name": "Board"}` and scope `task_wide`. The evidence classifier
treats only `kind: "board"` (not a PRP owner kind) as board-owned, so
the arbiter binds this person-owned blocker to the reporting agent, and
no link is recorded
> - So every Codex W2Q task still loses the answer
(`issue_not_in_progress`) and needs Retry
> - This pull request classifies a `user` owner as a person
(board-bound) and records the link from the reported owner kind, not
from the spelling of the owner
> - The benefit is that the person's answer reaches the agent whatever
owner shape the model uses

## Linked Issues or Issue Description

Refs #20931

**What happened?**
Restart-chaos wave w-20261111T230717-s5201 on master 9c56e546 (Daytona
warm, Codex, no fault): all 3 W2Q tasks (t01, t05, t09) failed. Each
continuation after the answer (runs 8864fa72, 814b2cd0, 757c1c8f) was
cancelled with `issue_not_in_progress (current status: blocked)`. Each
agent blocked with `owner: {"kind": "user", "name": "Board"}`, scope
`task_wide` (reason codes `missing_required_answer`,
`awaiting_codename_answer`). The decision was `task_wide_blocker_bound`
with the unblock owner `{agentId}`, so `questionsAwaitedByAgentBlock`
recorded no `awaitingInteractionIds`.

**What did you expect to happen?**
The answer continues the task, as it does for a board-bound blocker.

**Steps to reproduce**
1. A Codex agent asks a question; the person comments before answering.
2. The agent calls `paperclip_block` with a `user` owner and scope
`task_wide`.
3. The person answers. The continuation is cancelled with
`issue_not_in_progress`.

## What Changed

- `evidence-classifier.ts`: new `nativeBlockerOwnerKind` and an optional
`ownerKind` on the blocker assessment (`agent`, `user`, `system`,
`external`, legacy `board`, or `unspecified`). A `user` owner is now
board-owned, like `board`: a person resolves it, so it is not bound to
the reporting agent.
- `status-arbiter.ts`: both agent-reported blocker decisions
(`task_wide_blocker_bound`, `current_track_blocker_waiting`) carry
`blockerOwnerKind`.
- `blocked-question-answer.ts` and the status committer:
`questionsAwaitedByAgentBlock` records the agent's own pending,
undismissed questions when the reported owner is one a person resolves:
`user`, `external`, legacy `board`, or no recognized kind. `agent` and
`system` owners and dependency blocks
(`durable_dependency_blocker_bound`) stay excluded. A decision without
`blockerOwnerKind` keeps the previous board-bound rule. All other guards
are unchanged: asked by this agent, answered by a person, decision still
current, `statusVersion` match, guarded checkout.
- `doc/execution-semantics.md` describes the rule.

## Verification

- `server/src/__tests__/heartbeat-blocked-question-answer.test.ts` now
builds the block decision with the real `classifyNativeEvidence` and
`arbitrateNativeStatus` from the exact Codex report shape, commits it
with `commitNativeStatusDecision`, and delivers the answer through
`heartbeat.wakeup`.
- Red on master (product hunks removed, test kept): `user` task-wide
fails because the unblock owner is `{ agentId }` instead of `board`;
`external` task-wide fails with `status: cancelled, errorCode:
issue_not_in_progress`.
- Green: 10/10. `user` task-wide, `external` task-wide and `user`
current-track are delivered. `agent` and `system` owners still cancel
with `issue_not_in_progress`, as do the earlier negative cases.
- `cd server && pnpm exec vitest run
src/services/native-runtime/status-arbiter.test.ts
src/services/native-runtime/evidence-classifier.test.ts
src/services/native-runtime/historical-task-questions.test.ts
src/modules/run-dispatch`: 257 passed. `recovery-observability`,
`heartbeat-issue-liveness-escalation`, `native-completion-feedback`,
`paperclip-control-plane-port`: all passed.
- `cd server && pnpm exec tsc --noEmit -p tsconfig.json`: passes.
- Chaos: see the comment below.

## Risks

- A task-wide blocker that names a person (`user`) no longer binds its
unblock descriptor to the reporting agent and no longer wakes that agent
about its own blocker. The board owns it, as for a current-track
blocker.
- An `external` owner can now link to the agent's open question. If that
blocker was about something else, the answer resumes the agent and it
reports the blocker again.
- Upgrade: `ownerKind` and `blockerOwnerKind` are optional. A decision
built by an older release keeps the previous rule. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:50:00 -06:00
DottaandClaude Opus 5.5 87fe4d9793 fix(workspaces): retry a workspace_finalize record write that loses a deadlock (#20971)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a run ends, Paperclip records a `workspace_finalize` workspace
operation. Dependent work reads that record to know if the workspace is
safe to use.
> - The record insert has foreign keys to `execution_workspaces`,
`companies`, `heartbeat_runs` and `issues`. Postgres takes a key-share
lock on each parent row, in trigger order: workspace, company, run,
issue.
> - The native finalizer and other writers lock the issue row and then
the run row (the order that #20819 documents). When such a transaction
holds the issue and then asks for the run, while the finalize insert
holds the run and waits for the issue, Postgres detects a deadlock and
aborts one side.
> - In restart-chaos wave `w-20261111T205259-s6301` (task t01, run
`e7129ec7`) the insert lost. The log says "failed to record
workspace_finalize=failed operation; dependents may remain gated". The
failed-finalize record was lost.
> - The insert's lock order comes from the foreign keys, so the code
cannot reorder it without a new explicit lock order that other writers
(for example the remote-workspace release path, workspace then run)
could invert again. A deadlock victim is safe to run again, because its
transaction rolled back completely.
> - This pull request retries the finalize record write on lock timeout,
deadlock and serialization failure (55P03, 40P01, 40001), with the same
bounded policy that #20850 added for the provider-pack admission write.
> - The benefit is that a short lock conflict can no longer delete the
record that releases dependent work.

## Linked Issues or Issue Description

**What happened?**
Run `e7129ec7` failed its native session recovery
(`runner_harness_state_mismatch: backup_without_reusable_lease`). The
failure path called `recordWorkspaceFinalize("failed")`. The insert into
`workspace_operations` failed with `PostgresError: deadlock detected`
(40P01). The server logged that dependents may remain gated. The run
then settled as failed with `native_session_interrupted`; the next
`finalizeNativeRun` call logged `native_finalization_missing` because
that run had no proposed result.

**Expected behavior**
A transient deadlock does not lose the finalize record. The write runs
again and the record is saved.

**Steps to reproduce**
1. Transaction A: lock the issue row `FOR UPDATE`.
2. Start a `workspace_finalize` record for a run on that issue. Its
insert takes key-share on the workspace, company and run rows, then
waits for the issue.
3. Transaction A: lock the run row `FOR UPDATE`.
4. Postgres reports a deadlock. Before this change the insert is the
victim and the record is lost.

**Additional context**
The new test reproduces this with two real transactions on embedded
Postgres. It gives transaction A a longer `deadlock_timeout`, so the
insert is always the victim, as on the chaos run.

Refs #20850
Refs #20819

## What Changed

- New `server/src/services/transient-lock-contention.ts`:
`postgresErrorCode`, `isTransientLockContention` and
`retryTransientLockContention` (5 attempts, 100 ms × attempt backoff,
only 55P03, 40P01 and 40001).
- `workspaceOperationService.createRecorder().recordOperation` wraps its
`workspace_finalize` state writes (the running-row insert and the
terminal update) in that retry. Each attempt is a new transaction, or a
new savepoint when the service runs inside a caller's transaction.
- `native-provider-pack-admission.ts` uses the shared helper. Its
behavior and its exported `nativeProviderPackAdmissionFailureCode` name
do not change.
- New test in
`server/src/__tests__/workspace-operations-reconciliation.test.ts`: a
real two-transaction deadlock between an issue-then-run locker and the
finalize insert.

## Verification

- Red on master: `cd server && npx vitest run
src/__tests__/workspace-operations-reconciliation.test.ts -t "loses a
deadlock"` failed with `deadlock detected`, code `40P01`, on the same
`insert into "workspace_operations"` query as the chaos log.
- Green: the same test passes; I ran it 3 times. The full file and the
provider-pack admission tests pass (49 tests).
`native-result-commit-work.test.ts` passes.
- `npx tsc --noEmit -p server` passes.
- I stopped `pnpm verify` after it selected 2533 test files; required CI
supplies broad coverage.
- Chaos: no new cell. The existing W2 × K1 cell (Codex, Daytona warm)
reached this path when the restart fault broke native session recovery.
I did not rerun a wave; both wave slots were busy.

## Risks

- Low risk. Only the `workspace_finalize` state writes retry, and only
on the three contention codes. Other errors throw as before.
- A deadlock victim's transaction is fully rolled back, so the retried
insert (same operation id) and the retried update are idempotent.
- The worst added delay is about 1 s of backoff across 5 attempts, on
the failure path only.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:09:57 -06:00
DottaandClaude Opus 5.5 827bd21017 fix(recovery): retry lost-lease preparations instead of holding them for reconciliation (#20936)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Run recovery restarts work that stops for infrastructure reasons,
and keeps saved files safe
> - A legacy Runner run can lose its controller lease while it prepares
the workspace, before any provider work starts (`controller_lease_lost`
with `executionRecovery: { kind: "bootstrap", providerWorkStarted: false
}`)
> - Such a run has no unsaved provider state, but recovery still blocked
the task: either a saved-files hold that ends in `source_unavailable`,
or an immediate "No live execution path" block after one failed
continuation retry
> - Recovery must work without a person, and a shown Retry must work
> - This pull request lets lost-lease preparations use the existing
interrupted-preparation settlement, and lets the bounded stranded-task
sweep retry a continuation that failed before provider work
> - The benefit is that these tasks retry automatically, and tasks that
an older release already blocked recover after deploy

## Linked Issues or Issue Description

Refs #20924 (fixes the lease renewal starvation that caused the lost
leases; this PR fixes what recovery did afterwards).

**What happened?**
A staging burst of lost controller leases stopped many legacy Runner
runs while they prepared the workspace. No provider work started. Most
of the affected tasks then became blocked. One shape kept a saved-files
hold (`legacy_execution_requires_reconciliation`). Its file recovery
ended `exhausted` with `source_unavailable`, because the workspace had
already retired the prepared epoch back to the last save. The hold never
settled. The other shape was a continuation retry that also lost its
lease. The release path blocked the task at once with "No live execution
path", although `controller_lease_lost` is a transient cause with a
3-attempt budget in the stranded-task sweep.

**Expected behavior**
A run that fails before provider work starts retries automatically,
within a bounded budget. It never needs reconciliation. If the prepared
workspace is gone, the task continues from the last committed save.

**Steps to reproduce**
1. Start a legacy Runner run with a remote workspace.
2. Make the controller lose its lease while the run prepares (before
provider dispatch).
3. Let restore recovery record a saved-files hold, and let the workspace
retire the prepared epoch.
4. See that the hold stays `resolved/blocked` with `source_unavailable`,
and the task stays blocked.

**Paperclip version or commit**
master at c1edb9ec02.

**Agent adapter(s) involved**
Not adapter-specific (core bug). Seen with `paperclip_runner` (Daytona
sandbox).

## What Changed

- `server/src/services/cancelled-native-startup.ts`:
`isInterruptedNativePreparationCandidate` accepts a failed
`controller_lease_lost` run with the bootstrap certificate
(`providerWorkStarted: false`), the same as a certified `process_lost`
run. The error label alone still grants nothing. This lets the existing
`legacy_interrupted_preparation_hold` settlement, terminal-time proof,
status receipt, and `interrupted_preparation` retry lane apply to these
runs.
- `server/src/modules/wake-queue/*`: the release snapshot carries
`retryableBeforeProviderWork` (bootstrap certificate plus a
`transient_infra` continuation classification). When an automatic
continuation retry failed for such a cause, release returns `released`
instead of blocking. The stranded-task sweep then retries with its
existing backoff and 3-attempt cap, and blocks only after that budget is
spent. Non-retryable causes, uninvokable agents, and failures after
provider work still block at once.
- `doc/execution-semantics.md`: documents both rules.
- No migration. Holds saved by the running release are settled by the
existing ownership reconciler after deploy.

## Verification

- Red then green, in the staging shape (failed `controller_lease_lost`,
`renewal_deadline` lease-loss evidence, `restore_failed` with
`workspaceRestoreRecovery`, exhausted `verify_execution` with
`source_unavailable` / `use_alternative_strategy`, retired prepared
epoch):
- `src/__tests__/interrupted-preparation-hold-settle.test.ts`: before
the fix, both new cases fail (`settled: 0`). After the fix, the hold
settles as `false_positive`. With the status receipt, the task returns
to `in_progress` and one `interrupted_preparation` successor is queued.
For a hold saved by an older release without a receipt, the execution
gate clears and one Retry starts a run. A lost lease without the
no-provider-work certificate stays held.
- `src/modules/wake-queue/domain/policy.test.ts` and
`application/use-cases.test.ts`: before the policy change, the two new
release cases fail (the task is blocked). After it, a pre-provider
lease-loss retry failure is released to the sweep, and other failures
still block.
-
`src/modules/wake-queue/adapters/retryable-before-provider-work.test.ts`:
classification of the release snapshot fact.
- `pnpm verify` and server `tsc --noEmit` pass locally.

## Risks

- Lost-lease preparations now use the `interrupted_preparation` retry
lane. That lane is uncharged but bounded at 12 retries, and it does not
revive a chain whose failure budget is spent.
- A released continuation waits for the stranded-task sweep (30 s
reconstruction, plus backoff) instead of a direct requeue. The sweep's
budget bounds the chain.
- Settlement still needs every existing proof: the controller is gone or
settled, leases are released with stop receipts, there are no execution
events or native coordinator, and the prepared epoch is retired or the
files are repaired.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, in Claude Code with tool
use (shell, file edits, browser reads of staging with GET requests
only).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:30:32 -06:00
DottaandClaude Opus 5.5 c45f54ee74 fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown

## Linked Issues or Issue Description

**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.

**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:

1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.

The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.

**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.

**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.

## What Changed

- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.

## Verification

- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches 7e8901aa60, not current master, so each wave
ran this change cherry-picked onto 7e8901aa60:
- `w-20261111T210049-s6111` (first commit only): the close reason is now
logged: `close failed: Daytona sandbox lease 192606aa… is no longer
active.` The delete stayed 409 with that reason. The new check failed
the wave correctly.
- `w-20261111T222826-s6112` (all commits): two W2Q owners were released
with `sandbox was already destroyed`. The environment was deleted. 0
sandboxes left running. The warm-shutdown check passed. No 409 from a
warm session.
- Not verified: a wave where the graceful close fails and the forced
Launcher stop succeeds on a live sandbox (that path is covered by the
unit test). The wave's task failures (`T:first_run_succeeded`,
`R3:retry_needed`) are other open defects. They are also on master.

## Risks

- Environment delete now stops an idle Runner whose close did not
settle. The delete already has the operator's consent to destroy those
sandboxes. The workspace-export and restore-source guards on sandbox
destruction are unchanged. Instruction files are collected after the
confirmed stop, as in #20844.
- An owner is released without contact only when a lease for the same
company, environment and sandbox has a valid `destroyed` termination
receipt. A `stopped` receipt is not enough, because a stopped sandbox
can be resumed.
- `closeIdleWarmNativeSessions` results have a new `failures` field.
Callers that read only the counts are unchanged.
- No persisted state changes. Warm owners live only in controller
memory, so older saved state and mid-run deploys are not affected.
- The new harness check can fail a warm restart-fault wave that leaves a
warm session unclosed at controller shutdown. That is a real leaked
Runner.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs, chaos waves) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:12:21 -06:00
DottaandClaude Opus 5.5 9c56e5465c fix(native): keep the agent's own unanswered question open after a turn ends without a report (#20934)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can ask the person a question and wait for the answer.
Native task status comes from the server's status arbiter, not from
model prose
> - When a native turn ends without `paperclip_finish`, the Runner sends
a default report: `done`, objective satisfied, with the agent's final
message as the only evidence
> - A later comment from the person moves an open question into history
(#15392), so the question no longer gates completion
> - Together these close a task as `done` while the agent is still
waiting for its own answer and has produced nothing. No Retry appears
and nothing flags it
> - This pull request keeps such a task `in_review` on the agent's own
unanswered question when the only completion evidence is the agent's
final message
> - The benefit is that a "waiting for your answer" turn can no longer
become a silent false completion

## Linked Issues or Issue Description

Refs #15392

**What happened?**
Restart-chaos wave on master 7e8901aa60, OpenCode task t06 (W2Q), no
fault injected. Run fec9530f asks a question (interaction c2d3b3f2); the
task goes `in_review`. A board comment says "include NOTE-238ef87a in
the brief; do not write the brief yet; I will answer shortly". Run
f6cbefa8 replies "I will not write the brief until the board's answer
arrives. Ending turn to await the codename answer." and ends its turn
without `paperclip_finish`. The Runner's default report
(`provider:opencode:agent-message` as the only evidence,
`reportedWorkDisposition: done`) is accepted, and the low-risk claim
policy sets the task `done`. The question is still unanswered and the
`brief` document does not exist.

**What did you expect to happen?**
The task keeps waiting for the answer. The answer continues the task,
and the agent writes the brief.

**Steps to reproduce**
1. An agent asks a question on its task.
2. The person posts a comment before answering.
3. The agent ends its turn without a completion report, saying it waits
for the answer.
4. The task is `done`; the question stays pending.

## What Changed

- New `server/src/services/native-runtime/own-question-wait.ts`:
- `reportsOnlyProviderMessage(result)`: true when every evidence
reference in the report is `provider:<provider>:agent-message` and the
report has no artifact and no verification. This is the shape of the
Runner's default report for a turn that ended without
`paperclip_finish`.
- `pendingOwnQuestionId(db, …)`: the agent's own pending, undismissed
ordinary question (`ask_user_questions`, not a governed tool, secret or
connection request) on the task.
- `finalizeNativeRun` (`native-run-finalizer.ts`): when the proposed
decision is `done`, the task is not an Agent Chat conversation, the
report cites only the agent's final message, and the agent's own
question is still pending, the decision becomes `in_review` with reason
`own_question_unanswered` and releases the checkout. The question stays
pending, so the person's answer continues the task through the existing
answer path.
- Dismissal (`POST /issues/:id/interactions/:interactionId/dismiss`,
`dismissInteraction`): new `enqueueOwnQuestionDismissalWake` runs inside
the dismissal transaction, only when the card moves from pending to
dismissed. When the task is `in_review` on an applied
`own_question_unanswered` decision of the assignee's run, the dismissed
card is the assignee's own question, and no other pending, undismissed
question of that agent remains, it saves a durable wake intent
(`issue_question_dismissed`, idempotency key per question) in the
existing native status wake outbox (`enqueueWake`, now exported). The
outbox dispatcher delivers and retries it, so the agent continues
without the answer. Dismissal of any other request keeps the no-wake
rule.
- `assignmentDenial` (`agent-interaction-delivery.ts`): an
`issue_question_dismissed` wake counts as a person's wake when its
`dismissedInteractionId` names the recipient's own question on that task
with a `dismissedByUserId`, so a recipient that refuses agent
assignments still receives it.
- `doc/execution-semantics.md` describes the exception to the
historical-question rule.

## Verification

- New case in
`server/src/services/native-runtime/historical-task-questions.test.ts`
("keeps the agent's own unanswered question open after an unreported
turn"), run through the real `PaperclipControlPlanePort.completeRun` and
`finalizeNativeRun` with a low-risk `agent_claim_policy` contract, as on
staging:
- Red on master (finalizer hunk removed, test kept): `unreported` fails,
`expected 'done' to be 'in_review'`.
- Green: `unreported` rests `in_review` with `own_question_unanswered`
and the question still pending. `unreported-dismissed`,
`unreported-answered` and `verified` (work-product evidence) still rest
`done`, so the #15392 behavior is unchanged.
- Dismissal after finalization (same test): the release check is null
while a second own question is open and returns the agent once both are
dismissed. A repeated dismissal saves no second intent; the saved intent
is a queued `issue_question_dismissed` outbox row. Delivery test
`delivers a dismissal wake only for a person's dismissal of the
recipient's question` (30 passed). Route test `saves the dismissal wake
intent in the dismissal transaction only when the card is dismissed`
(147 passed).
- `cd server && pnpm exec vitest run
src/services/native-runtime/historical-task-questions.test.ts
src/services/native-runtime/native-run-finalizer`: all passed.
- `cd server && pnpm exec vitest run
src/services/native-runtime/paperclip-control-plane-port.test.ts
src/services/native-runtime/status-arbiter.test.ts`: 119 passed.
- `cd server && pnpm exec tsc --noEmit -p tsconfig.json`: passes.
- Chaos, Daytona OpenCode, seed 6201, W2Q K0, 4 tasks (wave
w-20261111T211421-s6201, run from master 7e8901aa60 plus this change
because the pinned runner image must match runner sources): no false
completion. t02 and t04 PASS (in_review -> in_progress -> done after the
answer). t01 failed `R3:retry_needed` through the blocked path fixed
separately in #20931 (not in this build). t03 timed out because its
rewake OpenCode turn stayed active for 20 minutes without ending
(provider turn never finished; this change acts only at finalization,
which never ran). The unreported-done path did not occur in this sample,
so the red/green proof for this change is the integration test. Before:
wave w-20261111T200718-s6201 t06 closed done with the question
unanswered.

## Risks

- Low. Only a `done` decision whose report cites nothing but the agent's
final message changes, and only while the same agent's own ordinary
question is pending and undismissed. Explicit completion reports with
their own evidence keep the current rule. A person can still dismiss the
question to release it.
- An agent that really finished without `paperclip_finish` while its own
old question is open now rests `in_review` instead of `done`. The open
question is visible on the task; answering or dismissing it settles the
task.
- No schema change. A result saved by an older release is finalized by
the same rule because it reads only the saved report and interaction
rows.
- Related observation (not changed here): in 5 Claude/OpenCode W2/W2C
cells the agent wrote "the task remains in_progress as instructed" yet
the task became `done` (Claude t06, wave s6101). The run's
`paperclip_finish` input itself sent `reportedWorkDisposition: "done"`,
`objectiveSatisfied: true`, with the deferred phase listed as
`blocksCompletion: false`. The server did not map a yielded disposition
to done. That is model behavior on a contradictory report, a different
root from this change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:50:28 -06:00